Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5315

Critical performance issu on maven2 jobs during site generation.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • maven-plugin
    • None
    • Solaris 10,Glassfish v2, hudson 1.336,maven 2.2.1
      machine: SunOS hostname 5.10 Generic_137111-08 sun4u sparc SUNW,Sun-Fire-V490

      I have a large project which I am trying to build a site for. In order to have hudson collect unittest,checkstyle,findbugs etc. trends. However there appears to be a critical performance difference between running the site as a maven2 job and as a freestyle job.

      My build looks like this:

      • 1 parent reactor pom
      • 57 modules in a mix of jar,war,ear and a assembly zip module
      • Some of these modules have unittest
      • Some of these modules contains antrun tasks needed for webservice (server and client),weblogic mbean,weblogic workshop frontend. most of these antruns are wrapped in a "if" task to avoid needless regenerations (like javac skipping compile of class files are up2date)

      I think the problem comes from the large number to "install" lifecycles the report cycles starts in order to be sure the unitests has been run.

      Here are the performance I am seeing running site on this job

      • manual maven on my laptop (Lenovo Thinkpad T61P 2.4 ghz) 30-40 minutes
      • hudson freestyle project: 90 minutes
      • hudson maven2 job (1. attempt): build failed after 11 hours, most likely because og missing dependency (most likely another job was deploying in nexus repo)
      • hudson maven2 job (2. attempt): build appears to be hanging and was killed after 17 hours, the build appear to be approx halfway.

      To give an idea of the general performance, a normal install runs like this:

      • laptop: 4-5 minutes
      • hudson maven2 job: 15 minutes

      So in normal install the build server takes approx 3 times as long. and during site this differece is the same for freestyle projects, but maven2 projects seem take forever.

        1. freestyle_log.txt
          24 kB
        2. freestyle-project.nps
          542 kB
        3. javacore.20100129.023743.364726.txt
          1.96 MB
        4. javacore.20100129.211530.491682.txt
          1.81 MB
        5. javacore.20100131.220945.491684.txt
          2.00 MB
        6. javacore.20100201.114114.524534.txt
          558 kB
        7. javacore.20100201.114955.524534.txt
          558 kB
        8. javacore.20100201.120315.524534.txt
          558 kB
        9. maven2_log.txt
          33 kB
        10. maven2-project.nps
          783 kB

          [JENKINS-5315] Critical performance issu on maven2 jobs during site generation.

          When the build seems to be taking forever, please go to the build page on Hudson, then click "monitor Maven process", and grab the thread dump of the Maven process and attach it here. If you can, do it several times with some time apart so that we can have multiple samples.

          Also, look at the console output as it appears, and show us where a long delay occurs. That helps us find where the problem is.

          Finally, do you do distributed builds? If so, tell us how you connect your master and slave, and if it has enough bandwidth.

          Kohsuke Kawaguchi added a comment - When the build seems to be taking forever, please go to the build page on Hudson, then click "monitor Maven process", and grab the thread dump of the Maven process and attach it here. If you can, do it several times with some time apart so that we can have multiple samples. Also, look at the console output as it appears, and show us where a long delay occurs. That helps us find where the problem is. Finally, do you do distributed builds? If so, tell us how you connect your master and slave, and if it has enough bandwidth.

          Chris Graham added a comment -

          I have a similar but different story to tell.

          I have 222 poms in one project that produces 58 ear files.

          On the build machine, if I run a build via telnet, it takes 222 minutes.

          Hudson M2 jobs have taken 312, 325 and 362 minutes.

          I also experience complete hangs in Hudson. Hudson is running on AIX 5.3 under WAS 6.1 FP19.

          When it hangs, I am unable to open any Hudson page. I am also unable to stop the server. It's lost all communication. The only way to recover from this situation is to manually kill the process, and then restart the server.

          I'm not using distributed builds.

          I am also using the Locks and Latches plugin - I'm not sure if that is related, so I'll try disabling that for a while to see if makes any difference.

          -Chris

          Chris Graham added a comment - I have a similar but different story to tell. I have 222 poms in one project that produces 58 ear files. On the build machine, if I run a build via telnet, it takes 222 minutes. Hudson M2 jobs have taken 312, 325 and 362 minutes. I also experience complete hangs in Hudson. Hudson is running on AIX 5.3 under WAS 6.1 FP19. When it hangs, I am unable to open any Hudson page. I am also unable to stop the server. It's lost all communication. The only way to recover from this situation is to manually kill the process, and then restart the server. I'm not using distributed builds. I am also using the Locks and Latches plugin - I'm not sure if that is related, so I'll try disabling that for a while to see if makes any difference. -Chris

          Chris Graham added a comment -

          I've found the stuck maven process:

          ps -ef | grep java:
          build 524534 491684 0 15:34:11 - 29:45 /was61/WebSphere/AppServer/java/bin//java -Xmx1024M -cp /hudson/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.327.jar:/opt/apache-maven-2.0.9/boot/classworlds-1.1.jar hudson.maven.agent.Main /opt/apache-maven-2.0.9 /was61/WebSphere/AppServer/profiles/AppSrv01/installedApps/localhostCell01/hudson-237_war.ear/hudson-237.war/WEB-INF/lib/remoting-1.327.jar /hudson/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.327.jar 49828

          (ignore the reference to hudson-237.war - I miss typed the app name when I manually installed it. IT IS 1.327).

          kill -3 a few times, a few minutes apart give me the three core files attached.

          I hope this helps.

          -Chris

          Chris Graham added a comment - I've found the stuck maven process: ps -ef | grep java: build 524534 491684 0 15:34:11 - 29:45 /was61/WebSphere/AppServer/java/bin//java -Xmx1024M -cp /hudson/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.327.jar:/opt/apache-maven-2.0.9/boot/classworlds-1.1.jar hudson.maven.agent.Main /opt/apache-maven-2.0.9 /was61/WebSphere/AppServer/profiles/AppSrv01/installedApps/localhostCell01/hudson-237_war.ear/hudson-237.war/WEB-INF/lib/remoting-1.327.jar /hudson/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.327.jar 49828 (ignore the reference to hudson-237.war - I miss typed the app name when I manually installed it. IT IS 1.327). kill -3 a few times, a few minutes apart give me the three core files attached. I hope this helps. -Chris

          Chris Graham added a comment -

          I just re-ran the same project as a freestyle one. It completed in 217 minutes.

          -Chris

          Chris Graham added a comment - I just re-ran the same project as a freestyle one. It completed in 217 minutes. -Chris

          Chris Graham added a comment -

          It appears that the Hudson websphere instance ran out of memory (256Mb!) when the maven job appears to hang.
          So I do not know which is causing the error? Is Hudson simply running out of memory and loosing contact with the Maven job, or is the Maven job hanging and killing the server?

          I don't know which.

          But I've found some coredumps of the websphere instance that was running when the maven job died. I think that the previous javacore files were from the same process that was spawned by these failing websphere instanaces.

          -Chris

          Chris Graham added a comment - It appears that the Hudson websphere instance ran out of memory (256Mb!) when the maven job appears to hang. So I do not know which is causing the error? Is Hudson simply running out of memory and loosing contact with the Maven job, or is the Maven job hanging and killing the server? I don't know which. But I've found some coredumps of the websphere instance that was running when the maven job died. I think that the previous javacore files were from the same process that was spawned by these failing websphere instanaces. -Chris

          lynggaard added a comment -

          freestyle_log.txt shows the freestyle project running at 34 seconds
          maven2_log.txt shows a maven2 project running in 1m 4seconds or almost twice as slow

          Both projects have used the "Timestamp console output" plugin.

          The project in question is a small 2 module maven reactor, each module has 1-2 java classes and one of them depends on the other so nothing fancy.

          The maven target being run is "-e -o install site"

          lynggaard added a comment - freestyle_log.txt shows the freestyle project running at 34 seconds maven2_log.txt shows a maven2 project running in 1m 4seconds or almost twice as slow Both projects have used the "Timestamp console output" plugin. The project in question is a small 2 module maven reactor, each module has 1-2 java classes and one of them depends on the other so nothing fancy. The maven target being run is "-e -o install site"

          lynggaard added a comment -

          Netbeans 6.9.1 Profiling snapshots for two similar projects.Tests done on hudson 1.384 and the newest plugins.

          I can't seem to find the cause,most places just seems to be waiting but perhaps people with better hudson knowledge can find the issue

          lynggaard added a comment - Netbeans 6.9.1 Profiling snapshots for two similar projects.Tests done on hudson 1.384 and the newest plugins. I can't seem to find the cause,most places just seems to be waiting but perhaps people with better hudson knowledge can find the issue

            Unassigned Unassigned
            lynggaard lynggaard
            Votes:
            6 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: