• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • maven-plugin
    • None
    • We're running Jenkins on a RHEL 5.2, 32 bits box, with Java 1.6.0_04. The build agent runs on a RHEL 5.4, 64 bits box, with Java 1.6.0_23.

      Hi. We upgraded our old Hudson server to Jenkins 1.409.2 on Sep, 20th. Since then we're facing bigger build times, like twice the time it used to take with Hudson.

      We're running Jenkins on a RHEL 5.2, 32 bits box, with Java 1.6.0_04. The build agent runs on a RHEL 5.4, 64 bits box, with Java 1.6.0_23.

      I'm attaching the System_Information_Jenkins.html page which contains the configuration of our Jenkins server.

      I'm also attaching the config.xml file of a particular job which shows the problem well.

      I'm also attaching the etics-6.1.x-es_76_Console.html file with the Console output of a particular build. The output lines are prefixed with the time of day (via the timestamper plugin) so that we can see where most of the time was spent. As you can see, the build started at 10:47:17 and ended at 11:26:36, having taken 39 minutes and 19 seconds in the whole. However, as you can see at the end of the log, maven tells us that the build took only 11 minutes and 3 seconds. Most of the remaining 28 minutes and 16 seconds was spent between the lines prefixed with times 10:47:54 and 11:15:21.

      I don't know what the Jenkins server or the agent were doing during this quiet period. I tried to see which processes were busy during this period and the only strange thing I noticed was that there was a process in the agent machine very busy, which was this:

      /l/disk0/ipscate/tools/java/jdk1.6.0_23/bin/java -jar	slave.jar
      

      I straced it during a few seconds and saw that it was busy opening lots of files under the /l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd directory. In fact, in a short period of about 15 or 20 seconds it opened more than 40000 files, some of them being repeatedly opened thousands of times. These were the most often opened ones:

      # sort /tmp/trace | uniq -c | sort -n | tail -5
         3502 32319 open("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/_maven.repositories", O_RDONLY) = 16
         3502 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/cpqd-public-bom-6.5.2.pom", {st_mode=S_IFREG|0644, st_size=45211, ...}) = 0
         3793 32319 open("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/parent/cpqd-super-parent/6.5.2/cpqd-super-parent-6.5.2.pom", O_RDONLY) = 16
         3793 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/parent/cpqd-super-parent/6.5.2/cpqd-super-parent-6.5.2.pom", {st_mode=S_IFREG|0644, st_size=14792, ...}) = 0
         7004 32319 stat("/l/disk0/ipscate/ip/nexus/.m2/repository/br/com/cpqd/public/cpqd-public-bom/6.5.2/_maven.repositories", {st_mode=S_IFREG|0644, st_size=163, ...}) = 0
      

      I'm not sure this behaviour is directly related with the delay I was seeing in the build, but it doesn't seem right to me.

      Do you have any suggestion as to how can I go about investigating this?

        1. config.xml
          3 kB
        2. etics-6.1.x-es_76_Console.html
          1.19 MB
        3. fine-stack-traces.zip
          958 kB
        4. JENKINS-11362_stack-traces.zip
          32 kB
        5. m3.log.gz
          8 kB
        6. simplificado.zip
          3.05 MB
        7. simplificado-config.xml
          2 kB
        8. System_Information_Jenkins.html
          24 kB

          [JENKINS-11362] Strange delay during a maven job build

          Jesse Glick added a comment -

          Found http://jira.codehaus.org/browse/MNG-5312 as the root cause. The hudson.maven.PomInfo constructor calls MavenProject.getParent and this can be exceedingly slow. Options for Jenkins include

          1. Wait for a Maven release (3.0.5?) with the performance problem fixed.
          2. Bundle a patched version of maven-core with the plugin.
          3. Stop calling getParent from PomInfo, at least at the user's option. But this might cause Jenkins to be missing dependency information needed for build triggers: an update to a parent POM (other than the relativePath default of ../pom.xml) would not be "noticed" by modules using it.

          Provisionally going with option #1, hence closing this issue since we would anyway bundle any new Maven release with the plugin when it becomes available. But I could work on #2 or #3 instead if anyone thinks it would be a good idea. Unclear if many users are affected by this issue; the impact is on projects using lots of <scope>import</scope>.

          Jesse Glick added a comment - Found http://jira.codehaus.org/browse/MNG-5312 as the root cause. The hudson.maven.PomInfo constructor calls MavenProject.getParent and this can be exceedingly slow. Options for Jenkins include Wait for a Maven release (3.0.5?) with the performance problem fixed. Bundle a patched version of maven-core with the plugin. Stop calling getParent from PomInfo , at least at the user's option. But this might cause Jenkins to be missing dependency information needed for build triggers: an update to a parent POM (other than the relativePath default of ../pom.xml ) would not be "noticed" by modules using it. Provisionally going with option #1, hence closing this issue since we would anyway bundle any new Maven release with the plugin when it becomes available. But I could work on #2 or #3 instead if anyone thinks it would be a good idea. Unclear if many users are affected by this issue; the impact is on projects using lots of <scope>import</scope> .

          Jesse Glick added a comment -

          Note: 3.1.0-alpha-1 with this fix has been proposed. Looks like it will take close to a year to get this simple fix into an official release, sigh.

          Jesse Glick added a comment - Note: 3.1.0-alpha-1 with this fix has been proposed. Looks like it will take close to a year to get this simple fix into an official release, sigh.

          Jesse Glick added a comment -

          3.1.0 finally released.

          Jesse Glick added a comment - 3.1.0 finally released.

          Jesse Glick added a comment -

          Playing with a Maven 3.1.0 upgrade but it is not trivial, due to Aether and SLF4J changes. Created maven-3.1.0 branches in lib-jenkins-maven-embedder and jenkins. @olamy do you want to pick up the rest, specifically making Maven3Builder and EmbedderLoggerImpl compilable and testing the result?

          Jesse Glick added a comment - Playing with a Maven 3.1.0 upgrade but it is not trivial, due to Aether and SLF4J changes. Created maven-3.1.0 branches in lib-jenkins-maven-embedder and jenkins . @olamy do you want to pick up the rest, specifically making Maven3Builder and EmbedderLoggerImpl compilable and testing the result?

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          pom.xml
          http://jenkins-ci.org/commit/maven-plugin/ebf86b4567dfec37c9e21a4491ff7ca27a77d0c9
          Log:
          Using lib-jenkins-maven-embedder version with workaround for https://jira.codehaus.org/browse/MNG-5125.
          I still hope this gives some ease regarding JENKINS-11362
          Originally-Committed-As: f8774f0599850a1c920c4da9e058009b77219703

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: pom.xml http://jenkins-ci.org/commit/maven-plugin/ebf86b4567dfec37c9e21a4491ff7ca27a77d0c9 Log: Using lib-jenkins-maven-embedder version with workaround for https://jira.codehaus.org/browse/MNG-5125 . I still hope this gives some ease regarding JENKINS-11362 Originally-Committed-As: f8774f0599850a1c920c4da9e058009b77219703

          Two weeks ago we've upgraded to 1.530 the Jenkins instance that was causing us this problem and it seems to be working ok. The Maven 3.1.0 release finally incorporated the commits that solved the problem (e778ea6~3..e778ea6) and when it was incorporated in Jenkins 1.526 the problem has finally been solved.

          Big thanks to kutzi and to jglick who worked on this.

          Gustavo Chaves added a comment - Two weeks ago we've upgraded to 1.530 the Jenkins instance that was causing us this problem and it seems to be working ok. The Maven 3.1.0 release finally incorporated the commits that solved the problem (e778ea6~3..e778ea6) and when it was incorporated in Jenkins 1.526 the problem has finally been solved. Big thanks to kutzi and to jglick who worked on this.

          Amgad Hanafy added a comment - - edited

          I have same issue when running maven build on slave node
          Jenkins ver. 1.580
          maven 3.2.1
          master node : AIX 7.1
          Slave node windows 7 64bit

          please advise

          Amgad Hanafy added a comment - - edited I have same issue when running maven build on slave node Jenkins ver. 1.580 maven 3.2.1 master node : AIX 7.1 Slave node windows 7 64bit please advise

          Amgad Hanafy added a comment -

          still have this issue

          Amgad Hanafy added a comment - still have this issue

          I'd say unless it is a regression it must be another problem because the bug was fixed on Jenkins 1.526.

          Are you using maven style or free style jobs?

          A while ago we decided to ditch maven style and go for free style jobs due to other considerations. However, since only maven style jobs use the maven bundled on Jenkins to preprocess POMs it may be the source of your problems too. If you can, try to use a free style job instead and see if it solves the delay problem.

          Gustavo Chaves added a comment - I'd say unless it is a regression it must be another problem because the bug was fixed on Jenkins 1.526. Are you using maven style or free style jobs? A while ago we decided to ditch maven style and go for free style jobs due to other considerations . However, since only maven style jobs use the maven bundled on Jenkins to preprocess POMs it may be the source of your problems too. If you can, try to use a free style job instead and see if it solves the delay problem.

          Jesse Glick added a comment -

          amgad whatever issue you are seeing may or may not be related. Would need to be separately diagnosed, or filed with a reproducible test case. Anyway you need to specify the Maven plugin version at a bare minimum.

          Jesse Glick added a comment - amgad whatever issue you are seeing may or may not be related. Would need to be separately diagnosed, or filed with a reproducible test case. Anyway you need to specify the Maven plugin version at a bare minimum.

            olamy Olivier Lamy
            gnustavo Gustavo Chaves
            Votes:
            3 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: