Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19712

[Fingerprint] Excessive Heap consumption during "Waiting for Jenkins to finish collecting data"

    XMLWordPrintable

Details

    Description

      Fingerprinting has always been a bit pain in the ... that's why i thought we could improve performance of our maven build jobs by upgrading from 1.480.3 to 1.509.3 which promised some fixes regarding Fingerprinting.

      Before the upgrade (using 1.480.3) we experienced Fingerprinting to take time between 5min and 15min on average for builds of the size of between 5 to 30 modules.

      Actually it got way worse after upgrade to 1.509.3:
      i observed huge amounts of data being loaded into heap which hit into old gen space pretty much instantly because of the huge amount being loaded into heap. so we ended up with running full GCs as long as there were builds in the state of "Waiting for Jenkins to finish collecting data".

      we then downgraded to 1.509.2 which worked out pretty well so far.

      we observed these problems with:

      • jenkins 1.509.3
      • tomcat 6
      • jdk 6
      • jvm-options: -XX:MaxPermSize=512M -Xmx6G -Xms6G -XX:NewRatio=4 -XX:SurvivorRatio=6 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
      • centos 5.5 final

      we also observed many blocked threads during Fingerprinting-phases. unfortunately i didnt keep the whole thread dump log output, but the were blocked on hudson.model.Fingerprint.add(Fingerprint.java:717)

      I attached some screenshots of our jvm-monitoring where you can clearly see the difference between 1.509.3 (many Full GC runs) and 1.509.2 (zero Full GC runs)
      i also tuned the NewRatio jvm-option a bit when i downgraded to 1.509.2

      I have to say that this issue is kind of a big hit to us, we won't be able to upgrade to any jenkins version containing these Fingerprint code changes within 1.509.3 as long as fingerprinting is still a mandatory step!

      Attachments

        Issue Links

          Activity

            mcklaus Klaus Azesberger created issue -
            seder Stefan Prietl made changes -
            Field Original Value New Value
            Attachment jenkins_collectingdata_issue_cpuutil.png [ 24528 ]
            Attachment jenkins_collectingdata_issue_cpuload.png [ 24529 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-11333 [ JENKINS-11333 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-18417 [ JENKINS-18417 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 151230 ] JNJira + In-Review [ 177890 ]

            People

              Unassigned Unassigned
              mcklaus Klaus Azesberger
              Votes:
              13 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: