Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Critical
    • Resolution: Fixed
    • core

    Description

      Jenkins starts using 100% CPU after a few days. Using jstack I see several threads trying to write compressed output, and apparently not changing over time:

      • java.util.zip.Deflater.deflate(byte[], int, int) @bci=55, line=322 (Compiled frame; information may be imprecise)
      • java.util.zip.DeflaterOutputStream.deflate() @bci=14, line=176 (Compiled frame)
      • java.util.zip.DeflaterOutputStream.write(byte[], int, int) @bci=108, line=135 (Compiled frame)
      • java.util.zip.GZIPOutputStream.write(byte[], int, int) @bci=4, line=89 (Compiled frame)
      • org.kohsuke.stapler.compression.FilterServletOutputStream.write(byte[], int, int) @bci=7, line=31 (Compiled frame)
      • sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=220 (Interpreted frame)
      • sun.nio.cs.StreamEncoder.implClose() @bci=84, line=315 (Interpreted frame)
      • sun.nio.cs.StreamEncoder.close() @bci=18, line=148 (Interpreted frame)
      • java.io.OutputStreamWriter.close() @bci=4, line=233 (Interpreted frame)
      • java.io.PrintWriter.close() @bci=21, line=312 (Interpreted frame)
      • org.kohsuke.stapler.compression.CompressionFilter.reportException(java.lang.Exception, javax.servlet.http.HttpServletResponse) @bci=112, line=77 (Interpreted frame)
      • org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=53 (Compiled frame)
      • winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
      • winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
      • hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Compiled frame)
      • winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
      • winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
      • winstone.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=483, line=331 (Compiled frame)
      • winstone.RequestHandlerThread.processRequest(winstone.WebAppConfiguration, winstone.WinstoneRequest, winstone.WinstoneResponse, java.lang.String) @bci=38, line=215 (Compiled frame)
      • winstone.RequestHandlerThread.run() @bci=631, line=138 (Compiled frame)
      • java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
      • java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
      • java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
      • winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Compiled frame)
      • java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame)
      • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)
      • java.lang.Thread.run() @bci=11, line=679 (Interpreted frame)

      I'm suspecting but not 100% sure that these threads are in an infinite loop (livelocked). I'm struggling to see what other threads might be doing this.

      This JVM was not started with debugging enabled to attach a debugger for analysis. I've enabled it now. Stack traces attached as files below.

      Attachments

        1. jenkins.stacktrace.1
          53 kB
        2. jenkins.stacktrace.2
          51 kB
        3. jenkins.stacktrace.3
          46 kB
        4. thread-dump-prod.txt
          134 kB
        5. thread-dump-reproduce-1.txt
          72 kB
        6. thread-dump-reproduce-2.txt
          72 kB
        7. thread-dump-reproduce-3.txt
          72 kB

        Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-19473 failed to archive slave artifacts. Unexpected end of ZLIB input stream

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-14361 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-18513 Slowness in Jenkins

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-19346 100% on one CPU

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-13625 ERR_CONTENT_DECODING_FAILED returned on testResults and console output after Jenkins reload

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-17349 CPU on master node keeps increasing and never come down

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 65 older comments
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/hudson/maven/AbstractMavenProcessFactory.java
            src/main/java/hudson/maven/AggregatingClassLoader.java
            src/main/java/hudson/maven/Maven3Builder.java
            src/main/java/hudson/maven/Maven3ProcessFactory.java
            src/main/java/hudson/maven/MavenBuildInformation.java
            src/main/java/hudson/maven/MavenEmbedderRequest.java
            src/main/java/hudson/maven/MavenModuleSet.java
            src/main/java/hudson/maven/MavenModuleSetBuild.java
            src/main/java/hudson/maven/MavenProbeAction.java
            src/main/java/hudson/maven/MavenProcessFactory.java
            src/main/java/hudson/maven/MavenUtil.java
            src/main/java/hudson/maven/PlexusModuleContributor.java
            src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java
            src/main/java/hudson/maven/reporters/MavenFingerprinter.java
            src/main/java/hudson/maven/reporters/TestMojo.java
            src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties
            src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties
            src/main/resources/hudson/maven/Messages.properties
            src/main/resources/hudson/maven/Messages_es.properties
            src/main/resources/hudson/maven/Messages_zh_TW.properties
            src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties
            src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
            src/test/java/hudson/maven/reporters/TestMojoTest.java
            http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e
            Log:
            Merge branch 'master' into JENKINS-14362-jzlib

            Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/hudson/maven/AbstractMavenProcessFactory.java src/main/java/hudson/maven/AggregatingClassLoader.java src/main/java/hudson/maven/Maven3Builder.java src/main/java/hudson/maven/Maven3ProcessFactory.java src/main/java/hudson/maven/MavenBuildInformation.java src/main/java/hudson/maven/MavenEmbedderRequest.java src/main/java/hudson/maven/MavenModuleSet.java src/main/java/hudson/maven/MavenModuleSetBuild.java src/main/java/hudson/maven/MavenProbeAction.java src/main/java/hudson/maven/MavenProcessFactory.java src/main/java/hudson/maven/MavenUtil.java src/main/java/hudson/maven/PlexusModuleContributor.java src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java src/main/java/hudson/maven/reporters/MavenFingerprinter.java src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties src/main/resources/hudson/maven/Messages.properties src/main/resources/hudson/maven/Messages_es.properties src/main/resources/hudson/maven/Messages_zh_TW.properties src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e Log: Merge branch 'master' into JENKINS-14362 -jzlib Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/resources/hudson/maven/Messages.properties
            src/main/resources/hudson/maven/Messages_de.properties
            src/main/resources/hudson/maven/RedeployPublisher/config_de.properties
            src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties
            http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6
            Log:
            Merge branch 'master' into JENKINS-14362-jzlib

            Conflicts:
            core/pom.xml

            Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/resources/hudson/maven/Messages.properties src/main/resources/hudson/maven/Messages_de.properties src/main/resources/hudson/maven/RedeployPublisher/config_de.properties src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6 Log: Merge branch 'master' into JENKINS-14362 -jzlib Conflicts: core/pom.xml Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5
            bbonn bbonn added a comment -

            Hi There,

            Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.

            bbonn bbonn added a comment - Hi There, Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.
            jglick Jesse Glick added a comment -

            It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.

            jglick Jesse Glick added a comment - It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.
            danielcranford Daniel added a comment -

            FWIW, this appears to be the same bug as this JDK bug https://bugs.openjdk.java.net/browse/JDK-8193682

            danielcranford Daniel added a comment - FWIW, this appears to be the same bug as this JDK bug  https://bugs.openjdk.java.net/browse/JDK-8193682

            People

              lmcazra Audrey Azra
              gcc Chris Wilson
              Votes:
              44 Vote for this issue
              Watchers:
              62 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: