Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Critical
    • Resolution: Fixed
    • core

    Description

      Jenkins starts using 100% CPU after a few days. Using jstack I see several threads trying to write compressed output, and apparently not changing over time:

      • java.util.zip.Deflater.deflate(byte[], int, int) @bci=55, line=322 (Compiled frame; information may be imprecise)
      • java.util.zip.DeflaterOutputStream.deflate() @bci=14, line=176 (Compiled frame)
      • java.util.zip.DeflaterOutputStream.write(byte[], int, int) @bci=108, line=135 (Compiled frame)
      • java.util.zip.GZIPOutputStream.write(byte[], int, int) @bci=4, line=89 (Compiled frame)
      • org.kohsuke.stapler.compression.FilterServletOutputStream.write(byte[], int, int) @bci=7, line=31 (Compiled frame)
      • sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=220 (Interpreted frame)
      • sun.nio.cs.StreamEncoder.implClose() @bci=84, line=315 (Interpreted frame)
      • sun.nio.cs.StreamEncoder.close() @bci=18, line=148 (Interpreted frame)
      • java.io.OutputStreamWriter.close() @bci=4, line=233 (Interpreted frame)
      • java.io.PrintWriter.close() @bci=21, line=312 (Interpreted frame)
      • org.kohsuke.stapler.compression.CompressionFilter.reportException(java.lang.Exception, javax.servlet.http.HttpServletResponse) @bci=112, line=77 (Interpreted frame)
      • org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=53 (Compiled frame)
      • winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
      • winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
      • hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Compiled frame)
      • winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame)
      • winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame)
      • winstone.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=483, line=331 (Compiled frame)
      • winstone.RequestHandlerThread.processRequest(winstone.WebAppConfiguration, winstone.WinstoneRequest, winstone.WinstoneResponse, java.lang.String) @bci=38, line=215 (Compiled frame)
      • winstone.RequestHandlerThread.run() @bci=631, line=138 (Compiled frame)
      • java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
      • java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame)
      • java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame)
      • winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Compiled frame)
      • java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame)
      • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame)
      • java.lang.Thread.run() @bci=11, line=679 (Interpreted frame)

      I'm suspecting but not 100% sure that these threads are in an infinite loop (livelocked). I'm struggling to see what other threads might be doing this.

      This JVM was not started with debugging enabled to attach a debugger for analysis. I've enabled it now. Stack traces attached as files below.

      Attachments

        1. jenkins.stacktrace.1
          53 kB
        2. jenkins.stacktrace.2
          51 kB
        3. jenkins.stacktrace.3
          46 kB
        4. thread-dump-prod.txt
          134 kB
        5. thread-dump-reproduce-1.txt
          72 kB
        6. thread-dump-reproduce-2.txt
          72 kB
        7. thread-dump-reproduce-3.txt
          72 kB

        Issue Links

          Activity

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/hudson/maven/AbstractMavenProcessFactory.java
            src/main/java/hudson/maven/AggregatingClassLoader.java
            src/main/java/hudson/maven/Maven3Builder.java
            src/main/java/hudson/maven/Maven3ProcessFactory.java
            src/main/java/hudson/maven/MavenBuildInformation.java
            src/main/java/hudson/maven/MavenEmbedderRequest.java
            src/main/java/hudson/maven/MavenModuleSet.java
            src/main/java/hudson/maven/MavenModuleSetBuild.java
            src/main/java/hudson/maven/MavenProbeAction.java
            src/main/java/hudson/maven/MavenProcessFactory.java
            src/main/java/hudson/maven/MavenUtil.java
            src/main/java/hudson/maven/PlexusModuleContributor.java
            src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java
            src/main/java/hudson/maven/reporters/MavenFingerprinter.java
            src/main/java/hudson/maven/reporters/TestMojo.java
            src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties
            src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties
            src/main/resources/hudson/maven/Messages.properties
            src/main/resources/hudson/maven/Messages_es.properties
            src/main/resources/hudson/maven/Messages_zh_TW.properties
            src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties
            src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
            src/test/java/hudson/maven/reporters/TestMojoTest.java
            http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e
            Log:
            Merge branch 'master' into JENKINS-14362-jzlib

            Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/hudson/maven/AbstractMavenProcessFactory.java src/main/java/hudson/maven/AggregatingClassLoader.java src/main/java/hudson/maven/Maven3Builder.java src/main/java/hudson/maven/Maven3ProcessFactory.java src/main/java/hudson/maven/MavenBuildInformation.java src/main/java/hudson/maven/MavenEmbedderRequest.java src/main/java/hudson/maven/MavenModuleSet.java src/main/java/hudson/maven/MavenModuleSetBuild.java src/main/java/hudson/maven/MavenProbeAction.java src/main/java/hudson/maven/MavenProcessFactory.java src/main/java/hudson/maven/MavenUtil.java src/main/java/hudson/maven/PlexusModuleContributor.java src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java src/main/java/hudson/maven/reporters/MavenFingerprinter.java src/main/java/hudson/maven/reporters/TestMojo.java src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties src/main/resources/hudson/maven/Messages.properties src/main/resources/hudson/maven/Messages_es.properties src/main/resources/hudson/maven/Messages_zh_TW.properties src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java src/test/java/hudson/maven/reporters/TestMojoTest.java http://jenkins-ci.org/commit/maven-plugin/14863a0498e0e6c389f2779b9150e9280b363d3e Log: Merge branch 'master' into JENKINS-14362 -jzlib Originally-Committed-As: dedb24ef034a04fea288eaa50089e8e774714e99

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/resources/hudson/maven/Messages.properties
            src/main/resources/hudson/maven/Messages_de.properties
            src/main/resources/hudson/maven/RedeployPublisher/config_de.properties
            src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties
            http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6
            Log:
            Merge branch 'master' into JENKINS-14362-jzlib

            Conflicts:
            core/pom.xml

            Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/resources/hudson/maven/Messages.properties src/main/resources/hudson/maven/Messages_de.properties src/main/resources/hudson/maven/RedeployPublisher/config_de.properties src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties http://jenkins-ci.org/commit/maven-plugin/d75a8c218b6ffd24fe2ddea8953ffaf22da63bc6 Log: Merge branch 'master' into JENKINS-14362 -jzlib Conflicts: core/pom.xml Originally-Committed-As: b7e2d6127f77711681113a020681508d57622ea5
            bbonn bbonn added a comment -

            Hi There,

            Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.

            bbonn bbonn added a comment - Hi There, Just verifying. I see this bug has a 1.509.4 fixed label, but on Jenkins-CI.org, it is not listed in the change log until 1.520. Can someone please validate where the fix is? Thanks a lot.
            jglick Jesse Glick added a comment -

            It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.

            jglick Jesse Glick added a comment - It was first fixed in trunk in 1.520, then backported to 1.509.4. http://jenkins-ci.org/changelog-stable neglects to mention it, but that changelog is generally unreliable anyway.
            danielcranford Daniel added a comment -

            FWIW, this appears to be the same bug as this JDK bugĀ https://bugs.openjdk.java.net/browse/JDK-8193682

            danielcranford Daniel added a comment - FWIW, this appears to be the same bug as this JDK bugĀ  https://bugs.openjdk.java.net/browse/JDK-8193682

            People

              lmcazra Audrey Azra
              gcc Chris Wilson
              Votes:
              44 Vote for this issue
              Watchers:
              62 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: