Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17349

CPU on master node keeps increasing and never come down

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • all-changes-plugin

      We are using Jenkins ver. 1.480.3, upgraded on Mar 1st.
      During the past month, the CPU usage continuously creasing...
      The 1st week: <5%
      The 2ed week: ~=17%
      The 3ed week: ~= 25%,
      And today it increasd to ~= 44%!!!

        1. CPU.png
          CPU.png
          18 kB
        2. CPU incease pic.jpg
          CPU incease pic.jpg
          43 kB
        3. JavaMelody__CNRDGPS_3_26_13 (1).pdf
          805 kB
        4. JavaMelody__CNRDGPS_4_8_13.pdf
          813 kB
        5. percent http errors.png
          percent http errors.png
          23 kB

          [JENKINS-17349] CPU on master node keeps increasing and never come down

          sharon xia added a comment -

          After the JVM arguments changed to -Xmx2048m -XX:MaxPermSize=512m, there is still a CPU increase after several days run.

          sharon xia added a comment - After the JVM arguments changed to -Xmx2048m -XX:MaxPermSize=512m, there is still a CPU increase after several days run.

          sharon xia added a comment -

          In the middle of the picture, the CPU decreased dramatically after a system restart to let new parameters take effect. However, after about one week, the cpu increased to ~= 6%.

          sharon xia added a comment - In the middle of the picture, the CPU decreased dramatically after a system restart to let new parameters take effect. However, after about one week, the cpu increased to ~= 6%.

          evernat added a comment -

          Ok for MaxPermSize.

          At the end of the first report (cpu = 31%), there were 4 "RequestHandler[...]" threads with very high cpu usage. They were executing "java.util.zip.Deflater.deflateBytes(Native Method)".
          At the end of the second report (cpu = 6%), there was 1 "RequestHandlerThread2944" thread with high cpu usage. It was executing "java.util.zip.Deflater.deflateBytes(Native Method)".

          If you look at the threads at the bottom of the report today, is "RequestHandlerThread2944" still there and executing "java.util.zip.Deflater.deflateBytes(Native Method)"?
          Are there other threads like this doing the same thing today?

          If yes, what are the stack-traces of these threads? You may click on "Dump threads as text" below the threads table.

          Note that you can kill those threads with the red button at the right of the table, it may decrease the cpu used. But that's not a long term solution.

          evernat added a comment - Ok for MaxPermSize. At the end of the first report (cpu = 31%), there were 4 "RequestHandler [...] " threads with very high cpu usage. They were executing "java.util.zip.Deflater.deflateBytes(Native Method)". At the end of the second report (cpu = 6%), there was 1 "RequestHandlerThread 2944 " thread with high cpu usage. It was executing "java.util.zip.Deflater.deflateBytes(Native Method)". If you look at the threads at the bottom of the report today, is "RequestHandlerThread 2944 " still there and executing "java.util.zip.Deflater.deflateBytes(Native Method)"? Are there other threads like this doing the same thing today? If yes, what are the stack-traces of these threads? You may click on "Dump threads as text" below the threads table. Note that you can kill those threads with the red button at the right of the table, it may decrease the cpu used. But that's not a long term solution.

          Michael Tharp added a comment -

          I'm seeing this on 1.511 but have not yet tried adjusting launch options. Each "step" is another thread consuming 100% CPU. The one thread I have stuck right now is also in java.util.zip.Deflater.deflateBytes.

          Michael Tharp added a comment - I'm seeing this on 1.511 but have not yet tried adjusting launch options. Each "step" is another thread consuming 100% CPU. The one thread I have stuck right now is also in java.util.zip.Deflater.deflateBytes.

          Roger Myung added a comment -

          We have this issue too.
          Two flags that we've tried unsuccessfully are
          --handlerCountMax=100 --handlerCountMaxIdle=20

          I'm curious if there is a performance difference between Winstone and Tomcat. I was planning on moving to Tomcat to try to separate the web server process from the core Jenkins process.

          Roger Myung added a comment - We have this issue too. Two flags that we've tried unsuccessfully are --handlerCountMax=100 --handlerCountMaxIdle=20 I'm curious if there is a performance difference between Winstone and Tomcat. I was planning on moving to Tomcat to try to separate the web server process from the core Jenkins process.

          Daniel Pinyol added a comment -

          We have this issue too with 1.514

          Daniel Pinyol added a comment - We have this issue too with 1.514

          I have this issue with 1.515 as well. Forgive my ignorance, but how is limiting the memory with "-Xmx2048m -XX:MaxPermSize=512m" going to limit CPU usage?

          Martin-Louis Bright added a comment - I have this issue with 1.515 as well. Forgive my ignorance, but how is limiting the memory with "-Xmx2048m -XX:MaxPermSize=512m" going to limit CPU usage?

          Yinghua Wang added a comment -

          I have the same issue on the latest 1.517. The master CPU usage keeps increasing to 400% until restart the Jenkins service - it drops to 95%.

          No such problem on 1.502 before I upgrade to 1.517.

          Yinghua Wang added a comment - I have the same issue on the latest 1.517. The master CPU usage keeps increasing to 400% until restart the Jenkins service - it drops to 95%. No such problem on 1.502 before I upgrade to 1.517.

          evernat added a comment -

          This issue may be related to the JENKINS-14362 issue, where Jesse Glick has submitted in the comments, a patched Jenkins 1.513 for anyone to test.

          See https://issues.jenkins-ci.org/browse/JENKINS-14362?focusedCommentId=179526&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-179526

          evernat added a comment - This issue may be related to the JENKINS-14362 issue, where Jesse Glick has submitted in the comments, a patched Jenkins 1.513 for anyone to test. See https://issues.jenkins-ci.org/browse/JENKINS-14362?focusedCommentId=179526&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-179526

          evernat added a comment -

          resolving this issue as fixed, since the related issue JENKINS-14362 was resolved as fixed (for Jenkins v1.520)

          evernat added a comment - resolving this issue as fixed, since the related issue JENKINS-14362 was resolved as fixed (for Jenkins v1.520)

            wolfs Stefan Wolf
            sharon_xia sharon xia
            Votes:
            6 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: