Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17349

CPU on master node keeps increasing and never come down

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • all-changes-plugin

      We are using Jenkins ver. 1.480.3, upgraded on Mar 1st.
      During the past month, the CPU usage continuously creasing...
      The 1st week: <5%
      The 2ed week: ~=17%
      The 3ed week: ~= 25%,
      And today it increasd to ~= 44%!!!

        1. CPU.png
          CPU.png
          18 kB
        2. CPU incease pic.jpg
          CPU incease pic.jpg
          43 kB
        3. JavaMelody__CNRDGPS_3_26_13 (1).pdf
          805 kB
        4. JavaMelody__CNRDGPS_4_8_13.pdf
          813 kB
        5. percent http errors.png
          percent http errors.png
          23 kB

          [JENKINS-17349] CPU on master node keeps increasing and never come down

          sharon xia created issue -

          evernat added a comment - - edited

          I have joined 2 interesting graphics from the PDF: % CPU for 1 month and % http errors for 1 month.

          It is strange to me that % CPU increases suddenly, about each week. (Note: this graphic is the %CPU of the Jenkins process, and not of the OS.)
          And in the second graph, it seems that you have many http errors lately.

          From the joined PDF:
          It can be seen in the http statistics, that you have some downloads of very large artifacts (100 to 400 Mb) taking a long time.
          And most importantly, in the "Detailed system information" from the PDF, it says:
          Perm gen memory : 80 Mb / 82 Mb, which is quite full.

          I suggest first to add a jvm parameter, probably in JENKINS_HOME/jenkins.xml: -XX:MaxPermSize=256m
          And restart Jenkins.

          evernat added a comment - - edited I have joined 2 interesting graphics from the PDF: % CPU for 1 month and % http errors for 1 month. It is strange to me that % CPU increases suddenly, about each week. (Note: this graphic is the %CPU of the Jenkins process, and not of the OS.) And in the second graph, it seems that you have many http errors lately. From the joined PDF: It can be seen in the http statistics, that you have some downloads of very large artifacts (100 to 400 Mb) taking a long time. And most importantly, in the "Detailed system information" from the PDF, it says: Perm gen memory : 80 Mb / 82 Mb, which is quite full. I suggest first to add a jvm parameter, probably in JENKINS_HOME/jenkins.xml: -XX:MaxPermSize=256m And restart Jenkins.
          evernat made changes -
          Attachment New: CPU.png [ 23339 ]
          evernat made changes -
          Attachment New: percent http errors.png [ 23340 ]

          sharon xia added a comment -

          Good catch!
          Today I found another thing: There is a slave node(used to do our daily build), in which we have many branch build jobs connected to it.

          I found the sudden increases were caused when a branch build started, and the CPU didn't decreased even if the branch build finished. However, during this week we might have >20 branch builds and only 1 of 2 builds caused the cpu sudden increase.

          If I disconnected the slave build node, then the cup will decrease from the plug-in, althought from task manager the java.exe still keeps high. And after I reconnect the slave build node back, the cpu went back to high.

          sharon xia added a comment - Good catch! Today I found another thing: There is a slave node(used to do our daily build), in which we have many branch build jobs connected to it. I found the sudden increases were caused when a branch build started, and the CPU didn't decreased even if the branch build finished. However, during this week we might have >20 branch builds and only 1 of 2 builds caused the cpu sudden increase. If I disconnected the slave build node, then the cup will decrease from the plug-in, althought from task manager the java.exe still keeps high. And after I reconnect the slave build node back, the cpu went back to high.

          sharon xia added a comment -

          Perm gen memory : 80 Mb / 82 Mb, which is quite full.
          This should not be the direct reason that CPU incresed, correct? I checked it will changes from 80Mb to 81Mb, 82Mb and turned to be 80Mb again...

          sharon xia added a comment - Perm gen memory : 80 Mb / 82 Mb, which is quite full. This should not be the direct reason that CPU incresed, correct? I checked it will changes from 80Mb to 81Mb, 82Mb and turned to be 80Mb again...

          evernat added a comment -

          The limited perm gen may cause some cascading problems, for example in GC and in class loading, and perhaps introduce an overhead in CPU.

          If your large artifacts are copied between node and master or sent from master to browsers, this may also be a cause of high cpu usage. In fact, there are some other Jenkins issues, with some threads currently executing "java.util.zip.Deflater.deflateBytes", like your threads.

          The recent fix of JENKINS-7813, in the future Jenkins v1.509, may help perhaps for communication between node and master.

          evernat added a comment - The limited perm gen may cause some cascading problems, for example in GC and in class loading, and perhaps introduce an overhead in CPU. If your large artifacts are copied between node and master or sent from master to browsers, this may also be a cause of high cpu usage. In fact, there are some other Jenkins issues, with some threads currently executing "java.util.zip.Deflater.deflateBytes", like your threads. The recent fix of JENKINS-7813 , in the future Jenkins v1.509, may help perhaps for communication between node and master.

          sharon xia added a comment -

          Thanks evernat!

          I've modified the configuration file and restart the server(restart Jenkins service won't take effect), and the CPU goes back to <5%, let's wait for longer time to check whether it will address the CPU increase issue.

          JVM arguments: -Xrs
          -Xmx2048m
          -XX:MaxPermSize=512m
          -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle

          And the large artifact download are expected users' behavior. Currently the download speed is fast so I don't think there is an issue.

          sharon xia added a comment - Thanks evernat! I've modified the configuration file and restart the server(restart Jenkins service won't take effect), and the CPU goes back to <5%, let's wait for longer time to check whether it will address the CPU increase issue. JVM arguments: -Xrs -Xmx2048m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle And the large artifact download are expected users' behavior. Currently the download speed is fast so I don't think there is an issue.

          sharon xia added a comment -

          JavaMelody pdf after memory inceased.

          sharon xia added a comment - JavaMelody pdf after memory inceased.
          sharon xia made changes -
          Attachment New: JavaMelody__CNRDGPS_4_8_13.pdf [ 23413 ]

            wolfs Stefan Wolf
            sharon_xia sharon xia
            Votes:
            6 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: