Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21649

High CPU Usage on master while slaves working (multijob)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: multijob-plugin
    • Labels:
      None
    • Environment:
      Ubuntu 12.04, Oracle Java 7
      java version "1.7.0_51"
      Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
      Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
    • Similar Issues:

      Description

      When a slave is running a build in a multijob plugin, the monitoring thread on the master eats up 100% of a CPU core.
      This scales linearly-- so when I have 4 parallel jobs running, jenkins master is using 400% cpu.

      Attaching screen cap of profiler output, as well as YourKit profiler snapshot.

        Attachments

          Activity

          Hide
          ralf Ralf added a comment -

          I have made the same observation using:
          Jenkins: 1.549
          Multijob Plugin: 1.11

          Show
          ralf Ralf added a comment - I have made the same observation using: Jenkins: 1.549 Multijob Plugin: 1.11
          Hide
          patriks Patrik Schalin added a comment -

          Seeing the same on Jenkins 1.532.1 with MultiJob PLugin 1.12.

          Typically its the pool threads that are heavy on CPU...

          "pool-26-thread-1" prio=5 RUNNABLE
          	java.util.Hashtable.get(Unknown Source)
          	java.text.NumberFormat.getInstance(Unknown Source)
          	java.text.NumberFormat.getInstance(Unknown Source)
          	java.text.MessageFormat.subformat(Unknown Source)
          	java.text.MessageFormat.format(Unknown Source)
          	java.text.Format.format(Unknown Source)
          	java.text.MessageFormat.format(Unknown Source)
          	org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139)
          	hudson.Messages.Util_second(Messages.java:405)
          	hudson.Util.getTimeSpanString(Util.java:682)
          	hudson.model.Run.getDurationString(Run.java:697)
          	com.tikal.jenkins.plugins.multijob.MultiJobBuilder.updateSubBuild(MultiJobBuilder.java:292)
          	com.tikal.jenkins.plugins.multijob.MultiJobBuilder.access$000(MultiJobBuilder.java:60)
          	com.tikal.jenkins.plugins.multijob.MultiJobBuilder$SubJobWorker.run(MultiJobBuilder.java:200)
          	java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          	java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	java.lang.Thread.run(Unknown Source)
          
          "pool-26-thread-2" prio=5 WAITING
          	hudson.remoting.AsyncFutureImpl.isCancelled(AsyncFutureImpl.java:64)
          	com.tikal.jenkins.plugins.multijob.MultiJobBuilder$SubJobWorker.run(MultiJobBuilder.java:194)
          	java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
          	java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	java.lang.Thread.run(Unknown Source)
          
          Show
          patriks Patrik Schalin added a comment - Seeing the same on Jenkins 1.532.1 with MultiJob PLugin 1.12. Typically its the pool threads that are heavy on CPU... "pool-26-thread-1" prio=5 RUNNABLE java.util.Hashtable.get(Unknown Source) java.text.NumberFormat.getInstance(Unknown Source) java.text.NumberFormat.getInstance(Unknown Source) java.text.MessageFormat.subformat(Unknown Source) java.text.MessageFormat.format(Unknown Source) java.text.Format.format(Unknown Source) java.text.MessageFormat.format(Unknown Source) org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139) hudson.Messages.Util_second(Messages.java:405) hudson.Util.getTimeSpanString(Util.java:682) hudson.model.Run.getDurationString(Run.java:697) com.tikal.jenkins.plugins.multijob.MultiJobBuilder.updateSubBuild(MultiJobBuilder.java:292) com.tikal.jenkins.plugins.multijob.MultiJobBuilder.access$000(MultiJobBuilder.java:60) com.tikal.jenkins.plugins.multijob.MultiJobBuilder$SubJobWorker.run(MultiJobBuilder.java:200) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.lang. Thread .run(Unknown Source) "pool-26-thread-2" prio=5 WAITING hudson.remoting.AsyncFutureImpl.isCancelled(AsyncFutureImpl.java:64) com.tikal.jenkins.plugins.multijob.MultiJobBuilder$SubJobWorker.run(MultiJobBuilder.java:194) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.lang. Thread .run(Unknown Source)
          Hide
          ddumont Dan Dumont added a comment -

          There's more information in JENKINS-21798

          The thread trace is misleading, as the threads round-robin the work.

          Show
          ddumont Dan Dumont added a comment - There's more information in JENKINS-21798 The thread trace is misleading, as the threads round-robin the work.
          Hide
          hagzag hagzag added a comment -

          Itai/Alex could you please look into this, looks like a show stopper to me (if we ca re-produce).

          Tx

          Show
          hagzag hagzag added a comment - Itai/Alex could you please look into this, looks like a show stopper to me (if we ca re-produce). Tx
          Hide
          patriks Patrik Schalin added a comment -

          Seem to be a fix here: https://github.com/jenkinsci/tikal-multijob-plugin/pull/49

          Is that good? Any ETA on this fix, 1.13?

          Show
          patriks Patrik Schalin added a comment - Seem to be a fix here: https://github.com/jenkinsci/tikal-multijob-plugin/pull/49 Is that good? Any ETA on this fix, 1.13?
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Nicolas Morey-Chaisemartin
          Path:
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuilder.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/f9a88fd8a6b094b4e817bcc4a0fc19ab91be2ddb
          Log:
          MultiJobBuilder.java: Sleep in the subtask polling loop

          Without the sleep, the thread keeps polling and updating the subtask which causes a very high CPU usage.
          This fixes JENKINS-21649 and JENKINS-21798

          Signed-off-by: Nicolas Morey-Chaisemartin <nmorey@kalray.eu>

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas Morey-Chaisemartin Path: src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuilder.java http://jenkins-ci.org/commit/tikal-multijob-plugin/f9a88fd8a6b094b4e817bcc4a0fc19ab91be2ddb Log: MultiJobBuilder.java: Sleep in the subtask polling loop Without the sleep, the thread keeps polling and updating the subtask which causes a very high CPU usage. This fixes JENKINS-21649 and JENKINS-21798 Signed-off-by: Nicolas Morey-Chaisemartin <nmorey@kalray.eu>
          Hide
          hagzag hagzag added a comment -

          Merged PR #49 from nmorey/update-subbuild-poll

          Show
          hagzag hagzag added a comment - Merged PR #49 from nmorey/update-subbuild-poll

            People

            Assignee:
            itai itai Or
            Reporter:
            smolyn greg smolyn
            Votes:
            3 Vote for this issue
            Watchers:
            10 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: