Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27041

Thousands of threads created by thousands of GitHub pushes

XMLWordPrintable

      A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

      "Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
    java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       at java.lang.Object.wait(Object.java:503)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
       - locked <0x…> (a hudson.slaves.WorkspaceList)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
       - locked <0x…> (a hudson.slaves.WorkspaceList)
       at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
       at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
       at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
       at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
       at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
       at java.util.concurrent.FutureTask.run(FutureTask.java:262)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)

      Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

      Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

      So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

          [JENKINS-27041] Thousands of threads created by thousands of GitHub pushes

          Ryan Campbell created issue -

          Daniel Beck added a comment - - edited

          Isn't this covered by the global config option Max # of concurrent polling that appears once you have 10+ top level items?

          Related: https://github.com/jenkinsci/jenkins/pull/1230#discussion_r12500721

          Daniel Beck added a comment - - edited Isn't this covered by the global config option Max # of concurrent polling that appears once you have 10+ top level items? Related: https://github.com/jenkinsci/jenkins/pull/1230#discussion_r12500721
          Jesse Glick made changes -
          Labels New: performance
          Jesse Glick made changes -
          Description Original: A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

          {{{code}}}
          "Waiting to acquire /scratch/jenkins/workspace/SS-JOB : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x00007f084019e800 nid=0x3229 in Object.wait() [0x00007f083497a000]
              java.lang.Thread.State: WAITING (on object monitor)
                 at java.lang.Object.wait(Native Method)
                 at java.lang.Object.wait(Object.java:503)
                 at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
                 - locked &lt;0x0000000773632e50&gt; (a hudson.slaves.WorkspaceList)
                 at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
                 - locked &lt;0x0000000773632e50&gt; (a hudson.slaves.WorkspaceList)
                 at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
                 at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
                 at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
                 at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
                 at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
                 at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
                 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                 at java.lang.Thread.run(Thread.java:745)
          {{{code}}}

          Looking at the logs, there were 1150 GitHub push notifications for SS-JOB over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

          Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

          So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.
          		 New: A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

          {code:none}
          "Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
              java.lang.Thread.State: WAITING (on object monitor)
                 at java.lang.Object.wait(Native Method)
                 at java.lang.Object.wait(Object.java:503)
                 at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
                 - locked <0x…> (a hudson.slaves.WorkspaceList)
                 at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
                 - locked <0x…> (a hudson.slaves.WorkspaceList)
                 at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
                 at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
                 at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
                 at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
                 at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
                 at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
                 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
                 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                 at java.util.concurrent.FutureTask.run(FutureTask.java:262)
                 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                 at java.lang.Thread.run(Thread.java:745)
          {code}

          Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

          Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

          So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

          Kanstantsin Shautsou added a comment -

          Is it still actual? We have in TODO change polling algo that will be lighter.

          Kanstantsin Shautsou added a comment - Is it still actual? We have in TODO change polling algo that will be lighter.

          Jesse Glick added a comment -

          Isn't this covered by the global config option Max # of concurrent polling

          Probably not, since here polling is being called directly from the trigger, rather than simply asking for polling to be scheduled as SCMTrigger does. At least IIUC. And GitHubPushTrigger is still using MasterComputer.threadPoolForRemoting rather than Timer.get() which would bound the number of threads in use concurrently. It is also not using SequentialExecutionQueue correctly, because it fails to override Runnable.equals to reflect the identity of the Job.

          Jesse Glick added a comment - Isn't this covered by the global config option Max # of concurrent polling Probably not, since here polling is being called directly from the trigger, rather than simply asking for polling to be scheduled as SCMTrigger does. At least IIUC. And GitHubPushTrigger is still using MasterComputer.threadPoolForRemoting rather than Timer.get() which would bound the number of threads in use concurrently. It is also not using SequentialExecutionQueue correctly, because it fails to override Runnable.equals to reflect the identity of the Job .
          Jesse Glick made changes -
          Link New: This issue is related to JENKINS-22456 [ JENKINS-22456 ]

          Kanstantsin Shautsou added a comment -

          jglick queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.

          Kanstantsin Shautsou added a comment - jglick queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.

          Kanstantsin Shautsou added a comment -

          It using SequentialExecutionQueue that is in Descriptor and wraps remoteThreadPool.

          Kanstantsin Shautsou added a comment - It using SequentialExecutionQueue that is in Descriptor and wraps remoteThreadPool.

          Kanstantsin Shautsou added a comment -

          Btw, see stacktrace.

          Kanstantsin Shautsou added a comment - Btw, see stacktrace.

            recampbell Ryan Campbell
            recampbell Ryan Campbell
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: