[JENKINS-27041] Thousands of threads created by thousands of GitHub pushes

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: github-plugin
Labels:
- performance
Environment:
1.580.2

Similar Issues:
Powered by SuggestiMate

Show

A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

"Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
    java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       at java.lang.Object.wait(Object.java:503)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
       - locked <0x…> (a hudson.slaves.WorkspaceList)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
       - locked <0x…> (a hudson.slaves.WorkspaceList)
       at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
       at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
       at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
       at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
       at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
       at java.util.concurrent.FutureTask.run(FutureTask.java:262)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)

Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

is related to

JENKINS-22456 github plugin too many open files unclosed github-polling.log filehandles causes all builds to fail

Resolved

Ryan Campbell created issue - 2015-02-19 21:34

Jesse Glick made changes - 2015-02-19 21:51

Labels

New: performance

Jesse Glick made changes - 2015-02-19 21:53

Description

Original: A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

{{{code}}}
"Waiting to acquire /scratch/jenkins/workspace/SS-JOB : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x00007f084019e800 nid=0x3229 in Object.wait() [0x00007f083497a000]
    java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       at java.lang.Object.wait(Object.java:503)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
       - locked <0x0000000773632e50> (a hudson.slaves.WorkspaceList)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
       - locked <0x0000000773632e50> (a hudson.slaves.WorkspaceList)
       at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
       at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
       at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
       at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
       at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
       at java.util.concurrent.FutureTask.run(FutureTask.java:262)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)
{{{code}}}

Looking at the logs, there were 1150 GitHub push notifications for SS-JOB over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

New: A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

{code:none}
"Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
    java.lang.Thread.State: WAITING (on object monitor)
       at java.lang.Object.wait(Native Method)
       at java.lang.Object.wait(Object.java:503)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
       - locked <0x…> (a hudson.slaves.WorkspaceList)
       at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
       - locked <0x…> (a hudson.slaves.WorkspaceList)
       at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
       at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
       at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
       at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
       at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
       at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
       at java.util.concurrent.FutureTask.run(FutureTask.java:262)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
       at java.lang.Thread.run(Thread.java:745)
{code}

Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

Jesse Glick made changes - 2015-10-09 14:21

Link

New: This issue is related to ~~JENKINS-22456~~ [ ~~JENKINS-22456~~ ]

R. Tyler Croy made changes - 2016-07-25 23:51

Workflow

Original: JNJira [ 161233 ]

New: JNJira + In-Review [ 180622 ]

Jenkins

Details

Description

Attachments

Issue Links

Activity

People

Dates