Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27041

Thousands of threads created by thousands of GitHub pushes

      A Jenkins instance using this plugin was found to have more than 4400 threads with this stack trace:

      "Waiting to acquire /…/workspace/… : Computer.threadPoolForRemoting [#37]" daemon prio=10 tid=0x… nid=0x… in Object.wait() [0x…]
          java.lang.Thread.State: WAITING (on object monitor)
             at java.lang.Object.wait(Native Method)
             at java.lang.Object.wait(Object.java:503)
             at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:245)
             - locked <0x…> (a hudson.slaves.WorkspaceList)
             at hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:224)
             - locked <0x…> (a hudson.slaves.WorkspaceList)
             at hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1446)
             at hudson.model.AbstractProject._poll(AbstractProject.java:1425)
             at hudson.model.AbstractProject.poll(AbstractProject.java:1336)
             at com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:81)
             at com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:106)
             at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
             at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
             at java.util.concurrent.FutureTask.run(FutureTask.java:262)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
             at java.lang.Thread.run(Thread.java:745)
      

      Looking at the logs, there were 1150 GitHub push notifications for this job over 12 hours. Each created a polling thread and competed for a lock on the workspace. Each polling request took 1-6 seconds.

      Such a high thread count had other knock on effects, such as high CPU usage when counting threads.

      So we should figure out a way to either eliminate the duplicate polling or just keep the thread count down.

          [JENKINS-27041] Thousands of threads created by thousands of GitHub pushes

          Kanstantsin Shautsou added a comment - - edited

          But in the current state when GH trigger calls GIT polling, that has custom algorithm for detecting new commits (that may never end triggering builds), gh plugin triggering probably should be merged per job.

          Imho let's better migrate to commitNotify, WDYT?

          Kanstantsin Shautsou added a comment - - edited But in the current state when GH trigger calls GIT polling, that has custom algorithm for detecting new commits (that may never end triggering builds), gh plugin triggering probably should be merged per job. Imho let's better migrate to commitNotify, WDYT?

          Jesse Glick added a comment -

          You can coalesce pushBy users but only try to lock the workspace for polling once I guess. In other words, if you have received a thousand POSTs from three users altogether, but are still waiting in WorkspaceList.acquire, just let Git polling run once, and if changes are found, schedule one build with a CauseAction (scheduleBuild2) with multiple GitHubPushCause links. Or modify GitHubPushCause to support a set of users.

          What I am unsure about is what happens if someone configures GitSCM with a wildcard branch spec (against my advice—use multibranch instead!); in that case there might have been commits to multiple branches, requires a corresponding number of builds to be scheduled. Now for /git/notifyCommit this is handled by JenkinsAbstractProjectListener passing RevisionParameterAction, which is a QueueAction so it forces multiple builds to arise. It seems that GitHubPushTrigger handles this case only insofar as every push results in an attempt to schedule a queue item, even when only one branch is configured, as found in this issue.

          Jesse Glick added a comment - You can coalesce pushBy users but only try to lock the workspace for polling once I guess. In other words, if you have received a thousand POSTs from three users altogether, but are still waiting in WorkspaceList.acquire , just let Git polling run once, and if changes are found, schedule one build with a CauseAction ( scheduleBuild2 ) with multiple GitHubPushCause links. Or modify GitHubPushCause to support a set of users. What I am unsure about is what happens if someone configures GitSCM with a wildcard branch spec (against my advice—use multibranch instead!); in that case there might have been commits to multiple branches, requires a corresponding number of builds to be scheduled. Now for /git/notifyCommit this is handled by JenkinsAbstractProjectListener passing RevisionParameterAction , which is a QueueAction so it forces multiple builds to arise. It seems that GitHubPushTrigger handles this case only insofar as every push results in an attempt to schedule a queue item, even when only one branch is configured, as found in this issue.

          Kanstantsin Shautsou added a comment - - edited

          Atm github push plugin ONLY kicks scm polling. So if you configured wildcard it will go to git scm polling algo (that sucks JENKINS-30475 JENKINS-30345). The only thing that gh do -> kick existed poll. So in theory switch from GHpush to generic pollscm will do the same.
          I see 3 ways of triggering jobs:
          1) kik poll()
          2) put in queue with predefined actions and parameters (like github-pullrequest-plugin do atm)
          3) send to branchNotify() commitNotify() or whatever else listener exists in git plugin

          Kanstantsin Shautsou added a comment - - edited Atm github push plugin ONLY kicks scm polling. So if you configured wildcard it will go to git scm polling algo (that sucks JENKINS-30475 JENKINS-30345 ). The only thing that gh do -> kick existed poll. So in theory switch from GHpush to generic pollscm will do the same. I see 3 ways of triggering jobs: 1) kik poll() 2) put in queue with predefined actions and parameters (like github-pullrequest-plugin do atm) 3) send to branchNotify() commitNotify() or whatever else listener exists in git plugin

          If user configured wildcard, then GHpush trigger shouldn't bother with it's calculation and 2) is bad variant (but will work for other planned trigger types).
          1) Kicking polling is not efficient, so 3) sounds like the right variant.

          Kanstantsin Shautsou added a comment - If user configured wildcard, then GHpush trigger shouldn't bother with it's calculation and 2) is bad variant (but will work for other planned trigger types). 1) Kicking polling is not efficient, so 3) sounds like the right variant.

          Kirill Merkushev added a comment - - edited

          notifyCommit required enabled scm trigger. More than that - it will trigger all jobs with enabled scm trigger (even without GH trigger). And we can just remove trigger from plugin and handle triggering from root action

          So it changes existing use case of this plugin and brakes all existed jobs with only GH push trigger. Also with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".

          Kirill Merkushev added a comment - - edited notifyCommit required enabled scm trigger. More than that - it will trigger all jobs with enabled scm trigger (even without GH trigger). And we can just remove trigger from plugin and handle triggering from root action So it changes existing use case of this plugin and brakes all existed jobs with only GH push trigger. Also with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".

          Daniel Beck added a comment - - edited

          integer

          queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo.

          Have your action implement Queue.QueueAction to control this behavior.

          (Update) Noticed only now this is obsolete as Jesse already addressed it. Sorry about that.

          Daniel Beck added a comment - - edited integer queue shouldn't merge the same job with different actions. Either you will lose all builds for different commits in one job/repo. Have your action implement Queue.QueueAction to control this behavior. ( Update ) Noticed only now this is obsolete as Jesse already addressed it. Sorry about that.

          Jesse Glick added a comment -

          notifyCommit required enabled scm trigger.

          Sure, though the schedule may be empty. Or @daily, which is wise anyway.

          it will trigger all jobs with enabled scm trigger (even without GH trigger)

          Only those whose SCM URL matches the notified URL.

          we can just remove trigger from plugin and handle triggering from root action

          No, the purpose of the GH-specific trigger IIUC is to interpret the web hook format that GH sends natively, since you have no opportunity to set up a custom push event script.

          with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks".

          I am not sure what your point is here.

          Jesse Glick added a comment - notifyCommit required enabled scm trigger. Sure, though the schedule may be empty. Or @daily , which is wise anyway. it will trigger all jobs with enabled scm trigger (even without GH trigger) Only those whose SCM URL matches the notified URL. we can just remove trigger from plugin and handle triggering from root action No, the purpose of the GH-specific trigger IIUC is to interpret the web hook format that GH sends natively, since you have no opportunity to set up a custom push event script. with notifyCommit you can disable unwanted triggering only with checkbox "Ignore post commit hooks". I am not sure what your point is here.

          points to not use notifyCommit:
          1. It breaks all existing setups (you should explicitly enable scm trigger, which usually not enabled)
          2. If you don't want to trigger job by notifyCommit you should explicitly turn it off with additional git behaviour option (most of people don't know about this option) - it maybe in case of nightly builds, when we check git only at exact time to start heavy build.

          so it should be implemented without notifyCommit. Maybe we should use GH events api to poll state changing

          Kirill Merkushev added a comment - points to not use notifyCommit: 1. It breaks all existing setups (you should explicitly enable scm trigger, which usually not enabled) 2. If you don't want to trigger job by notifyCommit you should explicitly turn it off with additional git behaviour option (most of people don't know about this option) - it maybe in case of nightly builds, when we check git only at exact time to start heavy build. so it should be implemented without notifyCommit. Maybe we should use GH events api to poll state changing

          1. Probably can be done optional, default in GH trigger as new, existed like it was.

          Kanstantsin Shautsou added a comment - 1. Probably can be done optional, default in GH trigger as new, existed like it was.

          Damien Ryan added a comment -

          +1 for any improvements around this area. I've currently got rapidly increasing threads every time there's a github push and these only come down slowly when the machine stops being busy (which is rare). Eventually, java hits the thread limit and then either one of the slaves drops its job or the whole thing dies and requires a reboot:

          Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070]
          java.lang.Object.wait(Native Method)
          java.lang.Object.wait(Object.java:503)
          hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:255)
          hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:234)
          hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1467)
          hudson.model.AbstractProject._poll(AbstractProject.java:1444)
          hudson.model.AbstractProject.poll(AbstractProject.java:1355)
          com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:73)
          com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:98)
          hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
          java.util.concurrent.FutureTask.run(FutureTask.java:262)
          java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          java.lang.Thread.run(Thread.java:745)
          Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070] 
          

          Damien Ryan added a comment - +1 for any improvements around this area. I've currently got rapidly increasing threads every time there's a github push and these only come down slowly when the machine stops being busy (which is rare). Eventually, java hits the thread limit and then either one of the slaves drops its job or the whole thing dies and requires a reboot: Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070] java.lang. Object .wait(Native Method) java.lang. Object .wait( Object .java:503) hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:255) hudson.slaves.WorkspaceList.acquire(WorkspaceList.java:234) hudson.model.AbstractProject.pollWithWorkspace(AbstractProject.java:1467) hudson.model.AbstractProject._poll(AbstractProject.java:1444) hudson.model.AbstractProject.poll(AbstractProject.java:1355) com.cloudbees.jenkins.GitHubPushTrigger$1.runPolling(GitHubPushTrigger.java:73) com.cloudbees.jenkins.GitHubPushTrigger$1.run(GitHubPushTrigger.java:98) hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask.run(FutureTask.java:262) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang. Thread .run( Thread .java:745) Waiting to acquire C:\dev : Computer.threadPoolForRemoting [#1070]

            recampbell Ryan Campbell
            recampbell Ryan Campbell
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: