Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58684

Linux max task resource exhaustion after excessive timer thread creation

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • sse-gateway-plugin
    • Jenkins 2.176.2 using Ubuntu 18.04 package file from pkg.jenkins.io/debian-stable
      AMD Threadripper 32-core/64-core machine with 128GB of RAM
      Single master node with 20 executors

      Jenkins is exhausting resources on my machine seemingly to create 10k + timer threads. This thread creation then hits the systemd imposed 15% of task limit set by the kernel (/proc/sys/kernel/pid_max = 131072) which is 19660 by default on my system.

      This particular java heap dump shows that 4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader -> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.

      Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with Ubuntu 18.04 as of 2019.07.23.

       

      To dig deeper, after the resource limit is hit, threads fail to create and Jenkins becomes unstable until restarted, logs contain many instances of the following:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
      java.lang.OutOfMemoryError: unable to create new native thread
      

       
      NetData on the system shows that if it doesn't hit the thread limit it will spike and often recover correctly suggesting the threads are cleaned up at some point if it doesn't derail the process first.

      I've increased the systemd limit on the system and the problem hasn't derailed the jenkins process in a few days by adding the following to the systemd service file:

      TasksMax=32768
      

      I have been capturing heaps with some automation tools and catch several heaps exceeding 20k threads in the last few days signifying that the process is still occurring and likely affecting other users, they probably aren't noticing though.

       

      Not sure if this is related to a plugin (listed below with version) or Jenkins core code?

       

      [
        {
          "plugin": "Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api)",
          "version": "4.5.5-3.0"
        },
        {
          "plugin": "Authentication Tokens API Plugin (authentication-tokens)",
          "version": "1.3"
        },
        {
          "plugin": "Authorize Project (authorize-project)",
          "version": "1.3.0"
        },
        {
          "plugin": "Autofavorite for Blue Ocean (blueocean-autofavorite)",
          "version": "1.2.4"
        },
        {
          "plugin": "Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source)",
          "version": "2.4.5"
        },
        {
          "plugin": "Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "Bitbucket Plugin (bitbucket)",
          "version": "1.1.10"
        },
        {
          "plugin": "Blue Ocean (blueocean)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Core JS (blueocean-core-js)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Executor Info (blueocean-executor-info)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Pipeline Editor (blueocean-pipeline-editor)",
          "version": "1.17.0"
        },
        {
          "plugin": "Branch API Plugin (branch-api)",
          "version": "2.5.3"
        },
        {
          "plugin": "Build Timeout (build-timeout)",
          "version": "1.19"
        },
        {
          "plugin": "Command Agent Launcher Plugin (command-launcher)",
          "version": "1.3"
        },
        {
          "plugin": "Common API for Blue Ocean (blueocean-commons)",
          "version": "1.17.0"
        },
        {
          "plugin": "Conditional BuildStep (conditional-buildstep)",
          "version": "1.3.6"
        },
        {
          "plugin": "Config API for Blue Ocean (blueocean-config)",
          "version": "1.17.0"
        },
        {
          "plugin": "Copy Artifact Plugin (copyartifact)",
          "version": "1.42.1"
        },
        {
          "plugin": "Credentials Binding Plugin (credentials-binding)",
          "version": "1.19"
        },
        {
          "plugin": "Credentials Plugin (credentials)",
          "version": "2.2.0"
        },
        {
          "plugin": "Dashboard for Blue Ocean (blueocean-dashboard)",
          "version": "1.17.0"
        },
        {
          "plugin": "Design Language (jenkins-design-language)",
          "version": "1.17.0"
        },
        {
          "plugin": "Display URL API (display-url-api)",
          "version": "2.3.1"
        },
        {
          "plugin": "Display URL for Blue Ocean (blueocean-display-url)",
          "version": "2.3.0"
        },
        {
          "plugin": "Docker Commons Plugin (docker-commons)",
          "version": "1.15"
        },
        {
          "plugin": "Docker Pipeline (docker-workflow)",
          "version": "1.18"
        },
        {
          "plugin": "Durable Task Plugin (durable-task)",
          "version": "1.30"
        },
        {
          "plugin": "Email Extension Plugin (email-ext)",
          "version": "2.66"
        },
        {
          "plugin": "Events API for Blue Ocean (blueocean-events)",
          "version": "1.17.0"
        },
        {
          "plugin": "External Monitor Job Type Plugin (external-monitor-job)",
          "version": "1.7"
        },
        {
          "plugin": "Favorite (favorite)",
          "version": "2.3.2"
        },
        {
          "plugin": "Folders Plugin (cloudbees-folder)",
          "version": "6.9"
        },
        {
          "plugin": "GIT server Plugin (git-server)",
          "version": "1.7"
        },
        {
          "plugin": "Git Pipeline for Blue Ocean (blueocean-git-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "Git client plugin (git-client)",
          "version": "3.0.0-rc"
        },
        {
          "plugin": "Git plugin (git)",
          "version": "4.0.0-rc"
        },
        {
          "plugin": "GitHub API Plugin (github-api)",
          "version": "1.95"
        },
        {
          "plugin": "GitHub Branch Source Plugin (github-branch-source)",
          "version": "2.5.4"
        },
        {
          "plugin": "GitHub Pipeline for Blue Ocean (blueocean-github-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "GitHub plugin (github)",
          "version": "1.29.4"
        },
        {
          "plugin": "Google Login Plugin (google-login)",
          "version": "1.6"
        },
        {
          "plugin": "Gradle Plugin (gradle)",
          "version": "1.33"
        },
        {
          "plugin": "HTML Publisher plugin (htmlpublisher)",
          "version": "1.18"
        },
        {
          "plugin": "HTTP Request Plugin (http_request)",
          "version": "1.8.23"
        },
        {
          "plugin": "Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api)",
          "version": "2.1.7-1.0"
        },
        {
          "plugin": "Icon Shim Plugin (icon-shim)",
          "version": "2.0.3"
        },
        {
          "plugin": "JIRA Integration for Blue Ocean (blueocean-jira)",
          "version": "1.17.0"
        },
        {
          "plugin": "JIRA plugin (jira)",
          "version": "3.0.8"
        },
        {
          "plugin": "JSch dependency plugin (jsch)",
          "version": "0.1.55"
        },
        {
          "plugin": "JUnit Plugin (junit)",
          "version": "1.28"
        },
        {
          "plugin": "JWT for Blue Ocean (blueocean-jwt)",
          "version": "1.17.0"
        },
        {
          "plugin": "Jackson 2 API Plugin (jackson2-api)",
          "version": "2.9.9.1"
        },
        {
          "plugin": "JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor)",
          "version": "1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: Handlebars bundle plugin (handlebars)",
          "version": "1.1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: Moment.js bundle plugin (momentjs)",
          "version": "1.1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached)",
          "version": "1.2.1"
        },
        {
          "plugin": "Javadoc Plugin (javadoc)",
          "version": "1.5"
        },
        {
          "plugin": "LDAP Plugin (ldap)",
          "version": "1.20"
        },
        {
          "plugin": "Lockable Resources plugin (lockable-resources)",
          "version": "2.5"
        },
        {
          "plugin": "Mailer Plugin (mailer)",
          "version": "1.23"
        },
        {
          "plugin": "MapDB API Plugin (mapdb-api)",
          "version": "1.0.9.0"
        },
        {
          "plugin": "Matrix Authorization Strategy Plugin (matrix-auth)",
          "version": "2.4.2"
        },
        {
          "plugin": "Matrix Project Plugin (matrix-project)",
          "version": "1.14"
        },
        {
          "plugin": "Maven Integration plugin (maven-plugin)",
          "version": "3.3"
        },
        {
          "plugin": "Mercurial plugin (mercurial)",
          "version": "2.7"
        },
        {
          "plugin": "Metrics Plugin (metrics)",
          "version": "4.0.2.5"
        },
        {
          "plugin": "OWASP Markup Formatter Plugin (antisamy-markup-formatter)",
          "version": "1.5"
        },
        {
          "plugin": "Oracle Java SE Development Kit Installer Plugin (jdk-tool)",
          "version": "1.3"
        },
        {
          "plugin": "PAM Authentication plugin (pam-auth)",
          "version": "1.5.1"
        },
        {
          "plugin": "Personalization for Blue Ocean (blueocean-personalization)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline (workflow-aggregator)",
          "version": "2.6"
        },
        {
          "plugin": "Pipeline Graph Analysis Plugin (pipeline-graph-analysis)",
          "version": "1.10"
        },
        {
          "plugin": "Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline Utility Steps (pipeline-utility-steps)",
          "version": "2.3.0"
        },
        {
          "plugin": "Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline: API (workflow-api)",
          "version": "2.35"
        },
        {
          "plugin": "Pipeline: Basic Steps (workflow-basic-steps)",
          "version": "2.18"
        },
        {
          "plugin": "Pipeline: Build Step (pipeline-build-step)",
          "version": "2.9"
        },
        {
          "plugin": "Pipeline: Declarative (pipeline-model-definition)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Declarative Agent API (pipeline-model-declarative-agent)",
          "version": "1.1.1"
        },
        {
          "plugin": "Pipeline: Declarative Extension Points API (pipeline-model-extensions)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: GitHub Groovy Libraries (pipeline-github-lib)",
          "version": "1.0"
        },
        {
          "plugin": "Pipeline: Groovy (workflow-cps)",
          "version": "2.72"
        },
        {
          "plugin": "Pipeline: Input Step (pipeline-input-step)",
          "version": "2.10"
        },
        {
          "plugin": "Pipeline: Job (workflow-job)",
          "version": "2.33"
        },
        {
          "plugin": "Pipeline: Milestone Step (pipeline-milestone-step)",
          "version": "1.3.1"
        },
        {
          "plugin": "Pipeline: Model API (pipeline-model-api)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Multibranch (workflow-multibranch)",
          "version": "2.21"
        },
        {
          "plugin": "Pipeline: Nodes and Processes (workflow-durable-task-step)",
          "version": "2.32"
        },
        {
          "plugin": "Pipeline: REST API Plugin (pipeline-rest-api)",
          "version": "2.11"
        },
        {
          "plugin": "Pipeline: SCM Step (workflow-scm-step)",
          "version": "2.9"
        },
        {
          "plugin": "Pipeline: Shared Groovy Libraries (workflow-cps-global-lib)",
          "version": "2.14"
        },
        {
          "plugin": "Pipeline: Stage Step (pipeline-stage-step)",
          "version": "2.3"
        },
        {
          "plugin": "Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Stage View Plugin (pipeline-stage-view)",
          "version": "2.11"
        },
        {
          "plugin": "Pipeline: Step API (workflow-step-api)",
          "version": "2.20"
        },
        {
          "plugin": "Pipeline: Supporting APIs (workflow-support)",
          "version": "3.3"
        },
        {
          "plugin": "Plain Credentials Plugin (plain-credentials)",
          "version": "1.5"
        },
        {
          "plugin": "Pub-Sub \"light\" Bus (pubsub-light)",
          "version": "1.12"
        },
        {
          "plugin": "REST API for Blue Ocean (blueocean-rest)",
          "version": "1.17.0"
        },
        {
          "plugin": "REST Implementation for Blue Ocean (blueocean-rest-impl)",
          "version": "1.17.0"
        },
        {
          "plugin": "Resource Disposer Plugin (resource-disposer)",
          "version": "0.13"
        },
        {
          "plugin": "Run Condition Plugin (run-condition)",
          "version": "1.2"
        },
        {
          "plugin": "SCM API Plugin (scm-api)",
          "version": "2.6.3"
        },
        {
          "plugin": "SSH Agent Plugin (ssh-agent)",
          "version": "1.17"
        },
        {
          "plugin": "SSH Credentials Plugin (ssh-credentials)",
          "version": "1.17.1"
        },
        {
          "plugin": "SSH Slaves plugin (ssh-slaves)",
          "version": "1.30.1"
        },
        {
          "plugin": "Script Security Plugin (script-security)",
          "version": "1.61"
        },
        {
          "plugin": "Server Sent Events (SSE) Gateway Plugin (sse-gateway)",
          "version": "1.18"
        },
        {
          "plugin": "Skip Notifications Trait plugin (skip-notifications-trait)",
          "version": "1.0.3"
        },
        {
          "plugin": "Structs Plugin (structs)",
          "version": "1.19"
        },
        {
          "plugin": "Subversion Plug-in (subversion)",
          "version": "2.12.2"
        },
        {
          "plugin": "Timestamper (timestamper)",
          "version": "1.10"
        },
        {
          "plugin": "Token Macro Plugin (token-macro)",
          "version": "2.8"
        },
        {
          "plugin": "Trilead API Plugin (trilead-api)",
          "version": "1.0.3"
        },
        {
          "plugin": "Variant Plugin (variant)",
          "version": "1.2"
        },
        {
          "plugin": "WMI Windows Agents Plugin (windows-slaves)",
          "version": "1.4"
        },
        {
          "plugin": "Web for Blue Ocean (blueocean-web)",
          "version": "1.17.0"
        },
        {
          "plugin": "Workspace Cleanup Plugin (ws-cleanup)",
          "version": "0.37"
        },
        {
          "plugin": "bouncycastle API Plugin (bouncycastle-api)",
          "version": "2.17"
        },
        {
          "plugin": "i18n for Blue Ocean (blueocean-i18n)",
          "version": "1.17.0"
        }
      ]
      

          [JENKINS-58684] Linux max task resource exhaustion after excessive timer thread creation

          Sverre Moe added a comment -

          Cleaned up, how?
          We also have experienced this problem that past few months.
          Recently we upgraded our Jenkins server Linux distribution from SLES12.3 to SLES15.0. This was done by creating a new VM and copied over the JENKINS_HOME. It has been running for 4 days now and we have not gotten the OutOfMemoryError. More time will tell if it is truly stable again.

          Sverre Moe added a comment - Cleaned up, how? We also have experienced this problem that past few months. Recently we upgraded our Jenkins server Linux distribution from SLES12.3 to SLES15.0. This was done by creating a new VM and copied over the JENKINS_HOME. It has been running for 4 days now and we have not gotten the OutOfMemoryError. More time will tell if it is truly stable again.

          djviking Ad "cleaned up": Sorry for my lack of clarity => due to the OOM there were remaining running Docker Containers that I had to stop and remove; and it also appeared that about 2xx "EventDispatcher.retryProcessor" threads were remaining inside Jenkins Java process...

          Other notes (~15h after Jenkins master restart):

          • This morning no "EventDispatcher.retryProcessor" threads are there (or remaining) yet.
          • Interestingly there are 25 "org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep" threads, even though just one pipeline build is running (without any parallelism)

          Reinhold Füreder added a comment - djviking Ad "cleaned up": Sorry for my lack of clarity => due to the OOM there were remaining running Docker Containers that I had to stop and remove; and it also appeared that about 2xx "EventDispatcher.retryProcessor" threads were remaining inside Jenkins Java process... Other notes (~15h after Jenkins master restart): This morning no "EventDispatcher.retryProcessor" threads are there (or remaining) yet. Interestingly there are 25 "org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep" threads, even though just one pipeline build is running (without any parallelism)

          Olivier Lamy added a comment -

          Yes please try last snapshot from  https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/sse-gateway/1.20-SNAPSHOT/

           reinholdfuereder djviking  please tell us what is the result 

          We are close to merge the PR just need to another change and test it with BlueOcean full build.

          Thanks for your patience (and your report  )

           

          Olivier Lamy added a comment - Yes please try last snapshot from   https://repo.jenkins-ci.org/snapshots/org/jenkins-ci/plugins/sse-gateway/1.20-SNAPSHOT/   reinholdfuereder djviking   please tell us what is the result  We are close to merge the PR just need to another change and test it with BlueOcean full build. Thanks for your patience (and your report  )  

          Just chiming in here:

          I've been running into this issue for a few weeks on my old Mac I've got set up as a CI server; at some point after using the Blue Ocean UI I'd start seeing the thread count on Jenkins rise rapidly from its usual ~70ish into the thousands and then I'd start getting 'could not create native thread' errors and other strange side-effects.  I had been avoiding Blue Ocean for the last week or so due to this.

          I went ahead and tried the sse-gateway 1.20-SNAPSHOT posted yesterday and went back to using Blue Ocean and am happy to report I've not observed the thread leak since.  I'll keep my eye on it but so far things are looking good.  Thanks everyone who has been looking into this!

          Eric Froemling added a comment - Just chiming in here: I've been running into this issue for a few weeks on my old Mac I've got set up as a CI server; at some point after using the Blue Ocean UI I'd start seeing the thread count on Jenkins rise rapidly from its usual ~70ish into the thousands and then I'd start getting 'could not create native thread' errors and other strange side-effects.  I had been avoiding Blue Ocean for the last week or so due to this. I went ahead and tried the sse-gateway 1.20-SNAPSHOT posted yesterday and went back to using Blue Ocean and am happy to report I've not observed the thread leak since.  I'll keep my eye on it but so far things are looking good.  Thanks everyone who has been looking into this!

          Olivier Lamy added a comment -

          1.20 version has been released

          Olivier Lamy added a comment - 1.20 version has been released

          olamy Sorry for my delayed feedback: AFAIK (no user complaints in my company – at least none that I know of...) the snapshot build of sse-gateway worked fine; and on Jenkins server neither remaining "EventDispatcher.retryProcessor" threads, nor thousands of them at the same time. => Looks good

          Reinhold Füreder added a comment - olamy Sorry for my delayed feedback: AFAIK (no user complaints in my company – at least none that I know of...) the snapshot build of sse-gateway worked fine; and on Jenkins server neither remaining "EventDispatcher.retryProcessor" threads, nor thousands of them at the same time. => Looks good

          Kyle added a comment -

          Updated on my server, thanks for all the hard work!

          Kyle added a comment - Updated on my server, thanks for all the hard work!

          Søren Friis added a comment -

          There seems to be a problem with Showing Blue Ocean in MS Edge after this update.
          https://issues.jenkins-ci.org/browse/JENKINS-59291

           

          Søren Friis added a comment - There seems to be a problem with Showing Blue Ocean in MS Edge after this update. https://issues.jenkins-ci.org/browse/JENKINS-59291  

          Pradeep Singh added a comment -

          Thread count accelerated to sky high (~22k) after upgrading Jenkins from 2.17 to 2.190.3.

          We had analysed thread dump which didn't help us to find the solution although, it showed couple of thread in blocked state.

          After upgrading Jenkins to LTS 2.204.4 solved this issue.

          Pradeep Singh added a comment - Thread count accelerated to sky high (~22k) after upgrading Jenkins from 2.17 to 2.190.3. We had analysed thread dump which didn't help us to find the solution although, it showed couple of thread in blocked state. After upgrading Jenkins to LTS  2.204.4  solved this issue.

          Karl Shultz added a comment -

          I apologize for the noise just now, I must have hit a hotkey that assigned this to me. I've switched it back.

          Karl Shultz added a comment - I apologize for the noise just now, I must have hit a hotkey that assigned this to me. I've switched it back.

            olamy Olivier Lamy
            2bluesc Kyle
            Votes:
            9 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: