Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58684

Linux max task resource exhaustion after excessive timer thread creation

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: sse-gateway-plugin
    • Labels:
    • Environment:
      Jenkins 2.176.2 using Ubuntu 18.04 package file from pkg.jenkins.io/debian-stable
      AMD Threadripper 32-core/64-core machine with 128GB of RAM
      Single master node with 20 executors
    • Similar Issues:

      Description

      Jenkins is exhausting resources on my machine seemingly to create 10k + timer threads. This thread creation then hits the systemd imposed 15% of task limit set by the kernel (/proc/sys/kernel/pid_max = 131072) which is 19660 by default on my system.

      This particular java heap dump shows that 4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader -> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.

      Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with Ubuntu 18.04 as of 2019.07.23.

       

      To dig deeper, after the resource limit is hit, threads fail to create and Jenkins becomes unstable until restarted, logs contain many instances of the following:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
      java.lang.OutOfMemoryError: unable to create new native thread
      

       
      NetData on the system shows that if it doesn't hit the thread limit it will spike and often recover correctly suggesting the threads are cleaned up at some point if it doesn't derail the process first.

      I've increased the systemd limit on the system and the problem hasn't derailed the jenkins process in a few days by adding the following to the systemd service file:

      TasksMax=32768
      

      I have been capturing heaps with some automation tools and catch several heaps exceeding 20k threads in the last few days signifying that the process is still occurring and likely affecting other users, they probably aren't noticing though.

       

      Not sure if this is related to a plugin (listed below with version) or Jenkins core code?

       

      [
        {
          "plugin": "Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api)",
          "version": "4.5.5-3.0"
        },
        {
          "plugin": "Authentication Tokens API Plugin (authentication-tokens)",
          "version": "1.3"
        },
        {
          "plugin": "Authorize Project (authorize-project)",
          "version": "1.3.0"
        },
        {
          "plugin": "Autofavorite for Blue Ocean (blueocean-autofavorite)",
          "version": "1.2.4"
        },
        {
          "plugin": "Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source)",
          "version": "2.4.5"
        },
        {
          "plugin": "Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "Bitbucket Plugin (bitbucket)",
          "version": "1.1.10"
        },
        {
          "plugin": "Blue Ocean (blueocean)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Core JS (blueocean-core-js)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Executor Info (blueocean-executor-info)",
          "version": "1.17.0"
        },
        {
          "plugin": "Blue Ocean Pipeline Editor (blueocean-pipeline-editor)",
          "version": "1.17.0"
        },
        {
          "plugin": "Branch API Plugin (branch-api)",
          "version": "2.5.3"
        },
        {
          "plugin": "Build Timeout (build-timeout)",
          "version": "1.19"
        },
        {
          "plugin": "Command Agent Launcher Plugin (command-launcher)",
          "version": "1.3"
        },
        {
          "plugin": "Common API for Blue Ocean (blueocean-commons)",
          "version": "1.17.0"
        },
        {
          "plugin": "Conditional BuildStep (conditional-buildstep)",
          "version": "1.3.6"
        },
        {
          "plugin": "Config API for Blue Ocean (blueocean-config)",
          "version": "1.17.0"
        },
        {
          "plugin": "Copy Artifact Plugin (copyartifact)",
          "version": "1.42.1"
        },
        {
          "plugin": "Credentials Binding Plugin (credentials-binding)",
          "version": "1.19"
        },
        {
          "plugin": "Credentials Plugin (credentials)",
          "version": "2.2.0"
        },
        {
          "plugin": "Dashboard for Blue Ocean (blueocean-dashboard)",
          "version": "1.17.0"
        },
        {
          "plugin": "Design Language (jenkins-design-language)",
          "version": "1.17.0"
        },
        {
          "plugin": "Display URL API (display-url-api)",
          "version": "2.3.1"
        },
        {
          "plugin": "Display URL for Blue Ocean (blueocean-display-url)",
          "version": "2.3.0"
        },
        {
          "plugin": "Docker Commons Plugin (docker-commons)",
          "version": "1.15"
        },
        {
          "plugin": "Docker Pipeline (docker-workflow)",
          "version": "1.18"
        },
        {
          "plugin": "Durable Task Plugin (durable-task)",
          "version": "1.30"
        },
        {
          "plugin": "Email Extension Plugin (email-ext)",
          "version": "2.66"
        },
        {
          "plugin": "Events API for Blue Ocean (blueocean-events)",
          "version": "1.17.0"
        },
        {
          "plugin": "External Monitor Job Type Plugin (external-monitor-job)",
          "version": "1.7"
        },
        {
          "plugin": "Favorite (favorite)",
          "version": "2.3.2"
        },
        {
          "plugin": "Folders Plugin (cloudbees-folder)",
          "version": "6.9"
        },
        {
          "plugin": "GIT server Plugin (git-server)",
          "version": "1.7"
        },
        {
          "plugin": "Git Pipeline for Blue Ocean (blueocean-git-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "Git client plugin (git-client)",
          "version": "3.0.0-rc"
        },
        {
          "plugin": "Git plugin (git)",
          "version": "4.0.0-rc"
        },
        {
          "plugin": "GitHub API Plugin (github-api)",
          "version": "1.95"
        },
        {
          "plugin": "GitHub Branch Source Plugin (github-branch-source)",
          "version": "2.5.4"
        },
        {
          "plugin": "GitHub Pipeline for Blue Ocean (blueocean-github-pipeline)",
          "version": "1.17.0"
        },
        {
          "plugin": "GitHub plugin (github)",
          "version": "1.29.4"
        },
        {
          "plugin": "Google Login Plugin (google-login)",
          "version": "1.6"
        },
        {
          "plugin": "Gradle Plugin (gradle)",
          "version": "1.33"
        },
        {
          "plugin": "HTML Publisher plugin (htmlpublisher)",
          "version": "1.18"
        },
        {
          "plugin": "HTTP Request Plugin (http_request)",
          "version": "1.8.23"
        },
        {
          "plugin": "Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api)",
          "version": "2.1.7-1.0"
        },
        {
          "plugin": "Icon Shim Plugin (icon-shim)",
          "version": "2.0.3"
        },
        {
          "plugin": "JIRA Integration for Blue Ocean (blueocean-jira)",
          "version": "1.17.0"
        },
        {
          "plugin": "JIRA plugin (jira)",
          "version": "3.0.8"
        },
        {
          "plugin": "JSch dependency plugin (jsch)",
          "version": "0.1.55"
        },
        {
          "plugin": "JUnit Plugin (junit)",
          "version": "1.28"
        },
        {
          "plugin": "JWT for Blue Ocean (blueocean-jwt)",
          "version": "1.17.0"
        },
        {
          "plugin": "Jackson 2 API Plugin (jackson2-api)",
          "version": "2.9.9.1"
        },
        {
          "plugin": "JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor)",
          "version": "1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: Handlebars bundle plugin (handlebars)",
          "version": "1.1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: Moment.js bundle plugin (momentjs)",
          "version": "1.1.1"
        },
        {
          "plugin": "JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached)",
          "version": "1.2.1"
        },
        {
          "plugin": "Javadoc Plugin (javadoc)",
          "version": "1.5"
        },
        {
          "plugin": "LDAP Plugin (ldap)",
          "version": "1.20"
        },
        {
          "plugin": "Lockable Resources plugin (lockable-resources)",
          "version": "2.5"
        },
        {
          "plugin": "Mailer Plugin (mailer)",
          "version": "1.23"
        },
        {
          "plugin": "MapDB API Plugin (mapdb-api)",
          "version": "1.0.9.0"
        },
        {
          "plugin": "Matrix Authorization Strategy Plugin (matrix-auth)",
          "version": "2.4.2"
        },
        {
          "plugin": "Matrix Project Plugin (matrix-project)",
          "version": "1.14"
        },
        {
          "plugin": "Maven Integration plugin (maven-plugin)",
          "version": "3.3"
        },
        {
          "plugin": "Mercurial plugin (mercurial)",
          "version": "2.7"
        },
        {
          "plugin": "Metrics Plugin (metrics)",
          "version": "4.0.2.5"
        },
        {
          "plugin": "OWASP Markup Formatter Plugin (antisamy-markup-formatter)",
          "version": "1.5"
        },
        {
          "plugin": "Oracle Java SE Development Kit Installer Plugin (jdk-tool)",
          "version": "1.3"
        },
        {
          "plugin": "PAM Authentication plugin (pam-auth)",
          "version": "1.5.1"
        },
        {
          "plugin": "Personalization for Blue Ocean (blueocean-personalization)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline (workflow-aggregator)",
          "version": "2.6"
        },
        {
          "plugin": "Pipeline Graph Analysis Plugin (pipeline-graph-analysis)",
          "version": "1.10"
        },
        {
          "plugin": "Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline Utility Steps (pipeline-utility-steps)",
          "version": "2.3.0"
        },
        {
          "plugin": "Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl)",
          "version": "1.17.0"
        },
        {
          "plugin": "Pipeline: API (workflow-api)",
          "version": "2.35"
        },
        {
          "plugin": "Pipeline: Basic Steps (workflow-basic-steps)",
          "version": "2.18"
        },
        {
          "plugin": "Pipeline: Build Step (pipeline-build-step)",
          "version": "2.9"
        },
        {
          "plugin": "Pipeline: Declarative (pipeline-model-definition)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Declarative Agent API (pipeline-model-declarative-agent)",
          "version": "1.1.1"
        },
        {
          "plugin": "Pipeline: Declarative Extension Points API (pipeline-model-extensions)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: GitHub Groovy Libraries (pipeline-github-lib)",
          "version": "1.0"
        },
        {
          "plugin": "Pipeline: Groovy (workflow-cps)",
          "version": "2.72"
        },
        {
          "plugin": "Pipeline: Input Step (pipeline-input-step)",
          "version": "2.10"
        },
        {
          "plugin": "Pipeline: Job (workflow-job)",
          "version": "2.33"
        },
        {
          "plugin": "Pipeline: Milestone Step (pipeline-milestone-step)",
          "version": "1.3.1"
        },
        {
          "plugin": "Pipeline: Model API (pipeline-model-api)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Multibranch (workflow-multibranch)",
          "version": "2.21"
        },
        {
          "plugin": "Pipeline: Nodes and Processes (workflow-durable-task-step)",
          "version": "2.32"
        },
        {
          "plugin": "Pipeline: REST API Plugin (pipeline-rest-api)",
          "version": "2.11"
        },
        {
          "plugin": "Pipeline: SCM Step (workflow-scm-step)",
          "version": "2.9"
        },
        {
          "plugin": "Pipeline: Shared Groovy Libraries (workflow-cps-global-lib)",
          "version": "2.14"
        },
        {
          "plugin": "Pipeline: Stage Step (pipeline-stage-step)",
          "version": "2.3"
        },
        {
          "plugin": "Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata)",
          "version": "1.3.9"
        },
        {
          "plugin": "Pipeline: Stage View Plugin (pipeline-stage-view)",
          "version": "2.11"
        },
        {
          "plugin": "Pipeline: Step API (workflow-step-api)",
          "version": "2.20"
        },
        {
          "plugin": "Pipeline: Supporting APIs (workflow-support)",
          "version": "3.3"
        },
        {
          "plugin": "Plain Credentials Plugin (plain-credentials)",
          "version": "1.5"
        },
        {
          "plugin": "Pub-Sub \"light\" Bus (pubsub-light)",
          "version": "1.12"
        },
        {
          "plugin": "REST API for Blue Ocean (blueocean-rest)",
          "version": "1.17.0"
        },
        {
          "plugin": "REST Implementation for Blue Ocean (blueocean-rest-impl)",
          "version": "1.17.0"
        },
        {
          "plugin": "Resource Disposer Plugin (resource-disposer)",
          "version": "0.13"
        },
        {
          "plugin": "Run Condition Plugin (run-condition)",
          "version": "1.2"
        },
        {
          "plugin": "SCM API Plugin (scm-api)",
          "version": "2.6.3"
        },
        {
          "plugin": "SSH Agent Plugin (ssh-agent)",
          "version": "1.17"
        },
        {
          "plugin": "SSH Credentials Plugin (ssh-credentials)",
          "version": "1.17.1"
        },
        {
          "plugin": "SSH Slaves plugin (ssh-slaves)",
          "version": "1.30.1"
        },
        {
          "plugin": "Script Security Plugin (script-security)",
          "version": "1.61"
        },
        {
          "plugin": "Server Sent Events (SSE) Gateway Plugin (sse-gateway)",
          "version": "1.18"
        },
        {
          "plugin": "Skip Notifications Trait plugin (skip-notifications-trait)",
          "version": "1.0.3"
        },
        {
          "plugin": "Structs Plugin (structs)",
          "version": "1.19"
        },
        {
          "plugin": "Subversion Plug-in (subversion)",
          "version": "2.12.2"
        },
        {
          "plugin": "Timestamper (timestamper)",
          "version": "1.10"
        },
        {
          "plugin": "Token Macro Plugin (token-macro)",
          "version": "2.8"
        },
        {
          "plugin": "Trilead API Plugin (trilead-api)",
          "version": "1.0.3"
        },
        {
          "plugin": "Variant Plugin (variant)",
          "version": "1.2"
        },
        {
          "plugin": "WMI Windows Agents Plugin (windows-slaves)",
          "version": "1.4"
        },
        {
          "plugin": "Web for Blue Ocean (blueocean-web)",
          "version": "1.17.0"
        },
        {
          "plugin": "Workspace Cleanup Plugin (ws-cleanup)",
          "version": "0.37"
        },
        {
          "plugin": "bouncycastle API Plugin (bouncycastle-api)",
          "version": "2.17"
        },
        {
          "plugin": "i18n for Blue Ocean (blueocean-i18n)",
          "version": "1.17.0"
        }
      ]
      

        Attachments

          Issue Links

            Activity

            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Olivier Lamy Sorry for my delayed feedback: AFAIK (no user complaints in my company – at least none that I know of...) the snapshot build of sse-gateway worked fine; and on Jenkins server neither remaining "EventDispatcher.retryProcessor" threads, nor thousands of them at the same time. => Looks good

            Show
            reinholdfuereder Reinhold Füreder added a comment - Olivier Lamy Sorry for my delayed feedback: AFAIK (no user complaints in my company – at least none that I know of...) the snapshot build of sse-gateway worked fine; and on Jenkins server neither remaining "EventDispatcher.retryProcessor" threads, nor thousands of them at the same time. => Looks good
            Hide
            2bluesc Kyle Manna added a comment -

            Updated on my server, thanks for all the hard work!

            Show
            2bluesc Kyle Manna added a comment - Updated on my server, thanks for all the hard work!
            Hide
            sorenfriis Søren Friis added a comment -

            There seems to be a problem with Showing Blue Ocean in MS Edge after this update.
            https://issues.jenkins-ci.org/browse/JENKINS-59291

             

            Show
            sorenfriis Søren Friis added a comment - There seems to be a problem with Showing Blue Ocean in MS Edge after this update. https://issues.jenkins-ci.org/browse/JENKINS-59291  
            Hide
            prsingh Pradeep Singh added a comment -

            Thread count accelerated to sky high (~22k) after upgrading Jenkins from 2.17 to 2.190.3.

            We had analysed thread dump which didn't help us to find the solution although, it showed couple of thread in blocked state.

            After upgrading Jenkins to LTS 2.204.4 solved this issue.

            Show
            prsingh Pradeep Singh added a comment - Thread count accelerated to sky high (~22k) after upgrading Jenkins from 2.17 to 2.190.3. We had analysed thread dump which didn't help us to find the solution although, it showed couple of thread in blocked state. After upgrading Jenkins to LTS  2.204.4  solved this issue.
            Hide
            kshultz Karl Shultz added a comment -

            I apologize for the noise just now, I must have hit a hotkey that assigned this to me. I've switched it back.

            Show
            kshultz Karl Shultz added a comment - I apologize for the noise just now, I must have hit a hotkey that assigned this to me. I've switched it back.

              People

              Assignee:
              olamy Olivier Lamy
              Reporter:
              2bluesc Kyle Manna
              Votes:
              9 Vote for this issue
              Watchers:
              18 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: