Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58684

Linux max task resource exhaustion after excessive timer thread creation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • sse-gateway-plugin
    • Jenkins 2.176.2 using Ubuntu 18.04 package file from pkg.jenkins.io/debian-stable
      AMD Threadripper 32-core/64-core machine with 128GB of RAM
      Single master node with 20 executors

      Jenkins is exhausting resources on my machine seemingly to create 10k + timer threads. This thread creation then hits the systemd imposed 15% of task limit set by the kernel (/proc/sys/kernel/pid_max = 131072) which is 19660 by default on my system.

      This particular java heap dump shows that 4675 of the total 4877 match the expression Timer-.* and have the Context Class Loader -> org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$CleanGroovyClassLoader.

      Bottom right corner of my Jenkins UI confirms I'm running the latest version (Jenkins ver. 2.176.2)with all plugins updated on OpenJDK Runtime Environment (build 1.8.0_222-b05) with Ubuntu 18.04 as of 2019.07.23.

       

      To dig deeper, after the resource limit is hit, threads fail to create and Jenkins becomes unstable until restarted, logs contain many instances of the following:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@621715d8 failed
java.lang.OutOfMemoryError: unable to create new native thread

       
      NetData on the system shows that if it doesn't hit the thread limit it will spike and often recover correctly suggesting the threads are cleaned up at some point if it doesn't derail the process first.

      I've increased the systemd limit on the system and the problem hasn't derailed the jenkins process in a few days by adding the following to the systemd service file:

      TasksMax=32768

      I have been capturing heaps with some automation tools and catch several heaps exceeding 20k threads in the last few days signifying that the process is still occurring and likely affecting other users, they probably aren't noticing though.

       

      Not sure if this is related to a plugin (listed below with version) or Jenkins core code?

       

      [
  {
    "plugin": "Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api)",
    "version": "4.5.5-3.0"
  },
  {
    "plugin": "Authentication Tokens API Plugin (authentication-tokens)",
    "version": "1.3"
  },
  {
    "plugin": "Authorize Project (authorize-project)",
    "version": "1.3.0"
  },
  {
    "plugin": "Autofavorite for Blue Ocean (blueocean-autofavorite)",
    "version": "1.2.4"
  },
  {
    "plugin": "Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source)",
    "version": "2.4.5"
  },
  {
    "plugin": "Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline)",
    "version": "1.17.0"
  },
  {
    "plugin": "Bitbucket Plugin (bitbucket)",
    "version": "1.1.10"
  },
  {
    "plugin": "Blue Ocean (blueocean)",
    "version": "1.17.0"
  },
  {
    "plugin": "Blue Ocean Core JS (blueocean-core-js)",
    "version": "1.17.0"
  },
  {
    "plugin": "Blue Ocean Executor Info (blueocean-executor-info)",
    "version": "1.17.0"
  },
  {
    "plugin": "Blue Ocean Pipeline Editor (blueocean-pipeline-editor)",
    "version": "1.17.0"
  },
  {
    "plugin": "Branch API Plugin (branch-api)",
    "version": "2.5.3"
  },
  {
    "plugin": "Build Timeout (build-timeout)",
    "version": "1.19"
  },
  {
    "plugin": "Command Agent Launcher Plugin (command-launcher)",
    "version": "1.3"
  },
  {
    "plugin": "Common API for Blue Ocean (blueocean-commons)",
    "version": "1.17.0"
  },
  {
    "plugin": "Conditional BuildStep (conditional-buildstep)",
    "version": "1.3.6"
  },
  {
    "plugin": "Config API for Blue Ocean (blueocean-config)",
    "version": "1.17.0"
  },
  {
    "plugin": "Copy Artifact Plugin (copyartifact)",
    "version": "1.42.1"
  },
  {
    "plugin": "Credentials Binding Plugin (credentials-binding)",
    "version": "1.19"
  },
  {
    "plugin": "Credentials Plugin (credentials)",
    "version": "2.2.0"
  },
  {
    "plugin": "Dashboard for Blue Ocean (blueocean-dashboard)",
    "version": "1.17.0"
  },
  {
    "plugin": "Design Language (jenkins-design-language)",
    "version": "1.17.0"
  },
  {
    "plugin": "Display URL API (display-url-api)",
    "version": "2.3.1"
  },
  {
    "plugin": "Display URL for Blue Ocean (blueocean-display-url)",
    "version": "2.3.0"
  },
  {
    "plugin": "Docker Commons Plugin (docker-commons)",
    "version": "1.15"
  },
  {
    "plugin": "Docker Pipeline (docker-workflow)",
    "version": "1.18"
  },
  {
    "plugin": "Durable Task Plugin (durable-task)",
    "version": "1.30"
  },
  {
    "plugin": "Email Extension Plugin (email-ext)",
    "version": "2.66"
  },
  {
    "plugin": "Events API for Blue Ocean (blueocean-events)",
    "version": "1.17.0"
  },
  {
    "plugin": "External Monitor Job Type Plugin (external-monitor-job)",
    "version": "1.7"
  },
  {
    "plugin": "Favorite (favorite)",
    "version": "2.3.2"
  },
  {
    "plugin": "Folders Plugin (cloudbees-folder)",
    "version": "6.9"
  },
  {
    "plugin": "GIT server Plugin (git-server)",
    "version": "1.7"
  },
  {
    "plugin": "Git Pipeline for Blue Ocean (blueocean-git-pipeline)",
    "version": "1.17.0"
  },
  {
    "plugin": "Git client plugin (git-client)",
    "version": "3.0.0-rc"
  },
  {
    "plugin": "Git plugin (git)",
    "version": "4.0.0-rc"
  },
  {
    "plugin": "GitHub API Plugin (github-api)",
    "version": "1.95"
  },
  {
    "plugin": "GitHub Branch Source Plugin (github-branch-source)",
    "version": "2.5.4"
  },
  {
    "plugin": "GitHub Pipeline for Blue Ocean (blueocean-github-pipeline)",
    "version": "1.17.0"
  },
  {
    "plugin": "GitHub plugin (github)",
    "version": "1.29.4"
  },
  {
    "plugin": "Google Login Plugin (google-login)",
    "version": "1.6"
  },
  {
    "plugin": "Gradle Plugin (gradle)",
    "version": "1.33"
  },
  {
    "plugin": "HTML Publisher plugin (htmlpublisher)",
    "version": "1.18"
  },
  {
    "plugin": "HTTP Request Plugin (http_request)",
    "version": "1.8.23"
  },
  {
    "plugin": "Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api)",
    "version": "2.1.7-1.0"
  },
  {
    "plugin": "Icon Shim Plugin (icon-shim)",
    "version": "2.0.3"
  },
  {
    "plugin": "JIRA Integration for Blue Ocean (blueocean-jira)",
    "version": "1.17.0"
  },
  {
    "plugin": "JIRA plugin (jira)",
    "version": "3.0.8"
  },
  {
    "plugin": "JSch dependency plugin (jsch)",
    "version": "0.1.55"
  },
  {
    "plugin": "JUnit Plugin (junit)",
    "version": "1.28"
  },
  {
    "plugin": "JWT for Blue Ocean (blueocean-jwt)",
    "version": "1.17.0"
  },
  {
    "plugin": "Jackson 2 API Plugin (jackson2-api)",
    "version": "2.9.9.1"
  },
  {
    "plugin": "JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor)",
    "version": "1.1"
  },
  {
    "plugin": "JavaScript GUI Lib: Handlebars bundle plugin (handlebars)",
    "version": "1.1.1"
  },
  {
    "plugin": "JavaScript GUI Lib: Moment.js bundle plugin (momentjs)",
    "version": "1.1.1"
  },
  {
    "plugin": "JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached)",
    "version": "1.2.1"
  },
  {
    "plugin": "Javadoc Plugin (javadoc)",
    "version": "1.5"
  },
  {
    "plugin": "LDAP Plugin (ldap)",
    "version": "1.20"
  },
  {
    "plugin": "Lockable Resources plugin (lockable-resources)",
    "version": "2.5"
  },
  {
    "plugin": "Mailer Plugin (mailer)",
    "version": "1.23"
  },
  {
    "plugin": "MapDB API Plugin (mapdb-api)",
    "version": "1.0.9.0"
  },
  {
    "plugin": "Matrix Authorization Strategy Plugin (matrix-auth)",
    "version": "2.4.2"
  },
  {
    "plugin": "Matrix Project Plugin (matrix-project)",
    "version": "1.14"
  },
  {
    "plugin": "Maven Integration plugin (maven-plugin)",
    "version": "3.3"
  },
  {
    "plugin": "Mercurial plugin (mercurial)",
    "version": "2.7"
  },
  {
    "plugin": "Metrics Plugin (metrics)",
    "version": "4.0.2.5"
  },
  {
    "plugin": "OWASP Markup Formatter Plugin (antisamy-markup-formatter)",
    "version": "1.5"
  },
  {
    "plugin": "Oracle Java SE Development Kit Installer Plugin (jdk-tool)",
    "version": "1.3"
  },
  {
    "plugin": "PAM Authentication plugin (pam-auth)",
    "version": "1.5.1"
  },
  {
    "plugin": "Personalization for Blue Ocean (blueocean-personalization)",
    "version": "1.17.0"
  },
  {
    "plugin": "Pipeline (workflow-aggregator)",
    "version": "2.6"
  },
  {
    "plugin": "Pipeline Graph Analysis Plugin (pipeline-graph-analysis)",
    "version": "1.10"
  },
  {
    "plugin": "Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api)",
    "version": "1.17.0"
  },
  {
    "plugin": "Pipeline Utility Steps (pipeline-utility-steps)",
    "version": "2.3.0"
  },
  {
    "plugin": "Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl)",
    "version": "1.17.0"
  },
  {
    "plugin": "Pipeline: API (workflow-api)",
    "version": "2.35"
  },
  {
    "plugin": "Pipeline: Basic Steps (workflow-basic-steps)",
    "version": "2.18"
  },
  {
    "plugin": "Pipeline: Build Step (pipeline-build-step)",
    "version": "2.9"
  },
  {
    "plugin": "Pipeline: Declarative (pipeline-model-definition)",
    "version": "1.3.9"
  },
  {
    "plugin": "Pipeline: Declarative Agent API (pipeline-model-declarative-agent)",
    "version": "1.1.1"
  },
  {
    "plugin": "Pipeline: Declarative Extension Points API (pipeline-model-extensions)",
    "version": "1.3.9"
  },
  {
    "plugin": "Pipeline: GitHub Groovy Libraries (pipeline-github-lib)",
    "version": "1.0"
  },
  {
    "plugin": "Pipeline: Groovy (workflow-cps)",
    "version": "2.72"
  },
  {
    "plugin": "Pipeline: Input Step (pipeline-input-step)",
    "version": "2.10"
  },
  {
    "plugin": "Pipeline: Job (workflow-job)",
    "version": "2.33"
  },
  {
    "plugin": "Pipeline: Milestone Step (pipeline-milestone-step)",
    "version": "1.3.1"
  },
  {
    "plugin": "Pipeline: Model API (pipeline-model-api)",
    "version": "1.3.9"
  },
  {
    "plugin": "Pipeline: Multibranch (workflow-multibranch)",
    "version": "2.21"
  },
  {
    "plugin": "Pipeline: Nodes and Processes (workflow-durable-task-step)",
    "version": "2.32"
  },
  {
    "plugin": "Pipeline: REST API Plugin (pipeline-rest-api)",
    "version": "2.11"
  },
  {
    "plugin": "Pipeline: SCM Step (workflow-scm-step)",
    "version": "2.9"
  },
  {
    "plugin": "Pipeline: Shared Groovy Libraries (workflow-cps-global-lib)",
    "version": "2.14"
  },
  {
    "plugin": "Pipeline: Stage Step (pipeline-stage-step)",
    "version": "2.3"
  },
  {
    "plugin": "Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata)",
    "version": "1.3.9"
  },
  {
    "plugin": "Pipeline: Stage View Plugin (pipeline-stage-view)",
    "version": "2.11"
  },
  {
    "plugin": "Pipeline: Step API (workflow-step-api)",
    "version": "2.20"
  },
  {
    "plugin": "Pipeline: Supporting APIs (workflow-support)",
    "version": "3.3"
  },
  {
    "plugin": "Plain Credentials Plugin (plain-credentials)",
    "version": "1.5"
  },
  {
    "plugin": "Pub-Sub \"light\" Bus (pubsub-light)",
    "version": "1.12"
  },
  {
    "plugin": "REST API for Blue Ocean (blueocean-rest)",
    "version": "1.17.0"
  },
  {
    "plugin": "REST Implementation for Blue Ocean (blueocean-rest-impl)",
    "version": "1.17.0"
  },
  {
    "plugin": "Resource Disposer Plugin (resource-disposer)",
    "version": "0.13"
  },
  {
    "plugin": "Run Condition Plugin (run-condition)",
    "version": "1.2"
  },
  {
    "plugin": "SCM API Plugin (scm-api)",
    "version": "2.6.3"
  },
  {
    "plugin": "SSH Agent Plugin (ssh-agent)",
    "version": "1.17"
  },
  {
    "plugin": "SSH Credentials Plugin (ssh-credentials)",
    "version": "1.17.1"
  },
  {
    "plugin": "SSH Slaves plugin (ssh-slaves)",
    "version": "1.30.1"
  },
  {
    "plugin": "Script Security Plugin (script-security)",
    "version": "1.61"
  },
  {
    "plugin": "Server Sent Events (SSE) Gateway Plugin (sse-gateway)",
    "version": "1.18"
  },
  {
    "plugin": "Skip Notifications Trait plugin (skip-notifications-trait)",
    "version": "1.0.3"
  },
  {
    "plugin": "Structs Plugin (structs)",
    "version": "1.19"
  },
  {
    "plugin": "Subversion Plug-in (subversion)",
    "version": "2.12.2"
  },
  {
    "plugin": "Timestamper (timestamper)",
    "version": "1.10"
  },
  {
    "plugin": "Token Macro Plugin (token-macro)",
    "version": "2.8"
  },
  {
    "plugin": "Trilead API Plugin (trilead-api)",
    "version": "1.0.3"
  },
  {
    "plugin": "Variant Plugin (variant)",
    "version": "1.2"
  },
  {
    "plugin": "WMI Windows Agents Plugin (windows-slaves)",
    "version": "1.4"
  },
  {
    "plugin": "Web for Blue Ocean (blueocean-web)",
    "version": "1.17.0"
  },
  {
    "plugin": "Workspace Cleanup Plugin (ws-cleanup)",
    "version": "0.37"
  },
  {
    "plugin": "bouncycastle API Plugin (bouncycastle-api)",
    "version": "2.17"
  },
  {
    "plugin": "i18n for Blue Ocean (blueocean-i18n)",
    "version": "1.17.0"
  }
]

        1. d119bce8-b89a-4ece-9327-a7fbfc791ff9.png
          454 kB
          Kyle
        2. image-2019-08-05-17-17-46-130.png
          96 kB
          Colin Chapman
        3. Jenkins_thread.jpg
          39 kB
          Pradeep Singh
        4. logs-jenkins-06.08.txt
          70 kB
          Colin Chapman
        5. Screenshot from 2019-07-22 15-35-03.png
          952 kB
          Kyle
        6. Screenshot from 2019-08-05 13-27-54.png
          484 kB
          Kyle
        7. Screenshot from 2019-08-20 11-29-37.png
          84 kB
          Kyle
        8. thread-dump-ft.png
          43 kB
          Colin Chapman

            olamy Olivier Lamy
            2bluesc Kyle
            Votes:
            9 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: