Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48685

Deadlock when running a Multijob with multiple slaves

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • multijob-plugin
    • None

      After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock.

      We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up.

      Attached is the deadlock stack traces from a thread dump.

      Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307

Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
- waiting to lock <0x000000008cdab698> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)

Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
- waiting to lock <0x000000008d744a90> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)

      We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible.

        1. jenkins-deadlock.txt
          88 kB

          [JENKINS-48685] Deadlock when running a Multijob with multiple slaves

          Mark Ketchum created issue -
          Mark Ketchum made changes -
          Environment New: Jenkins
          =======

          Version details
          ---------------

            * Version: `2.89.2`
            * Mode: WAR
            * Url: https://bld-ria.corp.pingidentity.com/
            * Java
                - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
                - Vendor: Oracle Corporation
                - Version: 1.8.0_151
                - Maximum memory: 1.62 GB (1743257600)
                - Allocated memory: 719.00 MB (753926144)
                - Free memory: 547.74 MB (574348208)
                - In-use memory: 171.26 MB (179577936)
                - GC strategy: ParallelGC
            * Java Runtime Specification
                - Name: Java Platform API Specification
                - Vendor: Oracle Corporation
                - Version: 1.8
            * JVM Specification
                - Name: Java Virtual Machine Specification
                - Vendor: Oracle Corporation
                - Version: 1.8
            * JVM Implementation
                - Name: OpenJDK 64-Bit Server VM
                - Vendor: Oracle Corporation
                - Version: 25.151-b12
            * Operating system
                - Name: Linux
                - Architecture: amd64
                - Version: 4.4.0-104-generic
                - Distribution: Ubuntu 16.04.3 LTS
            * Process ID: 1519 (0x5ef)
            * Process started: 2017-12-21 19:56:58.300+0000
            * Process uptime: 40 sec
            * JVM startup parameters:
                - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`
                - Classpath: `/usr/share/jenkins/jenkins.war`
                - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`
                - arg[0]: `-Djava.awt.headless=true`

          Important configuration
          ---------------

            * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`
            * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`
            * CSRF Protection: true
            * Initialization Milestone: Completed initialization

          Active Plugins
          --------------

            * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'
            * analysis-core:1.93 'Static Analysis Utilities'
            * ant:1.7 'Ant Plugin'
            * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'
            * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'
            * authentication-tokens:1.3 'Authentication Tokens API Plugin'
            * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'
            * aws-java-sdk:1.11.248 'Amazon Web Services SDK'
            * bouncycastle-api:2.16.2 'bouncycastle API Plugin'
            * branch-api:2.0.16 'Branch API Plugin'
            * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'
            * build-timeout:1.19 'Build Timeout'
            * build-token-root:1.4 'Build Authorization Token Root Plugin'
            * built-on-column:1.1 'built-on-column'
            * cloudbees-folder:6.2.1 'Folders Plugin'
            * command-launcher:1.2 'Command Agent Launcher Plugin'
            * conditional-buildstep:1.3.6 'Conditional BuildStep'
            * copyartifact:1.39 'Copy Artifact Plugin'
            * credentials:2.1.16 'Credentials Plugin'
            * credentials-binding:1.13 'Credentials Binding Plugin'
            * display-url-api:2.2.0 'Display URL API'
            * docker-commons:1.10 'Docker Commons Plugin'
            * docker-workflow:1.14 'Docker Pipeline'
            * durable-task:1.17 'Durable Task Plugin'
            * ec2:1.38 'Amazon EC2 plugin'
            * email-ext:2.61 'Email Extension Plugin'
            * envinject:2.1.5 'Environment Injector Plugin'
            * envinject-api:1.4 'EnvInject API Plugin'
            * external-monitor-job:1.7 'External Monitor Job Type Plugin'
            * ez-templates:1.3.1 'EZ Templates'
            * findbugs:4.71 'FindBugs Plug-in'
            * git:3.6.4 'Jenkins Git plugin'
            * git-client:2.7.0 'Jenkins Git client plugin'
            * git-server:1.7 'Jenkins GIT server Plugin'
            * github:1.28.1 'GitHub plugin'
            * github-api:1.90 'GitHub API Plugin'
            * github-branch-source:2.3.2 'GitHub Branch Source Plugin'
            * github-organization-folder:1.6 'GitHub Organization Folder Plugin'
            * gradle:1.28 'Gradle Plugin'
            * groovy-postbuild:2.3.1 'Groovy Postbuild'
            * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'
            * icon-shim:2.0.3 'Icon Shim Plugin'
            * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'
            * jacoco:2.2.1 'Jenkins JaCoCo plugin'
            * javadoc:1.4 'Javadoc Plugin'
            * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'
            * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'
            * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'
            * jsch:0.1.54.1 'Jenkins JSch dependency plugin'
            * junit:1.23 'JUnit Plugin'
            * ldap:1.18 'LDAP Plugin'
            * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'
            * mailer:1.20 'Jenkins Mailer Plugin'
            * mapdb-api:1.0.9.0 'MapDB API Plugin'
            * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'
            * matrix-project:1.12 'Matrix Project Plugin'
            * maven-plugin:3.0 'Maven Integration plugin'
            * metrics:3.1.2.10 'Metrics Plugin'
            * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'
            * node-iterator-api:1.5.0 'Node Iterator API Plugin'
            * notification:1.12 'Jenkins Notification plugin'
            * pam-auth:1.3 'PAM Authentication plugin'
            * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'
            * pipeline-build-step:2.6 'Pipeline: Build Step'
            * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'
            * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'
            * pipeline-input-step:2.8 'Pipeline: Input Step'
            * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'
            * pipeline-model-api:1.2.5 'Pipeline: Model API'
            * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'
            * pipeline-model-definition:1.2.5 'Pipeline: Declarative'
            * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'
            * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'
            * pipeline-stage-step:2.3 'Pipeline: Stage Step'
            * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'
            * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'
            * plain-credentials:1.4 'Plain Credentials Plugin'
            * python:1.3 'Python Plugin'
            * resource-disposer:0.8 'Resource Disposer Plugin'
            * run-condition:1.0 'Run Condition Plugin'
            * saml:1.0.4 'SAML Plugin'
            * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'
            * scm-api:2.2.6 'SCM API Plugin'
            * script-security:1.39 'Script Security Plugin'
            * simple-theme-plugin:0.3 'Simple Theme Plugin'
            * slack:2.3 'Slack Notification Plugin'
            * ssh-credentials:1.13 'SSH Credentials Plugin'
            * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'
            * structs:1.10 'Structs Plugin'
            * subversion:2.10.2 'Jenkins Subversion Plug-in'
            * support-core:2.44 'Support Core Plugin'
            * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'
            * timestamper:1.8.9 'Timestamper'
            * token-macro:2.3 'Token Macro Plugin'
            * unleash:2.3.0 'Unleash Maven Plugin'
            * windows-slaves:1.3.1 'Windows Slaves Plugin'
            * workflow-aggregator:2.5 'Pipeline'
            * workflow-api:2.24 'Pipeline: API'
            * workflow-basic-steps:2.6 'Pipeline: Basic Steps'
            * workflow-cps:2.42 'Pipeline: Groovy'
            * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'
            * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'
            * workflow-job:2.16 'Pipeline: Job'
            * workflow-multibranch:2.16 'Pipeline: Multibranch'
            * workflow-scm-step:2.6 'Pipeline: SCM Step'
            * workflow-step-api:2.14 'Pipeline: Step API'
            * workflow-support:2.16 'Pipeline: Supporting APIs'
            * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'
          Mark Ketchum made changes -
          Environment Original: Jenkins
          =======

          Version details
          ---------------

            * Version: `2.89.2`
            * Mode: WAR
            * Url: https://bld-ria.corp.pingidentity.com/
            * Java
                - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
                - Vendor: Oracle Corporation
                - Version: 1.8.0_151
                - Maximum memory: 1.62 GB (1743257600)
                - Allocated memory: 719.00 MB (753926144)
                - Free memory: 547.74 MB (574348208)
                - In-use memory: 171.26 MB (179577936)
                - GC strategy: ParallelGC
            * Java Runtime Specification
                - Name: Java Platform API Specification
                - Vendor: Oracle Corporation
                - Version: 1.8
            * JVM Specification
                - Name: Java Virtual Machine Specification
                - Vendor: Oracle Corporation
                - Version: 1.8
            * JVM Implementation
                - Name: OpenJDK 64-Bit Server VM
                - Vendor: Oracle Corporation
                - Version: 25.151-b12
            * Operating system
                - Name: Linux
                - Architecture: amd64
                - Version: 4.4.0-104-generic
                - Distribution: Ubuntu 16.04.3 LTS
            * Process ID: 1519 (0x5ef)
            * Process started: 2017-12-21 19:56:58.300+0000
            * Process uptime: 40 sec
            * JVM startup parameters:
                - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`
                - Classpath: `/usr/share/jenkins/jenkins.war`
                - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`
                - arg[0]: `-Djava.awt.headless=true`

          Important configuration
          ---------------

            * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`
            * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`
            * CSRF Protection: true
            * Initialization Milestone: Completed initialization

          Active Plugins
          --------------

            * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'
            * analysis-core:1.93 'Static Analysis Utilities'
            * ant:1.7 'Ant Plugin'
            * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'
            * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'
            * authentication-tokens:1.3 'Authentication Tokens API Plugin'
            * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'
            * aws-java-sdk:1.11.248 'Amazon Web Services SDK'
            * bouncycastle-api:2.16.2 'bouncycastle API Plugin'
            * branch-api:2.0.16 'Branch API Plugin'
            * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'
            * build-timeout:1.19 'Build Timeout'
            * build-token-root:1.4 'Build Authorization Token Root Plugin'
            * built-on-column:1.1 'built-on-column'
            * cloudbees-folder:6.2.1 'Folders Plugin'
            * command-launcher:1.2 'Command Agent Launcher Plugin'
            * conditional-buildstep:1.3.6 'Conditional BuildStep'
            * copyartifact:1.39 'Copy Artifact Plugin'
            * credentials:2.1.16 'Credentials Plugin'
            * credentials-binding:1.13 'Credentials Binding Plugin'
            * display-url-api:2.2.0 'Display URL API'
            * docker-commons:1.10 'Docker Commons Plugin'
            * docker-workflow:1.14 'Docker Pipeline'
            * durable-task:1.17 'Durable Task Plugin'
            * ec2:1.38 'Amazon EC2 plugin'
            * email-ext:2.61 'Email Extension Plugin'
            * envinject:2.1.5 'Environment Injector Plugin'
            * envinject-api:1.4 'EnvInject API Plugin'
            * external-monitor-job:1.7 'External Monitor Job Type Plugin'
            * ez-templates:1.3.1 'EZ Templates'
            * findbugs:4.71 'FindBugs Plug-in'
            * git:3.6.4 'Jenkins Git plugin'
            * git-client:2.7.0 'Jenkins Git client plugin'
            * git-server:1.7 'Jenkins GIT server Plugin'
            * github:1.28.1 'GitHub plugin'
            * github-api:1.90 'GitHub API Plugin'
            * github-branch-source:2.3.2 'GitHub Branch Source Plugin'
            * github-organization-folder:1.6 'GitHub Organization Folder Plugin'
            * gradle:1.28 'Gradle Plugin'
            * groovy-postbuild:2.3.1 'Groovy Postbuild'
            * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'
            * icon-shim:2.0.3 'Icon Shim Plugin'
            * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'
            * jacoco:2.2.1 'Jenkins JaCoCo plugin'
            * javadoc:1.4 'Javadoc Plugin'
            * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'
            * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'
            * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'
            * jsch:0.1.54.1 'Jenkins JSch dependency plugin'
            * junit:1.23 'JUnit Plugin'
            * ldap:1.18 'LDAP Plugin'
            * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'
            * mailer:1.20 'Jenkins Mailer Plugin'
            * mapdb-api:1.0.9.0 'MapDB API Plugin'
            * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'
            * matrix-project:1.12 'Matrix Project Plugin'
            * maven-plugin:3.0 'Maven Integration plugin'
            * metrics:3.1.2.10 'Metrics Plugin'
            * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'
            * node-iterator-api:1.5.0 'Node Iterator API Plugin'
            * notification:1.12 'Jenkins Notification plugin'
            * pam-auth:1.3 'PAM Authentication plugin'
            * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'
            * pipeline-build-step:2.6 'Pipeline: Build Step'
            * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'
            * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'
            * pipeline-input-step:2.8 'Pipeline: Input Step'
            * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'
            * pipeline-model-api:1.2.5 'Pipeline: Model API'
            * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'
            * pipeline-model-definition:1.2.5 'Pipeline: Declarative'
            * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'
            * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'
            * pipeline-stage-step:2.3 'Pipeline: Stage Step'
            * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'
            * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'
            * plain-credentials:1.4 'Plain Credentials Plugin'
            * python:1.3 'Python Plugin'
            * resource-disposer:0.8 'Resource Disposer Plugin'
            * run-condition:1.0 'Run Condition Plugin'
            * saml:1.0.4 'SAML Plugin'
            * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'
            * scm-api:2.2.6 'SCM API Plugin'
            * script-security:1.39 'Script Security Plugin'
            * simple-theme-plugin:0.3 'Simple Theme Plugin'
            * slack:2.3 'Slack Notification Plugin'
            * ssh-credentials:1.13 'SSH Credentials Plugin'
            * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'
            * structs:1.10 'Structs Plugin'
            * subversion:2.10.2 'Jenkins Subversion Plug-in'
            * support-core:2.44 'Support Core Plugin'
            * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'
            * timestamper:1.8.9 'Timestamper'
            * token-macro:2.3 'Token Macro Plugin'
            * unleash:2.3.0 'Unleash Maven Plugin'
            * windows-slaves:1.3.1 'Windows Slaves Plugin'
            * workflow-aggregator:2.5 'Pipeline'
            * workflow-api:2.24 'Pipeline: API'
            * workflow-basic-steps:2.6 'Pipeline: Basic Steps'
            * workflow-cps:2.42 'Pipeline: Groovy'
            * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'
            * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'
            * workflow-job:2.16 'Pipeline: Job'
            * workflow-multibranch:2.16 'Pipeline: Multibranch'
            * workflow-scm-step:2.6 'Pipeline: SCM Step'
            * workflow-step-api:2.14 'Pipeline: Step API'
            * workflow-support:2.16 'Pipeline: Supporting APIs'
            * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'
          		 New: Jenkins
          =======

          Version details
          ---------------

            * Version: `2.89.2`
            * Mode: WAR
            * Url: *****
            * Java
                - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
                - Vendor: Oracle Corporation
                - Version: 1.8.0_151
                - Maximum memory: 1.62 GB (1743257600)
                - Allocated memory: 719.00 MB (753926144)
                - Free memory: 547.74 MB (574348208)
                - In-use memory: 171.26 MB (179577936)
                - GC strategy: ParallelGC
            * Java Runtime Specification
                - Name: Java Platform API Specification
                - Vendor: Oracle Corporation
                - Version: 1.8
            * JVM Specification
                - Name: Java Virtual Machine Specification
                - Vendor: Oracle Corporation
                - Version: 1.8
            * JVM Implementation
                - Name: OpenJDK 64-Bit Server VM
                - Vendor: Oracle Corporation
                - Version: 25.151-b12
            * Operating system
                - Name: Linux
                - Architecture: amd64
                - Version: 4.4.0-104-generic
                - Distribution: Ubuntu 16.04.3 LTS
            * Process ID: 1519 (0x5ef)
            * Process started: 2017-12-21 19:56:58.300+0000
            * Process uptime: 40 sec
            * JVM startup parameters:
                - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`
                - Classpath: `/usr/share/jenkins/jenkins.war`
                - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`
                - arg[0]: `-Djava.awt.headless=true`

          Important configuration
          ---------------

            * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`
            * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`
            * CSRF Protection: true
            * Initialization Milestone: Completed initialization

          Active Plugins
          --------------

            * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'
            * analysis-core:1.93 'Static Analysis Utilities'
            * ant:1.7 'Ant Plugin'
            * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'
            * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'
            * authentication-tokens:1.3 'Authentication Tokens API Plugin'
            * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'
            * aws-java-sdk:1.11.248 'Amazon Web Services SDK'
            * bouncycastle-api:2.16.2 'bouncycastle API Plugin'
            * branch-api:2.0.16 'Branch API Plugin'
            * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'
            * build-timeout:1.19 'Build Timeout'
            * build-token-root:1.4 'Build Authorization Token Root Plugin'
            * built-on-column:1.1 'built-on-column'
            * cloudbees-folder:6.2.1 'Folders Plugin'
            * command-launcher:1.2 'Command Agent Launcher Plugin'
            * conditional-buildstep:1.3.6 'Conditional BuildStep'
            * copyartifact:1.39 'Copy Artifact Plugin'
            * credentials:2.1.16 'Credentials Plugin'
            * credentials-binding:1.13 'Credentials Binding Plugin'
            * display-url-api:2.2.0 'Display URL API'
            * docker-commons:1.10 'Docker Commons Plugin'
            * docker-workflow:1.14 'Docker Pipeline'
            * durable-task:1.17 'Durable Task Plugin'
            * ec2:1.38 'Amazon EC2 plugin'
            * email-ext:2.61 'Email Extension Plugin'
            * envinject:2.1.5 'Environment Injector Plugin'
            * envinject-api:1.4 'EnvInject API Plugin'
            * external-monitor-job:1.7 'External Monitor Job Type Plugin'
            * ez-templates:1.3.1 'EZ Templates'
            * findbugs:4.71 'FindBugs Plug-in'
            * git:3.6.4 'Jenkins Git plugin'
            * git-client:2.7.0 'Jenkins Git client plugin'
            * git-server:1.7 'Jenkins GIT server Plugin'
            * github:1.28.1 'GitHub plugin'
            * github-api:1.90 'GitHub API Plugin'
            * github-branch-source:2.3.2 'GitHub Branch Source Plugin'
            * github-organization-folder:1.6 'GitHub Organization Folder Plugin'
            * gradle:1.28 'Gradle Plugin'
            * groovy-postbuild:2.3.1 'Groovy Postbuild'
            * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'
            * icon-shim:2.0.3 'Icon Shim Plugin'
            * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'
            * jacoco:2.2.1 'Jenkins JaCoCo plugin'
            * javadoc:1.4 'Javadoc Plugin'
            * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'
            * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'
            * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'
            * jsch:0.1.54.1 'Jenkins JSch dependency plugin'
            * junit:1.23 'JUnit Plugin'
            * ldap:1.18 'LDAP Plugin'
            * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'
            * mailer:1.20 'Jenkins Mailer Plugin'
            * mapdb-api:1.0.9.0 'MapDB API Plugin'
            * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'
            * matrix-project:1.12 'Matrix Project Plugin'
            * maven-plugin:3.0 'Maven Integration plugin'
            * metrics:3.1.2.10 'Metrics Plugin'
            * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'
            * node-iterator-api:1.5.0 'Node Iterator API Plugin'
            * notification:1.12 'Jenkins Notification plugin'
            * pam-auth:1.3 'PAM Authentication plugin'
            * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'
            * pipeline-build-step:2.6 'Pipeline: Build Step'
            * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'
            * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'
            * pipeline-input-step:2.8 'Pipeline: Input Step'
            * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'
            * pipeline-model-api:1.2.5 'Pipeline: Model API'
            * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'
            * pipeline-model-definition:1.2.5 'Pipeline: Declarative'
            * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'
            * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'
            * pipeline-stage-step:2.3 'Pipeline: Stage Step'
            * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'
            * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'
            * plain-credentials:1.4 'Plain Credentials Plugin'
            * python:1.3 'Python Plugin'
            * resource-disposer:0.8 'Resource Disposer Plugin'
            * run-condition:1.0 'Run Condition Plugin'
            * saml:1.0.4 'SAML Plugin'
            * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'
            * scm-api:2.2.6 'SCM API Plugin'
            * script-security:1.39 'Script Security Plugin'
            * simple-theme-plugin:0.3 'Simple Theme Plugin'
            * slack:2.3 'Slack Notification Plugin'
            * ssh-credentials:1.13 'SSH Credentials Plugin'
            * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'
            * structs:1.10 'Structs Plugin'
            * subversion:2.10.2 'Jenkins Subversion Plug-in'
            * support-core:2.44 'Support Core Plugin'
            * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'
            * timestamper:1.8.9 'Timestamper'
            * token-macro:2.3 'Token Macro Plugin'
            * unleash:2.3.0 'Unleash Maven Plugin'
            * windows-slaves:1.3.1 'Windows Slaves Plugin'
            * workflow-aggregator:2.5 'Pipeline'
            * workflow-api:2.24 'Pipeline: API'
            * workflow-basic-steps:2.6 'Pipeline: Basic Steps'
            * workflow-cps:2.42 'Pipeline: Groovy'
            * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'
            * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'
            * workflow-job:2.16 'Pipeline: Job'
            * workflow-multibranch:2.16 'Pipeline: Multibranch'
            * workflow-scm-step:2.6 'Pipeline: SCM Step'
            * workflow-step-api:2.14 'Pipeline: Step API'
            * workflow-support:2.16 'Pipeline: Supporting APIs'
            * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'
          Mark Ketchum made changes -
          Description Original: After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock.

          We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up.

          Attached is the deadlock stack traces from a thread dump.

          We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible.

          {code:java}
          Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307

          Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED
          stackTrace:
          java.lang.Thread.State: BLOCKED (on object monitor)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
          - waiting to lock <0x000000008cdab698> (a hudson.model.RunMap)
          at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
          at hudson.model.Run.fromExternalizableId(Run.java:2345)
          at hudson.model.Run$Replacer.readResolve(Run.java:1937)

          Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED
          stackTrace:
          java.lang.Thread.State: BLOCKED (on object monitor)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
          - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap)
          at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
          at hudson.model.Run.fromExternalizableId(Run.java:2345)
          at hudson.model.Run$Replacer.readResolve(Run.java:1937)
          at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source){code}           		New: After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock.

          We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up.

          Attached is the deadlock stack traces from a thread dump.

          {code:java}
          Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307

          Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED
          stackTrace:
          java.lang.Thread.State: BLOCKED (on object monitor)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
          - waiting to lock <0x000000008cdab698> (a hudson.model.RunMap)
          at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
          at hudson.model.Run.fromExternalizableId(Run.java:2345)
          at hudson.model.Run$Replacer.readResolve(Run.java:1937)

          Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED
          stackTrace:
          java.lang.Thread.State: BLOCKED (on object monitor)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
          - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap)
          at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
          at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
          at hudson.model.Run.fromExternalizableId(Run.java:2345)
          at hudson.model.Run$Replacer.readResolve(Run.java:1937)
          at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source){code}

          We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible.
          Allan BURDAJEWICZ made changes -
          Link New: This issue relates to JENKINS-49328 [ JENKINS-49328 ]
          Jesse Glick made changes -
          Priority Original: Minor [ 4 ] New: Critical [ 2 ]
          Jesse Glick made changes -
          Component/s Original: core [ 15593 ]
          Jesse Glick made changes -
          Resolution New: Duplicate [ 3 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Julien Duchesne made changes -
          Attachment New: dump.txt [ 41362 ]
          Attachment New: exception.txt [ 41363 ]
          Julien Duchesne made changes -
          Comment [ I don't think this is fixed by PR #129 (https://github.com/jenkinsci/tikal-multijob-plugin/pull/129) in JENKINS-49328. I merged in that PR into a custom version yesterday and I still got a deadlock during the night. Here is a thread dump and an exception I get about every second only since I upgraded Jenkins to 2.89.3.

          [^exception.txt]

          [^dump.txt]

            ]
          Julien Duchesne made changes -
          Attachment Original: exception.txt [ 41363 ]

            Unassigned Unassigned
            ketchumm Mark Ketchum
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: