• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Component/s: multijob-plugin
• Labels:

  None
• Environment:

  Hide

  Jenkins
  =======
  
  Version details
  ---------------
  
    * Version: `2.89.2`
    * Mode: WAR
    * Url: *****
    * Java
        - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
        - Vendor: Oracle Corporation
        - Version: 1.8.0_151
        - Maximum memory: 1.62 GB (1743257600)
        - Allocated memory: 719.00 MB (753926144)
        - Free memory: 547.74 MB (574348208)
        - In-use memory: 171.26 MB (179577936)
        - GC strategy: ParallelGC
    * Java Runtime Specification
        - Name: Java Platform API Specification
        - Vendor: Oracle Corporation
        - Version: 1.8
    * JVM Specification
        - Name: Java Virtual Machine Specification
        - Vendor: Oracle Corporation
        - Version: 1.8
    * JVM Implementation
        - Name: OpenJDK 64-Bit Server VM
        - Vendor: Oracle Corporation
        - Version: 25.151-b12
    * Operating system
        - Name: Linux
        - Architecture: amd64
        - Version: 4.4.0-104-generic
        - Distribution: Ubuntu 16.04.3 LTS
    * Process ID: 1519 (0x5ef)
    * Process started: 2017-12-21 19:56:58.300+0000
    * Process uptime: 40 sec
    * JVM startup parameters:
        - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`
        - Classpath: `/usr/share/jenkins/jenkins.war`
        - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`
        - arg[0]: `-Djava.awt.headless=true`
  
  Important configuration
  ---------------
  
    * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`
    * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`
    * CSRF Protection: true
    * Initialization Milestone: Completed initialization
  
  Active Plugins
  --------------
  
    * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'
    * analysis-core:1.93 'Static Analysis Utilities'
    * ant:1.7 'Ant Plugin'
    * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'
    * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'
    * authentication-tokens:1.3 'Authentication Tokens API Plugin'
    * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'
    * aws-java-sdk:1.11.248 'Amazon Web Services SDK'
    * bouncycastle-api:2.16.2 'bouncycastle API Plugin'
    * branch-api:2.0.16 'Branch API Plugin'
    * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'
    * build-timeout:1.19 'Build Timeout'
    * build-token-root:1.4 'Build Authorization Token Root Plugin'
    * built-on-column:1.1 'built-on-column'
    * cloudbees-folder:6.2.1 'Folders Plugin'
    * command-launcher:1.2 'Command Agent Launcher Plugin'
    * conditional-buildstep:1.3.6 'Conditional BuildStep'
    * copyartifact:1.39 'Copy Artifact Plugin'
    * credentials:2.1.16 'Credentials Plugin'
    * credentials-binding:1.13 'Credentials Binding Plugin'
    * display-url-api:2.2.0 'Display URL API'
    * docker-commons:1.10 'Docker Commons Plugin'
    * docker-workflow:1.14 'Docker Pipeline'
    * durable-task:1.17 'Durable Task Plugin'
    * ec2:1.38 'Amazon EC2 plugin'
    * email-ext:2.61 'Email Extension Plugin'
    * envinject:2.1.5 'Environment Injector Plugin'
    * envinject-api:1.4 'EnvInject API Plugin'
    * external-monitor-job:1.7 'External Monitor Job Type Plugin'
    * ez-templates:1.3.1 'EZ Templates'
    * findbugs:4.71 'FindBugs Plug-in'
    * git:3.6.4 'Jenkins Git plugin'
    * git-client:2.7.0 'Jenkins Git client plugin'
    * git-server:1.7 'Jenkins GIT server Plugin'
    * github:1.28.1 'GitHub plugin'
    * github-api:1.90 'GitHub API Plugin'
    * github-branch-source:2.3.2 'GitHub Branch Source Plugin'
    * github-organization-folder:1.6 'GitHub Organization Folder Plugin'
    * gradle:1.28 'Gradle Plugin'
    * groovy-postbuild:2.3.1 'Groovy Postbuild'
    * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'
    * icon-shim:2.0.3 'Icon Shim Plugin'
    * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'
    * jacoco:2.2.1 'Jenkins JaCoCo plugin'
    * javadoc:1.4 'Javadoc Plugin'
    * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'
    * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'
    * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'
    * jsch:0.1.54.1 'Jenkins JSch dependency plugin'
    * junit:1.23 'JUnit Plugin'
    * ldap:1.18 'LDAP Plugin'
    * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'
    * mailer:1.20 'Jenkins Mailer Plugin'
    * mapdb-api:1.0.9.0 'MapDB API Plugin'
    * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'
    * matrix-project:1.12 'Matrix Project Plugin'
    * maven-plugin:3.0 'Maven Integration plugin'
    * metrics:3.1.2.10 'Metrics Plugin'
    * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'
    * node-iterator-api:1.5.0 'Node Iterator API Plugin'
    * notification:1.12 'Jenkins Notification plugin'
    * pam-auth:1.3 'PAM Authentication plugin'
    * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'
    * pipeline-build-step:2.6 'Pipeline: Build Step'
    * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'
    * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'
    * pipeline-input-step:2.8 'Pipeline: Input Step'
    * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'
    * pipeline-model-api:1.2.5 'Pipeline: Model API'
    * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'
    * pipeline-model-definition:1.2.5 'Pipeline: Declarative'
    * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'
    * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'
    * pipeline-stage-step:2.3 'Pipeline: Stage Step'
    * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'
    * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'
    * plain-credentials:1.4 'Plain Credentials Plugin'
    * python:1.3 'Python Plugin'
    * resource-disposer:0.8 'Resource Disposer Plugin'
    * run-condition:1.0 'Run Condition Plugin'
    * saml:1.0.4 'SAML Plugin'
    * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'
    * scm-api:2.2.6 'SCM API Plugin'
    * script-security:1.39 'Script Security Plugin'
    * simple-theme-plugin:0.3 'Simple Theme Plugin'
    * slack:2.3 'Slack Notification Plugin'
    * ssh-credentials:1.13 'SSH Credentials Plugin'
    * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'
    * structs:1.10 'Structs Plugin'
    * subversion:2.10.2 'Jenkins Subversion Plug-in'
    * support-core:2.44 'Support Core Plugin'
    * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'
    * timestamper:1.8.9 'Timestamper'
    * token-macro:2.3 'Token Macro Plugin'
    * unleash:2.3.0 'Unleash Maven Plugin'
    * windows-slaves:1.3.1 'Windows Slaves Plugin'
    * workflow-aggregator:2.5 'Pipeline'
    * workflow-api:2.24 'Pipeline: API'
    * workflow-basic-steps:2.6 'Pipeline: Basic Steps'
    * workflow-cps:2.42 'Pipeline: Groovy'
    * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'
    * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'
    * workflow-job:2.16 'Pipeline: Job'
    * workflow-multibranch:2.16 'Pipeline: Multibranch'
    * workflow-scm-step:2.6 'Pipeline: SCM Step'
    * workflow-step-api:2.14 'Pipeline: Step API'
    * workflow-support:2.16 'Pipeline: Supporting APIs'
    * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'
  

  Show

  Jenkins ======= Version details ---------------   * Version: `2.89.2`   * Mode: WAR   * Url: *****   * Java       - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`       - Vendor: Oracle Corporation       - Version: 1.8.0_151       - Maximum memory: 1.62 GB (1743257600)       - Allocated memory: 719.00 MB (753926144)       - Free memory: 547.74 MB (574348208)       - In-use memory: 171.26 MB (179577936)       - GC strategy: ParallelGC   * Java Runtime Specification       - Name: Java Platform API Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Specification       - Name: Java Virtual Machine Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Implementation       - Name: OpenJDK 64-Bit Server VM       - Vendor: Oracle Corporation       - Version: 25.151-b12   * Operating system       - Name: Linux       - Architecture: amd64       - Version: 4.4.0-104-generic       - Distribution: Ubuntu 16.04.3 LTS   * Process ID: 1519 (0x5ef)   * Process started: 2017-12-21 19:56:58.300+0000   * Process uptime: 40 sec   * JVM startup parameters:       - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`       - Classpath: `/usr/share/jenkins/jenkins.war`       - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`       - arg[0]: `-Djava.awt.headless=true` Important configuration ---------------   * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`   * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`   * CSRF Protection: true   * Initialization Milestone: Completed initialization Active Plugins --------------   * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'   * analysis-core:1.93 'Static Analysis Utilities'   * ant:1.7 'Ant Plugin'   * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'   * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'   * authentication-tokens:1.3 'Authentication Tokens API Plugin'   * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'   * aws-java-sdk:1.11.248 'Amazon Web Services SDK'   * bouncycastle-api:2.16.2 'bouncycastle API Plugin'   * branch-api:2.0.16 'Branch API Plugin'   * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'   * build-timeout:1.19 'Build Timeout'   * build-token-root:1.4 'Build Authorization Token Root Plugin'   * built-on-column:1.1 'built-on-column'   * cloudbees-folder:6.2.1 'Folders Plugin'   * command-launcher:1.2 'Command Agent Launcher Plugin'   * conditional-buildstep:1.3.6 'Conditional BuildStep'   * copyartifact:1.39 'Copy Artifact Plugin'   * credentials:2.1.16 'Credentials Plugin'   * credentials-binding:1.13 'Credentials Binding Plugin'   * display-url-api:2.2.0 'Display URL API'   * docker-commons:1.10 'Docker Commons Plugin'   * docker-workflow:1.14 'Docker Pipeline'   * durable-task:1.17 'Durable Task Plugin'   * ec2:1.38 'Amazon EC2 plugin'   * email-ext:2.61 'Email Extension Plugin'   * envinject:2.1.5 'Environment Injector Plugin'   * envinject-api:1.4 'EnvInject API Plugin'   * external-monitor-job:1.7 'External Monitor Job Type Plugin'   * ez-templates:1.3.1 'EZ Templates'   * findbugs:4.71 'FindBugs Plug-in'   * git:3.6.4 'Jenkins Git plugin'   * git-client:2.7.0 'Jenkins Git client plugin'   * git-server:1.7 'Jenkins GIT server Plugin'   * github:1.28.1 'GitHub plugin'   * github-api:1.90 'GitHub API Plugin'   * github-branch-source:2.3.2 'GitHub Branch Source Plugin'   * github-organization-folder:1.6 'GitHub Organization Folder Plugin'   * gradle:1.28 'Gradle Plugin'   * groovy-postbuild:2.3.1 'Groovy Postbuild'   * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'   * icon-shim:2.0.3 'Icon Shim Plugin'   * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'   * jacoco:2.2.1 'Jenkins JaCoCo plugin'   * javadoc:1.4 'Javadoc Plugin'   * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'   * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'   * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'   * jsch:0.1.54.1 'Jenkins JSch dependency plugin'   * junit:1.23 'JUnit Plugin'   * ldap:1.18 'LDAP Plugin'   * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'   * mailer:1.20 'Jenkins Mailer Plugin'   * mapdb-api:1.0.9.0 'MapDB API Plugin'   * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'   * matrix-project:1.12 'Matrix Project Plugin'   * maven-plugin:3.0 'Maven Integration plugin'   * metrics:3.1.2.10 'Metrics Plugin'   * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'   * node-iterator-api:1.5.0 'Node Iterator API Plugin'   * notification:1.12 'Jenkins Notification plugin'   * pam-auth:1.3 'PAM Authentication plugin'   * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'   * pipeline-build-step:2.6 'Pipeline: Build Step'   * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'   * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'   * pipeline-input-step:2.8 'Pipeline: Input Step'   * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'   * pipeline-model-api:1.2.5 'Pipeline: Model API'   * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'   * pipeline-model-definition:1.2.5 'Pipeline: Declarative'   * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'   * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'   * pipeline-stage-step:2.3 'Pipeline: Stage Step'   * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'   * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'   * plain-credentials:1.4 'Plain Credentials Plugin'   * python:1.3 'Python Plugin'   * resource-disposer:0.8 'Resource Disposer Plugin'   * run-condition:1.0 'Run Condition Plugin'   * saml:1.0.4 'SAML Plugin'   * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'   * scm-api:2.2.6 'SCM API Plugin'   * script-security:1.39 'Script Security Plugin'   * simple-theme-plugin:0.3 'Simple Theme Plugin'   * slack:2.3 'Slack Notification Plugin'   * ssh-credentials:1.13 'SSH Credentials Plugin'   * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'   * structs:1.10 'Structs Plugin'   * subversion:2.10.2 'Jenkins Subversion Plug-in'   * support-core:2.44 'Support Core Plugin'   * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'   * timestamper:1.8.9 'Timestamper'   * token-macro:2.3 'Token Macro Plugin'   * unleash:2.3.0 'Unleash Maven Plugin'   * windows-slaves:1.3.1 'Windows Slaves Plugin'   * workflow-aggregator:2.5 'Pipeline'   * workflow-api:2.24 'Pipeline: API'   * workflow-basic-steps:2.6 'Pipeline: Basic Steps'   * workflow-cps:2.42 'Pipeline: Groovy'   * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'   * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'   * workflow-job:2.16 'Pipeline: Job'   * workflow-multibranch:2.16 'Pipeline: Multibranch'   * workflow-scm-step:2.6 'Pipeline: SCM Step'   * workflow-step-api:2.14 'Pipeline: Step API'   * workflow-support:2.16 'Pipeline: Supporting APIs'   * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'

1. jenkins-deadlock.txt

   88 kB

   2017-12-21 17:18

[JENKINS-48685] Deadlock when running a Multijob with multiple slaves

Mark Ketchum created issue - 2017-12-21 20:07

Mark Ketchum made changes - 2017-12-21 20:09

Environment

New: Jenkins ======= Version details ---------------   * Version: `2.89.2`   * Mode: WAR   * Url: https://bld-ria.corp.pingidentity.com/   * Java       - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`       - Vendor: Oracle Corporation       - Version: 1.8.0_151       - Maximum memory: 1.62 GB (1743257600)       - Allocated memory: 719.00 MB (753926144)       - Free memory: 547.74 MB (574348208)       - In-use memory: 171.26 MB (179577936)       - GC strategy: ParallelGC   * Java Runtime Specification       - Name: Java Platform API Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Specification       - Name: Java Virtual Machine Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Implementation       - Name: OpenJDK 64-Bit Server VM       - Vendor: Oracle Corporation       - Version: 25.151-b12   * Operating system       - Name: Linux       - Architecture: amd64       - Version: 4.4.0-104-generic       - Distribution: Ubuntu 16.04.3 LTS   * Process ID: 1519 (0x5ef)   * Process started: 2017-12-21 19:56:58.300+0000   * Process uptime: 40 sec   * JVM startup parameters:       - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`       - Classpath: `/usr/share/jenkins/jenkins.war`       - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`       - arg[0]: `-Djava.awt.headless=true` Important configuration ---------------   * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`   * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`   * CSRF Protection: true   * Initialization Milestone: Completed initialization Active Plugins --------------   * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'   * analysis-core:1.93 'Static Analysis Utilities'   * ant:1.7 'Ant Plugin'   * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'   * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'   * authentication-tokens:1.3 'Authentication Tokens API Plugin'   * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'   * aws-java-sdk:1.11.248 'Amazon Web Services SDK'   * bouncycastle-api:2.16.2 'bouncycastle API Plugin'   * branch-api:2.0.16 'Branch API Plugin'   * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'   * build-timeout:1.19 'Build Timeout'   * build-token-root:1.4 'Build Authorization Token Root Plugin'   * built-on-column:1.1 'built-on-column'   * cloudbees-folder:6.2.1 'Folders Plugin'   * command-launcher:1.2 'Command Agent Launcher Plugin'   * conditional-buildstep:1.3.6 'Conditional BuildStep'   * copyartifact:1.39 'Copy Artifact Plugin'   * credentials:2.1.16 'Credentials Plugin'   * credentials-binding:1.13 'Credentials Binding Plugin'   * display-url-api:2.2.0 'Display URL API'   * docker-commons:1.10 'Docker Commons Plugin'   * docker-workflow:1.14 'Docker Pipeline'   * durable-task:1.17 'Durable Task Plugin'   * ec2:1.38 'Amazon EC2 plugin'   * email-ext:2.61 'Email Extension Plugin'   * envinject:2.1.5 'Environment Injector Plugin'   * envinject-api:1.4 'EnvInject API Plugin'   * external-monitor-job:1.7 'External Monitor Job Type Plugin'   * ez-templates:1.3.1 'EZ Templates'   * findbugs:4.71 'FindBugs Plug-in'   * git:3.6.4 'Jenkins Git plugin'   * git-client:2.7.0 'Jenkins Git client plugin'   * git-server:1.7 'Jenkins GIT server Plugin'   * github:1.28.1 'GitHub plugin'   * github-api:1.90 'GitHub API Plugin'   * github-branch-source:2.3.2 'GitHub Branch Source Plugin'   * github-organization-folder:1.6 'GitHub Organization Folder Plugin'   * gradle:1.28 'Gradle Plugin'   * groovy-postbuild:2.3.1 'Groovy Postbuild'   * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'   * icon-shim:2.0.3 'Icon Shim Plugin'   * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'   * jacoco:2.2.1 'Jenkins JaCoCo plugin'   * javadoc:1.4 'Javadoc Plugin'   * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'   * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'   * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'   * jsch:0.1.54.1 'Jenkins JSch dependency plugin'   * junit:1.23 'JUnit Plugin'   * ldap:1.18 'LDAP Plugin'   * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'   * mailer:1.20 'Jenkins Mailer Plugin'   * mapdb-api:1.0.9.0 'MapDB API Plugin'   * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'   * matrix-project:1.12 'Matrix Project Plugin'   * maven-plugin:3.0 'Maven Integration plugin'   * metrics:3.1.2.10 'Metrics Plugin'   * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'   * node-iterator-api:1.5.0 'Node Iterator API Plugin'   * notification:1.12 'Jenkins Notification plugin'   * pam-auth:1.3 'PAM Authentication plugin'   * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'   * pipeline-build-step:2.6 'Pipeline: Build Step'   * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'   * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'   * pipeline-input-step:2.8 'Pipeline: Input Step'   * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'   * pipeline-model-api:1.2.5 'Pipeline: Model API'   * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'   * pipeline-model-definition:1.2.5 'Pipeline: Declarative'   * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'   * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'   * pipeline-stage-step:2.3 'Pipeline: Stage Step'   * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'   * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'   * plain-credentials:1.4 'Plain Credentials Plugin'   * python:1.3 'Python Plugin'   * resource-disposer:0.8 'Resource Disposer Plugin'   * run-condition:1.0 'Run Condition Plugin'   * saml:1.0.4 'SAML Plugin'   * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'   * scm-api:2.2.6 'SCM API Plugin'   * script-security:1.39 'Script Security Plugin'   * simple-theme-plugin:0.3 'Simple Theme Plugin'   * slack:2.3 'Slack Notification Plugin'   * ssh-credentials:1.13 'SSH Credentials Plugin'   * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'   * structs:1.10 'Structs Plugin'   * subversion:2.10.2 'Jenkins Subversion Plug-in'   * support-core:2.44 'Support Core Plugin'   * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'   * timestamper:1.8.9 'Timestamper'   * token-macro:2.3 'Token Macro Plugin'   * unleash:2.3.0 'Unleash Maven Plugin'   * windows-slaves:1.3.1 'Windows Slaves Plugin'   * workflow-aggregator:2.5 'Pipeline'   * workflow-api:2.24 'Pipeline: API'   * workflow-basic-steps:2.6 'Pipeline: Basic Steps'   * workflow-cps:2.42 'Pipeline: Groovy'   * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'   * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'   * workflow-job:2.16 'Pipeline: Job'   * workflow-multibranch:2.16 'Pipeline: Multibranch'   * workflow-scm-step:2.6 'Pipeline: SCM Step'   * workflow-step-api:2.14 'Pipeline: Step API'   * workflow-support:2.16 'Pipeline: Supporting APIs'   * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'

Mark Ketchum made changes - 2017-12-21 20:10

Environment

Original: Jenkins ======= Version details ---------------   * Version: `2.89.2`   * Mode: WAR   * Url: https://bld-ria.corp.pingidentity.com/   * Java       - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`       - Vendor: Oracle Corporation       - Version: 1.8.0_151       - Maximum memory: 1.62 GB (1743257600)       - Allocated memory: 719.00 MB (753926144)       - Free memory: 547.74 MB (574348208)       - In-use memory: 171.26 MB (179577936)       - GC strategy: ParallelGC   * Java Runtime Specification       - Name: Java Platform API Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Specification       - Name: Java Virtual Machine Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Implementation       - Name: OpenJDK 64-Bit Server VM       - Vendor: Oracle Corporation       - Version: 25.151-b12   * Operating system       - Name: Linux       - Architecture: amd64       - Version: 4.4.0-104-generic       - Distribution: Ubuntu 16.04.3 LTS   * Process ID: 1519 (0x5ef)   * Process started: 2017-12-21 19:56:58.300+0000   * Process uptime: 40 sec   * JVM startup parameters:       - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`       - Classpath: `/usr/share/jenkins/jenkins.war`       - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`       - arg[0]: `-Djava.awt.headless=true` Important configuration ---------------   * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`   * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`   * CSRF Protection: true   * Initialization Milestone: Completed initialization Active Plugins --------------   * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'   * analysis-core:1.93 'Static Analysis Utilities'   * ant:1.7 'Ant Plugin'   * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'   * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'   * authentication-tokens:1.3 'Authentication Tokens API Plugin'   * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'   * aws-java-sdk:1.11.248 'Amazon Web Services SDK'   * bouncycastle-api:2.16.2 'bouncycastle API Plugin'   * branch-api:2.0.16 'Branch API Plugin'   * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'   * build-timeout:1.19 'Build Timeout'   * build-token-root:1.4 'Build Authorization Token Root Plugin'   * built-on-column:1.1 'built-on-column'   * cloudbees-folder:6.2.1 'Folders Plugin'   * command-launcher:1.2 'Command Agent Launcher Plugin'   * conditional-buildstep:1.3.6 'Conditional BuildStep'   * copyartifact:1.39 'Copy Artifact Plugin'   * credentials:2.1.16 'Credentials Plugin'   * credentials-binding:1.13 'Credentials Binding Plugin'   * display-url-api:2.2.0 'Display URL API'   * docker-commons:1.10 'Docker Commons Plugin'   * docker-workflow:1.14 'Docker Pipeline'   * durable-task:1.17 'Durable Task Plugin'   * ec2:1.38 'Amazon EC2 plugin'   * email-ext:2.61 'Email Extension Plugin'   * envinject:2.1.5 'Environment Injector Plugin'   * envinject-api:1.4 'EnvInject API Plugin'   * external-monitor-job:1.7 'External Monitor Job Type Plugin'   * ez-templates:1.3.1 'EZ Templates'   * findbugs:4.71 'FindBugs Plug-in'   * git:3.6.4 'Jenkins Git plugin'   * git-client:2.7.0 'Jenkins Git client plugin'   * git-server:1.7 'Jenkins GIT server Plugin'   * github:1.28.1 'GitHub plugin'   * github-api:1.90 'GitHub API Plugin'   * github-branch-source:2.3.2 'GitHub Branch Source Plugin'   * github-organization-folder:1.6 'GitHub Organization Folder Plugin'   * gradle:1.28 'Gradle Plugin'   * groovy-postbuild:2.3.1 'Groovy Postbuild'   * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'   * icon-shim:2.0.3 'Icon Shim Plugin'   * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'   * jacoco:2.2.1 'Jenkins JaCoCo plugin'   * javadoc:1.4 'Javadoc Plugin'   * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'   * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'   * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'   * jsch:0.1.54.1 'Jenkins JSch dependency plugin'   * junit:1.23 'JUnit Plugin'   * ldap:1.18 'LDAP Plugin'   * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'   * mailer:1.20 'Jenkins Mailer Plugin'   * mapdb-api:1.0.9.0 'MapDB API Plugin'   * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'   * matrix-project:1.12 'Matrix Project Plugin'   * maven-plugin:3.0 'Maven Integration plugin'   * metrics:3.1.2.10 'Metrics Plugin'   * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'   * node-iterator-api:1.5.0 'Node Iterator API Plugin'   * notification:1.12 'Jenkins Notification plugin'   * pam-auth:1.3 'PAM Authentication plugin'   * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'   * pipeline-build-step:2.6 'Pipeline: Build Step'   * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'   * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'   * pipeline-input-step:2.8 'Pipeline: Input Step'   * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'   * pipeline-model-api:1.2.5 'Pipeline: Model API'   * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'   * pipeline-model-definition:1.2.5 'Pipeline: Declarative'   * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'   * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'   * pipeline-stage-step:2.3 'Pipeline: Stage Step'   * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'   * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'   * plain-credentials:1.4 'Plain Credentials Plugin'   * python:1.3 'Python Plugin'   * resource-disposer:0.8 'Resource Disposer Plugin'   * run-condition:1.0 'Run Condition Plugin'   * saml:1.0.4 'SAML Plugin'   * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'   * scm-api:2.2.6 'SCM API Plugin'   * script-security:1.39 'Script Security Plugin'   * simple-theme-plugin:0.3 'Simple Theme Plugin'   * slack:2.3 'Slack Notification Plugin'   * ssh-credentials:1.13 'SSH Credentials Plugin'   * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'   * structs:1.10 'Structs Plugin'   * subversion:2.10.2 'Jenkins Subversion Plug-in'   * support-core:2.44 'Support Core Plugin'   * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'   * timestamper:1.8.9 'Timestamper'   * token-macro:2.3 'Token Macro Plugin'   * unleash:2.3.0 'Unleash Maven Plugin'   * windows-slaves:1.3.1 'Windows Slaves Plugin'   * workflow-aggregator:2.5 'Pipeline'   * workflow-api:2.24 'Pipeline: API'   * workflow-basic-steps:2.6 'Pipeline: Basic Steps'   * workflow-cps:2.42 'Pipeline: Groovy'   * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'   * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'   * workflow-job:2.16 'Pipeline: Job'   * workflow-multibranch:2.16 'Pipeline: Multibranch'   * workflow-scm-step:2.6 'Pipeline: SCM Step'   * workflow-step-api:2.14 'Pipeline: Step API'   * workflow-support:2.16 'Pipeline: Supporting APIs'   * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'

New: Jenkins ======= Version details ---------------   * Version: `2.89.2`   * Mode: WAR   * Url: *****   * Java       - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`       - Vendor: Oracle Corporation       - Version: 1.8.0_151       - Maximum memory: 1.62 GB (1743257600)       - Allocated memory: 719.00 MB (753926144)       - Free memory: 547.74 MB (574348208)       - In-use memory: 171.26 MB (179577936)       - GC strategy: ParallelGC   * Java Runtime Specification       - Name: Java Platform API Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Specification       - Name: Java Virtual Machine Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Implementation       - Name: OpenJDK 64-Bit Server VM       - Vendor: Oracle Corporation       - Version: 25.151-b12   * Operating system       - Name: Linux       - Architecture: amd64       - Version: 4.4.0-104-generic       - Distribution: Ubuntu 16.04.3 LTS   * Process ID: 1519 (0x5ef)   * Process started: 2017-12-21 19:56:58.300+0000   * Process uptime: 40 sec   * JVM startup parameters:       - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`       - Classpath: `/usr/share/jenkins/jenkins.war`       - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`       - arg[0]: `-Djava.awt.headless=true` Important configuration ---------------   * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`   * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`   * CSRF Protection: true   * Initialization Milestone: Completed initialization Active Plugins --------------   * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'   * analysis-core:1.93 'Static Analysis Utilities'   * ant:1.7 'Ant Plugin'   * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'   * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'   * authentication-tokens:1.3 'Authentication Tokens API Plugin'   * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'   * aws-java-sdk:1.11.248 'Amazon Web Services SDK'   * bouncycastle-api:2.16.2 'bouncycastle API Plugin'   * branch-api:2.0.16 'Branch API Plugin'   * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'   * build-timeout:1.19 'Build Timeout'   * build-token-root:1.4 'Build Authorization Token Root Plugin'   * built-on-column:1.1 'built-on-column'   * cloudbees-folder:6.2.1 'Folders Plugin'   * command-launcher:1.2 'Command Agent Launcher Plugin'   * conditional-buildstep:1.3.6 'Conditional BuildStep'   * copyartifact:1.39 'Copy Artifact Plugin'   * credentials:2.1.16 'Credentials Plugin'   * credentials-binding:1.13 'Credentials Binding Plugin'   * display-url-api:2.2.0 'Display URL API'   * docker-commons:1.10 'Docker Commons Plugin'   * docker-workflow:1.14 'Docker Pipeline'   * durable-task:1.17 'Durable Task Plugin'   * ec2:1.38 'Amazon EC2 plugin'   * email-ext:2.61 'Email Extension Plugin'   * envinject:2.1.5 'Environment Injector Plugin'   * envinject-api:1.4 'EnvInject API Plugin'   * external-monitor-job:1.7 'External Monitor Job Type Plugin'   * ez-templates:1.3.1 'EZ Templates'   * findbugs:4.71 'FindBugs Plug-in'   * git:3.6.4 'Jenkins Git plugin'   * git-client:2.7.0 'Jenkins Git client plugin'   * git-server:1.7 'Jenkins GIT server Plugin'   * github:1.28.1 'GitHub plugin'   * github-api:1.90 'GitHub API Plugin'   * github-branch-source:2.3.2 'GitHub Branch Source Plugin'   * github-organization-folder:1.6 'GitHub Organization Folder Plugin'   * gradle:1.28 'Gradle Plugin'   * groovy-postbuild:2.3.1 'Groovy Postbuild'   * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'   * icon-shim:2.0.3 'Icon Shim Plugin'   * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'   * jacoco:2.2.1 'Jenkins JaCoCo plugin'   * javadoc:1.4 'Javadoc Plugin'   * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'   * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'   * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'   * jsch:0.1.54.1 'Jenkins JSch dependency plugin'   * junit:1.23 'JUnit Plugin'   * ldap:1.18 'LDAP Plugin'   * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'   * mailer:1.20 'Jenkins Mailer Plugin'   * mapdb-api:1.0.9.0 'MapDB API Plugin'   * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'   * matrix-project:1.12 'Matrix Project Plugin'   * maven-plugin:3.0 'Maven Integration plugin'   * metrics:3.1.2.10 'Metrics Plugin'   * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'   * node-iterator-api:1.5.0 'Node Iterator API Plugin'   * notification:1.12 'Jenkins Notification plugin'   * pam-auth:1.3 'PAM Authentication plugin'   * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'   * pipeline-build-step:2.6 'Pipeline: Build Step'   * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'   * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'   * pipeline-input-step:2.8 'Pipeline: Input Step'   * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'   * pipeline-model-api:1.2.5 'Pipeline: Model API'   * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'   * pipeline-model-definition:1.2.5 'Pipeline: Declarative'   * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'   * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'   * pipeline-stage-step:2.3 'Pipeline: Stage Step'   * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'   * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'   * plain-credentials:1.4 'Plain Credentials Plugin'   * python:1.3 'Python Plugin'   * resource-disposer:0.8 'Resource Disposer Plugin'   * run-condition:1.0 'Run Condition Plugin'   * saml:1.0.4 'SAML Plugin'   * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'   * scm-api:2.2.6 'SCM API Plugin'   * script-security:1.39 'Script Security Plugin'   * simple-theme-plugin:0.3 'Simple Theme Plugin'   * slack:2.3 'Slack Notification Plugin'   * ssh-credentials:1.13 'SSH Credentials Plugin'   * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'   * structs:1.10 'Structs Plugin'   * subversion:2.10.2 'Jenkins Subversion Plug-in'   * support-core:2.44 'Support Core Plugin'   * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'   * timestamper:1.8.9 'Timestamper'   * token-macro:2.3 'Token Macro Plugin'   * unleash:2.3.0 'Unleash Maven Plugin'   * windows-slaves:1.3.1 'Windows Slaves Plugin'   * workflow-aggregator:2.5 'Pipeline'   * workflow-api:2.24 'Pipeline: API'   * workflow-basic-steps:2.6 'Pipeline: Basic Steps'   * workflow-cps:2.42 'Pipeline: Groovy'   * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'   * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'   * workflow-job:2.16 'Pipeline: Job'   * workflow-multibranch:2.16 'Pipeline: Multibranch'   * workflow-scm-step:2.6 'Pipeline: SCM Step'   * workflow-step-api:2.14 'Pipeline: Step API'   * workflow-support:2.16 'Pipeline: Supporting APIs'   * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'

Mark Ketchum made changes - 2017-12-21 20:10

Description

Original: After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock. We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up. Attached is the deadlock stack traces from a thread dump. We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible. {code:java} Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008cdab698> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source){code}

New: After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock. We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up. Attached is the deadlock stack traces from a thread dump. {code:java} Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008cdab698> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source){code} We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible.

Collapse comment: Oleg Nenashev added a comment - 2017-12-22 08:39

Oleg Nenashev added a comment - 2017-12-22 08:39

Here is a root cause thread from the dump:

Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
- waiting to lock <0x000000008d744a90> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
at hudson.util.XStream2.unmarshal(XStream2.java:114)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
at hudson.XmlFile.unmarshal(XmlFile.java:167)
at hudson.model.Run.reload(Run.java:336)
at hudson.model.Run.<init>(Run.java:324)
at hudson.model.AbstractBuild.<init>(AbstractBuild.java:173)
at hudson.model.Build.<init>(Build.java:104)
at com.tikal.jenkins.plugins.multijob.MultiJobBuild.<init>(MultiJobBuild.java:59)
at sun.reflect.GeneratedConstructorAccessor501.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165)
at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142)
at hudson.model.RunMap.retrieve(RunMap.java:224)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380)
- locked <0x000000008cdab698> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown

Needs investigation

Expand comment: Oleg Nenashev added a comment - 2017-12-22 08:39

Oleg Nenashev added a comment - 2017-12-22 08:39 Here is a root cause thread from the dump: Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED stackTrace: java.lang. Thread .State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71) at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85) at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80) at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:114) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:167) at hudson.model.Run.reload(Run.java:336) at hudson.model.Run.<init>(Run.java:324) at hudson.model.AbstractBuild.<init>(AbstractBuild.java:173) at hudson.model.Build.<init>(Build.java:104) at com.tikal.jenkins.plugins.multijob.MultiJobBuild.<init>(MultiJobBuild.java:59) at sun.reflect.GeneratedConstructorAccessor501.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165) at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142) at hudson.model.RunMap.retrieve(RunMap.java:224) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380) - locked <0x000000008cdab698> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Needs investigation

Allan BURDAJEWICZ made changes - 2018-02-02 03:02

Link

New: This issue relates to JENKINS-49328 [ JENKINS-49328 ]

Collapse comment: Allan BURDAJEWICZ added a comment - 2018-02-02 03:11

Allan BURDAJEWICZ added a comment - 2018-02-02 03:11

oleg_nenashev jglick Could this be related to JENKINS-49328 and nested references ? We see the `hudson.model.Run$Replacer.readResolve` in the stacktrace. I have also seen a case were the hudson.model.RunMap#retrieve fails with a StackOverflowError, showing the same stacktrace as we can see here.

Expand comment: Allan BURDAJEWICZ added a comment - 2018-02-02 03:11

Allan BURDAJEWICZ added a comment - 2018-02-02 03:11 oleg_nenashev jglick Could this be related to JENKINS-49328 and nested references ? We see the `hudson.model.Run$Replacer.readResolve` in the stacktrace. I have also seen a case were the hudson.model.RunMap#retrieve fails with a StackOverflowError, showing the same stacktrace as we can see here.

Jesse Glick made changes - 2018-02-02 08:03

Priority

Original: Minor [ 4 ]

New: Critical [ 2 ]

Jesse Glick made changes - 2018-02-02 08:03

Component/s

Original: core [ 15593 ]

Jesse Glick made changes - 2018-02-02 08:04

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Load more newer events