-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
Jenkins
=======
Version details
---------------
* Version: `2.89.2`
* Mode: WAR
* Url: *****
* Java
- Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
- Vendor: Oracle Corporation
- Version: 1.8.0_151
- Maximum memory: 1.62 GB (1743257600)
- Allocated memory: 719.00 MB (753926144)
- Free memory: 547.74 MB (574348208)
- In-use memory: 171.26 MB (179577936)
- GC strategy: ParallelGC
* Java Runtime Specification
- Name: Java Platform API Specification
- Vendor: Oracle Corporation
- Version: 1.8
* JVM Specification
- Name: Java Virtual Machine Specification
- Vendor: Oracle Corporation
- Version: 1.8
* JVM Implementation
- Name: OpenJDK 64-Bit Server VM
- Vendor: Oracle Corporation
- Version: 25.151-b12
* Operating system
- Name: Linux
- Architecture: amd64
- Version: 4.4.0-104-generic
- Distribution: Ubuntu 16.04.3 LTS
* Process ID: 1519 (0x5ef)
* Process started: 2017-12-21 19:56:58.300+0000
* Process uptime: 40 sec
* JVM startup parameters:
- Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`
- Classpath: `/usr/share/jenkins/jenkins.war`
- Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`
- arg[0]: `-Djava.awt.headless=true`
Important configuration
---------------
* Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`
* Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`
* CSRF Protection: true
* Initialization Milestone: Completed initialization
Active Plugins
--------------
* ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'
* analysis-core:1.93 'Static Analysis Utilities'
* ant:1.7 'Ant Plugin'
* antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'
* apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'
* authentication-tokens:1.3 'Authentication Tokens API Plugin'
* aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'
* aws-java-sdk:1.11.248 'Amazon Web Services SDK'
* bouncycastle-api:2.16.2 'bouncycastle API Plugin'
* branch-api:2.0.16 'Branch API Plugin'
* build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'
* build-timeout:1.19 'Build Timeout'
* build-token-root:1.4 'Build Authorization Token Root Plugin'
* built-on-column:1.1 'built-on-column'
* cloudbees-folder:6.2.1 'Folders Plugin'
* command-launcher:1.2 'Command Agent Launcher Plugin'
* conditional-buildstep:1.3.6 'Conditional BuildStep'
* copyartifact:1.39 'Copy Artifact Plugin'
* credentials:2.1.16 'Credentials Plugin'
* credentials-binding:1.13 'Credentials Binding Plugin'
* display-url-api:2.2.0 'Display URL API'
* docker-commons:1.10 'Docker Commons Plugin'
* docker-workflow:1.14 'Docker Pipeline'
* durable-task:1.17 'Durable Task Plugin'
* ec2:1.38 'Amazon EC2 plugin'
* email-ext:2.61 'Email Extension Plugin'
* envinject:2.1.5 'Environment Injector Plugin'
* envinject-api:1.4 'EnvInject API Plugin'
* external-monitor-job:1.7 'External Monitor Job Type Plugin'
* ez-templates:1.3.1 'EZ Templates'
* findbugs:4.71 'FindBugs Plug-in'
* git:3.6.4 'Jenkins Git plugin'
* git-client:2.7.0 'Jenkins Git client plugin'
* git-server:1.7 'Jenkins GIT server Plugin'
* github:1.28.1 'GitHub plugin'
* github-api:1.90 'GitHub API Plugin'
* github-branch-source:2.3.2 'GitHub Branch Source Plugin'
* github-organization-folder:1.6 'GitHub Organization Folder Plugin'
* gradle:1.28 'Gradle Plugin'
* groovy-postbuild:2.3.1 'Groovy Postbuild'
* handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'
* icon-shim:2.0.3 'Icon Shim Plugin'
* jackson2-api:2.8.10.1 'Jackson 2 API Plugin'
* jacoco:2.2.1 'Jenkins JaCoCo plugin'
* javadoc:1.4 'Javadoc Plugin'
* jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'
* jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'
* jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'
* jsch:0.1.54.1 'Jenkins JSch dependency plugin'
* junit:1.23 'JUnit Plugin'
* ldap:1.18 'LDAP Plugin'
* m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'
* mailer:1.20 'Jenkins Mailer Plugin'
* mapdb-api:1.0.9.0 'MapDB API Plugin'
* matrix-auth:2.2 'Matrix Authorization Strategy Plugin'
* matrix-project:1.12 'Matrix Project Plugin'
* maven-plugin:3.0 'Maven Integration plugin'
* metrics:3.1.2.10 'Metrics Plugin'
* momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'
* node-iterator-api:1.5.0 'Node Iterator API Plugin'
* notification:1.12 'Jenkins Notification plugin'
* pam-auth:1.3 'PAM Authentication plugin'
* parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'
* pipeline-build-step:2.6 'Pipeline: Build Step'
* pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'
* pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'
* pipeline-input-step:2.8 'Pipeline: Input Step'
* pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'
* pipeline-model-api:1.2.5 'Pipeline: Model API'
* pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'
* pipeline-model-definition:1.2.5 'Pipeline: Declarative'
* pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'
* pipeline-rest-api:2.9 'Pipeline: REST API Plugin'
* pipeline-stage-step:2.3 'Pipeline: Stage Step'
* pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'
* pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'
* plain-credentials:1.4 'Plain Credentials Plugin'
* python:1.3 'Python Plugin'
* resource-disposer:0.8 'Resource Disposer Plugin'
* run-condition:1.0 'Run Condition Plugin'
* saml:1.0.4 'SAML Plugin'
* sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'
* scm-api:2.2.6 'SCM API Plugin'
* script-security:1.39 'Script Security Plugin'
* simple-theme-plugin:0.3 'Simple Theme Plugin'
* slack:2.3 'Slack Notification Plugin'
* ssh-credentials:1.13 'SSH Credentials Plugin'
* ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'
* structs:1.10 'Structs Plugin'
* subversion:2.10.2 'Jenkins Subversion Plug-in'
* support-core:2.44 'Support Core Plugin'
* throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'
* timestamper:1.8.9 'Timestamper'
* token-macro:2.3 'Token Macro Plugin'
* unleash:2.3.0 'Unleash Maven Plugin'
* windows-slaves:1.3.1 'Windows Slaves Plugin'
* workflow-aggregator:2.5 'Pipeline'
* workflow-api:2.24 'Pipeline: API'
* workflow-basic-steps:2.6 'Pipeline: Basic Steps'
* workflow-cps:2.42 'Pipeline: Groovy'
* workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'
* workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'
* workflow-job:2.16 'Pipeline: Job'
* workflow-multibranch:2.16 'Pipeline: Multibranch'
* workflow-scm-step:2.6 'Pipeline: SCM Step'
* workflow-step-api:2.14 'Pipeline: Step API'
* workflow-support:2.16 'Pipeline: Supporting APIs'
* ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'
Jenkins ======= Version details --------------- * Version: `2.89.2` * Mode: WAR * Url: ***** * Java - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre` - Vendor: Oracle Corporation - Version: 1.8.0_151 - Maximum memory: 1.62 GB (1743257600) - Allocated memory: 719.00 MB (753926144) - Free memory: 547.74 MB (574348208) - In-use memory: 171.26 MB (179577936) - GC strategy: ParallelGC * Java Runtime Specification - Name: Java Platform API Specification - Vendor: Oracle Corporation - Version: 1.8 * JVM Specification - Name: Java Virtual Machine Specification - Vendor: Oracle Corporation - Version: 1.8 * JVM Implementation - Name: OpenJDK 64-Bit Server VM - Vendor: Oracle Corporation - Version: 25.151-b12 * Operating system - Name: Linux - Architecture: amd64 - Version: 4.4.0-104-generic - Distribution: Ubuntu 16.04.3 LTS * Process ID: 1519 (0x5ef) * Process started: 2017-12-21 19:56:58.300+0000 * Process uptime: 40 sec * JVM startup parameters: - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes` - Classpath: `/usr/share/jenkins/jenkins.war` - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib` - arg[0]: `-Djava.awt.headless=true` Important configuration --------------- * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm` * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy` * CSRF Protection: true * Initialization Milestone: Completed initialization Active Plugins -------------- * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin' * analysis-core:1.93 'Static Analysis Utilities' * ant:1.7 'Ant Plugin' * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin' * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin' * authentication-tokens:1.3 'Authentication Tokens API Plugin' * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin' * aws-java-sdk:1.11.248 'Amazon Web Services SDK' * bouncycastle-api:2.16.2 'bouncycastle API Plugin' * branch-api:2.0.16 'Branch API Plugin' * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View' * build-timeout:1.19 'Build Timeout' * build-token-root:1.4 'Build Authorization Token Root Plugin' * built-on-column:1.1 'built-on-column' * cloudbees-folder:6.2.1 'Folders Plugin' * command-launcher:1.2 'Command Agent Launcher Plugin' * conditional-buildstep:1.3.6 'Conditional BuildStep' * copyartifact:1.39 'Copy Artifact Plugin' * credentials:2.1.16 'Credentials Plugin' * credentials-binding:1.13 'Credentials Binding Plugin' * display-url-api:2.2.0 'Display URL API' * docker-commons:1.10 'Docker Commons Plugin' * docker-workflow:1.14 'Docker Pipeline' * durable-task:1.17 'Durable Task Plugin' * ec2:1.38 'Amazon EC2 plugin' * email-ext:2.61 'Email Extension Plugin' * envinject:2.1.5 'Environment Injector Plugin' * envinject-api:1.4 'EnvInject API Plugin' * external-monitor-job:1.7 'External Monitor Job Type Plugin' * ez-templates:1.3.1 'EZ Templates' * findbugs:4.71 'FindBugs Plug-in' * git:3.6.4 'Jenkins Git plugin' * git-client:2.7.0 'Jenkins Git client plugin' * git-server:1.7 'Jenkins GIT server Plugin' * github:1.28.1 'GitHub plugin' * github-api:1.90 'GitHub API Plugin' * github-branch-source:2.3.2 'GitHub Branch Source Plugin' * github-organization-folder:1.6 'GitHub Organization Folder Plugin' * gradle:1.28 'Gradle Plugin' * groovy-postbuild:2.3.1 'Groovy Postbuild' * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin' * icon-shim:2.0.3 'Icon Shim Plugin' * jackson2-api:2.8.10.1 'Jackson 2 API Plugin' * jacoco:2.2.1 'Jenkins JaCoCo plugin' * javadoc:1.4 'Javadoc Plugin' * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin' * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin' * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin' * jsch:0.1.54.1 'Jenkins JSch dependency plugin' * junit:1.23 'JUnit Plugin' * ldap:1.18 'LDAP Plugin' * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in' * mailer:1.20 'Jenkins Mailer Plugin' * mapdb-api:1.0.9.0 'MapDB API Plugin' * matrix-auth:2.2 'Matrix Authorization Strategy Plugin' * matrix-project:1.12 'Matrix Project Plugin' * maven-plugin:3.0 'Maven Integration plugin' * metrics:3.1.2.10 'Metrics Plugin' * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin' * node-iterator-api:1.5.0 'Node Iterator API Plugin' * notification:1.12 'Jenkins Notification plugin' * pam-auth:1.3 'PAM Authentication plugin' * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin' * pipeline-build-step:2.6 'Pipeline: Build Step' * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries' * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin' * pipeline-input-step:2.8 'Pipeline: Input Step' * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step' * pipeline-model-api:1.2.5 'Pipeline: Model API' * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API' * pipeline-model-definition:1.2.5 'Pipeline: Declarative' * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API' * pipeline-rest-api:2.9 'Pipeline: REST API Plugin' * pipeline-stage-step:2.3 'Pipeline: Stage Step' * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata' * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin' * plain-credentials:1.4 'Plain Credentials Plugin' * python:1.3 'Python Plugin' * resource-disposer:0.8 'Resource Disposer Plugin' * run-condition:1.0 'Run Condition Plugin' * saml:1.0.4 'SAML Plugin' * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin' * scm-api:2.2.6 'SCM API Plugin' * script-security:1.39 'Script Security Plugin' * simple-theme-plugin:0.3 'Simple Theme Plugin' * slack:2.3 'Slack Notification Plugin' * ssh-credentials:1.13 'SSH Credentials Plugin' * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin' * structs:1.10 'Structs Plugin' * subversion:2.10.2 'Jenkins Subversion Plug-in' * support-core:2.44 'Support Core Plugin' * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in' * timestamper:1.8.9 'Timestamper' * token-macro:2.3 'Token Macro Plugin' * unleash:2.3.0 'Unleash Maven Plugin' * windows-slaves:1.3.1 'Windows Slaves Plugin' * workflow-aggregator:2.5 'Pipeline' * workflow-api:2.24 'Pipeline: API' * workflow-basic-steps:2.6 'Pipeline: Basic Steps' * workflow-cps:2.42 'Pipeline: Groovy' * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries' * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes' * workflow-job:2.16 'Pipeline: Job' * workflow-multibranch:2.16 'Pipeline: Multibranch' * workflow-scm-step:2.6 'Pipeline: SCM Step' * workflow-step-api:2.14 'Pipeline: Step API' * workflow-support:2.16 'Pipeline: Supporting APIs' * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'
After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock.
We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up.
Attached is the deadlock stack traces from a thread dump.
Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008cdab698> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible.
- relates to
-
-
- Closed
-
