[JENKINS-48685] Deadlock when running a Multijob with multiple slaves - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Critical
Resolution: Duplicate
Component/s: multijob-plugin
Labels:
None
Environment:

Hide
Jenkins
=======

Version details
---------------

  * Version: `2.89.2`
  * Mode: WAR
  * Url: *****
  * Java
      - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
      - Vendor: Oracle Corporation
      - Version: 1.8.0_151
      - Maximum memory: 1.62 GB (1743257600)
      - Allocated memory: 719.00 MB (753926144)
      - Free memory: 547.74 MB (574348208)
      - In-use memory: 171.26 MB (179577936)
      - GC strategy: ParallelGC
  * Java Runtime Specification
      - Name: Java Platform API Specification
      - Vendor: Oracle Corporation
      - Version: 1.8
  * JVM Specification
      - Name: Java Virtual Machine Specification
      - Vendor: Oracle Corporation
      - Version: 1.8
  * JVM Implementation
      - Name: OpenJDK 64-Bit Server VM
      - Vendor: Oracle Corporation
      - Version: 25.151-b12
  * Operating system
      - Name: Linux
      - Architecture: amd64
      - Version: 4.4.0-104-generic
      - Distribution: Ubuntu 16.04.3 LTS
  * Process ID: 1519 (0x5ef)
  * Process started: 2017-12-21 19:56:58.300+0000
  * Process uptime: 40 sec
  * JVM startup parameters:
      - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`
      - Classpath: `/usr/share/jenkins/jenkins.war`
      - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`
      - arg[0]: `-Djava.awt.headless=true`

Important configuration
---------------

  * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`
  * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`
  * CSRF Protection: true
  * Initialization Milestone: Completed initialization

Active Plugins
--------------

  * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'
  * analysis-core:1.93 'Static Analysis Utilities'
  * ant:1.7 'Ant Plugin'
  * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'
  * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'
  * authentication-tokens:1.3 'Authentication Tokens API Plugin'
  * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'
  * aws-java-sdk:1.11.248 'Amazon Web Services SDK'
  * bouncycastle-api:2.16.2 'bouncycastle API Plugin'
  * branch-api:2.0.16 'Branch API Plugin'
  * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'
  * build-timeout:1.19 'Build Timeout'
  * build-token-root:1.4 'Build Authorization Token Root Plugin'
  * built-on-column:1.1 'built-on-column'
  * cloudbees-folder:6.2.1 'Folders Plugin'
  * command-launcher:1.2 'Command Agent Launcher Plugin'
  * conditional-buildstep:1.3.6 'Conditional BuildStep'
  * copyartifact:1.39 'Copy Artifact Plugin'
  * credentials:2.1.16 'Credentials Plugin'
  * credentials-binding:1.13 'Credentials Binding Plugin'
  * display-url-api:2.2.0 'Display URL API'
  * docker-commons:1.10 'Docker Commons Plugin'
  * docker-workflow:1.14 'Docker Pipeline'
  * durable-task:1.17 'Durable Task Plugin'
  * ec2:1.38 'Amazon EC2 plugin'
  * email-ext:2.61 'Email Extension Plugin'
  * envinject:2.1.5 'Environment Injector Plugin'
  * envinject-api:1.4 'EnvInject API Plugin'
  * external-monitor-job:1.7 'External Monitor Job Type Plugin'
  * ez-templates:1.3.1 'EZ Templates'
  * findbugs:4.71 'FindBugs Plug-in'
  * git:3.6.4 'Jenkins Git plugin'
  * git-client:2.7.0 'Jenkins Git client plugin'
  * git-server:1.7 'Jenkins GIT server Plugin'
  * github:1.28.1 'GitHub plugin'
  * github-api:1.90 'GitHub API Plugin'
  * github-branch-source:2.3.2 'GitHub Branch Source Plugin'
  * github-organization-folder:1.6 'GitHub Organization Folder Plugin'
  * gradle:1.28 'Gradle Plugin'
  * groovy-postbuild:2.3.1 'Groovy Postbuild'
  * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'
  * icon-shim:2.0.3 'Icon Shim Plugin'
  * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'
  * jacoco:2.2.1 'Jenkins JaCoCo plugin'
  * javadoc:1.4 'Javadoc Plugin'
  * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'
  * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'
  * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'
  * jsch:0.1.54.1 'Jenkins JSch dependency plugin'
  * junit:1.23 'JUnit Plugin'
  * ldap:1.18 'LDAP Plugin'
  * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'
  * mailer:1.20 'Jenkins Mailer Plugin'
  * mapdb-api:1.0.9.0 'MapDB API Plugin'
  * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'
  * matrix-project:1.12 'Matrix Project Plugin'
  * maven-plugin:3.0 'Maven Integration plugin'
  * metrics:3.1.2.10 'Metrics Plugin'
  * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'
  * node-iterator-api:1.5.0 'Node Iterator API Plugin'
  * notification:1.12 'Jenkins Notification plugin'
  * pam-auth:1.3 'PAM Authentication plugin'
  * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'
  * pipeline-build-step:2.6 'Pipeline: Build Step'
  * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'
  * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'
  * pipeline-input-step:2.8 'Pipeline: Input Step'
  * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'
  * pipeline-model-api:1.2.5 'Pipeline: Model API'
  * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'
  * pipeline-model-definition:1.2.5 'Pipeline: Declarative'
  * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'
  * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'
  * pipeline-stage-step:2.3 'Pipeline: Stage Step'
  * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'
  * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'
  * plain-credentials:1.4 'Plain Credentials Plugin'
  * python:1.3 'Python Plugin'
  * resource-disposer:0.8 'Resource Disposer Plugin'
  * run-condition:1.0 'Run Condition Plugin'
  * saml:1.0.4 'SAML Plugin'
  * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'
  * scm-api:2.2.6 'SCM API Plugin'
  * script-security:1.39 'Script Security Plugin'
  * simple-theme-plugin:0.3 'Simple Theme Plugin'
  * slack:2.3 'Slack Notification Plugin'
  * ssh-credentials:1.13 'SSH Credentials Plugin'
  * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'
  * structs:1.10 'Structs Plugin'
  * subversion:2.10.2 'Jenkins Subversion Plug-in'
  * support-core:2.44 'Support Core Plugin'
  * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'
  * timestamper:1.8.9 'Timestamper'
  * token-macro:2.3 'Token Macro Plugin'
  * unleash:2.3.0 'Unleash Maven Plugin'
  * windows-slaves:1.3.1 'Windows Slaves Plugin'
  * workflow-aggregator:2.5 'Pipeline'
  * workflow-api:2.24 'Pipeline: API'
  * workflow-basic-steps:2.6 'Pipeline: Basic Steps'
  * workflow-cps:2.42 'Pipeline: Groovy'
  * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'
  * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'
  * workflow-job:2.16 'Pipeline: Job'
  * workflow-multibranch:2.16 'Pipeline: Multibranch'
  * workflow-scm-step:2.6 'Pipeline: SCM Step'
  * workflow-step-api:2.14 'Pipeline: Step API'
  * workflow-support:2.16 'Pipeline: Supporting APIs'
  * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'

Show
Jenkins ======= Version details ---------------   * Version: `2.89.2`   * Mode: WAR   * Url: *****   * Java       - Home: `/usr/lib/jvm/java-8-openjdk-amd64/jre`       - Vendor: Oracle Corporation       - Version: 1.8.0_151       - Maximum memory: 1.62 GB (1743257600)       - Allocated memory: 719.00 MB (753926144)       - Free memory: 547.74 MB (574348208)       - In-use memory: 171.26 MB (179577936)       - GC strategy: ParallelGC   * Java Runtime Specification       - Name: Java Platform API Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Specification       - Name: Java Virtual Machine Specification       - Vendor: Oracle Corporation       - Version: 1.8   * JVM Implementation       - Name: OpenJDK 64-Bit Server VM       - Vendor: Oracle Corporation       - Version: 25.151-b12   * Operating system       - Name: Linux       - Architecture: amd64       - Version: 4.4.0-104-generic       - Distribution: Ubuntu 16.04.3 LTS   * Process ID: 1519 (0x5ef)   * Process started: 2017-12-21 19:56:58.300+0000   * Process uptime: 40 sec   * JVM startup parameters:       - Boot classpath: `/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes`       - Classpath: `/usr/share/jenkins/jenkins.war`       - Library path: `/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib`       - arg[0]: `-Djava.awt.headless=true` Important configuration ---------------   * Security realm: `org.jenkinsci.plugins.saml.SamlSecurityRealm`   * Authorization strategy: `hudson.security.FullControlOnceLoggedInAuthorizationStrategy`   * CSRF Protection: true   * Initialization Milestone: Completed initialization Active Plugins --------------   * ace-editor:1.1 'JavaScript GUI Lib: ACE Editor bundle plugin'   * analysis-core:1.93 'Static Analysis Utilities'   * ant:1.7 'Ant Plugin'   * antisamy-markup-formatter:1.5 'OWASP Markup Formatter Plugin'   * apache-httpcomponents-client-4-api:4.5.3-2.0 'Jenkins Apache HttpComponents Client 4.x API Plugin'   * authentication-tokens:1.3 'Authentication Tokens API Plugin'   * aws-credentials:1.23 'CloudBees Amazon Web Services Credentials Plugin'   * aws-java-sdk:1.11.248 'Amazon Web Services SDK'   * bouncycastle-api:2.16.2 'bouncycastle API Plugin'   * branch-api:2.0.16 'Branch API Plugin'   * build-monitor-plugin:1.12+build.201708172343 'Build Monitor View'   * build-timeout:1.19 'Build Timeout'   * build-token-root:1.4 'Build Authorization Token Root Plugin'   * built-on-column:1.1 'built-on-column'   * cloudbees-folder:6.2.1 'Folders Plugin'   * command-launcher:1.2 'Command Agent Launcher Plugin'   * conditional-buildstep:1.3.6 'Conditional BuildStep'   * copyartifact:1.39 'Copy Artifact Plugin'   * credentials:2.1.16 'Credentials Plugin'   * credentials-binding:1.13 'Credentials Binding Plugin'   * display-url-api:2.2.0 'Display URL API'   * docker-commons:1.10 'Docker Commons Plugin'   * docker-workflow:1.14 'Docker Pipeline'   * durable-task:1.17 'Durable Task Plugin'   * ec2:1.38 'Amazon EC2 plugin'   * email-ext:2.61 'Email Extension Plugin'   * envinject:2.1.5 'Environment Injector Plugin'   * envinject-api:1.4 'EnvInject API Plugin'   * external-monitor-job:1.7 'External Monitor Job Type Plugin'   * ez-templates:1.3.1 'EZ Templates'   * findbugs:4.71 'FindBugs Plug-in'   * git:3.6.4 'Jenkins Git plugin'   * git-client:2.7.0 'Jenkins Git client plugin'   * git-server:1.7 'Jenkins GIT server Plugin'   * github:1.28.1 'GitHub plugin'   * github-api:1.90 'GitHub API Plugin'   * github-branch-source:2.3.2 'GitHub Branch Source Plugin'   * github-organization-folder:1.6 'GitHub Organization Folder Plugin'   * gradle:1.28 'Gradle Plugin'   * groovy-postbuild:2.3.1 'Groovy Postbuild'   * handlebars:1.1.1 'JavaScript GUI Lib: Handlebars bundle plugin'   * icon-shim:2.0.3 'Icon Shim Plugin'   * jackson2-api:2.8.10.1 'Jackson 2 API Plugin'   * jacoco:2.2.1 'Jenkins JaCoCo plugin'   * javadoc:1.4 'Javadoc Plugin'   * jenkins-multijob-plugin:1.28 'Jenkins Multijob plugin'   * jobConfigHistory:2.18 'Jenkins Job Configuration History Plugin'   * jquery-detached:1.2.1 'JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin'   * jsch:0.1.54.1 'Jenkins JSch dependency plugin'   * junit:1.23 'JUnit Plugin'   * ldap:1.18 'LDAP Plugin'   * m2release:0.14.0 'Jenkins Maven Release Plug-in Plug-in'   * mailer:1.20 'Jenkins Mailer Plugin'   * mapdb-api:1.0.9.0 'MapDB API Plugin'   * matrix-auth:2.2 'Matrix Authorization Strategy Plugin'   * matrix-project:1.12 'Matrix Project Plugin'   * maven-plugin:3.0 'Maven Integration plugin'   * metrics:3.1.2.10 'Metrics Plugin'   * momentjs:1.1.1 'JavaScript GUI Lib: Moment.js bundle plugin'   * node-iterator-api:1.5.0 'Node Iterator API Plugin'   * notification:1.12 'Jenkins Notification plugin'   * pam-auth:1.3 'PAM Authentication plugin'   * parameterized-trigger:2.35.2 'Jenkins Parameterized Trigger plugin'   * pipeline-build-step:2.6 'Pipeline: Build Step'   * pipeline-github-lib:1.0 'Pipeline: GitHub Groovy Libraries'   * pipeline-graph-analysis:1.5 'Pipeline Graph Analysis Plugin'   * pipeline-input-step:2.8 'Pipeline: Input Step'   * pipeline-milestone-step:1.3.1 'Pipeline: Milestone Step'   * pipeline-model-api:1.2.5 'Pipeline: Model API'   * pipeline-model-declarative-agent:1.1.1 'Pipeline: Declarative Agent API'   * pipeline-model-definition:1.2.5 'Pipeline: Declarative'   * pipeline-model-extensions:1.2.5 'Pipeline: Declarative Extension Points API'   * pipeline-rest-api:2.9 'Pipeline: REST API Plugin'   * pipeline-stage-step:2.3 'Pipeline: Stage Step'   * pipeline-stage-tags-metadata:1.2.5 'Pipeline: Stage Tags Metadata'   * pipeline-stage-view:2.9 'Pipeline: Stage View Plugin'   * plain-credentials:1.4 'Plain Credentials Plugin'   * python:1.3 'Python Plugin'   * resource-disposer:0.8 'Resource Disposer Plugin'   * run-condition:1.0 'Run Condition Plugin'   * saml:1.0.4 'SAML Plugin'   * sauce-ondemand:1.170 'Jenkins Sauce OnDemand plugin'   * scm-api:2.2.6 'SCM API Plugin'   * script-security:1.39 'Script Security Plugin'   * simple-theme-plugin:0.3 'Simple Theme Plugin'   * slack:2.3 'Slack Notification Plugin'   * ssh-credentials:1.13 'SSH Credentials Plugin'   * ssh-slaves:1.23 *(update available)* 'Jenkins SSH Slaves plugin'   * structs:1.10 'Structs Plugin'   * subversion:2.10.2 'Jenkins Subversion Plug-in'   * support-core:2.44 'Support Core Plugin'   * throttle-concurrents:2.0.1 'Jenkins Throttle Concurrent Builds Plug-in'   * timestamper:1.8.9 'Timestamper'   * token-macro:2.3 'Token Macro Plugin'   * unleash:2.3.0 'Unleash Maven Plugin'   * windows-slaves:1.3.1 'Windows Slaves Plugin'   * workflow-aggregator:2.5 'Pipeline'   * workflow-api:2.24 'Pipeline: API'   * workflow-basic-steps:2.6 'Pipeline: Basic Steps'   * workflow-cps:2.42 'Pipeline: Groovy'   * workflow-cps-global-lib:2.9 'Pipeline: Shared Groovy Libraries'   * workflow-durable-task-step:2.17 'Pipeline: Nodes and Processes'   * workflow-job:2.16 'Pipeline: Job'   * workflow-multibranch:2.16 'Pipeline: Multibranch'   * workflow-scm-step:2.6 'Pipeline: SCM Step'   * workflow-step-api:2.14 'Pipeline: Step API'   * workflow-support:2.16 'Pipeline: Supporting APIs'   * ws-cleanup:0.34 'Jenkins Workspace Cleanup Plugin'

Similar Issues:

Show

Description

After upgrading from 2.73.3 to 2.89.2 our Jenkins has started to experience deadlock.

We use the Multijob plugin to run any number of other jobs that extend a common template. When the Multijob kicks off, it will spin up as many AWS slaves as it needs to run all of the child jobs in parallel (Test-Suites in the stack trace). Every time we run one of these Multijob jobs, Jenkins locks up.

Attached is the deadlock stack traces from a thread dump.

Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 is in deadlock with Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307

Executor #4 for Big Box (r4.2xlarge) (i-05a4635a2e6e063cf) : executing Test-Suites/test-suite-1 #1165 - priority:5 - threadId:0x00007f8fe4118800 - nativeId:0x3455 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
- waiting to lock <0x000000008cdab698> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)

Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
- waiting to lock <0x000000008d744a90> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)

We tried downgrading Jenkins again, but we had already updated all of the other plugins and after downgrading the majority of the plugins were not compatible.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

jenkins-deadlock.txt
88 kB
2017-12-21 17:18

Issue Links

relates to

JENKINS-49328 MultiJob has a forbidden nested reference to a model object

Closed

Activity

Ascending order - Click to sort in descending order

Oleg Nenashev added a comment - 2017-12-22 08:39

Here is a root cause thread from the dump:

Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369)
- waiting to lock <0x000000008d744a90> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71)
at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85)
at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80)
at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393)
at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331)
at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
at hudson.util.XStream2.unmarshal(XStream2.java:114)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
at hudson.XmlFile.unmarshal(XmlFile.java:167)
at hudson.model.Run.reload(Run.java:336)
at hudson.model.Run.<init>(Run.java:324)
at hudson.model.AbstractBuild.<init>(AbstractBuild.java:173)
at hudson.model.Build.<init>(Build.java:104)
at com.tikal.jenkins.plugins.multijob.MultiJobBuild.<init>(MultiJobBuild.java:59)
at sun.reflect.GeneratedConstructorAccessor501.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165)
at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142)
at hudson.model.RunMap.retrieve(RunMap.java:224)
at hudson.model.RunMap.retrieve(RunMap.java:57)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380)
- locked <0x000000008cdab698> (a hudson.model.RunMap)
at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926)
at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137)
at hudson.model.Run.fromExternalizableId(Run.java:2345)
at hudson.model.Run$Replacer.readResolve(Run.java:1937)
at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown

Needs investigation

Oleg Nenashev added a comment - 2017-12-22 08:39 Here is a root cause thread from the dump: Executor #2 for Big Box (r4.2xlarge) (i-057d9fdd7076c7c10) : executing Test-Suites/test-suite-2 #1307 - priority:5 - threadId:0x00007f8ff868e000 - nativeId:0x32e9 - state:BLOCKED stackTrace: java.lang. Thread .State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:369) - waiting to lock <0x000000008d744a90> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:271) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readItem(AbstractCollectionConverter.java:71) at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:85) at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:80) at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:76) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:393) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:331) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:114) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:167) at hudson.model.Run.reload(Run.java:336) at hudson.model.Run.<init>(Run.java:324) at hudson.model.AbstractBuild.<init>(AbstractBuild.java:173) at hudson.model.Build.<init>(Build.java:104) at com.tikal.jenkins.plugins.multijob.MultiJobBuild.<init>(MultiJobBuild.java:59) at sun.reflect.GeneratedConstructorAccessor501.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:165) at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:142) at hudson.model.RunMap.retrieve(RunMap.java:224) at hudson.model.RunMap.retrieve(RunMap.java:57) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500) at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380) - locked <0x000000008cdab698> (a hudson.model.RunMap) at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:926) at hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:137) at hudson.model.Run.fromExternalizableId(Run.java:2345) at hudson.model.Run$Replacer.readResolve(Run.java:1937) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Needs investigation

Allan BURDAJEWICZ added a comment - 2018-02-02 03:11

oleg_nenashev jglick Could this be related to ~~JENKINS-49328~~ and nested references ? We see the `hudson.model.Run$Replacer.readResolve` in the stacktrace. I have also seen a case were the hudson.model.RunMap#retrieve fails with a StackOverflowError, showing the same stacktrace as we can see here.

Allan BURDAJEWICZ added a comment - 2018-02-02 03:11 oleg_nenashev jglick Could this be related to JENKINS-49328 and nested references ? We see the `hudson.model.Run$Replacer.readResolve` in the stacktrace. I have also seen a case were the hudson.model.RunMap#retrieve fails with a StackOverflowError, showing the same stacktrace as we can see here.

People

Assignee:: Unassigned

Reporter:: Mark Ketchum

Votes:: 1 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2017-12-21 20:07

Updated:: 2018-02-07 12:08

Resolved:: 2018-02-02 08:04