-
Bug
-
Resolution: Unresolved
-
Major
-
Jenkins: 2.277.2
Jenkins JVM: Oracle OpenJDK 64-Bit Server VM 1.8.0_282 (25.282-b08)
OS: Linux version 4.4.0-210-generic (buildd@lgw01-amd64-009) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12) )
---
ansicolor:1.0.0
momentjs:1.1.1
saferestart:0.3
jdk-tool:1.5
docker-workflow:1.26
junit:1.50
script-security:1.77
github-api:1.123
antisamy-markup-formatter:2.1
pipeline-model-definition:1.8.5
github:1.33.1
git:4.7.2
javadoc:1.6
workflow-multibranch:2.24
workflow-basic-steps:2.23
ssh-credentials:1.18.1
workflow-support:3.8
ant:1.11
pipeline-model-api:1.8.5
conditional-buildstep:1.4.1
git-client:3.7.2
bouncycastle-api:2.20
cloudbees-folder:6.15
openstack-cloud:2.58
postbuild-task:1.9
ws-cleanup:0.39
sidebar-link:1.12.0
leastload:3.0.0
lockable-resources:2.11
build-name-setter:2.2.0
durable-task:1.37
multiple-scms:0.8
pipeline-input-step:2.12
cloud-stats:0.27
powershell:1.5
workflow-step-api:2.23
publish-over:0.22
handlebars:3.0.8
cloudbees-disk-usage-simple:0.10
matrix-auth:2.6.7
popper-api:1.16.1-2
pipeline-utility-steps:2.8.0
build-timeout:1.20
run-condition:1.5
docker-plugin:1.2.2
apache-httpcomponents-client-4-api:4.5.13-1.0
plain-credentials:1.7
jobConfigHistory:2.27
workflow-cps-global-lib:2.20
envinject:2.4.0
build-flow-plugin:0.20
display-url-api:2.3.5
authentication-tokens:1.4
simple-theme-plugin:0.6
maven-plugin:3.12
workflow-job:2.41
pipeline-stage-view:2.19
slack:2.48
workflow-scm-step:2.13
snakeyaml-api:1.29.1
job-import-plugin:3.4
appscan:1.0.7
teamconcert:1.1.9.5-SNAPSHOT (private-7318df45-jenkins)
workflow-api:2.45
groovy:2.4
command-launcher:1.6
credentials:2.5
next-build-number:1.6
matrix-project:1.18
token-macro:2.15
role-strategy:3.1.1
gradle:1.36
log-parser:2.1
pipeline-stage-step:2.5
label-linked-jobs:6.0.1
jackson2-api:2.12.3
extended-read-permission:3.2
ibm-application-security:1.3.2
bootstrap5-api:5.0.1-2
jsch:0.1.55.2
docker-commons:1.17
mailer:1.34
font-awesome-api:5.15.3-3
pipeline-graph-analysis:1.11
rebuild:1.32
caffeine-api:2.9.1-23.v51c4e2c879c8
checks-api:1.7.0
Parameterized-Remote-Trigger:3.1.5.1
okhttp-api:3.14.9
ivy:2.1
groovy-label-assignment:1.2.0
credentials-binding:1.25
windows-slaves:1.8
ssh-slaves:1.31.5
scm-api:2.6.4
artifactory:3.11.4
publish-over-ssh:1.22
echarts-api:5.1.2-2
test-results-analyzer:0.3.5
trilead-api:1.0.13
pipeline-milestone-step:1.3.2
jquery3-api:3.6.0-1
ace-editor:1.1
ssh-agent:1.22
timestamper:1.13
workflow-aggregator:2.6
envinject-api:1.7
parameterized-trigger:2.41
workflow-durable-task-step:2.39
pam-auth:1.6
pipeline-stage-tags-metadata:1.8.5
external-monitor-job:1.4
purge-build-queue-plugin:1.0
saml:2.0.6
popper2-api:2.5.4-2
bootstrap4-api:4.6.0-3
workflow-cps:2.92
pipeline-rest-api:2.19
config-file-provider:3.8.0
ldap:2.7
pipeline-build-step:2.13
docker-java-api:3.1.5.2
pipeline-model-extensions:1.8.5
plugin-util-api:2.3.0
structs:1.23
git-server:1.9
branch-api:2.6.4
resource-disposer:0.16Jenkins: 2.277.2 Jenkins JVM: Oracle OpenJDK 64-Bit Server VM 1.8.0_282 (25.282-b08) OS: Linux version 4.4.0-210-generic ( buildd@lgw01-amd64-009 ) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12) ) --- ansicolor:1.0.0 momentjs:1.1.1 saferestart:0.3 jdk-tool:1.5 docker-workflow:1.26 junit:1.50 script-security:1.77 github-api:1.123 antisamy-markup-formatter:2.1 pipeline-model-definition:1.8.5 github:1.33.1 git:4.7.2 javadoc:1.6 workflow-multibranch:2.24 workflow-basic-steps:2.23 ssh-credentials:1.18.1 workflow-support:3.8 ant:1.11 pipeline-model-api:1.8.5 conditional-buildstep:1.4.1 git-client:3.7.2 bouncycastle-api:2.20 cloudbees-folder:6.15 openstack-cloud:2.58 postbuild-task:1.9 ws-cleanup:0.39 sidebar-link:1.12.0 leastload:3.0.0 lockable-resources:2.11 build-name-setter:2.2.0 durable-task:1.37 multiple-scms:0.8 pipeline-input-step:2.12 cloud-stats:0.27 powershell:1.5 workflow-step-api:2.23 publish-over:0.22 handlebars:3.0.8 cloudbees-disk-usage-simple:0.10 matrix-auth:2.6.7 popper-api:1.16.1-2 pipeline-utility-steps:2.8.0 build-timeout:1.20 run-condition:1.5 docker-plugin:1.2.2 apache-httpcomponents-client-4-api:4.5.13-1.0 plain-credentials:1.7 jobConfigHistory:2.27 workflow-cps-global-lib:2.20 envinject:2.4.0 build-flow-plugin:0.20 display-url-api:2.3.5 authentication-tokens:1.4 simple-theme-plugin:0.6 maven-plugin:3.12 workflow-job:2.41 pipeline-stage-view:2.19 slack:2.48 workflow-scm-step:2.13 snakeyaml-api:1.29.1 job-import-plugin:3.4 appscan:1.0.7 teamconcert:1.1.9.5-SNAPSHOT (private-7318df45-jenkins) workflow-api:2.45 groovy:2.4 command-launcher:1.6 credentials:2.5 next-build-number:1.6 matrix-project:1.18 token-macro:2.15 role-strategy:3.1.1 gradle:1.36 log-parser:2.1 pipeline-stage-step:2.5 label-linked-jobs:6.0.1 jackson2-api:2.12.3 extended-read-permission:3.2 ibm-application-security:1.3.2 bootstrap5-api:5.0.1-2 jsch:0.1.55.2 docker-commons:1.17 mailer:1.34 font-awesome-api:5.15.3-3 pipeline-graph-analysis:1.11 rebuild:1.32 caffeine-api:2.9.1-23.v51c4e2c879c8 checks-api:1.7.0 Parameterized-Remote-Trigger:3.1.5.1 okhttp-api:3.14.9 ivy:2.1 groovy-label-assignment:1.2.0 credentials-binding:1.25 windows-slaves:1.8 ssh-slaves:1.31.5 scm-api:2.6.4 artifactory:3.11.4 publish-over-ssh:1.22 echarts-api:5.1.2-2 test-results-analyzer:0.3.5 trilead-api:1.0.13 pipeline-milestone-step:1.3.2 jquery3-api:3.6.0-1 ace-editor:1.1 ssh-agent:1.22 timestamper:1.13 workflow-aggregator:2.6 envinject-api:1.7 parameterized-trigger:2.41 workflow-durable-task-step:2.39 pam-auth:1.6 pipeline-stage-tags-metadata:1.8.5 external-monitor-job:1.4 purge-build-queue-plugin:1.0 saml:2.0.6 popper2-api:2.5.4-2 bootstrap4-api:4.6.0-3 workflow-cps:2.92 pipeline-rest-api:2.19 config-file-provider:3.8.0 ldap:2.7 pipeline-build-step:2.13 docker-java-api:3.1.5.2 pipeline-model-extensions:1.8.5 plugin-util-api:2.3.0 structs:1.23 git-server:1.9 branch-api:2.6.4 resource-disposer:0.16
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access /threadDump for our Jenkins server while a hang is in progress, we see a stack trace like the following for the hanging executor thread:
"Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429)
The problem seems to be a wait() call that never returns from AsyncFutureImpl.java:79 (Jenkins core) to TriggerBuilder.java:155 (Parameterized Trigger plugin).
Bizarrely, the hanging continues even after the node running the parent has gone offline:
The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64).
The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default):
- Parent [Freestyle]:
- Discard old builds → Max # of builds to keep: 30
- Restrict where this project can be run: iibdev-x-rhel-hur-fyre-discard-87671
- Build periodically: * * * * *
- Build step: Trigger/call builds on other projects: Child
- Block until the triggered projects finish their builds
- Build on the same node
- Build step → Execute shell:
echo "Got past downstream"
- Child [Pipeline]:
- Do not allow the pipeline to resume if the master restarts
- Disable Rebuilding for this job
- Pipeline script:
println("foobar")- Use Groovy Sandbox
[JENKINS-66091] Parameterized Trigger intermittently hangs on wait() call from AsyncFutureImpl.java:79
Arnie Alpenbeach
created issue -
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access https://hyc-iib-jenkins.swg-devops.com/threadDump while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !image-2021-07-08-11-44-34-035.png! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and therefore difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{{{* * * * *}}}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and therefore difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{{{* * * * *}}}} ** Build step: Trigger/call builds on other projects: {{Child}} * ** *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and therefore difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{{{* * * * *}}}} ** Build step: Trigger/call builds on other projects: {{Child}} * ** *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and therefore difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{{{* * * * *}}}} ** Build step: Trigger/call builds on other projects: {{Child}} * ** *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and therefore difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{{{* * * * *}}}} ** Build step: Trigger/call builds on other projects: {{Child}} * ** *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* * ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* * ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName()// Set this relabeling job instance's name: currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* * ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName() currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* * ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName() currentBuild.setDisplayName("Node: ${node_name}") {code} * ** *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName() currentBuild.setDisplayName("Node: ${node_name}") {code} *** Use Groovy Sandbox |
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step: Changes build name: {{#${BUILD_NUMBER}: Foo}} ** Build step: Execute shell: <trivial echo script> * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} def node = currentBuild.getRawBuild().getExecutor().getOwner().getNode() def node_name = node.getNodeName() currentBuild.setDisplayName("Node: ${node_name}") {code} *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step → Execute shell: {code:bash} echo "Got past downstream" {code} * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} println("foobar") {code} *** Use Groovy Sandbox |
Arnie Alpenbeach
made changes -
| Priority | Original: Minor [ 4 ] | New: Major [ 3 ] |
Arnie Alpenbeach
made changes -
| Description |
Original:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access [https://hyc-iib-jenkins.swg-devops.com/threadDump] while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step → Execute shell: {code:bash} echo "Got past downstream" {code} * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} println("foobar") {code} *** Use Groovy Sandbox |
New:
We use the Parameterized Trigger plugin to launch a downstream Pipeline job (child) from an upstream Freestyle job (parent). The parent is configured to launch the child on the same single-executor node as the parent. Intermittently, the parent will hang waiting for the child to return control to it after the child completes.
If we access /threadDump for our Jenkins server while a hang is in progress, we see a stack trace like the following for the hanging executor thread: {code:java} "Executor #0 for iibdev-x-rhel-hur-fyre-discard-87671 : executing ib000_Linux_PPCLE_Packager #2489" Id=101251 Group=main WAITING on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Native Method) - waiting on hudson.model.queue.FutureImpl@278f4262 at java.lang.Object.wait(Object.java:502) at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:79) at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:155) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:803) at hudson.model.Build$BuildExecution.build(Build.java:197) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:513) at hudson.model.Run.execute(Run.java:1907) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429){code} The problem seems to be a wait() call that never returns from [AsyncFutureImpl.java:79|https://github.com/jenkinsci/remoting/blob/83d16d8a9157e00187fca9e8c67fa340b303f9c4/src/main/java/hudson/remoting/AsyncFutureImpl.java#L79] (Jenkins core) to [TriggerBuilder.java:155|https://github.com/jenkinsci/parameterized-trigger-plugin/blob/ca943ac3d89d8f7faab0cc14094fcd1f3a1aa085/src/main/java/hudson/plugins/parameterizedtrigger/TriggerBuilder.java#L155] (Parameterized Trigger plugin). Bizarrely, the hanging continues even after the node running the parent has gone offline: !hanging-continues-even-when-node-offline.png|id=cp-img! The problem has been reproduced on various Linux and Windows nodes: Red Hat Enterprise Linux 7.9 (x86_64, s390x, ppc64le) and Windows 2012 R2 (x86_64). The problem is very intermittent and difficult to reproduce - it has taken several weeks' worth of jobs running every minute of the day to reproduce the issue a handful of times. The problem has been reproduced with the following job pairing (assume all configuration options except those mentioned below left as the default): * Parent [Freestyle]: ** Discard old builds → Max # of builds to keep: {{30}} ** Restrict where this project can be run: {{iibdev-x-rhel-hur-fyre-discard-87671}} ** Build periodically: {{* * * * *}} ** Build step: Trigger/call builds on other projects: {{Child}} *** Block until the triggered projects finish their builds *** *Build on the same node* ** Build step → Execute shell: {code:bash} echo "Got past downstream" {code} * Child [Pipeline]: ** Do not allow the pipeline to resume if the master restarts ** Disable Rebuilding for this job ** Pipeline script: {code:java} println("foobar") {code} *** Use Groovy Sandbox |
Still happening with Jenkins v2.289.2.