Details
-
Type:
Bug
-
Status: Closed (View Workflow)
-
Priority:
Blocker
-
Resolution: Duplicate
-
Component/s: docker, docker-workflow-plugin, durable-task-plugin, pipeline
-
Labels:None
-
Environment:Ubuntu 18.04
Jenkins v. 2.190.1 (installed by apt-get, not in container)
Durable Task Plugin v. 1.33
-
Similar Issues:
Description
Tested on Durable Task Plugin v. 1.29 & v. 1.30 wasn't worked.
Durable Task Plugin has been "updated to v. 1.33". Still the problem isn't resolved.
PIPELINE:
pipeline {
agent {
docker
}
stages {
stage ("Test sh script in container") {
steps
}
}
}
ERROR:
[Pipeline] Start of Pipeline
[Pipeline] nodeRunning on buildserver in c:/jenkins/workspace/<project-name>
[Pipeline] {
[Pipeline] stage
[Pipeline]
{ (Declarative: Checkout SCM) [Pipeline] checkout [Pipeline] }[Pipeline] // stage
[Pipeline] withEnv
[Pipeline]
{ [Pipeline] isUnix[Pipeline] sh process apparently never started in c:/jenkins/workspace/<project-name>@tmp/durable-73ba6e89 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] }[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE
Attachments
Issue Links
- duplicates
-
-
- Resolved
-
- is caused by
-
-
- Resolved
-
Activity
Just noting that the title is a bit misleading. If I read this right, it looks like you have a failed shell step and that it has not been working on 1.29, 1.30, and 1.33. It's not that the upgrade to 1.33 has broken your pipeline correct?
Is there any way you can run your pipeline again with org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true and posting the output here? That might give a better idea of what's going on.
We're having same problem, but it's not yet clear what change caused the behavior. We are on durable task plugin v1.33, and Jenkins ver. 2.190.3, however we have not updated this recently to my knowledge (and I haven't been able to find plugin update history anywhere - any advice?).
The signature of the build failures is that the `withDockerContainer` step completes successfully, and the pipeline begins to run the sh command (the first step in the `withDockerContainer` clause) and there's no apparent activity (other than the seemingly normal output in the durable task plugin loggers) for 5 minutes, then the failure message as the OP here. Here is what the durable task loggers output during those 5 minutes:
Mar 05, 2020 7:03:57 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:03:57 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep still running in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop on
After 5 minutes, I then see:
Mar 05, 2020 7:05:12 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl=true Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[101:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@722b273b Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt'; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[102:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@460edded Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt'; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[103:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@6864647 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt'; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 686 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl=true Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 334 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl=true Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 189 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl=true
Nothing looks suspicious to me here, but I'm unfamiliar with this plugin before last night. Let me know if you can glean anything from this.
I will look into running with the LAUNCH_DIAGNOSTICS suggestion, but in the meantime, since the changelogs singe 1.29 for this plugin indicate instability, I'm inclined to revert and test.
Let me know if you have other suggestions.
Reverting DT plugin to v1.29, restarting Jenkins, and re building a test job that uses withDockerContainer { sh }, results in an error due to an unmet dependency between Durable Task and Pipeline: Nodes and Processes. So I went back to DT v1.33 since a lot of plugins depend on Nodes and Processes, and I'm not immediately sure how to resolve this dependency heck.
In the meantime, I have noticed that we are using PERFORMANCE_OPTIMIZED durability level for this job. That seems relevant, but I don't know enough to connect the dots. Also what seems relevant is that this behavior started after a reboot of the Jenkins master while a job downstream of this test job was running on a slave. From what I've read about Durable Task plugin, its purpose is to enable nodes to manage/monitor processes running on other nodes. This smells rotten. Please enlighten me if you can connect any of these dots.
UPDATE: I have changed the global durability setting to "MAX_SURVIVABILITY" from "PERFORMANCE_OPTIMIZED" and am still getting the same error.
Tim Black So the differences of 1.29 and 1.33 should be fairly negligible. What happened between all those release was the introduction of a binary wrapper (to replace the shell wrapper) for shell scripts to make them more "durable". However, there were varied compatibility issues (mostly with docker and non-x86 builds) that we made the original shell wrapper the default behavior again. Your logging shows that the original shell wrapper is indeed running. The biggest difference I can see between yours and the OP is that your script appears to be executing (writing output to the jenkins-log.txt) and terminating with 0 as opposed to failure.
Like I mentioned with OP, if you can run with LAUNCH_DIAGNOSTICS, that might help us get a better idea of what is going on.
For me, this issue turned out to be an empty environment variable set in Jenkins global properties. This.
When you run with LAUNCH_DIAGNOSTICS, the following line appears:
invalid argument "=" for "-e, --env" flag: invalid environment variable: =
What's happening is that empty environment variables cause the docker-workflow plugin to create a malformed docker run command. This issue is being tracked here: https://issues.jenkins-ci.org/browse/JENKINS-52264
The container does not run, so the threads that are monitoring the durable task script time out.
Will close this ticket in favor of the above mentioned ticket.
Issue is actually a problem in docker-workflow. The durable-task plugin error message is a symptom of the docker container failing to run.
I'm encountering the same error message on Ubuntu 18.04 except it seems to be breaking a git checkout step (checkout scm) and possibly other sh steps, not in a docker container. Also running durable-task v1.33 but also running lots of other plugins and I haven't tried reproducing the issue on clean system yet.
Update: Rebuilt Jenkins (with the same plugins and configuration) and now all of the pipelines work as expected, not sure what was causing it to break before.