Details
-
Bug
-
Status: Resolved (View Workflow)
-
Minor
-
Resolution: Fixed
-
None
-
kubernetes plugin v0.12 or current master
Jenkins 2.62
container step running in a _debian_ container
Description
podTemplate(name: "mypod", label: "label", containers: [ containerTemplate(name: 'debian', image: 'debian', ttyEnabled: true, command: 'cat', ) ]) { node("label") { container('debian') { sh 'for i in $(seq 1 1000); do echo $i; sleep 0.3; done' } } }
leads to
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Running on horst-nwmsn-32h5h in /home/jenkins/workspace/full
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[full] Running shell script
+ seq 1 1000
+ echo 1
1
+ sleep 0.3
+ echo 2
2
+ sleep 0.3
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Finished: FAILURE
Sometimes it fails a bit faster.
Might be related to the script being started with "nohup" now.
Attachments
Issue Links
- duplicates
-
JENKINS-47791 Eliminate ProcessLiveness
-
- Resolved
-
- links to
Activity
so it looks like I may have found something...
One theory I have, has to do with the logic for detecting if a process is still running...the plugin executes a "ps" inside the container while the main script is running...the result of the ps call is harvested from the output of the call and used to verify that the main script is alive...BUT it also exports all the env vars BEFORE running the "ps" command ... and some env vars may not be correctly escaped therefore causing some stream corruption between output and error and the EXITCODE is not harvested and thus it thinks the process is not running hence the "-1"
I found these traces in my logs:
WARNING: Unable to find "EXITCODE" in a valid identifier
WARNING: Unable to find "EXITCODE" in r kill -l [sigspec]
I took the combination of https://github.com/jenkinsci/kubernetes-plugin/pull/232 and https://github.com/jenkinsci/kubernetes-plugin/pull/218 to produce a new SNAPSHOT.
Have not had a "-1" since.
scoheb: good catch!
I will try to get the two pull requests merged quickly.
Still getting this from time to time:
WARNING: Error getting exit code
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecProc.join(ContainerExecProc.java:72)
at hudson.Proc.joinWithTimeout(Proc.java:170)
at org.jenkinsci.plugins.durabletask.ProcessLiveness._isAlive(ProcessLiveness.java:89)
at org.jenkinsci.plugins.durabletask.ProcessLiveness.isAlive(ProcessLiveness.java:73)
at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.exitStatus(BourneShellScript.java:198)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:322)
at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:289)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This one has nothing to do with the EXITCODE and invalid env vars...
I am now trying -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_FAILURE_TIMEOUT=120