Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46651

container step "script returned exit code -1"

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      kubernetes plugin v0.12 or current master
      Jenkins 2.62
      container step running in a _debian_ container
    • Similar Issues:

      Description

      podTemplate(name: "mypod", label: "label", containers: [
                              containerTemplate(name: 'debian',
                                      image: 'debian',
                                      ttyEnabled: true,
                                      command: 'cat',
                              )
      ]) {
          node("label") {
              container('debian') {
                  sh 'for i in $(seq 1 1000); do echo $i; sleep 0.3; done'
              }
          }
      }
      

      leads to

      [Pipeline] podTemplate
      [Pipeline] {
      [Pipeline] node
      Running on horst-nwmsn-32h5h in /home/jenkins/workspace/full
      [Pipeline] {
      [Pipeline] container
      [Pipeline] {
      [Pipeline] sh
      [full] Running shell script
      + seq 1 1000
      + echo 1
      1
      + sleep 0.3
      + echo 2
      2
      + sleep 0.3
      [Pipeline] }
      [Pipeline] // container
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // podTemplate
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -1
      Finished: FAILURE
      

      Sometimes it fails a bit faster.

      Might be related to the script being started with "nohup" now.

        Attachments

          Activity

          Hide
          scoheb Scott Hebert added a comment -

          I am now trying -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_FAILURE_TIMEOUT=120

          Show
          scoheb Scott Hebert added a comment - I am now trying -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_FAILURE_TIMEOUT=120
          Hide
          scoheb Scott Hebert added a comment -

          Martin Sander

          so it looks like I may have found something...

          One theory I have, has to do with the logic for detecting if a process is still running...the plugin executes a "ps" inside the container while the main script is running...the result of the ps call is harvested from the output of the call and used to verify that the main script is alive...BUT it also exports all the env vars BEFORE running the "ps" command ... and some env vars may not be correctly escaped therefore causing some stream corruption between output and error and the EXITCODE is not harvested and thus it thinks the process is not running hence the "-1"

          I found these traces in my logs:

          WARNING: Unable to find "EXITCODE" in a valid identifier
          WARNING: Unable to find "EXITCODE" in r kill -l [sigspec]

          I took the combination of https://github.com/jenkinsci/kubernetes-plugin/pull/232 and https://github.com/jenkinsci/kubernetes-plugin/pull/218 to produce a new SNAPSHOT.

          Have not had a "-1" since.

          Show
          scoheb Scott Hebert added a comment - Martin Sander so it looks like I may have found something... One theory I have, has to do with the logic for detecting if a process is still running...the plugin executes a "ps" inside the container while the main script is running...the result of the ps call is harvested from the output of the call and used to verify that the main script is alive...BUT it also exports all the env vars BEFORE running the "ps" command ... and some env vars may not be correctly escaped therefore causing some stream corruption between output and error and the EXITCODE is not harvested and thus it thinks the process is not running hence the "-1" I found these traces in my logs: WARNING: Unable to find "EXITCODE" in a valid identifier WARNING: Unable to find "EXITCODE" in r kill -l [sigspec] I took the combination of https://github.com/jenkinsci/kubernetes-plugin/pull/232 and https://github.com/jenkinsci/kubernetes-plugin/pull/218 to produce a new SNAPSHOT. Have not had a "-1" since.
          Hide
          0x89 Martin Sander added a comment -

          Scott Hebert: good catch!

          I will try to get the two pull requests merged quickly.

          Show
          0x89 Martin Sander added a comment - Scott Hebert : good catch! I will try to get the two pull requests merged quickly.
          Hide
          scoheb Scott Hebert added a comment -

          Martin Sander

          Still getting this from time to time:

          WARNING: Error getting exit code
          java.lang.InterruptedException
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
          at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
          at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecProc.join(ContainerExecProc.java:72)
          at hudson.Proc.joinWithTimeout(Proc.java:170)
          at org.jenkinsci.plugins.durabletask.ProcessLiveness._isAlive(ProcessLiveness.java:89)
          at org.jenkinsci.plugins.durabletask.ProcessLiveness.isAlive(ProcessLiveness.java:73)
          at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.exitStatus(BourneShellScript.java:198)
          at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:322)
          at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:289)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)

          This one has nothing to do with the EXITCODE and invalid env vars...

          Show
          scoheb Scott Hebert added a comment - Martin Sander Still getting this from time to time: WARNING: Error getting exit code java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecProc.join(ContainerExecProc.java:72) at hudson.Proc.joinWithTimeout(Proc.java:170) at org.jenkinsci.plugins.durabletask.ProcessLiveness._isAlive(ProcessLiveness.java:89) at org.jenkinsci.plugins.durabletask.ProcessLiveness.isAlive(ProcessLiveness.java:73) at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.exitStatus(BourneShellScript.java:198) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:322) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:289) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) This one has nothing to do with the EXITCODE and invalid env vars...
          Hide
          jglick Jesse Glick added a comment -

          Possibly solved by JENKINS-47791.

          Show
          jglick Jesse Glick added a comment - Possibly solved by  JENKINS-47791 .

            People

            Assignee:
            0x89 Martin Sander
            Reporter:
            0x89 Martin Sander
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: