Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59903

durable-task v1.31 breaks sh steps in pipeline when running in a Docker container

    • 1.33

      A pipeline like this:

      pipeline {
          agent {
              docker {
                  label 'docker'
                  image 'busybox'
              }
          }
          stages {
              stage("Test sh script in container") {
                  steps {
                    sh label: 'Echo "Hello World...', script: 'echo "Hello World!"'
                  }
              }
          }
      }
      

      Fails with this log:

      Running in Durability level: PERFORMANCE_OPTIMIZED
      [Pipeline] Start of Pipeline (hide)
      [Pipeline] node
      Running on docker-node in /...
      [Pipeline] {
      [Pipeline] isUnix
      [Pipeline] sh
      + docker inspect -f . busybox
      .
      [Pipeline] withDockerContainer
      got-legaci-3 does not seem to be running inside a container
      $ docker run -t -d -u 1002:1002 -w <<hidden>> busybox cat
      $ docker top 645fd28fda5fa3c61a4b49e8a38e46e0eec331ddf6037d3f77821dd6984a185f -eo pid,comm
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Test sh script in container)
      [Pipeline] sh (Echo "Hello World...)
      process apparently never started in /...
      (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      $ docker stop --time=1 645fd28fda5fa3c61a4b49e8a38e46e0eec331ddf6037d3f77821dd6984a185f
      $ docker rm -f 645fd28fda5fa3c61a4b49e8a38e46e0eec331ddf6037d3f77821dd6984a185f
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -2
      Finished: FAILURE
      

      Adding the -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true parameter gives this log:

      Running in Durability level: PERFORMANCE_OPTIMIZED
      [Pipeline] Start of Pipeline
      [Pipeline] node
      Running on docker-node in /...
      [Pipeline] {
      [Pipeline] isUnix
      [Pipeline] sh
      + docker inspect -f . busybox
      .
      [Pipeline] withDockerContainer
      got-legaci-3 does not seem to be running inside a container
      $ docker run -t -d -u 1002:1002 -w <<hidden>> busybox cat
      $ docker top 31b7474756f8ff5b1f0d12d0df952347e584b47113108d1f965adeeb0ee78e5e -eo pid,comm
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Test sh script in container)
      [Pipeline] sh (Echo "Hello World...)
      OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"/var/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64\": stat /var/jenkins/caches/durable-task/durable_task_monitor_1.31_unix_64: no such file or directory": unknown
      process apparently never started in /...
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      $ docker stop --time=1 31b7474756f8ff5b1f0d12d0df952347e584b47113108d1f965adeeb0ee78e5e
      $ docker rm -f 31b7474756f8ff5b1f0d12d0df952347e584b47113108d1f965adeeb0ee78e5e
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -2
      Finished: FAILURE
      

      Tested on three different Jenkins masters with similar, but no identical, configurations.

      Reverting to Durable Task Plugin v. 1.30 "solves" the problem.

          [JENKINS-59903] durable-task v1.31 breaks sh steps in pipeline when running in a Docker container

          Jesper Andersson created issue -
          Jesper Andersson made changes -
          Environment Original: Centos 7.7
          Jenkins ver. 2.190.1
          Durable Task Plugin v. 1.31
          New: Centos 7.7
          Jenkins ver. 2.190.1 (installed by yum)
          Durable Task Plugin v. 1.31

          Jesper Andersson added a comment - - edited

          A different workaround is adding args '-v /var/jenkins-legaci-lab/caches:/var/jenkins-legaci-lab/caches' to the docker{...} declaration in the pipeline.
          Like this:

          pipeline {
              agent {
                  docker {
                      label 'docker'
                      image 'busybox'
                      args '-v /var/jenkins/caches:/var/jenkins/caches'
                  }
              }
              stages {
                  stage("Test sh script in container") {
                      steps {
                        sh label: 'Echo "Hello World...', script: 'echo "Hello World!"'
                      }
                  }
              }
          }
          
          • Perhaps this should be solved in some declarative pipeline component?

          Jesper Andersson added a comment - - edited A different workaround is adding args '-v /var/jenkins-legaci-lab/caches:/var/jenkins-legaci-lab/caches' to the docker{... } declaration in the pipeline. Like this: pipeline { agent { docker { label 'docker' image 'busybox' args '-v /var/jenkins/caches:/var/jenkins/caches' } } stages { stage( "Test sh script in container" ) { steps { sh label: 'Echo "Hello World...' , script: 'echo " Hello World!"' } } } } Perhaps this should be solved in some declarative pipeline component?
          Aaron Pellman-Isaacs made changes -
          Link New: This issue is duplicated by JENKINS-59906 [ JENKINS-59906 ]

          Ivan Kurnosov added a comment -

          I confirm it affects me as well:

          Jenkins ver. 2.190.1, docker, kubernetes and other plugins: latest stable version.

          Jenkins runs on linux (inside a docker container)

          Ivan Kurnosov added a comment - I confirm it affects me as well: Jenkins ver. 2.190.1, docker, kubernetes and other plugins: latest stable version. Jenkins runs on linux (inside a docker container)
          Reinhold Füreder made changes -
          Link New: This issue is related to JENKINS-59907 [ JENKINS-59907 ]

          Also a problem when running Jenkins not in Docker and executing pipeline on Jenkins master (=> psst!); problem is definitely caused by new "Durable Task" plugin v1.31 bug:

          • Latest version of Jenkins core (v2.201) running on Ubuntu 16.04
          • The logs seem to indicate the problem already appears when trying to start the Docker container in the pipeline, but maybe the logs are just mangled?
            • See "// !!!" comment in belows build log...
          • Build log (proprietary):
            ...
            [Pipeline] // stage
            [Pipeline] stage
            [Pipeline] { (linkchecker)
            [Pipeline] script
            [Pipeline] {
            [Pipeline] withEnv
            [Pipeline] {
            [Pipeline] withDockerRegistry
            [Pipeline] {
            [Pipeline] isUnix
            [Pipeline] sh
            06:37:20  + docker inspect -f . ACME/linkchecker:5
            06:37:20  
            06:37:20  Error: No such object: ACME/linkchecker:5
            06:37:20  06:37:20.408218 durable_task_monitor.go:63: exit status 1              // !!!
            [Pipeline] isUnix
            [Pipeline] sh
            06:37:20  + docker inspect -f . dockerregistry.ACME.com/ACME/linkchecker:5
            06:37:20  .
            [Pipeline] withDockerContainer
            06:37:20  Jenkins does not seem to be running inside a container
            06:37:20  $ docker run -t -d -u 10112:10005 -w /var/lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline -v /var/lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline:/var/lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline:rw,z -v /var/lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline@tmp:/var/lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** dockerregistry.ACME.com/ACME/linkchecker:5 cat
            06:37:21  $ docker top 39f784ea27cbf6593fd40c1faaf04948daae94e97eb8ba42517f7c2f5e40c21e -eo pid,comm
            [Pipeline] {
            [Pipeline] sh
            06:42:27  process apparently never started in /var/lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline@tmp/durable-aed939a9      // !!!
            06:42:27  (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
            [Pipeline] }
            ...
            ERROR: script returned exit code -2
            Finished: FAILURE
            
          • Pipeline code (in shared Jenkins pipeline library):
            ...
              void execute(Closure configBody) {
                LinkCheckerRunDSL config = calcConfiguration(configBody)
            
                // Then build, based on the configuration provided:
                script.docker.withRegistry(Constants.ACME_DOCKER_REGISTRY_URL) {
                  script.docker.image(config.dockerImage).inside() { c ->
                    script.sh 'linkchecker --version'
            ...
            

          Reinhold Füreder added a comment - Also a problem when running Jenkins not in Docker and executing pipeline on Jenkins master (=> psst!); problem is definitely caused by new "Durable Task" plugin v1.31 bug: Latest version of Jenkins core (v2.201) running on Ubuntu 16.04 The logs seem to indicate the problem already appears when trying to start the Docker container in the pipeline, but maybe the logs are just mangled? See " // !!! " comment in belows build log... Build log (proprietary): ... [Pipeline] // stage [Pipeline] stage [Pipeline] { (linkchecker) [Pipeline] script [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withDockerRegistry [Pipeline] { [Pipeline] isUnix [Pipeline] sh 06:37:20 + docker inspect -f . ACME/linkchecker:5 06:37:20 06:37:20 Error: No such object: ACME/linkchecker:5 06:37:20 06:37:20.408218 durable_task_monitor.go:63: exit status 1 // !!! [Pipeline] isUnix [Pipeline] sh 06:37:20 + docker inspect -f . dockerregistry.ACME.com/ACME/linkchecker:5 06:37:20 . [Pipeline] withDockerContainer 06:37:20 Jenkins does not seem to be running inside a container 06:37:20 $ docker run -t -d -u 10112:10005 -w / var /lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline -v / var /lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline:/ var /lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline:rw,z -v / var /lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline@tmp:/ var /lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** dockerregistry.ACME.com/ACME/linkchecker:5 cat 06:37:21 $ docker top 39f784ea27cbf6593fd40c1faaf04948daae94e97eb8ba42517f7c2f5e40c21e -eo pid,comm [Pipeline] { [Pipeline] sh 06:42:27 process apparently never started in / var /lib/jenkins/workspace/Sandbox/ACME.linkCheckerPipeline@tmp/durable-aed939a9 // !!! 06:42:27 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS= true might make the problem clearer) [Pipeline] } ... ERROR: script returned exit code -2 Finished: FAILURE Pipeline code (in shared Jenkins pipeline library): ... void execute(Closure configBody) { LinkCheckerRunDSL config = calcConfiguration(configBody) // Then build, based on the configuration provided: script.docker.withRegistry(Constants.ACME_DOCKER_REGISTRY_URL) { script.docker.image(config.dockerImage).inside() { c -> script.sh 'linkchecker --version' ...
          Jakub L made changes -
          Link Original: This issue is related to JENKINS-59907 [ JENKINS-59907 ]
          Jesper Andersson made changes -
          Environment Original: Centos 7.7
          Jenkins ver. 2.190.1 (installed by yum)
          Durable Task Plugin v. 1.31
          New: Centos 7.7
          Jenkins ver. 2.190.1 (installed by yum, not in container)
          Durable Task Plugin v. 1.31

          Having the same issue with the 1.31 version of durable-task plugin. Had to rollback to 1.30.

          Constantin Bugneac added a comment - Having the same issue with the 1.31 version of durable-task plugin. Had to rollback to 1.30.

            carroll Carroll Chiou
            njesper Jesper Andersson
            Votes:
            34 Vote for this issue
            Watchers:
            48 Start watching this issue

              Created:
              Updated:
              Resolved: