Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60039

durable-task v1.33 breaks sh steps in pipeline when running in a Docker container

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Tested on Durable Task Plugin v. 1.29 & v. 1.30 wasn't worked.

      Durable Task Plugin has been "updated to v. 1.33". Still the problem isn't resolved.

       

      PIPELINE:

      pipeline {
          agent {
               docker

      {             label 'buildserver'             image 'busybox'             args '-v /var/lib/jenkins/caches:/jenkins/caches'            args '-v /var/lib/jenkins/workspace:/jenkins/workspace'            args '-u root'                  }

                  }
         stages {
            stage ("Test sh script in container") {
               steps

      {              sh label: 'Echo "Hello World...', script: 'echo "Hello World!"'                            }

                        }
                  }
          }

       

      ERROR:
      [Pipeline] Start of Pipeline

      [Pipeline] nodeRunning on buildserver in c:/jenkins/workspace/<project-name>

      [Pipeline] {

      [Pipeline] stage

      [Pipeline]

      { (Declarative: Checkout SCM) [Pipeline] checkout [Pipeline] }

      [Pipeline] // stage

      [Pipeline] withEnv

      [Pipeline]

      { [Pipeline] isUnix[Pipeline] sh process apparently never started in c:/jenkins/workspace/<project-name>@tmp/durable-73ba6e89 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) [Pipeline] }

      [Pipeline] // withEnv

      [Pipeline] }

      [Pipeline] // node

      [Pipeline] End of Pipeline

      ERROR: script returned exit code -2
      Finished: FAILURE

        Attachments

          Issue Links

            Activity

            Hide
            aikatz Alex Katz added a comment - - edited

            I'm encountering the same error message on Ubuntu 18.04 except it seems to be breaking a git checkout step (checkout scm) and possibly other sh steps, not in a docker container. Also running durable-task v1.33 but also running lots of other plugins and I haven't tried reproducing the issue on clean system yet.

            Update: Rebuilt Jenkins (with the same plugins and configuration) and now all of the pipelines work as expected, not sure what was causing it to break before.

            Show
            aikatz Alex Katz added a comment - - edited I'm encountering the same error message on Ubuntu 18.04 except it seems to be breaking a git checkout step ( checkout scm ) and possibly other sh steps, not in a docker container. Also running durable-task v1.33 but also running lots of other plugins and I haven't tried reproducing the issue on clean system yet. Update: Rebuilt Jenkins (with the same plugins and configuration) and now all of the pipelines work as expected, not sure what was causing it to break before.
            Hide
            carroll Carroll Chiou added a comment -

            Just noting that the title is a bit misleading. If I read this right, it looks like you have a failed shell step and that it has not been working on 1.29, 1.30, and 1.33. It's not that the upgrade to 1.33 has broken your pipeline correct?

            Is there any way you can run your pipeline again with org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true and posting the output here? That might give a better idea of what's going on.

            Show
            carroll Carroll Chiou added a comment - Just noting that the title is a bit misleading. If I read this right, it looks like you have a failed shell step and that it has not been working on 1.29, 1.30, and 1.33. It's not that the upgrade to 1.33 has broken your pipeline correct? Is there any way you can run your pipeline again with org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true and posting the output here? That might give a better idea of what's going on.
            Hide
            timblaktu Tim Black added a comment - - edited

            We're having same problem, but it's not yet clear what change caused the behavior. We are on durable task plugin v1.33, and Jenkins ver. 2.190.3, however we have not updated this recently to my knowledge (and I haven't been able to find plugin update history anywhere - any advice?).

            The signature of the build failures is that the `withDockerContainer` step completes successfully, and the pipeline begins to run the sh command (the first step in the `withDockerContainer` clause) and there's no apparent activity (other than the seemingly normal output in the durable task plugin loggers) for 5 minutes, then the failure message as the OP here. Here is what the durable task loggers output during those 5 minutes:

             

            Mar 05, 2020 7:03:57 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep  seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:03:57 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep still running in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop on

            After 5 minutes, I then see:

             

             

            Mar 05, 2020 7:05:12 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             calling close with nl=true
             Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[101:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@722b273b
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
             launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt'; wait) >& 2>&- &]
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             launched task
             Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[102:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@460edded
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
             launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt'; wait) >& 2>&- &]
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             launched task
             Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[103:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@6864647
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
             launching [nohup, sh, c, ({ while [ -d '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f' -a ! -f '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt'; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/script.sh' > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' 2>&1; echo $$? > '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt'; wait) >& 2>&- &]
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             launched task
             Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             creating filtered stream
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
             remote transcoding charset: null
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
             copied 686 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
             found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             calling close with nl=true
             Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             creating filtered stream
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
             remote transcoding charset: null
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
             copied 334 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
             found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             calling close with nl=true
             Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             seems to be online so using /var/lib/jenkins/workspace/TesiraFirmwareNG_develop
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             creating filtered stream
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
             remote transcoding charset: null
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask
             copied 189 bytes from /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript
             found exit code 0 in /var/lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f
             Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep
             calling close with nl=true
            

            Nothing looks suspicious to me here, but I'm unfamiliar with this plugin before last night. Let me know if you can glean anything from this.

            I will look into running with the LAUNCH_DIAGNOSTICS suggestion, but in the meantime, since the changelogs singe 1.29 for this plugin indicate instability, I'm inclined to revert and test.

            Let me know if you have other suggestions.

            Show
            timblaktu Tim Black added a comment - - edited We're having same problem, but it's not yet clear what change caused the behavior. We are on durable task plugin v1.33, and Jenkins ver. 2.190.3 , however we have not updated this recently to my knowledge (and I haven't been able to find plugin update history anywhere - any advice?). The signature of the build failures is that the `withDockerContainer` step completes successfully, and the pipeline begins to run the sh command (the first step in the `withDockerContainer` clause) and there's no apparent activity (other than the seemingly normal output in the durable task plugin loggers) for 5 minutes, then the failure message as the OP here. Here is what the durable task loggers output during those 5 minutes:   Mar 05, 2020 7:03:57 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep  seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:03:57 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep still running in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop on After 5 minutes, I then see:     Mar 05, 2020 7:05:12 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[101:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@722b273b Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859' -a ! -f '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' ; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/script.sh' > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt' 2>&1; echo $$? > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-result.txt' ; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[102:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@460edded Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0' -a ! -f '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' ; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/script.sh' > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt' 2>&1; echo $$? > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-result.txt' ; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINEST org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep JENKINS-34021: DurableTaskStep.Execution.listener present in CpsStepContext[103:sh]:OwnerTesiraFirmwareNG/develop/42:TesiraFirmwareNG/develop #42 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launching task against hudson.remoting.LocalChannel@41a7fb38 using hudson.Launcher$LocalLauncher@6864647 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript launching [nohup, sh, c, ({ while [ -d '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f' -a ! -f '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' ; sleep 3; done } & jsc=durable-97017349f4d4de030acf9aecc2fb49fa; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/script.sh' > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt' 2>&1; echo $$? > '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-result.txt' ; wait) >& 2>&- &] Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep launched task Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 686 bytes from / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-cc6ad859 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 334 bytes from / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a359e9b0 Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Mar 05, 2020 7:05:14 AM FINER org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep seems to be online so using / var /lib/jenkins/workspace/TesiraFirmwareNG_develop Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep creating filtered stream Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask remote transcoding charset: null Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.FileMonitoringTask copied 189 bytes from / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f/jenkins-log.txt Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.durabletask.BourneShellScript found exit code 0 in / var /lib/jenkins/workspace/TesiraFirmwareNG_develop-tmp/durable-a9f0797f Mar 05, 2020 7:05:14 AM FINE org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep calling close with nl= true Nothing looks suspicious to me here, but I'm unfamiliar with this plugin before last night. Let me know if you can glean anything from this. I will look into running with the LAUNCH_DIAGNOSTICS suggestion, but in the meantime, since the changelogs singe 1.29 for this plugin indicate instability, I'm inclined to revert and test. Let me know if you have other suggestions.
            Hide
            timblaktu Tim Black added a comment - - edited

            Reverting DT plugin to v1.29, restarting Jenkins, and re building a test job that uses withDockerContainer { sh }, results in an error due to an unmet dependency between Durable Task and Pipeline: Nodes and Processes. So I went back to DT v1.33 since a lot of plugins depend on Nodes and Processes, and I'm not immediately sure how to resolve this dependency heck.

            In the meantime, I have noticed that we are using PERFORMANCE_OPTIMIZED durability level for this job. That seems relevant, but I don't know enough to connect the dots. Also what seems relevant is that this behavior started after a reboot of the Jenkins master while a job downstream of this test job was running on a slave. From what I've read about Durable Task plugin, its purpose is to enable nodes to manage/monitor processes running on other nodes. This smells rotten. Please enlighten me if you can connect any of these dots. 

            UPDATE: I have changed the global durability setting to "MAX_SURVIVABILITY" from "PERFORMANCE_OPTIMIZED" and am still getting the same error.

            Show
            timblaktu Tim Black added a comment - - edited Reverting DT plugin to v1.29, restarting Jenkins, and re building a test job that uses withDockerContainer { sh }, results in an error due to an unmet dependency between Durable Task and Pipeline: Nodes and Processes. So I went back to DT v1.33 since a lot of plugins depend on Nodes and Processes, and I'm not immediately sure how to resolve this dependency heck. In the meantime, I have noticed that we are using PERFORMANCE_OPTIMIZED durability level for this job. That seems relevant, but I don't know enough to connect the dots. Also what seems relevant is that this behavior started after a reboot of the Jenkins master  while a job downstream of this test job was running on a slave. From what I've read about Durable Task plugin, its purpose is to enable nodes to manage/monitor processes running on other nodes. This smells rotten. Please enlighten me if you can connect any of these dots.  UPDATE: I have changed the global durability setting to "MAX_SURVIVABILITY" from "PERFORMANCE_OPTIMIZED" and am still getting the same error.
            Hide
            carroll Carroll Chiou added a comment -

            Tim Black So the differences of 1.29 and 1.33 should be fairly negligible. What happened between all those release was the introduction of a binary wrapper (to replace the shell wrapper) for shell scripts to make them more "durable". However, there were varied compatibility issues (mostly with docker and non-x86 builds) that we made the original shell wrapper the default behavior again. Your logging shows that the original shell wrapper is indeed running. The biggest difference I can see between yours and the OP is that your script appears to be executing (writing output to the jenkins-log.txt) and terminating with 0 as opposed to failure.

            Like I mentioned with OP, if you can run with LAUNCH_DIAGNOSTICS, that might help us get a better idea of what is going on.

            Show
            carroll Carroll Chiou added a comment - Tim Black So the differences of 1.29 and 1.33 should be fairly negligible. What happened between all those release was the introduction of a binary wrapper (to replace the shell wrapper) for shell scripts to make them more "durable". However, there were varied compatibility issues (mostly with docker and non-x86 builds) that we made the original shell wrapper the default behavior again. Your logging shows that the original shell wrapper is indeed running. The biggest difference I can see between yours and the OP is that your script appears to be executing (writing output to the jenkins-log.txt) and terminating with 0 as opposed to failure. Like I mentioned with OP, if you can run with LAUNCH_DIAGNOSTICS, that might help us get a better idea of what is going on.
            Hide
            timblaktu Tim Black added a comment -

            For me, this issue turned out to be an empty environment variable set in Jenkins global properties. This

            Show
            timblaktu Tim Black added a comment - For me, this issue turned out to be an empty environment variable set in Jenkins global properties. This . 
            Hide
            carroll Carroll Chiou added a comment -

            When you run with LAUNCH_DIAGNOSTICS, the following line appears:

            invalid argument "=" for "-e, --env" flag: invalid environment variable: =
            

            What's happening is that empty environment variables cause the docker-workflow plugin to create a malformed docker run command. This issue is being tracked here: https://issues.jenkins-ci.org/browse/JENKINS-52264
            The container does not run, so the threads that are monitoring the durable task script time out.
            Will close this ticket in favor of the above mentioned ticket.

            Show
            carroll Carroll Chiou added a comment - When you run with LAUNCH_DIAGNOSTICS, the following line appears: invalid argument "=" for "-e, --env" flag: invalid environment variable: = What's happening is that empty environment variables cause the docker-workflow plugin to create a malformed docker run command. This issue is being tracked here: https://issues.jenkins-ci.org/browse/JENKINS-52264 The container does not run, so the threads that are monitoring the durable task script time out. Will close this ticket in favor of the above mentioned ticket.
            Hide
            carroll Carroll Chiou added a comment -

            Issue is actually a problem in docker-workflow. The durable-task plugin error message is a symptom of the docker container failing to run.

            Show
            carroll Carroll Chiou added a comment - Issue is actually a problem in docker-workflow. The durable-task plugin error message is a symptom of the docker container failing to run.

              People

              Assignee:
              carroll Carroll Chiou
              Reporter:
              srinidhisivakumar srinidhi Sivakumar
              Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: