Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66935

Process Apparently never started durabletask v1.36

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • durable-task-plugin
    • None
    • Jenkins 2.277.4, with kubernetes 1.29.5 (AWS EKS, docker-based)

      Every now and then, we see pipelines failing due to the "process apparently never started..."-error. See also the snippet below. We have listened to the proposed solution, namely making use of the LAUNCH_DIAGNOSTICS flag, but that does not give any other leads. It actually printed nothing additional compared to what was already printed: "process apparently never started...".

      Next to that, we see this problem occuring multiple times a day. In the morning when load is generally relatively low, in the afternoon when there the load is skyhigh and everything in between. Therefore, this does not seem to be a usage-related issue.

      I have seen that there have been some changes in the past with the durable-task plugin, such as the renaming of several parameters (FORCE_BINARY_WRAPPER -> USE_BINARY_WRAPPER). Maybe we could make use of that? Could this problem be related to the configuration of Jenkins and plugins we are having? Or is it a purely technical issue?

       

      Thanks a lot!

       

       
      [2021-10-19T11:12:11.179Z] process apparently never started in /home/jenkins/agent/workspace/<some-branch-name>@tmp/durable-fbd89733
      https://jenkins2.verz.local/blue/organizations/jenkins/SDPJ%2Fsdp-jenkins2-image/detail/feature%2FSDP-4517_iac_health_plugin/3/pipeline/#step-55-log-2[2021-10-19T11:12:11.181Z] (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
      https://jenkins2.verz.local/blue/organizations/jenkins/SDPJ%2Fsdp-jenkins2-image/detail/feature%2FSDP-4517_iac_health_plugin/3/pipeline/#step-55-log-3script returned exit code -2

          [JENKINS-66935] Process Apparently never started durabletask v1.36

          Ron Matsliah added a comment - - edited

          Happening at my company also, Jenkins 2.346.1 on Amazon EKS with Durable task plugin 496 

          It's happening now and then without any pattern.

          This an example from the BourneShellScript log by timeframe:

           

          launching [nohup, sh, -c, { while [ -d '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b' -a \! -f '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt' ]; do touch '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-log.txt'; sleep 3; done } & jsc=durable-c710bd7a4abc79dec87ea1b9786199255bcad771e16f8e8d55f7765384605423; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe  '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/script.sh' > '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/output.txt' 2> '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-log.txt'; echo $$? > '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt.tmp'; mv '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt'; wait]
          starting check in /home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b
          found exit code -2 in /home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b
          found exit code 0 in /home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b
          

           

          Ron Matsliah added a comment - - edited Happening at my company also, Jenkins 2.346.1 on Amazon EKS with Durable task plugin 496  It's happening now and then without any pattern. This an example from the BourneShellScript log by timeframe:   launching [nohup, sh, -c, { while [ -d '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b' -a \! -f '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt' ]; do touch '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-log.txt' ; sleep 3; done } & jsc=durable-c710bd7a4abc79dec87ea1b9786199255bcad771e16f8e8d55f7765384605423; JENKINS_SERVER_COOKIE=$$jsc 'sh' -xe   '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/script.sh' > '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/output.txt' 2> '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-log.txt' ; echo $$? > '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt.tmp' ; mv '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt.tmp' '/home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b/jenkins-result.txt' ; wait] starting check in /home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b found exit code -2 in /home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b found exit code 0 in /home/jenkins/agent/workspace/service_name@tmp/durable-00ca5f5b  

            Unassigned Unassigned
            tomkennes Tom Kennes
            Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: