Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55308

intermittent "terminated" messages using sh in Pipelines

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • durable-task-plugin, workflow-job-plugin
    • None

      Testing Jenkins 2.138.2 LTS, Jenkins pipelines that use sh intermittently throw the following message in the console log …

       

      sh: line 1:  4449 Terminated              sleep 3

       

       … and sometimes this … 

       

      sh: line 1: 13136 Terminated              { while [ ( -d /proc/$pid -o ! -d /proc/$$ ) -a -d '/home/ec2-user/workspace/admin-smoke-test@tmp/durable-523481b0' -a ! -f '/home/ec2-user/workspace/admin-smoke-test@tmp/durable-523481b0/jenkins-result.txt' ]; do    touch '/home/ec2-user/workspace/admin-smoke-test@tmp/durable-523481b0/jenkins-log.txt'; sleep 3;{{done; }}}

       

      Jenkins master runs from a Docker image based on jenkins/jenkins:2.138.2-alpine with specific plugins baked into the image by /usr/local/bin/install-plugins.sh

      The message originates in durable-task-plugin, which must be a dependency of one of the plugins.txt plugins.  

      Two important observations:

      1) The issue does not occur when starting with the base jenkins/jenkins:2.138.2-alpine image and manually installing plugins via UI. That might suggest the issue is around how install-plugins.sh installs plugins and/or dependencies. 

      2) The issue does not occur on our production image, which is also 2.138.2-alpine + plugins built 2018-10-11. Rebuilding the the same image from the same Dockerfile results in different installed plugins. Makes me think results using install-plugins.sh are not deterministic.

          [JENKINS-55308] intermittent "terminated" messages using sh in Pipelines

          gc875 created issue -

          Jonas Lindström added a comment - - edited

          We run the jenkins-alpine docker image and we also get these messages. I tried switching to the Debian image (2.150.2) and the intermittent "terminated" message are still there, so I don't think it's specifically related to the OS.

          This seems to have been introduced after v1.22 of durable-task-plugin. I guess the error is lurking somewhere in this line in BourneShellScript.java:

                      cmd = String.format("pid=$$; { while [ \\( -d /proc/$pid -o \\! -d /proc/$$ \\) -a -d '%s' -a \\! -f '%s' ]; do touch '%s'; sleep 3; done } & jsc=%s; %s=$jsc %s '%s' > '%s' 2> '%s'; echo $? > '%s.tmp'; mv '%s.tmp' '%s'; wait", 
... more args ...

          It's difficult to downgrade to 1.22 since so many other plugins depend on later versions of this plugin.

          Jonas Lindström added a comment - - edited We run the jenkins-alpine docker image and we also get these messages. I tried switching to the Debian image (2.150.2) and the intermittent "terminated" message are still there, so I don't think it's specifically related to the OS. This seems to have been introduced after v1.22 of durable-task-plugin. I guess the error is lurking somewhere in this line in BourneShellScript.java: cmd = String .format( "pid=$$; { while [ \\( -d /proc/$pid -o \\! -d /proc/$$ \\) -a -d '%s' -a \\! -f '%s' ]; do touch '%s' ; sleep 3; done } & jsc=%s; %s=$jsc %s '%s' > '%s' 2> '%s' ; echo $? > '%s.tmp' ; mv '%s.tmp' '%s' ; wait" , ... more args ... It's difficult to downgrade to 1.22 since so many other plugins depend on later versions of this plugin.
          Jean-Paul G made changes -
          Component/s Original: docker [ 20834 ]

          Jean-Paul G added a comment - - edited

          Not specific to docker.

          Have the very same issue when running sh step with parallel, on ssh slave with Jenkins 2.156

          Jean-Paul G added a comment - - edited Not specific to docker. Have the very same issue when running sh step with parallel, on ssh slave with Jenkins 2.156

          Adam vonNieda added a comment -

          I've got the same issue when running sh step without parallel, on ssh slave with Jenkins 2.150.2 LTS, I'm not running on docker.

          Adam vonNieda added a comment - I've got the same issue when running sh step without parallel, on ssh slave with Jenkins 2.150.2 LTS, I'm not running on docker.

          Connor Yu added a comment -

          I am seeing this as well. Upgraded to 2.46.3 yesterday and started seeing this in my simple pipeline.  

          sh: line 1: 27323 Terminated              sleep 3

           

          Connor Yu added a comment - I am seeing this as well. Upgraded to 2.46.3 yesterday and started seeing this in my simple pipeline.  sh: line 1: 27323 Terminated sleep 3  

          Karl Parry added a comment -

          I am seeing this as well. We have just seen it randomly. We do get it when we run a custom clean up script that deletes the @tmp/@script/@script@tmp folders 

          sh: line 1: 27323 Terminated              sleep 3

          Karl Parry added a comment - I am seeing this as well. We have just seen it randomly. We do get it when we run a custom clean up script that deletes the @tmp/@script/@script@tmp folders  sh: line 1: 27323 Terminated sleep 3

          Matthew Hall added a comment -

          Also seeing this. Jenkins 2.150.3 and durable-task 1.29, which was an upgrade from 1.25.

          Matthew Hall added a comment - Also seeing this. Jenkins 2.150.3 and durable-task 1.29, which was an upgrade from 1.25.

          Mike Kozell added a comment -

          I upgraded Pipeline: Job (workflow-job) plugin from 2.25 to 2.31 and now see the issue.  I'm running durable-task 1.27.

          Mike Kozell added a comment - I upgraded Pipeline: Job (workflow-job) plugin from 2.25 to 2.31 and now see the issue.  I'm running durable-task 1.27.
          Devin Nusbaum made changes -
          Assignee Original: Carlos Sanchez [ csanchez ] New: Devin Nusbaum [ dnusbaum ]

            carroll Carroll Chiou
            gc875 gc875
            Votes:
            24 Vote for this issue
            Watchers:
            44 Start watching this issue

              Created:
              Updated: