Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49385

containers exit early in docker-workflow 1.15

XMLWordPrintable

      After updating to docker-workflow 1.15 all of my builds that folllow this pattern started exiting with status -1 or -2:

      stage ('Unit Tests') {
try {
   dir ("./somewhere/somepath") {
      sh "./somewhere/some.sh"
   }
   def userid = sh(script: 'grep Uid /proc/self/status | cut -f2', returnStdout: true).trim()
   image.inside("-v ${env.WORKSPACE}:/target -e USER=${env.USER} -e USERID=${userid} -e BRANCH_NAME=${BRANCH_NAME} -u ${userid} --link ...") { c ->
      sh "./somewhere/someother.sh"
      junit 'nosetests.xml'
      step([$class: 'CoberturaPublisher',
        /* ... */ ])
      sh "git clean -fdx"
    }
  } finally {
    sh "./somewhere/somefinal.sh || true"
  }
}

       

      All of the scripts here would run, but somewhere/someother.sh on some jobs would exit early and other jobs (different projects, same format) would complete successfully but still return a -1, such as:

      ERROR: script returned exit code -2

       

      Reverting to docker-workflow 1.14 alleviated the problems.

       

      I started with docker-engine 17.06 running on Debian Stretch and while diagnosing these problems upgraded to docker-ce (17.12.0-ce, build c97c6d6).  Upgrading docker made no difference.

       

          [JENKINS-49385] containers exit early in docker-workflow 1.15

          Scott Williams added a comment -

          I suspect that the issue is with "cat" being added to the end.  For example, if the entrypoint is something like:

          ENTRYPOINT ["command", "--arg1", "--arg2"]

          ...then I suspect that the docker run would then be parsed as "command --arg1 --arg2 cat", based on https://docs.docker.com/engine/reference/builder/#entrypoint . 

          The workaround would probably be overriding the --entrypoint for the "cat" check.

          Scott Williams added a comment - I suspect that the issue is with "cat" being added to the end.  For example, if the entrypoint is something like: ENTRYPOINT ["command", "--arg1", "--arg2"] ...then I suspect that the docker run would then be parsed as "command --arg1 --arg2 cat", based on https://docs.docker.com/engine/reference/builder/#entrypoint .  The workaround would probably be overriding the --entrypoint for the "cat" check.

          jamie norman added a comment - - edited

          This is completely breaking our build process, we rely on Docker to execute tools in out pipeline on the slaves.

          I have updated the priority to critical, not sure if that is the right process, but we can't see any of our containers being successful with this defect in place.

           

           

          jamie norman added a comment - - edited This is completely breaking our build process, we rely on Docker to execute tools in out pipeline on the slaves. I have updated the priority to critical, not sure if that is the right process, but we can't see any of our containers being successful with this defect in place.    

          Wojciech Duda added a comment -

          I encountered the same issue and had to revert Docker Workflow Plugin. A fix would be highly appreciated.

          Wojciech Duda added a comment - I encountered the same issue and had to revert Docker Workflow Plugin. A fix would be highly appreciated.

          Christoph Forster added a comment - - edited

          Why is this issue set to "resolved"? I don't see a 1.16 Plugin Version. Did the problem occur because of another Plugin or directly in Jenkins?

          Christoph Forster added a comment - - edited Why is this issue set to "resolved"? I don't see a 1.16 Plugin Version. Did the problem occur because of another Plugin or directly in Jenkins?

          Wojciech Duda added a comment -

          forster_ch I just saw that the bugfix was done as part of JENKINS-41278 and is released, version number is 1.15.1. Trying the update now.

          Wojciech Duda added a comment - forster_ch I just saw that the bugfix was done as part of JENKINS-41278 and is released, version number is 1.15.1. Trying the update now.

          Christoph Forster added a comment -

          Thanks for the Info. I'll try it.

          Christoph Forster added a comment - Thanks for the Info. I'll try it.

          Scott Williams added a comment -

          Sadly, 15.1 doesn't fix this issue at all.  The problem is that neither will work with an argument delimited ENTRYPOINT as the ENTRYPOINT gets placed at the beginning and the command is tacked on to the end. 

          $ docker run -t -d -u 0:0 -w /opt/jenkins/workspace/Zendesk-bulk -v /opt/jenkins/workspace/Zendesk-bulk:/opt/jenkins/workspace/Zendesk-bulk:rw,z -v /opt/jenkins/workspace/Zendesk-bulk@tmp:/opt/jenkins/workspace/Zendesk-bulk@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** a3af309c86bf5b2ad033f313de7dc875df1f71f1 cat
          $ docker top 1704cb59764ae62c736ffa4d78e1220bb0be3fb02931b54a4014a90f9ab0b920 -eo pid,comm

          This inevitably fails with a properly arg delimited ENTRYPOINT...
          java.io.IOException: Failed to run top '1704cb59764ae62c736ffa4d78e1220bb0be3fb02931b54a4014a90f9ab0b920'.
          Error response from daemon: Container 1704cb59764ae62c736ffa4d78e1220bb0be3fb02931b54a4014a90f9ab0b920 is not running
           

          Scott Williams added a comment - Sadly, 15.1 doesn't fix this issue at all.  The problem is that neither will work with an argument delimited ENTRYPOINT as the ENTRYPOINT gets placed at the beginning and the command is tacked on to the end.  $ docker run -t -d -u 0:0 -w /opt/jenkins/workspace/Zendesk-bulk -v /opt/jenkins/workspace/Zendesk-bulk:/opt/jenkins/workspace/Zendesk-bulk:rw,z -v /opt/jenkins/workspace/Zendesk-bulk@tmp:/opt/jenkins/workspace/Zendesk-bulk@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** a3af309c86bf5b2ad033f313de7dc875df1f71f1 cat $ docker top 1704cb59764ae62c736ffa4d78e1220bb0be3fb02931b54a4014a90f9ab0b920 -eo pid,comm This inevitably fails with a properly arg delimited ENTRYPOINT... java.io.IOException: Failed to run top '1704cb59764ae62c736ffa4d78e1220bb0be3fb02931b54a4014a90f9ab0b920'. Error response from daemon: Container 1704cb59764ae62c736ffa4d78e1220bb0be3fb02931b54a4014a90f9ab0b920 is not running  

          Scott Williams added a comment -

          Problem persists with 15.1 update.

          Scott Williams added a comment - Problem persists with 15.1 update.

          Carlos Cabrera added a comment -

          adding the following as per error log suggestion fixes for me on 15.1 

          args  "--entrypoint='' "

          Carlos Cabrera added a comment - adding the following as per error log suggestion fixes for me on 15.1  args  "--entrypoint='' "

          Nicolas De Loof added a comment -

          The "fix" is indeed about better reporting inability to run container with expected command, and describing in log how to override entrypoint with an empty command to just disable it, and let the configured command execute.

          Nicolas De Loof added a comment - The "fix" is indeed about better reporting inability to run container with expected command, and describing in log how to override entrypoint with an empty command to just disable it, and let the configured command execute.

            ndeloof Nicolas De Loof
            displague Marques Johansson
            Votes:
            6 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: