Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34289

docker.image.inside fails unexpectedly with Jenkinsfile

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Component/s: docker-workflow-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.651.1 
      CloudBees Docker Pipeline 1.4 
      Pipeline 2.0
      Docker 1.11.0
      RHEL 7.2
    • Similar Issues:

      Description

      With a simple Jenkinsfile when building, at some point it'll fail for no obvious reason.

      An example Jenkinsfile:

      def img = 'centos:7';
      
      node('docker') {
        stage "pulling";
        sh "docker pull ${img}"; // workaround for JENKINS-34288
      
        checkout scm;
      
        docker.image(img).inside {
          sh 'for i in $(seq 30); do sleep 1; echo $i; done';
          sh 'ls -alh --color';
        }
      }
      

      Partial output:

      [Pipeline] Run build steps inside a Docker container : Start
      $ docker run -t -d -u 995:993 -w /var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master -v /var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master:/var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master:rw -v /var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master@tmp:/var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** centos:7 cat
      [Pipeline] withDockerContainer {
      [Pipeline] sh
      [master] Running shell script
      ++ seq 30
      + for i in '$(seq 30)'
      + sleep 1
      [Pipeline] } //withDockerContainer
      $ docker stop 7fcbfd6ab39cf05257a43a774bd20b670bc39674a2047777fe603ee1a3162b10
      $ docker rm -f 7fcbfd6ab39cf05257a43a774bd20b670bc39674a2047777fe603ee1a3162b10
      [Pipeline] Run build steps inside a Docker container : End
      [Pipeline] } //node
      [Pipeline] Allocate node : End
      [Pipeline] End of Pipeline
      

        Attachments

          Issue Links

            Activity

            Hide
            docwhat Christian Höltje added a comment -

            Agreed. But it was definitely something in the change between 1.72.1 and 1.75. I should have closed it myself. Sorry.

            Show
            docwhat Christian Höltje added a comment - Agreed. But it was definitely something in the change between 1.72.1 and 1.75. I should have closed it myself. Sorry.
            Hide
            jglick Jesse Glick added a comment -

            No fix was made to address this problem, you just stopped running into it for some reason TBD.

            Show
            jglick Jesse Glick added a comment - No fix was made to address this problem, you just stopped running into it for some reason TBD.
            Hide
            docwhat Christian Höltje added a comment -

            I just upgraded github-organization-folder from version 1.2 to 1.3 and lo-and-behold! It works!

            W00T!

            I also upgraded github-api from 1.72.1 to 1.75 as well, but I doubt it impacted this.

            You guys are the greatest!

            Show
            docwhat Christian Höltje added a comment - I just upgraded github-organization-folder from version 1.2 to 1.3 and lo-and-behold! It works! W00T! I also upgraded github-api from 1.72.1 to 1.75 as well, but I doubt it impacted this. You guys are the greatest!
            Hide
            docwhat Christian Höltje added a comment -

            So I got some slaves up-and-running and the problem persists there. The slave.jar runs as the user "jenkins", not root. I suspect that if I ran it as root, it'd work like the master above.

            Show
            docwhat Christian Höltje added a comment - So I got some slaves up-and-running and the problem persists there. The slave.jar runs as the user "jenkins", not root. I suspect that if I ran it as root, it'd work like the master above.
            Hide
            docwhat Christian Höltje added a comment -

            If it helps, I was able to use auditd to see that execve() was called for the docker exec for the command in question and execve() returned 0 (which is not the exit code, but just says the syscall worked). So the command seems to be run by Jenkins. It just isn't getting the exit code back correctly (and gets -1 instead).

            Looking around, could this be related to JENKINS-25727 ? If I'm reading the code correctly, this is falling afoul of this code: https://github.com/jenkinsci/durable-task-plugin/blob/66d80d2b9761ebdb4f0d3bb7b9edb82357e33399/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L172-L174

            Show
            docwhat Christian Höltje added a comment - If it helps, I was able to use auditd to see that execve() was called for the docker exec for the command in question and execve() returned 0 (which is not the exit code, but just says the syscall worked). So the command seems to be run by Jenkins. It just isn't getting the exit code back correctly (and gets -1 instead). Looking around, could this be related to JENKINS-25727 ? If I'm reading the code correctly, this is falling afoul of this code: https://github.com/jenkinsci/durable-task-plugin/blob/66d80d2b9761ebdb4f0d3bb7b9edb82357e33399/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L172-L174

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              docwhat Christian Höltje
              Votes:
              3 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: