Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34289

docker.image.inside fails unexpectedly with Jenkinsfile

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Cannot Reproduce
    • docker-workflow-plugin
    • None
    • Jenkins 1.651.1 
      CloudBees Docker Pipeline 1.4 
      Pipeline 2.0
      Docker 1.11.0
      RHEL 7.2

    Description

      With a simple Jenkinsfile when building, at some point it'll fail for no obvious reason.

      An example Jenkinsfile:

      def img = 'centos:7';
      
      node('docker') {
        stage "pulling";
        sh "docker pull ${img}"; // workaround for JENKINS-34288
      
        checkout scm;
      
        docker.image(img).inside {
          sh 'for i in $(seq 30); do sleep 1; echo $i; done';
          sh 'ls -alh --color';
        }
      }
      

      Partial output:

      [Pipeline] Run build steps inside a Docker container : Start
      $ docker run -t -d -u 995:993 -w /var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master -v /var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master:/var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master:rw -v /var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master@tmp:/var/lib/jenkins/workspace/tron/docwhat-test-jenkinsfile/master@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** centos:7 cat
      [Pipeline] withDockerContainer {
      [Pipeline] sh
      [master] Running shell script
      ++ seq 30
      + for i in '$(seq 30)'
      + sleep 1
      [Pipeline] } //withDockerContainer
      $ docker stop 7fcbfd6ab39cf05257a43a774bd20b670bc39674a2047777fe603ee1a3162b10
      $ docker rm -f 7fcbfd6ab39cf05257a43a774bd20b670bc39674a2047777fe603ee1a3162b10
      [Pipeline] Run build steps inside a Docker container : End
      [Pipeline] } //node
      [Pipeline] Allocate node : End
      [Pipeline] End of Pipeline
      

      Attachments

        Issue Links

          Activity

            If it helps, I was able to use auditd to see that execve() was called for the docker exec for the command in question and execve() returned 0 (which is not the exit code, but just says the syscall worked). So the command seems to be run by Jenkins. It just isn't getting the exit code back correctly (and gets -1 instead).

            Looking around, could this be related to JENKINS-25727 ? If I'm reading the code correctly, this is falling afoul of this code: https://github.com/jenkinsci/durable-task-plugin/blob/66d80d2b9761ebdb4f0d3bb7b9edb82357e33399/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L172-L174

            docwhat Christian Höltje added a comment - If it helps, I was able to use auditd to see that execve() was called for the docker exec for the command in question and execve() returned 0 (which is not the exit code, but just says the syscall worked). So the command seems to be run by Jenkins. It just isn't getting the exit code back correctly (and gets -1 instead). Looking around, could this be related to JENKINS-25727 ? If I'm reading the code correctly, this is falling afoul of this code: https://github.com/jenkinsci/durable-task-plugin/blob/66d80d2b9761ebdb4f0d3bb7b9edb82357e33399/src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java#L172-L174

            So I got some slaves up-and-running and the problem persists there. The slave.jar runs as the user "jenkins", not root. I suspect that if I ran it as root, it'd work like the master above.

            docwhat Christian Höltje added a comment - So I got some slaves up-and-running and the problem persists there. The slave.jar runs as the user "jenkins", not root. I suspect that if I ran it as root, it'd work like the master above.

            I just upgraded github-organization-folder from version 1.2 to 1.3 and lo-and-behold! It works!

            W00T!

            I also upgraded github-api from 1.72.1 to 1.75 as well, but I doubt it impacted this.

            You guys are the greatest!

            docwhat Christian Höltje added a comment - I just upgraded github-organization-folder from version 1.2 to 1.3 and lo-and-behold! It works! W00T! I also upgraded github-api from 1.72.1 to 1.75 as well, but I doubt it impacted this. You guys are the greatest!
            jglick Jesse Glick added a comment -

            No fix was made to address this problem, you just stopped running into it for some reason TBD.

            jglick Jesse Glick added a comment - No fix was made to address this problem, you just stopped running into it for some reason TBD.

            Agreed. But it was definitely something in the change between 1.72.1 and 1.75. I should have closed it myself. Sorry.

            docwhat Christian Höltje added a comment - Agreed. But it was definitely something in the change between 1.72.1 and 1.75. I should have closed it myself. Sorry.

            People

              jglick Jesse Glick
              docwhat Christian Höltje
              Votes:
              3 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: