Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55577

docker agent using a custom Dockerfile is broken with docker buildkit

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • docker-workflow-plugin
    • ubuntu 16.04
      docker 18.09.0
      jenkins 2.150.1
      docker-workflow-plugin 1.17

      When enabling buildkit in docker (either via `/etc/docker/daemon.json` or a global env var in jenkins: DOCKER_BUILDKIT=1), the job fails just after building the docker image used for the agent:

       https://jenkins.io/doc/book/pipeline/docker/#dockerfile

      [Pipeline] // stage
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Agent Setup)
      [Pipeline] isUnix
      [Pipeline] readFile
      [Pipeline] sh
      [workspace] Running shell script
      + docker build -t xxxx -f Dockerfile .
      
      ... buildkit logs ...
      
      [Pipeline] dockerFingerprintFrom
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] End of Pipeline
      java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'
      	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.inspectRequiredField(DockerClient.java:220)
      	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:133)
      	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:85)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at hudson.security.ACL.impersonate(ACL.java:290)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

      Analysis:

      jenkins tries to get information about the base image used in the Dockerfile: `dockerFingerprintFrom`. It does so by assuming the base image is available in `docker images`, as it was just used to build the agent image.

      (This has a small timing issue if a system-wide `docker image prune` happens between the build and the call to `dockerFingerprintFrom`.)

      This worked great with the old docker builder, which effectively did a `docker pull` of the base image, but buildkit does not behave like that: it keeps things completely internally.

      $ cat Dockerfile
      FROM ubuntu:18.04
      $ DOCKER_BUILDKIT=1 docker build .
      [+] Building 1.7s (5/5) FINISHED                                                                                                                                                                                                             
       => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
       => => transferring context: 2B                                                                                                                                                                                                         0.0s
       => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
       => => transferring dockerfile: 37B                                                                                                                                                                                                     0.0s
       => [internal] load metadata for docker.io/library/ubuntu:18.04                                                                                                                                                                         1.6s
       => [1/1] FROM docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
       => => resolve docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
       => exporting to image                                                                                                                                                                                                                  0.0s
       => => exporting layers                                                                                                                                                                                                                 0.0s
       => => writing image sha256:645e081eb3fdb8c828216921411fc0c08335e4fd098c151b44aba797fa334839                                                                                                                                            0.0s
      $ docker inspect ubuntu:18.04
      []
      Error: No such object: ubuntu:18.04
      

          [JENKINS-55577] docker agent using a custom Dockerfile is broken with docker buildkit

          Faced the same issue.

          • jenkins version: 2.138.1
          • docker engine version: 18.09.2, API version: 1.39, Git commit: 6247962
          • docker-workflow-plugin version: 1.17

          Following step fails

          docker.withRegistry(registry) {
             docker.build(image_repo + ":$tag", '-f docker/Dockerfile .').push()
             docker.image(image_repo + ":$tag").push(shortTag)
          }
          

          Konstantin Shcherban added a comment - Faced the same issue. jenkins version: 2.138.1 docker engine version: 18.09.2, API version: 1.39, Git commit: 6247962 docker-workflow-plugin version: 1.17 Following step fails docker.withRegistry(registry) { docker.build(image_repo + ":$tag" , '-f docker/Dockerfile .' ).push() docker.image(image_repo + ":$tag" ).push(shortTag) }

          As a workaround, I simply disabled the whole function that tried to collect the base image id: https://github.com/Deepomatic/docker-workflow-plugin/commit/826b06d2188b01f9a91be5130bb9eec30831c731

          It's OK for me as I don't use any of these jenkins collected metadata; it should not be merged.

          I pushed the .hpi file I built there: https://github.com/Deepomatic/docker-workflow-plugin/releases/tag/deepomatic-docker-workflow-1.18-SNAPSHOT
          Built using these commands:

          docker run -it --rm -v $PWD:/app -v $PWD/.m2:/root/.m2 maven:3-jdk-8 bash
          cd /app
          mvn -DskipTests clean install
          # result at
          ls .m2/repository/org/jenkins-ci/plugins/docker-workflow/1.18-SNAPSHOT/docker-workflow-1.18-SNAPSHOT.hpi
          

          No issue encountered since we deployed that version more than a month ago.

          Thomas Riccardi added a comment - As a workaround, I simply disabled the whole function that tried to collect the base image id: https://github.com/Deepomatic/docker-workflow-plugin/commit/826b06d2188b01f9a91be5130bb9eec30831c731 It's OK for me as I don't use any of these jenkins collected metadata; it should not be merged. I pushed the .hpi file I built there: https://github.com/Deepomatic/docker-workflow-plugin/releases/tag/deepomatic-docker-workflow-1.18-SNAPSHOT Built using these commands: docker run -it --rm -v $PWD:/app -v $PWD/.m2:/root/.m2 maven:3-jdk-8 bash cd /app mvn -DskipTests clean install # result at ls .m2/repository/org/jenkins-ci/plugins/docker-workflow/1.18-SNAPSHOT/docker-workflow-1.18-SNAPSHOT.hpi No issue encountered since we deployed that version more than a month ago.

          Valentin Delaye added a comment - - edited

          Same issue for us Jenkins 2.164.1 and docker-workflow-plugin version: 1.17 and docker 18.09.3 with buildkit

           

          [Pipeline] End of Pipeline java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'
          
          

          Valentin Delaye added a comment - - edited Same issue for us Jenkins 2.164.1 and docker-workflow-plugin version: 1.17 and docker 18.09.3 with buildkit   [Pipeline] End of Pipeline java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'

          Instead of removing the functionality we just need to pull the last used image explicit before running docker build? Correct?

          Fabian Grutschus added a comment - Instead of removing the functionality we just need to pull the last used image explicit before running docker build ? Correct?

          jonesbusy
          This would be vulnerable to race-conditions: what if the image changed between the two actions ?

          Thomas Riccardi added a comment - jonesbusy This would be vulnerable to race-conditions: what if the image changed between the two actions ?

          Fabian Grutschus added a comment - - edited

          That true. So the only option would be to get the sha256 from the output and pull the image by that digest?

          Fabian Grutschus added a comment - - edited That true. So the only option would be to get the sha256 from the output and pull the image by that digest?

          Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...).
          I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...)

          Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?

          Thomas Riccardi added a comment - Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...). I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...) Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?

          buildctl has the option --export-cache=mode=max, which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.

          Fabian Grutschus added a comment - buildctl has the option --export-cache=mode=max , which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.

          anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.

          Peter Niederlag added a comment - anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.

          Michael Kriese added a comment - - edited

          This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released)

          https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350

          Michael Kriese added a comment - - edited This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released) https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350

            Unassigned Unassigned
            thomas_deepomatic Thomas Riccardi
            Votes:
            8 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: