Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55577

docker agent using a custom Dockerfile is broken with docker buildkit

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • docker-workflow-plugin
    • ubuntu 16.04
      docker 18.09.0
      jenkins 2.150.1
      docker-workflow-plugin 1.17

    Description

      When enabling buildkit in docker (either via `/etc/docker/daemon.json` or a global env var in jenkins: DOCKER_BUILDKIT=1), the job fails just after building the docker image used for the agent:

       https://jenkins.io/doc/book/pipeline/docker/#dockerfile

      [Pipeline] // stage
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Agent Setup)
      [Pipeline] isUnix
      [Pipeline] readFile
      [Pipeline] sh
      [workspace] Running shell script
      + docker build -t xxxx -f Dockerfile .
      
      ... buildkit logs ...
      
      [Pipeline] dockerFingerprintFrom
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] End of Pipeline
      java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'
      	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.inspectRequiredField(DockerClient.java:220)
      	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:133)
      	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:85)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at hudson.security.ACL.impersonate(ACL.java:290)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

      Analysis:

      jenkins tries to get information about the base image used in the Dockerfile: `dockerFingerprintFrom`. It does so by assuming the base image is available in `docker images`, as it was just used to build the agent image.

      (This has a small timing issue if a system-wide `docker image prune` happens between the build and the call to `dockerFingerprintFrom`.)

      This worked great with the old docker builder, which effectively did a `docker pull` of the base image, but buildkit does not behave like that: it keeps things completely internally.

      $ cat Dockerfile
      FROM ubuntu:18.04
      $ DOCKER_BUILDKIT=1 docker build .
      [+] Building 1.7s (5/5) FINISHED                                                                                                                                                                                                             
       => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
       => => transferring context: 2B                                                                                                                                                                                                         0.0s
       => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
       => => transferring dockerfile: 37B                                                                                                                                                                                                     0.0s
       => [internal] load metadata for docker.io/library/ubuntu:18.04                                                                                                                                                                         1.6s
       => [1/1] FROM docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
       => => resolve docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
       => exporting to image                                                                                                                                                                                                                  0.0s
       => => exporting layers                                                                                                                                                                                                                 0.0s
       => => writing image sha256:645e081eb3fdb8c828216921411fc0c08335e4fd098c151b44aba797fa334839                                                                                                                                            0.0s
      $ docker inspect ubuntu:18.04
      []
      Error: No such object: ubuntu:18.04
      

      Attachments

        Issue Links

          Activity

            fabiang Fabian Grutschus added a comment - - edited

            That true. So the only option would be to get the sha256 from the output and pull the image by that digest?

            fabiang Fabian Grutschus added a comment - - edited That true. So the only option would be to get the sha256 from the output and pull the image by that digest?

            Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...).
            I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...)

            Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?

            thomas_deepomatic Thomas Riccardi added a comment - Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...). I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...) Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?

            buildctl has the option --export-cache=mode=max, which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.

            fabiang Fabian Grutschus added a comment - buildctl has the option --export-cache=mode=max , which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.

            anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.

            jugglefish Peter Niederlag added a comment - anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.
            viceice Michael Kriese added a comment - - edited

            This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released)

            https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350

            viceice Michael Kriese added a comment - - edited This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released) https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350

            People

              Unassigned Unassigned
              thomas_deepomatic Thomas Riccardi
              Votes:
              8 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: