Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48038

"Could not find the history docker layer" when publishing build Info to artifactory

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • artifactory-plugin
    • Amazon Linux 4.9.58-18.55.amzn1.x86_64, Docker version 17.06.2-ce, Jenkins 2.89, Artifactory 5.5.2, Artifactory Plugin 2.13.1

      Description: I followed Jenkins Artifactory Plugin - Setting Up Docker Build Info and got

      Pushing image: mycompany-docker-local.jfrog.io/my-app:2dca83a
      Docker build-info captured on 'Jenkins' agent.
      Successfully pushed docker image: mycompany-docker-local.jfrog.io/my-app:2dca83a
      [Pipeline] publishBuildInfo
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      java.lang.IllegalStateException: Could not find the history docker layer: sha256:f88085956a84ef2f7a1e138ce82e3ac1a6d2c4c629c6d696643649810e202e7f for image: mycompany-docker-local.jfrog.io/my-app:2dca83a in Artifactory.

      The Image push is successful via the reverse proxy. I was able to verify that the reverse proxy is setup correctly on docker by stopping Jenkins and trying to push/pull images to/from the artifactory instance.

       

      My Jenkinsfile:

      node {
        def server = Artifactory.server 'mycompany.jfrog.io'
        def rtDocker = Artifactory.docker credentialsId: 'jfrog-deploy'
        
        def commit_id
        stage ('Preparation') {
          checkout scm
          sh "git rev-parse --short HEAD > .git/commit-id"
          commit_id = readFile('.git/commit-id').trim()
        }
        stage ('build docker image') {
          img = docker.build('my-app')
        }
        stage ('test') {
          docker.image('my-app').inside {
            sh 'cd /app'
            sh 'npm start &'
            sh 'npm test'
          }
        }
        stage('docker build/push') {
          docker.build("mycompany-docker-local.jfrog.io/my-app:${commit_id}")
          def buildInfo = rtDocker.push("mycompany-docker-local.jfrog.io/my-app:${commit_id}", 'my-app')
          server.publishBuildInfo buildInfo
        }
      }

          [JENKINS-48038] "Could not find the history docker layer" when publishing build Info to artifactory

          I have a similar issues when publishing info

          I also tried this 

          def buildInfo = Artifactory.newBuildInfo()
          rtDocker.push("${registry}/${debianversion}-${softwareversion}:latest", "${registry}", buildInfo)
          // Publish the build-info to Artifactory:
          server.publishBuildInfo buildInfo

          Here's the full stacktrace missing from this original ticket:

          [Pipeline] publishBuildInfo
          [Pipeline] }
          [Pipeline] // dir
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          java.lang.IllegalStateException: Could not find the history docker layer: sha256:4c09512a2cf008ee16230523fa91754c88b7af1361d53aa033969101b6a72661 for image: <registry>/stretch-python27:latest in Artifactory.
          	at org.jfrog.hudson.pipeline.docker.DockerImage.setDependenciesAndArtifacts(DockerImage.java:186)
          	at org.jfrog.hudson.pipeline.docker.DockerImage.generateBuildInfoModule(DockerImage.java:145)
          	at org.jfrog.hudson.pipeline.types.buildInfo.DockerBuildInfoHelper.generateBuildInfoModules(DockerBuildInfoHelper.java:37)
          	at org.jfrog.hudson.pipeline.types.buildInfo.BuildInfo.createDeployer(BuildInfo.java:200)
          	at org.jfrog.hudson.pipeline.types.buildInfo.BuildInfoAccessor.createDeployer(BuildInfoAccessor.java:86)
          	at org.jfrog.hudson.pipeline.steps.PublishBuildInfoStep$Execution.run(PublishBuildInfoStep.java:62)
          	at org.jfrog.hudson.pipeline.steps.PublishBuildInfoStep$Execution.run(PublishBuildInfoStep.java:39)
          	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:42)
          	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224)
          	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150)
          	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
          	at org.jfrog.hudson.pipeline.types.ArtifactoryServer.publishBuildInfo(ArtifactoryServer.java:135)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
          	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218)
          	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027)
          	at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47)
          	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
          	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
          	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
          	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
          	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
          	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153)
          	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157)
          	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
          	at WorkflowScript.run(WorkflowScript:38)
          	at ___cps.transform___(Native Method)
          	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
          	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
          	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
          	at sun.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          	at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
          	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
          	at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
          	at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
          	at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
          	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
          	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
          	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
          	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
          	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
          	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
          	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
          	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
          	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
          	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
          	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
          	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
          	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
          	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:748)

          (Don't mind <registry> )

          Florian Meskens added a comment - I have a similar issues when publishing info I also tried this  def buildInfo = Artifactory.newBuildInfo() rtDocker.push( "${registry}/${debianversion}-${softwareversion}:latest" , "${registry}" , buildInfo) // Publish the build-info to Artifactory: server.publishBuildInfo buildInfo Here's the full stacktrace missing from this original ticket: [Pipeline] publishBuildInfo [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline java.lang.IllegalStateException: Could not find the history docker layer: sha256:4c09512a2cf008ee16230523fa91754c88b7af1361d53aa033969101b6a72661 for image: <registry>/stretch-python27:latest in Artifactory. at org.jfrog.hudson.pipeline.docker.DockerImage.setDependenciesAndArtifacts(DockerImage.java:186) at org.jfrog.hudson.pipeline.docker.DockerImage.generateBuildInfoModule(DockerImage.java:145) at org.jfrog.hudson.pipeline.types.buildInfo.DockerBuildInfoHelper.generateBuildInfoModules(DockerBuildInfoHelper.java:37) at org.jfrog.hudson.pipeline.types.buildInfo.BuildInfo.createDeployer(BuildInfo.java:200) at org.jfrog.hudson.pipeline.types.buildInfo.BuildInfoAccessor.createDeployer(BuildInfoAccessor.java:86) at org.jfrog.hudson.pipeline.steps.PublishBuildInfoStep$Execution.run(PublishBuildInfoStep.java:62) at org.jfrog.hudson.pipeline.steps.PublishBuildInfoStep$Execution.run(PublishBuildInfoStep.java:39) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:42) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:224) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:150) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108) at org.jfrog.hudson.pipeline.types.ArtifactoryServer.publishBuildInfo(ArtifactoryServer.java:135) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1218) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1027) at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:153) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:157) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at WorkflowScript.run(WorkflowScript:38) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor286.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28) at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55) at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) (Don't mind <registry> )

          Hi Pradeep and Florian,

          Thank you for this information. We have seen this issue in the past and it was related to the Artifactory version. Upgrading to the latest Artifactory version solved the issue. Can you let us know if upgrading solved the issue for you? 

           

          Alexei Vainshtein added a comment - Hi Pradeep and Florian, Thank you for this information. We have seen this issue in the past and it was related to the Artifactory version. Upgrading to the latest Artifactory version solved the issue. Can you let us know if upgrading solved the issue for you?   

          We're running Artifactory 5.6.0 rev 50600900
          Is 5.6.2 fixing this?

          Florian Meskens added a comment - We're running Artifactory 5.6.0 rev 50600900 Is 5.6.2 fixing this?

          Hi Florian

           

          It should have worked on this version of Artifactory as well. Another possability that could occur that this layer is already presented in the _uploads folder within Artifactory. I will explain how the deployment of Docker images works: The layers first reaching Artifactory to a folder named _uploads that can be found in the following path: repositoryName/image/. After Artifactory finishes the processing of the information, those files are being automatically moved from to the tag folder. One of our assumptions that this layer is already located on your end within that folder. Can you please confirm that this _uploads folder exists for this image within Artifactory and share with us a screenshot of the artifacts located in that folder?  We want to check if just removing this _uploads folder and re-running the build could solve the issue.

          Alexei Vainshtein added a comment - Hi Florian   It should have worked on this version of Artifactory as well. Another possability that could occur that this layer is already presented in the _uploads folder within Artifactory. I will explain how the deployment of Docker images works: The layers first reaching Artifactory to a folder named  _uploads that can be found in the following path:  repositoryName/image/ . After Artifactory finishes the processing of the information, those files are being automatically moved from to the  tag folder. One of our assumptions that this layer is already located on your end within that folder. Can you please confirm that this _uploads folder exists for this image within Artifactory and share with us a screenshot of the artifacts located in that folder?  We want to check if just removing this _uploads folder and re-running the build could solve the issue.

          I did try removing the folder and re-running I still get the same error
          Fun part is that I see the sha that errors in the tree

          Florian Meskens added a comment - I did try removing the folder and re-running I still get the same error Fun part is that I see the sha that errors in the tree

          Alastair Houghton added a comment - - edited

          This problem still exists; we're using Artifactory 6.9.1, and I'm seeing similar behaviour. Interestingly it doesn't always fail in this particular manner; sometimes I'm told that Jenkins was unable to find the manifest.json file instead. I think the problem is actually a race condition between the Jenkins plugin and Artifactory itself; I think what's happening is that the Jenkins plugin uploads the Docker image to Artifactory, and then immediately tries to find the history layer in the image. If the Artifactory installation is fast enough, that's fine, but if the Jenkins plugin catches Artifactory just at the wrong moment (I assume while it's moving from _uploads to the final destination, or something of that nature), it can't find one or more of the files and so you get these error messages.

          Although this is listed as a "minor" problem, it's actually a pretty major bug in the Artifactory plug-in, because it makes doing

          def server = Artifactory.server 'myServer'
          def rtDocker = Artifactory.docker server:server
          
          def buildInfo = rtDocker.push("myImage:some-tag", "docker/myImage")
          
          server.publishBuildInfo buildInfo
          

          fail, either on the push step, or alternatively (if it fails to find the manifest, as opposed to the history layer) it will succeed but the build will show a single empty module with no artifacts, even though you can see that the Docker image has been uploaded successfully.

          (It might be worth adding that the images I'm using here are very large. That might well increase the window during which this can happen.)

          Alastair Houghton added a comment - - edited This problem still exists; we're using Artifactory 6.9.1, and I'm seeing similar behaviour. Interestingly it doesn't always  fail in this particular manner; sometimes I'm told that Jenkins was unable to find the manifest.json file instead. I think the problem is actually a race condition between the Jenkins plugin and Artifactory itself; I think what's happening is that the Jenkins plugin uploads the Docker image to Artifactory, and then immediately tries to find the history layer in the image. If the Artifactory installation is fast enough, that's fine, but if the Jenkins plugin catches Artifactory just at the wrong moment (I assume while it's moving from _uploads to the final destination, or something of that nature), it can't find one or more of the files and so you get these error messages. Although this is listed as a "minor" problem, it's actually a pretty major bug in the Artifactory plug-in, because it makes doing def server = Artifactory.server 'myServer' def rtDocker = Artifactory.docker server:server def buildInfo = rtDocker.push( "myImage:some-tag" , "docker/myImage" ) server.publishBuildInfo buildInfo fail, either on the push step, or alternatively (if it fails to find the manifest, as opposed to the history layer) it will succeed but the build will show a single empty module with no artifacts, even though you can see that the Docker image has been uploaded successfully. (It might be worth adding that the images I'm using here are very large. That might well increase the window during which this can happen.)

            eyalbe Eyal Ben Moshe
            pradeepvrd Pradeep Varadharajan
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: