-
Bug
-
Resolution: Unresolved
-
Major
-
RHEL 7.2, Docker version 1.12.3, Jenkins 2.19.1(running in Docker, no agents, only master), Artifactory 4.14.1, Artifactory Plugin 2.8.1
-
Powered by SuggestiMate
Description: I followed Jenkins Artifactory Plugin - Setting Up Docker Build Info and got
java.lang.IllegalStateException: Build info capturing failed for docker image: artifactory.mycompany.com/mycompany/myproject/myservice:0.0.1.c62f0c8.72 check build info proxy configuration.
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:104)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:61)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:40)
...
Docker/Server Setup:
- Instead of using only HTTP_PROXY in /etc/systemd/system/docker.service.d/http-proxy.conf, I also added NO_PROXY to be able to access to Artifactory artifactory.mycompany.com. Please note that adding or removing HTTPS_PROXY does not change anyting. None of the following configurations worked:
/etc/systemd/system/docker.service.d/http-proxy.conf
Environment="HTTP_PROXY=http://localhost:9765" "NO_PROXY=.mycompany.com" #Environment="HTTP_PROXY=http://localhost:9765" "HTTPS_PROXY=http://localhost:9765" "NO_PROXY=.mycompany.com" #Environment="HTTP_PROXY=http://10.74.200.52:9765" "NO_PROXY=localhost,127.0.0.0/8,.mycompany.com" #Environment="HTTP_PROXY=http://10.74.200.52:9765" "HTTPS_PROXY=http://10.74.200.52:9765" "NO_PROXY=localhost,127.0.0.0/8,.mycompany.com"
- Although Jenkins Artifactory Plugin - Setting Up Docker Build Info does not mention, I needed to execute sudo chmod 777 /var/run/docker.sock command to prevent following error:
javax.ws.rs.ProcessingException: org.newsclub.net.unix.AFUNIXSocketException: Permission denied (socket: /run/docker.sock) at com.github.dockerjava.jaxrs.connector.ApacheConnector.apply(ApacheConnector.java:484) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:246) at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:683) ... at com.github.dockerjava.core.command.InspectImageCmdImpl.exec(InspectImageCmdImpl.java:40) at org.jfrog.hudson.pipeline.docker.utils.DockerUtils.getImageIdFromTag(DockerUtils.java:33) at org.jfrog.hudson.pipeline.docker.utils.DockerAgentUtils$2.call(DockerAgentUtils.java:67) at org.jfrog.hudson.pipeline.docker.utils.DockerAgentUtils$2.call(DockerAgentUtils.java:65) at hudson.remoting.LocalChannel.call(LocalChannel.java:45) at org.jfrog.hudson.pipeline.docker.utils.DockerAgentUtils.registerImage(DockerAgentUtils.java:65)
Jenkins Setup:
- There is no Jenkins agent and Jenkins master runs in Docker container.
- To be able to run docker command in the container, I prefered Docker outside of Docker(dood) solution
- exposed port 9765 for Artifactory Plugin's internal proxy
- used volume for Jenkins's data
- ran Jenkins container as swarm service
- Followed Jenkins Artifactory Plugin - Setting Up Docker Build Info by choosing port 9765
FROM artifactory.mycompany.com/jenkins USER root # To be able to build Docker images within the container, we have chosen # Docker outside of Docker(dood) solution in preference to Docker in # Docker(dind). By using dood we are able to execute "docker" commands # without installing Docker into the container. See following # pages to get better understanding of all the benefits of dood: # - Jenkins with DooD (Docker outside of Docker): https://github.com/axltxl/docker-jenkins-dood # - Running Docker in Jenkins (in Docker): http://container-solutions.com/running-docker-in-jenkins-in-docker/ RUN apt-get update \ && apt-get install -y sudo \ && rm -rf /var/lib/apt/lists/* \ && echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers \ && printf '#!/bin/bash\nsudo docker.bin "$@"\n' > /usr/local/bin/docker \ && chmod +x /usr/local/bin/docker USER jenkins # Artifactory Plug-in's build-in proxy for Docker images: # https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Artifactory+Plugin+-+Setting+Up+Docker+Build+Info#JenkinsArtifactoryPlugin-SettingUpDockerBuildInfo-EnabletheJenkinsBuildInfoProxy EXPOSE 9765
docker build -f Dockerfile.myjenkins -t artifactory.mycompany.com/myjenkins . ... docker push artifactory.mycompany.com/myjenkins
docker service create --name myjenkins -p 8080:8080 -p 50000:50000 -p 9765:9765 --replicas 1 \ --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock,readonly \ --mount type=bind,src=$(which docker),dst=/usr/bin/docker.bin \ --mount type=volume,src=volume_myjenkins,dst=/var/jenkins_home \ artifactory.mkk.com.tr/myjenkins
Setup Test:
- On host machine;
- Connected to port 9765 successfully: telnet localhost 9765
- Executed following docker command successfully:
docker pull artifactory.mycompany.com/hello-world
- On Jenkins;
- Created a dummy job and executed the same docker command successfully.
- is blocked by
-
JENKINS-41059 Exception in thread "LittleProxy-JVM-shutdown-hook" java.lang.NoClassDefFoundError
-
- Open
-
[JENKINS-40037] Docker builds fail with "Build info capturing failed for docker image..." exception
Removing NO_PROXY=.mycompany.com did the trick! Thank you Roman.
I noticed I had given some incorrect information. NO_PROXY is needed to bypass our own proxy while accessing Docker Hub directly. I incorrectly said it was needed to access our Artifactory server.
Why I needed to access Docker Hub directly then? While following instructions under Make Sure Docker Works With the Artifactory Docker Registry of Jenkins Artifactory Plugin - Setting Up Docker Build Info page, docker pull hello-world command is required to succeed. Now, I understand that the important point here is the push step.
By the way;
- sudo chmod 777 /var/run/docker.sock is still needed. Is this a known issue that has to be mentioned in documentation, or am I doing someting wrong?
- When Jenkins container is running on a swarm cluster docker pull artifactory.mycompany.com/hello-world succeeds only on the machine Jenkins container runs because we use localhost in HTTP_PROXY url. Using dedicated host for Jenkins container and setting HTTP_PROXY=http://<jenkins_host>:<port> in http-proxy.conf on each swarm host do not seem to be a good solution. What do you think?
Thank you for your comments, we will try to improve the documentation so the setup would be easier.
Regarding your comments:
- You probably had to change permissions to /var/run/docker.sock because of your Jenkins/Docker setup, regularly Jenkins have permissions to access it but because Jenkins runs on Docker container you had to change permissions.
- Build info is captured via the proxy, so each node has to pass through it.
We found an interesting article describing how to configure proxy for Docker swarm:
https://technologyconversations.com/2016/08/01/integrating-proxy-with-docker-swarm-tour-around-docker-1-12-series/
Thank you very much for your valuable comments.
- I am going to add chmod 777 /var/run/docker.sock to docker service's configuration.
- I do not know what happened but telnet localhost <proxy_port> works on every swarm mode although Jenkins service runs on only one node. It seems swarm's internal load balancer directs the traffic successfully. Thus, following comment is not valid anymore:
When Jenkins container is running on a swarm cluster docker pull artifactory.mycompany.com/hello-world succeeds only on the machine Jenkins container runs because we use localhost in HTTP_PROXY url. Using dedicated host for Jenkins container and setting HTTP_PROXY=http://<jenkins_host>:<port> in http-proxy.conf on each swarm host do not seem to be a good solution...
Adding ExecStartPost worked for me:
[Service]
...
ExecStartPost=/usr/bin/chmod 777 /var/run/docker.sock
I have the same error.
My slave is on docker container and all slave can have acces to /var/run/docker.sock.
My jenkins-slave container is run on swarm.
All docker damemon have HTTP_PROXY=http://<jenkins_host>:<port>
If i disable artifacotry proxy all pull and push is KO
If i enable artifacotry all pull and push is OK
and
docker pull hello-world
docker tag hello-world:latest <my artifactory>/hello-world:latest
docker login docker-vsct.pkg.cloud.socrate.vsct.fr
docker push <my artifactyory>/hello-world:latest
Is ok on swarm
My pipeline
def server = Artifactory.server('artifactory_global')
def artDocker = Artifactory.docker credentialsId: 'admnexus'
image = docker.build("<my artifactyory>/hello-world:latest")
buildInfo = artDocker.push("<my artifactyory>/hello-world:latest", "my-registry")
The artDocker.push FAIL
java.lang.IllegalStateException: Build info capturing failed for docker image: <my artifactyory>/hello-world:latest check build info proxy configuration.
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:104)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:61)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:42)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
and this on Jenkins master LOG:
already completed CpsStepContext[23:dockerPushStep]:Ownerdev-heat/demo-app/41:dev-heat/demo-app #41
java.lang.IllegalStateException: delivered here
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:351)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onSuccess(CpsStepContext.java:331)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:42)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
at org.jfrog.hudson.pipeline.types.Docker.push(Docker.java:73)
at org.jfrog.hudson.pipeline.types.Docker.push(Docker.java:64)
at org.jfrog.hudson.pipeline.types.Docker.push(Docker.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrap.invoke(PojoMetaMethodSite.java:213)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:58)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Jan 12, 2017 2:04:47 PM INFO org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
new success: null
Jan 12, 2017 2:04:47 PM INFO org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
previously delivered here
java.lang.Throwable
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:339)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:327)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:103)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:61)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:42)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
All my jenkins-slave is run on the same swarm.
And all jenkins-slave is ephemeral (Docker once retention)
I think this is because jenkins master not search build info on master but on jenkins-slave agent.
The henkins-slave agent run on container is not used by swarm.
All my docker daemon are configured to use the jenkins master BuildInfo proxy because jenkins is deployed on the same swam than all jenkins-slave container.
Is it possible to disable start agent and use remote buildInfo proxy
Sébastien,
In the current Jenkins Artifactory Plugin release (2.8.2), when a docker push is performed on a specific agent, only the build-info proxy of this specific agents captures the image.
We have enhanced this functionality and in the coming release, all proxies (on all agents and the master) will have the image information. You'll be able to have your daemon proxy any of the build-info proxies (on the master and agents).
Here's a download link to snapshot version of the plugin, that already includes this functionality. Feel free to try it out:
https://repo.jfrog.org/artifactory/libs-snapshots-local/org/jenkins-ci/plugins/artifactory/2.8.3-SNAPSHOT/artifactory-2.8.3-SNAPSHOT.hpi
Additionally, we've just updated the documentation to show how you can set your docker daemon URL (this is already supported in 2.8.2):
https://wiki.jenkins-ci.org/display/JENKINS/Artifactory+-+Working+With+the+Pipeline+Jenkins+Plugin#Artifactory-WorkingWiththePipelineJenkinsPlugin-DockerBuildswithArtifactory
Running a standalone build-info proxy is currently not supported, but we're thinking of adding this functionality in the future.
Thank you for your reply; i have now a new error on my slave;
I open new issue.
I have updated artifactory-plugin to 2.9.0 and specified host docker daemon.
Now i have this error:
INFO: Pushing image: <private-repo>/vscloud/demoapp:dev using docker daemon host: tcp://10.207.19.193:4000
channel stopped
Connection terminated
Exception in thread "LittleProxy-JVM-shutdown-hook" java.lang.NoClassDefFoundError: io/netty/channel/group/ChannelMatchers
at io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:225)
at org.littleshoot.proxy.impl.DefaultHttpProxyServer.closeAllChannels(DefaultHttpProxyServer.java:435)
at org.littleshoot.proxy.impl.DefaultHttpProxyServer.doStop(DefaultHttpProxyServer.java:403)
at org.littleshoot.proxy.impl.DefaultHttpProxyServer.abort(DefaultHttpProxyServer.java:385)
at org.littleshoot.proxy.impl.DefaultHttpProxyServer$1.run(DefaultHttpProxyServer.java:141)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: io.netty.channel.group.ChannelMatchers
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:149)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 6 more
Slave JVM has not reported exit code. Is it still running?
ERROR: [01/18/17 11:45:32] [SSH] Error deleting file.
java.io.IOException: Sorry, this connection is closed.
at com.trilead.ssh2.transport.TransportManager.ensureConnected(TransportManager.java:587)
at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:660)
at com.trilead.ssh2.channel.ChannelManager.openSessionChannel(ChannelManager.java:572)
at com.trilead.ssh2.Session.<init>(Session.java:42)
at com.trilead.ssh2.Connection.openSession(Connection.java:1129)
at com.trilead.ssh2.Connection.exec(Connection.java:1551)
at hudson.plugins.sshslaves.SSHLauncher$3.run(SSHLauncher.java:1261)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Cannot read full block, EOF reached.
at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:81)
at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:693)
at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:489)
... 1 more
[01/18/17 11:45:32] [SSH] Connection closed.
What is the branch and git repo to show thecoming release, all proxies (on all agents and the master) ?
I have forked github master and i can see that only slave proxy agent is asked.
Sébastien,
Looks like you're experiencing an issue similar to this:
https://issues.jenkins-ci.org/browse/JENKINS-41059
The Jenkins Artifactory Plugin is managed here:
https://github.com/JFrogDev/jenkins-artifactory-plugin
On class DockerAgentUtils on method updateImageParentOnAgents
I can see that Jenkins.getInstance().getNodes() does not return master.
This is why it can not find buidInfo on master agent.
this Jenkins.getInstance().getChannel() can have access to master
After adding Log on BuildInfoHttpFiltersSource and JenkinsBuildInfoLog, I can see than proxy received buildInfo Data
But on rtDocker.push, this info is not retreived
Jan 23, 2017 9:34:23 AM INFO org.jfrog.hudson.pipeline.docker.proxy.BuildInfoHttpFiltersSource filterRequest
filterRequest: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
HEAD /v2/vscloud/demoapp/blobs/sha256:2c479166ffa8bfcc71460b960771b47046161ec97f8e6576da70ba202962db59 HTTP/1.1
Host: <my_artifactory_registry>
User-Agent: docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64 UpstreamClient(Jersey/2.23.1 (Apache HttpClient 4.5.1))
Authorization: Bearer AKCp2VobB17HMYFqxK9kcLm8jgfEbCAK8QCy7ju8q2cxnM9UvGQHhxzcDCtz2eNicvkdMGrVh
Connection: close
Jan 23, 2017 9:34:23 AM INFO org.jfrog.hudson.pipeline.docker.proxy.BuildInfoHttpFiltersSource filterRequest
filterRequest: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
CONNECT <my_artifactory_registry>:443 HTTP/1.1
Host: <my_artifactory_registry>:443
User-Agent: Go-http-client/1.1
Jan 23, 2017 9:34:23 AM INFO org.jfrog.hudson.pipeline.docker.proxy.BuildInfoHttpFiltersSource filterRequest
filterRequest: DefaultHttpRequest(decodeResult: success, version: HTTP/1.1)
PUT /v2/vscloud/demoapp/manifests/dev HTTP/1.1
Host: <my_artifactory_registry>
User-Agent: docker/1.12.3 go/go1.6.3 git-commit/6b644ec kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64 UpstreamClient(Jersey/2.23.1 (Apache HttpClient 4.5.1))
Content-Length: 738
Authorization: Bearer AKCp2VobB17HMYFqxK9kcLm8jgfEbCAK8QCy7ju8q2cxnM9UvGQHhxzcDCtz2eNicvkdMGrVh
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Accept-Encoding: gzip
Connection: close
Jan 23, 2017 9:34:23 AM INFO org.jfrog.hudson.pipeline.docker.proxy.BuildInfoFilterAdapter proxyToServerRequest
Proxy message received: {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config":
,
"layers": [
,
{ "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 1522511, "digest": "sha256:b94183b1beb0988d22083e5e60a94474d25be0d0dae4c57e8cb2e07a043a77be" } ]
}
Jan 23, 2017 9:34:23 AM INFO org.jfrog.hudson.util.JenkinsBuildInfoLog info
Docker build-info found on node vscloudJenkins-731fabf61c32-host-192-168-0-54.openstacklocal : false
Jan 23, 2017 9:34:23 AM INFO org.jfrog.hudson.util.JenkinsBuildInfoLog info
Docker build-info found on node Jenkins : false
Jan 23, 2017 9:34:23 AM WARNING org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
already completed CpsStepContext[34:dockerPushStep]:Ownerdev-heat/demo-app/100:dev-heat/demo-app #100
java.lang.IllegalStateException: delivered here
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:351)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onSuccess(CpsStepContext.java:331)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Jan 23, 2017 9:34:23 AM INFO org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
new success: null
Jan 23, 2017 9:34:23 AM INFO org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
previously delivered here
java.lang.Throwable
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:339)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:327)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:105)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:62)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Jan 23, 2017 9:34:23 AM INFO org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
earlier failure
java.lang.IllegalStateException: Build info capturing failed for docker image: docker-<my_artifactory_registry>/vscloud/demoapp:dev check build info proxy configuration.
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:105)
at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:62)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:260)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
A same job can sometimes works, but sometimes it failed
this does not depend on docker-daemon all is executed on the same daemon.
On each failed, I have this error
Jan 23, 2017 2:55:21 PM WARNING org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
already completed CpsStepContext[35:dockerPushStep]:Ownerdev-heat/demo/master/9:dev-heat/demo/master #9
java.lang.IllegalStateException: delivered here
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:351)
at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onSuccess(CpsStepContext.java:331)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
And this info message before FAILING:
(AWAITING_CHUNK) [id: 0xc94662c0, L:/172.18.0.2:2376 - R:/192.168.0.54:52894]: Connection to server became saturated, stopping reading
Jan 23, 2017 3:10:57 PM INFO org.littleshoot.proxy.impl.ClientToProxyConnection serverBecameWriteable
(AWAITING_CHUNK) [id: 0xc94662c0, L:/172.18.0.2:2376 - R:/192.168.0.54:52894]: All server connections writeable, resuming reading
Jan 23, 2017 3:10:57 PM INFO org.littleshoot.proxy.impl.ClientToProxyConnection serverBecameSaturated
(AWAITING_CHUNK) [id: 0xc94662c0, L:/172.18.0.2:2376 - R:/192.168.0.54:52894]: Connection to server became saturated, stopping reading
Jan 23, 2017 3:10:57 PM INFO org.littleshoot.proxy.impl.ClientToProxyConnection serverBecameWriteable
(AWAITING_CHUNK) [id: 0xc94662c0, L:/172.18.0.2:2376 - R:/192.168.0.54:52894]: All server connections writeable, resuming reading
sebglon,
We are trying to find the root cause for your issues, specifically the following message:
Connection to server became saturated, stopping reading
Could you share more details about your Pipeline script and the Docker image that you are pushing?
I have been following this thread for a while as I have a similar issue.
I receive the "Connection to server became saturated, stopping reading" as well, but do not have any full FS.. I can provide more information if needed.
For further investigation please contact our support team at support@jfrog.com
Please attach this Jira issue reference in the mail.
My jenkins master is on swarm cluster A
and my slave is spawn on a second swarm cluster B.
When i build image from my pipeline, i build on docker-in-Docker whit proxy configurer to point on my jenkins Master.
If i check on debug, i can see buidinfo on my master but have not acces to my Docker-in-Docker and not register ImageId and Image Tag on DockerAgentUtil.registerImage method.
Why this method check access on Docker Host?
Is it possible add nodo info on BuildInfo for using them on method updateImageParentOnAgents?
Because my Jenkins master have buildInfo but no acces to Docker-inDocker and jenkins slave have'n buidInfo and not updateImageParent.
The following exception indicates that the push command was executed without passing through the Build-info proxy.
java.lang.IllegalStateException: Build info capturing failed for docker image: artifactory.mycompany.com/mycompany/myproject/myservice:0.0.1.c62f0c8.72 check build info proxy configuration. at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:104) at org.jfrog.hudson.pipeline.steps.DockerPushStep$Execution.run(DockerPushStep.java:61) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousStepExecution.start(AbstractSynchronousStepExecution.java:40) ...
The simplest way to check whether the proxy configured correctly is to disable Build-info proxy (in Jenkins configuration or shutting down Jenkins completely) and trying to push or pull image from Artifactory Docker registry.
If the push/pull was successful proxy is not configured correctly.
In your configuration you have excluded Artifactory Docker registry from using the proxy.
Please try to remove the following from http-proxy.conf:
"NO_PROXY=.mycompany.com"