-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Jenkins 2.7.3, docker-workflow 1.7
-
Powered by SuggestiMate
When using either docker.withRegistry('', 'dockerhub') or withDockerRegistry(registry: [credentialsId: 'dockerhub']), a call to mycont.push() fails to authenticate with DockerHub:
withDockerRegistry(registry: [credentialsId: 'dockerhub']) { cont.push() cont.push('latest') }
leads to:
Proceeding [Pipeline] withDockerRegistry [Pipeline] { [Pipeline] sh [test] Running shell script + docker tag --force=true camptocamp/jenkins-test camptocamp/jenkins-test:latest unknown flag: --force See 'docker tag --help'. + docker tag camptocamp/jenkins-test camptocamp/jenkins-test:latest [Pipeline] sh [test] Running shell script + docker push camptocamp/jenkins-test:latest The push refers to a repository [docker.io/camptocamp/jenkins-test] 749689370cd8: Preparing unauthorized: authentication required [Pipeline] } [Pipeline] // withDockerRegistry [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline
I have the credentials set up properly in Jenkins, as shown in the attached screenshot, but Jenkins Credentials Manager says `This credential has not been recorded as used anywhere.`
- is blocked by
-
JENKINS-51395 withDockerRegistry([...]) syntax no longer accepted
-
- Resolved
-
-
JENKINS-51397 docker login being called without server-related environment variables
-
- Resolved
-
- is duplicated by
-
JENKINS-41051 withRegistry is not doing a "docker login"
-
- Resolved
-
-
JENKINS-48458 Multiple jobs running in parallel on the same agent using the Docker Workflow Plugin withRegistry results in overwritten credentials
-
- Resolved
-
-
JENKINS-32022 Auth hash for .docker/config.json is not correct format in jenkins 1.6+
-
- Resolved
-
- is related to
-
JENKINS-34958 Getting "Your Authorization Token has expired" when using ECR credentials
-
- Resolved
-
-
JENKINS-44143 ECR plugin: no basic auth credentials
-
- Closed
-
-
JENKINS-45851 Your Authorization Token is invalid.
-
- Closed
-
- relates to
-
JENKINS-33985 Credentials always points to https://index.docker.io/v1 even for private registry
-
- Resolved
-
-
JENKINS-51406 Integrate with new docker login APIs
-
- Open
-
-
JENKINS-51407 Integrate with new docker login APIs
-
- Open
-
-
JENKINS-51949 Docker agent in declarative pipeline failing to login to custom registry
-
- Open
-
-
JENKINS-28702 Clean up registry credentials
-
- Resolved
-
-
JENKINS-46321 JSONException: null object when 'auths' key is not exist in .docker/config.json file.
-
- Resolved
-
- links to
[JENKINS-38018] withDockerRegistry fails to authenticate with DockerHub
Actually, it still doesn't work. It worked because I had used the following code at some point:
withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: 'dockerhub', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) { sh 'docker login -u "$USERNAME" -p "$PASSWORD"' cont.push() }
and it didn't remove the credentials, so switching to `withDockerRegistry()` after that used the credentials still stored in `~/.docker`.
I've just experienced that `withDockerRegistry()` on clean slaves doesn't work, and had to revert to using `withCredentials()`.
withRegistry continues to work in the stock demo (using registry:0.9.1) but perhaps DockerHub changed their authentication mechanism? Need to check if this is reproducible from scratch.
Working for me using
docker.withRegistry('', '…my-hub-credentials-ID…') { writeFile file: 'Dockerfile', text: ''' FROM ubuntu RUN echo hello ''' docker.build('jglick/testing').push('latest') }
I can improve the step to print a message saying where it is storing your credentials (~/.dockercfg or ~/.docker/config.json), as well as to abort in case you specified a credentialsId which could not be resolved for some reason. I could also change it to run docker login, though I would rather first confirm a scenario where the current code does not work.
Need to check behavior with the 2.x registry.
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryEndpoint.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryToken.java
http://jenkins-ci.org/commit/docker-commons-plugin/702579ffe9001f4394e696dcc1d7990012ce97df
Log:
JENKINS-38018 Extend DockerRegistryEndpoint API to take a Launcher and TaskListener.
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryEndpoint.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryToken.java
http://jenkins-ci.org/commit/docker-commons-plugin/fb0aa29e60d3dfdd6d7bbc895a00075a397b00e4
Log:
Merge pull request #55 from jglick/registry-diag-JENKINS-38018
JENKINS-38018 Extend DockerRegistryEndpoint API to take a Launcher and TaskListener
Compare: https://github.com/jenkinsci/docker-commons-plugin/compare/78f6f39ce31b...fb0aa29e60d3
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStep.java
http://jenkins-ci.org/commit/docker-workflow-plugin/e3ef354e8fe9e044513471268f71f9149698bc19
Log:
JENKINS-38018 Call newly available API
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStep.java
http://jenkins-ci.org/commit/docker-workflow-plugin/0d391ed1ee5b71ade926fafc32e58f284270aae6
Log:
Merge pull request #72 from jglick/registry-diag-JENKINS-38018
JENKINS-38018 Call newly available API
Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/77241fcd8ad1...0d391ed1ee5b
Still seeing this with
- Docker plugin 0.16.2
- Credentials plugin 2.1.5
It may well be that I'm doing something wrong. The instructions are unclear.
Same problem.
Hint : Ensure you are using the right credentials for the repository you are referring to. Do check this, you can run from the Script console (/script ) something like :
println "cat ~/.docker/config.json".execute().text
My workaround is to instead of wrapping the docker related tasks into something like
def DOCKER_REGISTRY_URI="..." docker.withRegistry("${DOCKER_REGISTRY_URI}", ...){ // ... }
into something like
def DOCKER_REGISTRY_URI="..." withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: '...', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) { sh "docker login --password=${PASSWORD} --username=${USERNAME} ${DOCKER_REGISTRY_URI}" }
As mentioned I was unable to reproduce the issue with either DockerHub or a custom local registry. Until I can reproduce the issue from scratch, nothing will happen here.
Guys, I puzzled with the same problem couple days with kubenetes-plugin + docker common plugin. After investigation, I found:
- The .withRegistry('https://index.docker.io/v1/', 'your_docker_hub_login_credentialId'), depends what the base image run with which user (my example is root), it will create the credential (namely .dockercfg) in its home directory (e.g. /root/.dockercfg).
- But depends on your k8s nodes (debian jessie based) running with which version docker, it looks for /home/jenkins/.docker/config.json, with v2 syntax.
- Also if you just like me, sharing /var/run/docker.socket with docker within the build node, that probably also cause problem.
My dirty hack is, keep two files (.dockercfg and config.json) in sync by comparing which one is newer. I don't blame this or begging for solution myself since hooking up with /var/run/docker.socket, combining k8s-plugin + docker common plugin are all initiated by myself. Just share the story.
Thank you, zanhsieh! Your explanation helped me. jglick, this might help reproduce the problem:
- My build container is used to build the software, and also to build and push a new Docker image containing that software.
- The build container is built with a Dockerfile that has to install software using apt-get. As a result, I have "USER root" in the Dockerfile. But I never put "USER jenkins" after those install steps, so the build container is still running Jenkins slave commands as root.
- In my build container, $HOME is /home/jenkins but the uid is zero.
- docker.withRegistry() writes its authentication file to /root/.dockercfg, but docker push tries to read it from $HOME/.dockercfg.
...I solved my problem my putting "USER jenkins" at the end of my Dockerfile. This was not obvious from the feedback Jenkins/Docker gave me.
Jenkins will put .dockercfg into the directory given by the user.home according to the Jenkins agent JVM, so if this does not match the user of the actual docker commands then you will have problems.
I suspect that implementing JENKINS-28702 using CLI login/logout is the most straightforward fix.
jglick, so I have a decently reproducible case for a freestyle job as well where docker-commons needs to be updated(either with the CLI or with changing the .dockercfg file). It seems that with the 1.12 version of docker they changed the file which is needed to log you in from ~/.dockercfg to ~/.docker/config.json which means that it will not see your credentials. So if you have a Private repo in dockerhub it will not log you in properly.
Environment:
Jenkins 2.32.3(but I think jenkins version is irrelevant)
Docker-commons: 1.6
Docker version 1.12 on the machine running the job
Docker build and publish:1.3.2
Steps to reproduce:
Have a freestyle job
Have a private dockerhub repository
Have a step which is building a docker container and pushing it to that private dockerhub repo
Behavior:
It fails with a unauthenticated error because the ~/.dockercfg file no longer works. The docker build and publish step leverages the docker commons plugin to log into private repos so this is a good case to see the failure.
Expected Behavior
It would upload the docker container to your repo
Workaround:
If you log into that dockerhub repo through the CLI in a shell step before the build and publish step, it will create a ~/.docker/config.json file which will retain your credentials until the end of the build. Then the build and publish will work normally
So I suppose we could potentially create both files as a temporary fix until we move to the CLI?
What are your thoughts guys?
I already accepted the fact that if I require the old configuration syntax (e.g `.dockercfg`) I can use the `withDockerRegistry` provider and if I need the new syntax I can use run `docker login` with the correct credentials. But what I'm noticing now, is that if a `$HOME/.docker/config.json` file exists then `withDockerRegistry` appends to that instead ?
This behavior is really confusing. Would it be possible to add a parameter to `withDockerRegistry` that lets me specify the output path of the credentials file? It seems like it would solve a lot of confusion for people who want to control it, and otherwise fall back to the current logic that's already there.
The plan is to throw out the existing code and use docker login which would either work, or it is a bug report for docker.com.
I experience an issue when using my local Mac as a Jenkins Agent with authentication using "docker.withRegistry()"
Setup:
MacOSX 10.12.6
(as a docker container) Jenkins 2.60.2
Jenkins Agent: SSH agent to local MacOSX host
In the Jenkins job console I can see the log message the credentials get created and are put under my ssh user (local macosx user) but it fails to properly authenticate and I get the "unauthorized: authentication required" message.
If I switch to using a linux/CentOS 7 as a Jenkins SSH Agent I do not see this issue at all.
If I look closer at the "~/.docker/config.json" file there is a new entry for the Docker Registry URI but the credentials are in the file itself, all other records are empty for the Docker URIs. I think it's because natively Docker For Mac is using a credentials Store – "credsStore": "osxkeychain" and will not use in-line credentials.
If I look on the Linux Agent, it is not using a credential store and hence why I think it works properly when the credentials are in the file.
So there may be an issue when using a credential store and docker.withRegistry()..
jglick is there a timeframe for implementing the plan you referred to?
FWIW I remain unable to reproduce this issue. I created a private repository on DockerHub and successfully pushed to it using withRegistry from a fresh container (bind-mounted to my Docker socket but using a download of the latest docker binary); inside the container:
# cat /home/jenkins/.dockercfg {"https://index.docker.io/v1/": { "auth": "…", "email": "jglick" }}
Filed a fix for docker-workflow using normal username/password credentials (as opposed to, say, Amazon ECR credentials, which presumably were not the subject of this bug to begin with). The two other (freestyle) client plugins would need minor analogous changes to call the new API.
I see the exact same symptoms running docker on. Mac with Jenkins running in a container. I can push to docker hub from the command line but the push with registry fails as shown above. Is there a work around or do we need to wait for the plugin to be updated?
Read the older comments and used the CLI login as a work around as follows
stage('CLI workaround') { /* Workaround to address issue with credentials stored in Jenkins not * being passed correctly to the docker registry * - ref https://issues.jenkins-ci.org/browse/JENKINS-38018 */ withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: 'docker-hub-credentials', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) { sh 'docker login -u $USERNAME -p $PASSWORD https://index.docker.io/v1/' } }
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStep.java
src/main/resources/org/jenkinsci/plugins/docker/workflow/Docker.groovy
src/main/resources/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStep/config.jelly
src/test/java/org/jenkinsci/plugins/docker/workflow/WithContainerStepTest.java
http://jenkins-ci.org/commit/docker-workflow-plugin/6653988c2a18af31bc90bec0d23562daefacd1f4
Log:
JENKINS-38018 Use docker-login wherever possible for registry credentials.
Code changed in jenkins
User: Andrew Bayer
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStep.java
src/main/resources/org/jenkinsci/plugins/docker/workflow/Docker.groovy
src/main/resources/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStep/config.jelly
src/test/java/org/jenkinsci/plugins/docker/workflow/WithContainerStepTest.java
http://jenkins-ci.org/commit/docker-workflow-plugin/cf80d9e2f628392b79777abfb90371cbb5353e5c
Log:
Merge pull request #133 from jglick/login-JENKINS-38018
JENKINS-38018 Use docker-login wherever possible for registry credentials
Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/88211b996123...cf80d9e2f628
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryEndpoint.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryToken.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerServerEndpoint.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/KeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/NullKeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/RegistryKeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/ServerHostKeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/ServerKeyMaterialFactory.java
src/test/java/org/jenkinsci/plugins/docker/commons/util/SampleDockerBuilder.java
http://jenkins-ci.org/commit/docker-commons-plugin/a1352291ae31959dd3d68574ee24d2da91e1f468
Log:
JENKINS-38018 Use docker-login wherever possible for registry credentials.
Code changed in jenkins
User: Andrew Bayer
Path:
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryEndpoint.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerRegistryToken.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/DockerServerEndpoint.java
src/main/java/org/jenkinsci/plugins/docker/commons/credentials/KeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/NullKeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/RegistryKeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/ServerHostKeyMaterialFactory.java
src/main/java/org/jenkinsci/plugins/docker/commons/impl/ServerKeyMaterialFactory.java
src/test/java/org/jenkinsci/plugins/docker/commons/util/SampleDockerBuilder.java
http://jenkins-ci.org/commit/docker-commons-plugin/e00c616a847630fccf637e891000813a7b441ceb
Log:
Merge pull request #67 from jglick/login-JENKINS-38018
JENKINS-38018 Use docker-login wherever possible for registry credentials
Compare: https://github.com/jenkinsci/docker-commons-plugin/compare/0b0429d2ddaf...e00c616a8476
Fixed (pending release) in docker-workflow. docker-build-publish and docker-custom-build-environment would need minor code changes (switching from a now-deprecated API call) to pick up the fix.
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/test/java/org/jenkinsci/plugins/docker/workflow/DockerDSLTest.java
http://jenkins-ci.org/commit/docker-workflow-plugin/d56b6d232b56e8d59d6176697ae6a1b1ecd99455
Log:
Merge pull request #137 from jglick/release-JENKINS-38018
JENKINS-38018 Preparing for release
Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/cf80d9e2f628...d56b6d232b56
*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Fix released in docker-workflow 1.16. Probably I should file separate issues for the other two plugins, but I am too lazy.
hi, I'm using AWS ESC Repository with withDockerRegistry.
this pipeline is woking with 1.15.1
withDockerRegistry([credentialsId: 'AWS ECS Repository', url: 'https://my-repo-url']) { }
but I got NPE with 1.16. what happened?
java.lang.NullPointerException at org.jenkinsci.plugins.docker.workflow.RegistryEndpointStep$Execution.newKeyMaterialFactory(RegistryEndpointStep.java:81) at org.jenkinsci.plugins.docker.workflow.AbstractEndpointStepExecution.start(AbstractEndpointStepExecution.java:44) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at WorkflowScript.run(WorkflowScript:19) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:138) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:569) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:322) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:320) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:568) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:223) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:456) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:455) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:221) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:366) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:365) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:220) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withCredentialsBlock(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:404) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withCredentialsBlock(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:403) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:219) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:242) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:494) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:493) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:239) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:304) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:303) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:217) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:521) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:520) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(jar:file:/var/lib/jenkins/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:200) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
We are affected by the exact same problem reported by saisse
The pipeline code triggering it is
withDockerRegistry([credentialsId: 'gcr:hubrick-commons', url: 'https://eu.gcr.io']) {[...]}
Same as saisse here, but using k8s in AWS with a artifactory / jfrog repo.
I current workaround is
docker.withRegistry('url', 'credential') {
However that is undesirable as it exposes the username and password in the build log eg
Executing shell script inside container [dind] of pod [jenkins-slave-mxj9x-12m9q] Executing command: "docker" "login" "-u" "username" "-p" "password" "https://repo.example.com"
Same here with an internal Docker registry. Docker login now fails with "401 unauthorized" after upgrading to 1.16.
Reverting to 1.15.1 works, so this appears to be a regression with 1.16.
1.16 break builds,
with 1.15.1 this was working but not with 1.16
docker.withServer(host) { docker.withRegistry(repo, credentialsId ){
We also covered this problem with 1.16! Seems to be a bigger issue - hope a fix will be released soon... As workaround we also downgraded the docker pipeline plugin to 1.15.1 meanwhile.
withDockerRegistry([credentialsId: 'dockerhub.com']) { image.push("$imageTag") }
After fixing the NPE by changing from
withDockerRegistry([credentialsId: dockerRegistryCredentials, url: dockerRegistry]) {
to
withDockerRegistry(registry: [credentialsId: dockerRegistryCredentials, url: dockerRegistry], toolName: 'docker') {
we now get this error:
$ docker login -u ******** -p ******** [https://********:443] WARNING! Using --password via the CLI is insecure. Use --password-stdin. Warning: failed to get default registry endpoint from daemon (Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.37/info: dial unix /var/run/docker.sock: connect: permission denied). Using system default: https://index.docker.io/v1/ Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/auth: dial unix /var/run/docker.sock: connect: permission denied
schulzha Can you a point for me please?
Were you using the 1.16 version of the plugin, or a more recent build?
The reason I ask is the snippet generator http://jenkins:8080/pipeline-syntax/ (or similar depending on your environment) do not include the toolName or explicitly the registry:
timothy_clarke The error occurs with 1.16. This is what the snippet generator gives me:
// This step should not normally be used in your script. Consult the inline help for details. withDockerRegistry(registry: [credentialsId: 'bar', url: 'foo'], toolName: 'docker') { // some block }
I should add that our Jenkins host is not our docker host, so we are using this command inside a withDockerServer block:
script.withDockerServer([credentialsId: dockerServerCredentials, uri: dockerServer]) { script.withDockerRegistry(registry: [credentialsId: dockerRegistryCredentials, url: dockerRegistry], toolName: 'docker') { sh "docker build" sh "docker push" } }
AFAICT unrelated issues are being discussed here.
First, as originally reported by saisse, if you were using the syntax withDockerRegistry([…]), this seems to have stopped working after the introduction of the toolName parameter, I think due to a core Pipeline bug (though I had thought this was fixed years ago). Will investigate. To fix, simply be explicit about the registry: parameter name. (The DSL sugar docker.withRegistry internally calls the working syntax of the step.)
Second, vwbusguy, schulzha, and perhaps nicolasus are reporting unrelated issues ultimately due to a failure to run a docker login command. I cannot help you with that; something is broken in your environment. The new version of docker-workflow deliberately just calls the Docker CLI to log in, rather than attempting to fake the undocumented JSON authentication file format. Log in to your build node with a shell and run docker login with the same username and password until you track down the issue (for example, I saw a file permission issue on docker.sock).
timothy_clarke mentioned in passing a failure to mask passwords from the docker login command, which I have not seen and would not know how to reproduce. Seems to be some bug in the Kubernetes plugin, perhaps. File separately with complete steps to reproduce from scratch.
I just want to confirm that 1.16 is not working. My team had to downgrade the plugin back to 1.15.1 to get things rolling.
Just to comment, probably, as commented , there are multiple issues here, and first of all thank you all or the work and efforts done to make jenkins works with docker.
In my case I don't think it's an environment (only) problem, it works perfectly with plugin version 1.15.1 and it doesn't work at all with plugin version 1.16.
What I'm doing is a docker development cycle pipeline, that builds dockers inside a jenkins-ssh-slave (dockerized) running from a dockerized jenkins master (2.107.3).
This pipeline builds the docker images in the slave, connecting to the remote dockerd, and push the image to the private docker registry (nexus).
Well in this scenario, the `docker` command should take the DOCKER_HOST from the environment and authenticate to the docker private registry.
With the update this stopped to work
// run from base dir for folder builds and from tmp for url builds dir ("$runPwd") { echo "Build with server ${env.DOCKER_HOST} and registry ${env.CP_DOCKER_RELEASE}" docker.withServer(env.DOCKER_HOST) { docker.withRegistry(env.CP_DOCKER_RELEASE,env.DOCKER_CREDENTIALS ){ echo "Execute docker build -t $name:$version $params" sh "docker build -t $name:$version $params" // def image = docker.build ("$name:$version", params) } } }
After I downgrade version to 1.15.1 this works again with no other change.
If it's required a change in configuration or parameters I suppose it should be documented or have sensitive defaults to act as previous versions and continue to work as expected or be "adapted" with the new/changed configuration/parameters.
I f you need any help on variable values or Dockerfile's or conatiner o whatever may help don't hesitate to ask, I'll give whatever may help.
Thanks again,
Nico
nicolasus hmm, I suspect your issue is specific to the combination of withDockerServer and withDockerRegistry, something we have no tests for due to the technical difficulty of simulating these environments. I think I know what the issue is there and have filed it as JENKINS-51397.
nicolasus, the same here. After downgrading to version 1.15.1 everything works fine. Jenkins 2.107.3.
Please do not comment further on this issue. Check JENKINS-51395 and JENKINS-51397, or file a fresh bug and link it here if you are sure you have encountered a distinct issue.
Filed JENKINS-51406 & JENKINS-51407 for integration into other non-Pipeline plugins, so this issue can be considered strictly about Docker Pipeline.
jglick What docker login environment problem? Docker login works fine from CLI, ansible, or this plugin from 1.15.1 and previous. Between this and the breaking ENTRYPOINT in 1.15.x, I'm at my last straw with this plugin and possibly even Jenkins itself as Docker has become such a core of our operations. I'm happy to help test and possibly contribute code, but I just can't continue having my teams rely on a feature that has major regressions every update and the CloudBees devs keep dismissing the people reporting the issues and trying to fix them. It's never fun when we have to keep spending our time debugging and coding workarounds for this plugin rather than the code we're trying to test to begin with.
Nevermind, the user I was using didn't have rights to push to that repository… Sorry for the noise, everything works.