Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51949

Docker agent in declarative pipeline failing to login to custom registry

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • docker-workflow-plugin
    • Jenkins 2.124
      Docker Commons 1.13
      Docker Workflow 1.17

      We have an image that is stored in AWS ECR. Previously this step has been working but broke when we updated the Docker Workflow plugin from 1.15.1 to 1.17. We now get a `docker login failed` message.

      This is for a declarative pipeline and our configuration is as follows.

      stage('Docker step') {
           agent {
               docker {
                   image "xxxxxxxx"
                   registryUrl "https://xxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com"
                   registryCredentialsId "credentials-id"
               }
           }
       }

      When we roll the plugin back to 1.15.1 the issue no longer occurs.
      This may be related to JENKINS-38018

          [JENKINS-51949] Docker agent in declarative pipeline failing to login to custom registry

          Jarrett G added a comment -

          This breaks a key step in my build pipeline. Looks like the culprit is `src/main/java/org/jenkinsci/plugins/docker/commons/impl/RegistryKeyMaterialFactory.java`

              public KeyMaterial materialize() throws IOException, InterruptedException {
                  FilePath dockerConfig = createSecretsDirectory();
                  try {
                      // TODO on Docker 17.07+ use --password-stdin
                      EnvVars envWithConfig = new EnvVars(env);
                      envWithConfig.put("DOCKER_CONFIG", dockerConfig.getRemote());
                      if (launcher.launch().cmds(new ArgumentListBuilder(dockerExecutable, "login", "-u", username, "-p").add(password, true).add(endpoint)).envs(envWithConfig).stdout(listener).join() != 0) {
                          throw new AbortException("docker login failed");
                      }
                  } catch (IOException | InterruptedException x) {
                      try {
                          dockerConfig.deleteRecursive();
                      } catch (Exception x2) {
                          x.addSuppressed(x2);
                      }
                      throw x;
                  }
                  return new RegistryKeyMaterial(dockerConfig, new EnvVars("DOCKER_CONFIG", dockerConfig.getRemote()));
              }
          

          Looks like this was intentionally passed over, possibly due to some security issue. jglick - any reason why this hasn't been updated yet?

          Jarrett G added a comment - This breaks a key step in my build pipeline. Looks like the culprit is `src/main/java/org/jenkinsci/plugins/docker/commons/impl/RegistryKeyMaterialFactory.java` public KeyMaterial materialize() throws IOException, InterruptedException { FilePath dockerConfig = createSecretsDirectory(); try { // TODO on Docker 17.07+ use --password-stdin EnvVars envWithConfig = new EnvVars(env); envWithConfig.put( "DOCKER_CONFIG" , dockerConfig.getRemote()); if (launcher.launch().cmds( new ArgumentListBuilder(dockerExecutable, "login" , "-u" , username, "-p" ).add(password, true ).add(endpoint)).envs(envWithConfig).stdout(listener).join() != 0) { throw new AbortException( "docker login failed" ); } } catch (IOException | InterruptedException x) { try { dockerConfig.deleteRecursive(); } catch (Exception x2) { x.addSuppressed(x2); } throw x; } return new RegistryKeyMaterial(dockerConfig, new EnvVars( "DOCKER_CONFIG" , dockerConfig.getRemote())); } Looks like this was intentionally passed over, possibly due to some security issue. jglick  - any reason why this hasn't been updated yet?

          Jesse Glick added a comment -

          danielfosbery not sure offhand. No developer of this plugin that I know of tests against AWS ECR. It may have specialized requirements for running docker login that go beyond what a generic registry does. If in doubt, avoid use of agent docker and run Docker commands directly from sh.

          jarrettg any reason why what has not been updated yet?

          Jesse Glick added a comment - danielfosbery not sure offhand. No developer of this plugin that I know of tests against AWS ECR. It may have specialized requirements for running docker login that go beyond what a generic registry does. If in doubt, avoid use of agent docker and run Docker commands directly from sh . jarrettg any reason why what has not been updated yet?

          Jarrett G added a comment - - edited

          jglick - Wow, I did not describe that well. Sorry for the ambiguity  

           

          I meant that Docker throws an error and exits with a 1 if

          -p

          or

          --password

          is passed in as an arg. It looks like it only wants 

          --password-stdin

          now.

          Jarrett G added a comment - - edited jglick  - Wow, I did not describe that well. Sorry for the ambiguity     I meant that Docker throws an error and exits with a 1 if -p or --password is passed in as an arg. It looks like it only wants  --password-stdin now.

          sv savage added a comment -

          This is as close to my problem that I can find.  I am "new(1 month)" to jenkins.  I have an existing pipeline. that pulls a docker image from an insecure repo.  We are moving to a secure server.   I have copied the images to the new repo.  I can pull from the command line and it prompts from username and password and from then on I can pull and push images. 

          From Jenkins it fails to pull the image.    

          ```

          agent {
             docker {
                args "-v /srv/docker-helper/shares/bin/:/opt/pws/mnt/bin -v /srv/docker-helper/shares/cache/:/opt/pws/mnt/cache -v /srv/docker-helper/shares/inject-home.ro/:/opt/pws/mnt/inject-home:ro"
                image "${jobParams.DOCKER_IMAGE}"
                label "${jobParams.NODE_LABEL}"
                registryCredentialsId 'mig/****** (Jenkins Artifactory user - mig)'
                registryUrl "${jobParams.DOCKER_REGISTRY}"
             }
          }

          ```

          Where "mig/****** (Jenkins Artifactory user - mig)" i the name of a credential in jenkins.

           

          ```

          [Pipeline] withDockerRegistry
          [Pipeline]

          { [Pipeline] sh + docker inspect -f . mig Error: No such object: mig [Pipeline] sh + docker inspect -f . af01p.devtools.tst:6560/mig Error: No such object: af01p.devtools.tst:6560/mig [Pipeline] sh + docker pull af01p.devtools.tst:6560/mig Using default tag: latest Error response from daemon: unauthorized: The client does not have permission for manifest [Pipeline] }

          [Pipeline] // withDockerRegistry

          ```

          sv savage added a comment - This is as close to my problem that I can find.  I am "new(1 month)" to jenkins.  I have an existing pipeline. that pulls a docker image from an insecure repo.  We are moving to a secure server.   I have copied the images to the new repo.  I can pull from the command line and it prompts from username and password and from then on I can pull and push images.  From Jenkins it fails to pull the image.     ``` agent {    docker {       args "-v /srv/docker-helper/shares/bin/:/opt/pws/mnt/bin -v /srv/docker-helper/shares/cache/:/opt/pws/mnt/cache -v /srv/docker-helper/shares/inject-home.ro/:/opt/pws/mnt/inject-home:ro"       image "${jobParams.DOCKER_IMAGE}"       label "${jobParams.NODE_LABEL}"       registryCredentialsId 'mig/****** (Jenkins Artifactory user - mig)'       registryUrl "${jobParams.DOCKER_REGISTRY}"    } } ``` Where "mig/****** (Jenkins Artifactory user - mig)" i the name of a credential in jenkins.   ``` [Pipeline] withDockerRegistry [Pipeline] { [Pipeline] sh + docker inspect -f . mig Error: No such object: mig [Pipeline] sh + docker inspect -f . af01p.devtools.tst:6560/mig Error: No such object: af01p.devtools.tst:6560/mig [Pipeline] sh + docker pull af01p.devtools.tst:6560/mig Using default tag: latest Error response from daemon: unauthorized: The client does not have permission for manifest [Pipeline] } [Pipeline] // withDockerRegistry ```

          Jesse Glick added a comment -

          Again,

          If in doubt, avoid use of agent docker and run Docker commands directly from sh.

          Jesse Glick added a comment - Again, If in doubt, avoid use of agent docker and run Docker commands directly from sh .

            Unassigned Unassigned
            danielfosbery Daniel Fosbery
            Votes:
            6 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: