Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51615

Docker Compose with Amazon ECR Not Working on Remote Server

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • docker-workflow-plugin
    • None
    • Jenkins Version: 2.107.3
      Amazon ECR Plugin Version: 1.6

      Greetings, I am not sure if this is an issue with the Amazon ECR plugin or the docker workflow system, or something that is just not supported.  I have a working Amazon ECR / Docker setup going and I am able to publish to ECR from Jenkins like so: 

      node {
        stage ('Checkout') {
          checkout scm
        }
      
        stage('Build') {
          docker.withRegistry('https://XXXXX.dkr.ecr.us-west-2.amazonaws.com', 'ecr:us-west-2:JenkinsAWS') {
            withDockerContainer("XXXXX.dkr.ecr.us-west-2.amazonaws.com/node-builder") {
              // some build steps
            }
          }
        }
      
        stage('Docker Build') {
          sh '''
          docker build --file docker/Dockerfile --tag mycontainer .
          docker tag mycontainer XXXXX.dkr.ecr.us-west-2.amazonaws.com/mycontainer
          '''
        }
      
        stage('Docker Push') {
            docker.withRegistry('https://XXXXX.dkr.ecr.us-west-2.amazonaws.com', 'ecr:us-west-2:JenkinsAWS') {
              sh '''
              docker push XXXXX.dkr.ecr.us-west-2.amazonaws.com/mycontainer:latest
              '''
            }
          }
      }
      
      

      The above works as expected. However in another job, I am trying to run docker compose to "up" the container like this:

      node {
        stage ('Checkout') {
          checkout scm
        }
      
        stage('Compose Up') {
          docker.withServer('tcp://somehost:2376', 'JenkinsDockerClientCert') {
            docker.withRegistry('https://XXXXX.dkr.ecr.us-west-2.amazonaws.com', 'ecr:us-west-2:JenkinsAWS') {
              sh '''
              docker-compose -f docker-compose/docker-compose.yml -p myapp up -d
              '''
            }
          }
        }
      }
      

      This time I get the following output: 

      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] stage
      [Pipeline] { (Compose Up)
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] withDockerRegistry
      $ docker login -u AWS -p ******** https://XXXXX.dkr.ecr.us-west-2.amazonaws.com
      WARNING! Using --password via the CLI is insecure. Use --password-stdin.
      WARNING! Your password will be stored unencrypted in /var/jenkins/workspace/dev-tasks/Docker Compose Up@tmp/ddaeea78-470c-411a-a7d3-412fc18cdf0a/config.json.
      Configure a credential helper to remove this warning. See
      https://docs.docker.com/engine/reference/commandline/login/#credentials-store
      
      Login Succeeded
      [Pipeline] {
      [Pipeline] withDockerServer
      [Pipeline] {
      [Pipeline] sh
      [Docker Compose Up] Running shell script
      + docker-compose -f docker-compose/docker-compose.yml -p myapp up -d
      Pulling mycontainer (XXXXX.dkr.ecr.us-east-2.amazonaws.com/mycontainer)...
      Get https://XXXXX.dkr.ecr.us-east-2.amazonaws.com/v2/mycontainer/manifests/latest: no basic auth credentials
      [Pipeline] }
      [Pipeline] // withDockerServer
      [Pipeline] }
      [Pipeline] // withDockerRegistry
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      

      Docker Compose is obviously installed on the build agent, but we are pointing to a remote docker host. So we know docker compose is running on the build agent and that is probably where the ECR credentials are getting written.. hover the remote host does not seem to get the benefit of the "withRegistry" call. Perhaps I am doing something wrong.  Is this even a supported use case? If not are there any workarounds?

      Thanks.

       

          [JENKINS-51615] Docker Compose with Amazon ECR Not Working on Remote Server

          Is there any other information or trouble shooting I could do to help out with this issue? If there indeed is an issue and it is not simply operator error!

          Michael MacFadden added a comment - Is there any other information or trouble shooting I could do to help out with this issue? If there indeed is an issue and it is not simply operator error!

          Ivan Fernandez Calvo added a comment - - edited

          It is not related with Amazon ECR Plugin, but I think that it is the same issue in -JENKINS-44143- , it is related to old credentials in user docker configuration file, the file depends on the version of docker test something like that.

          node {
          //cleanup current user docker credentials
          sh 'rm ~/.dockercfg || true'
          sh 'rm ~/.docker/config.json || true'
          
          //configure registry
          docker.withRegistry('https://ID.ecr.eu-west-1.amazonaws.com', 'ecr:eu-west-1:86c8f5ec-1ce1-4e94-80c2-18e23bbd724a') {
          
          //build image
          def customImage = docker.build("my-image:${env.BUILD_ID}")
          
          //push image
          customImage.push()
          }

           

          Ivan Fernandez Calvo added a comment - - edited It is not related with Amazon ECR Plugin, but I think that it is the same issue in - JENKINS-44143 - , it is related to old credentials in user docker configuration file, the file depends on the version of docker test something like that. node { //cleanup current user docker credentials sh 'rm ~/.dockercfg || true ' sh 'rm ~/.docker/config.json || true ' //configure registry docker.withRegistry( 'https: //ID.ecr.eu-west-1.amazonaws.com' , 'ecr:eu-west-1:86c8f5ec-1ce1-4e94-80c2-18e23bbd724a' ) { //build image def customImage = docker.build( "my-image:${env.BUILD_ID}" ) //push image customImage.push() }  

          Yasuo Nakanishi added a comment - - edited

          Is it true that the cause is old credentials? I believe that the accessing URL is invalid.

          "Get https://XXXXX.dkr.ecr.us-east-2.amazonaws.com/v2/mycontainer/manifests/latest"

          Why has "v2" added to URL?

          My pipeline setting fails to login with Docker Pipeline with 1.17. same setting working with 1.15.1

           

          withDockerRegistry([credentialsId: 'AWS ECS Repository', url: 'https://ID.dkr.ecr.ap-northeast-1.amazonaws.com']) {
          }
          

          log with 1.17. 

          $ docker login -u AWS -p ******** https://ID.dkr.ecr.ap-northeast-1.amazonaws.com
          WARNING! Using --password via the CLI is insecure. Use --password-stdin.
          Error response from daemon: login attempt to https://ID.dkr.ecr.ap-northeast-1.amazonaws.com/v2/ failed with status: 403 Forbidden
          

          Likewise, I think that "v2" is added to the URL, and that is why it fails to login.

          Yasuo Nakanishi added a comment - - edited Is it true that the cause is old credentials? I believe that the accessing URL is invalid. "Get https://XXXXX.dkr.ecr.us-east-2.amazonaws.com/v2/mycontainer/manifests/latest " Why has "v2" added to URL? My pipeline setting fails to login with Docker Pipeline with 1.17. same setting working with 1.15.1   withDockerRegistry([credentialsId: 'AWS ECS Repository' , url: 'https: //ID.dkr.ecr.ap-northeast-1.amazonaws.com' ]) { } log with 1.17.  $ docker login -u AWS -p ******** https: //ID.dkr.ecr.ap-northeast-1.amazonaws.com WARNING! Using --password via the CLI is insecure. Use --password-stdin. Error response from daemon: login attempt to https: //ID.dkr.ecr.ap-northeast-1.amazonaws.com/v2/ failed with status: 403 Forbidden Likewise, I think that "v2" is added to the URL, and that is why it fails to login.

          Ivan Fernandez Calvo added a comment - - edited

          see JENKINS-51949, it is the same issue, it is not related to Amazon ECR Plugin

          Ivan Fernandez Calvo added a comment - - edited see  JENKINS-51949 , it is the same issue, it is not related to Amazon ECR Plugin

            Unassigned Unassigned
            mmacfadden Michael MacFadden
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: