-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins 2.4
Docker Build and Publish plugin 1.2.2 (+ PR #41)
In an attempt to start moving away from our self-hosted Docker Registry. I came across this plugin to make it easier to push to Amazon ECR. And after a (fairly) quick fix of the Docker Build and Publish plugin. Time had come to make that happen.
But instead I am getting the this error when it attempt to push. So something is wrong.
The push refers to a repository [somerepo.dkr.ecr.eu-west-1.amazonaws.com/imagename] 1b29323a75d2: Preparing 5bf87793f977: Preparing 5ccb950f635d: Preparing 965c3fc60463: Preparing f354df03c5c3: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 9523ecdf69b1: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 6d7b4f405a28: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 099efa904cb9: Preparing 8f83f19c7186: Preparing 1621d30a7846: Preparing e989ce4ed35e: Preparing ae30a2e42fe4: Preparing 461f75075df2: Preparing 5f70bf18a086: Preparing 5f70bf18a086: Preparing 6d7b4f405a28: Waiting 099efa904cb9: Waiting 8f83f19c7186: Waiting 1621d30a7846: Waiting e989ce4ed35e: Waiting ae30a2e42fe4: Waiting 461f75075df2: Waiting 5f70bf18a086: Waiting 9523ecdf69b1: Waiting f354df03c5c3: Image push failed f354df03c5c3: Image push failed 461f75075df2: Waiting ae30a2e42fe4: Waiting e989ce4ed35e: Waiting 1621d30a7846: Waiting 8f83f19c7186: Waiting 099efa904cb9: Waiting 6d7b4f405a28: Waiting 9523ecdf69b1: Waiting 5f70bf18a086: Waiting Error parsing HTTP response: invalid character 'Y' looking for beginning of value: "Your Authorization Token has expired. Please run 'aws ecr get-login' to fetch a new one." Build step 'Docker Build and Publish' marked build as failure
In the panel for updating the credentials I also get the message:
These credentials are valid but do not have access to the "AmazonEC2" service in the region "us-east-1". This message is not a problem if you need to access to other services or to other regions. Message: "You are not authorized to perform this operation. (UnauthorizedOperation)"
But I am using the AWS Managed policy "AmazonEC2ContainerRegistryPowerUser" to grant Jenkins access. And should pretty much have full access to all the ECR calls it needs. We do use it in 'eu-west-1' though. But the warning clearly states not to worry about it if we are not in that region.
- is related to
-
-
- Reopened
-
-
-
- Resolved
-
[JENKINS-34958] Getting "Your Authorization Token has expired" when using ECR credentials
Kristoffer Peterhänsel
created issue -
Vesa Alho
added a comment - I can confirm this issue. Works in us-east-1, but not eg. in eu-west-1. Faced this same issue today with this plugin.
Configuring ECR in us-east-1 region ( **.dkr.ecr.us-east-1.amazonaws.com/** ) the push succeeds.
But using ECR in eu-west-1 ( **.dkr.ecr.eu-west-1.amazonaws.com/** ) it fails: with an error:
error parsing HTTP 403 response body: unexpected end of JSON input: ""
Build step 'Docker Build and Publish' marked build as failure
It's the exact same job and settings, only changing the registry endpoint in the job.
Using the command line to issue a push with the token obtained by the jenkins job (set in ~/.docker/config.json) also fails, but using the aws cli to get the token, issuing a docker login and then push works fine regardless of the region.
That leads to me to believe the problem is somewhere in the token / credentials issue code as you mentioned.
Any ETA on the fix for this problem?
Docker version 1.11.1, build 5604cbe/1.11.1
Amazon ECR plugin: 1.3
CloudBees Docker Build and Publish plugin: 1.2.2
Docker Commons Plugin: 1.3.1
Luis Silva
added a comment - - edited Faced this same issue today with this plugin.
Configuring ECR in us-east-1 region ( ** .dkr.ecr.us-east-1.amazonaws.com/ ** ) the push succeeds.
But using ECR in eu-west-1 ( ** .dkr.ecr.eu-west-1.amazonaws.com/ ** ) it fails: with an error:
error parsing HTTP 403 response body: unexpected end of JSON input: ""
Build step 'Docker Build and Publish' marked build as failure
It's the exact same job and settings, only changing the registry endpoint in the job.
Using the command line to issue a push with the token obtained by the jenkins job (set in ~/.docker/config.json) also fails, but using the aws cli to get the token, issuing a docker login and then push works fine regardless of the region.
That leads to me to believe the problem is somewhere in the token / credentials issue code as you mentioned.
Any ETA on the fix for this problem?
Docker version 1.11.1, build 5604cbe/1.11.1
Amazon ECR plugin: 1.3
CloudBees Docker Build and Publish plugin: 1.2.2
Docker Commons Plugin: 1.3.1 
Tommy McNeely
added a comment - I confirm that we are also having this problem with us-west-2. I even tried doing the docker login command from ecr get-login as the jenkins user (and tested with a manual docker push), expecting that it might "see" that the credentials were already there, but it appears to overwrite the credentials regardless of their age/validity. I am thinking that the region might have to be assigned to the credential?
Tommy
Tommy McNeely
added a comment - I confirm that we are also having this problem with us-west-2. I even tried doing the docker login command from ecr get-login as the jenkins user (and tested with a manual docker push), expecting that it might "see" that the credentials were already there, but it appears to overwrite the credentials regardless of their age/validity. I am thinking that the region might have to be assigned to the credential?
Tommy 
Arron Woods
added a comment - We're using eu-west-1 and experiencing the same.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/ecr/AmazonECRClient.html
It looks like the AmazonEcrClient can accept a region ( AmazonEcrClient::configureRegion() ), but not sure where this would need to set in Jenkins?
Should the AwsCredentials plugin have a region parameter, or would it be set on the repository somewhere? If using the AWS url, the region could be extracted from the URL. Wouldn't help if using your own domain though.
Arron Woods
added a comment - We're using eu-west-1 and experiencing the same.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/ecr/AmazonECRClient.html
It looks like the AmazonEcrClient can accept a region ( AmazonEcrClient::configureRegion() ), but not sure where this would need to set in Jenkins?
Should the AwsCredentials plugin have a region parameter, or would it be set on the repository somewhere? If using the AWS url, the region could be extracted from the URL. Wouldn't help if using your own domain though. 
Alex Richards
added a comment - I'm also getting this issue when trying to use EU-WEST-1.
If I change my settings to use US-EAST-1 then it works ok.
Alex Richards
added a comment - I'm also getting this issue when trying to use EU-WEST-1.
If I change my settings to use US-EAST-1 then it works ok. The only way I could get around this issue was to add a shell command to a previous step with:
eval `aws ecr get-login --region eu-west-1`
Alex Richards
added a comment - - edited The only way I could get around this issue was to add a shell command to a previous step with:
eval `aws ecr get-login --region eu-west-1`
I tryed not to use the aws configuration on the server, so:
- add the AWS Credential to Jenkins
- in the build:
- Build Environment -> Use secret text(s) or file(s) -> AWS
- Build -> Execute Shell
export AWS_DEFAULT_REGION='eu-west-1' eval `aws ecr get-login`
- Build -> Docker Build & Publish -> do not put the registry credentials
In this way my build is ok!!!!
Hope it helped, I spent two days searching a solution!!!
| Workflow | Original: JNJira [ 171162 ] | New: JNJira + In-Review [ 184200 ] |
So I can't quite manually provoke this error message using the AWS CLI tool and docker. But it does appear that the credentials issued are regional. So it seems this plugin may need a way to set the desired region. But so far I haven't found the place in the AmazonECRClient where you can actually specify region...
Edit: And then of course I did find see it.