-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
- Jenkins v2.190.1 / Windows
- Artifactory Plugin v3.4.x
- Artifactory 6.12.2
-
Powered by SuggestiMate
After Updating to the artifactory plugin v3.4.1 all up- or downloads using the declarative steps fail by 401 (upload) / 403 (download) HTTP errors.
Both are working with v3.3.x and broken with 3.4.x.
According the Artifactory logs the steps do not authenticate.
Upload
rtUpload(serverId: 'artifactory-test', specPath: 'spec-ul.json') rtPublishBuildInfo(serverId: 'artifactory-test')
Fails with error 401:
Unauthorized Status code: 401 at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecDeploymentConsumer.consumerRun(SpecDeploymentConsumer.java:44) [...]
Download
rtDownload(serverId: 'artifactory-test', specPath: 'spec-dl.json')
Fails with error 403:
java.io.IOException: Failed to search artifact by the aql 'items.find({"repo": "msys2","path": {"$ne": "."}, "$or": [{"$and": [{"path": { "$match": "distrib"},"name": { "$match": "msys2-x86_64-latest.tar.xz"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")': HTTP/1.1 403 at org.jfrog.build.extractor.clientConfiguration.client.ArtifactoryDependenciesClient.getResponseStream(ArtifactoryDependenciesClient.java:136) [...]
Configuration
Artifactory is configured in the Jenkins settings (id artifactory-test in the examples) and useses the Credentials Plugin to access credentials.
The Test Connection button returns the correct Artifactory version.
Tested version of the plugin
Version | Result |
---|---|
3.3.0 | ![]() |
3.3.1 | ![]() |
3.3.2 | ![]() |
3.4.0 | ![]() |
3.4.1 | ![]() |
Workaround
Downgrade to a v3.3.x release.
- relates to
-
JENKINS-58902 Non-user-scoped credentials are not shown when build authentication is configured
-
- Open
-
-
JENKINS-55624 Authorize Projects plugin causes no git credentials to be found with 'Run as Specific User' Strategy is set
-
- Fixed but Unreleased
-
- links to
[JENKINS-59973] rtUpload / rtDownload fail with HTTP 401 / 403
I have manged to git bisect the causing commit: f2ca1492f64deb34b4cbb1ff286bd94411829bed
ethorsa, could you perhaps elaborate a bit about the credentials used on the repository you are trying to upload to?
I'm the author of that patch, and I have tried to reproduce it, without any luck. However offa seems to be able to reproduce it, so there might be something missing in the example I used to try and reproduce it.
See the link you provided for more information about what I have tried so far. In short, I have tried the maven-example from the jfrog/jenkins-examples repo, against a brand new installation of Artifactory-oss (version 6.16.0 rev 61600900), running in a container. I have configured a repository named 'libs-snapshot-local' (as this is what the example upload spec uses), where anonymous have manage permissions for both repo action and build action.
Thanks for reporting this issue.
We fix this issue in this commit:
https://github.com/jfrog/jenkins-artifactory-plugin/commit/879ba6cb7b5ba14c7ff3ffb4184e4fa249744eb5
Here is a snapshot with the fix:
We'll appreciate your feedback for that!
Unfortunately I still run into the error using the attached snapshot version.
I have setup a minimal example like this:
Pipeline Code
The pipeline code is integrated via multibranch pipeline (fetched from git repository). Beside the Git connection there's nothing configured in the multibranch pipeline project itself.
pipeline { agent any stages { stage("Test") { steps { rtDownload(serverId: 'artifactory-test', specPath: 'spec-dl.json') } } } }
spec-dl.json:
{ "files": [{ "pattern": "jenkins-artifactory-test/*", "target": "jenkins-artifactory-test/", "flat": "true" }] }
The git repository contains the Jenkinsfile and specfile. The job does not use folders – however, a multibranch pipeline itself is some kind of folder.
Artifactory repository
The related artifactory repository jenkins-artifactory-test is of generic type and has permissions set, so all user can do anything. Anonymous user don't have access though.
For this test it contains only a simple example.md file with some chars in it.
Jenkins Configuration
The Artifactory connection is specified in the Jenkins global configuration (Artifactory section):
- Use the Credentials Plugin: yes
- Server ID: artifactory-test (as used in the pipeline)
- Default Deployer Credentials:
- Credentials: A credential stored by the credentials Plugin (Scope: Global, contains username and associated Artifactory API key)
- Use Different Resolver Credentials: no
Pressing the Test Connection button shows "Found Artifactory 6.12.2".
Screenshot of the configuration attached.
Running the Pipeline
Executing the pipeline fails with following error (Blue Ocean, full log attached):
Using spec file: <Workspach Path>\spec-dl.json Searching for artifacts... Failed to search artifact by the aql 'items.find({"repo": "jenkins-artifactory-test","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")': HTTP/1.1 403
There are no related entries in the global Jenkins All Log. Are there any logs that could be helpful?
Versions
- Jenkins: v2.190.3
-
- Artifactory Plugin: Snapshot by yahaviz
- Credentials Plugin: 2.3.0
- Artifactory: 6.12.2
Unfortunately, I didn't manage to reproduce your issue.
I ran this project https://github.com/yahavi/jenkins-reproducer in multibranch pipeline and it worked.
Credentials plugin and Artifactory version as yours. "Allow Anonymous Access" is not checked.
I used this docker image for artifactory: https://bintray.com/jfrog/reg2/jfrog%3Aartifactory-oss/6.12.2
Repository: generic-local
Please look at Artifactory logs, specifically you need the access.log.
You can access Artifactory logs from the UI: https://www.jfrog.com/confluence/display/RTF/Artifactory+Log+Files#ArtifactoryLogFiles-ViewingLogFilesfromtheUI
Looking forward for your feedback!
To narrow down the possible root cause of the issue, can you please try sending this AQL to Artifactory using curl? If you get a 403 response using curl as well, we can take Jenkins out of the equation, and focus on why Artifactory rejects this request.
This is really wired ...
I have tested with Username + API key (Username Password) and API Key only (Secret Text) Credentials.
Interestingly the access.log shows only anonymous access when running the build:
[ACCEPTED LOGIN] for client : anonymous / <IP OF AGENT>.
The connection test in the global config uses the correct user though.
Sending the AQL shown in the error via curl responds the correct Json result. Here's my call:
curl --insecure \ -H 'X-JFrog-Art-Api:<API KEY>' \ -i "https://<HOST>/artifactory/api/search/aql" \ -X POST \ -H 'Content-Type:text/plain' \ --data 'items.find({"repo": "jenkins-artifactory-test","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")'
The API key is the same as configured in the jenkins credentials.
Using the rtServer() yields the same 403 error:
rtServer( id: 'Artifactory-1', url: 'https://<HOST>/artifactory', credentialsId: 'artifactory-test' // Username and api key configured by the credentials plugin ) rtDownload(serverId: 'Artifactory-1', specPath: 'spec-dl.json')
Ah, there might be the mismatch between my setup and yours. I wasn't aware that you were using API keys.
Hopefully I will be able to test that tonight.
My test has also been conducted with yahaviz's example repo: https://github.com/yahavi/jenkins-reproducer
To verify my simple setup outside of Jenkins, using my newly created API_KEY:
$ curl --insecure \ > -H 'X-JFrog-Art-Api:AKCp5e3VEwLXG..vqfWiw1nUUiuEKaMuYBn' \ > -i "http://localhost/artifactory/api/search/aql" \ > -X POST \ > -H 'Content-Type:text/plain' \ > --data 'items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")' HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Tue, 17 Dec 2019 22:16:13 GMT { "results" : [ { "repo" : "generic-local", "path" : ".", "name" : "hello", "type" : "file", "size" : 14, "actual_md5" : "bea8252ff4e80f41719ea13cdf007273", "actual_sha1" : "60fde9c2310b0d4cad4dab8d126b04387efba289" } ], "range" : { "start_pos" : 0, "end_pos" : 1, "total" : 1 } }
Used Versions
Jenkins ver. 2.190.3
Artifactory OSS 6.16.0 rev 61600900
Test matrix
artifactory-jenkins-plugin | Anonymous (Credentials set to '- none -') | User/password | User/API_KEY | API_KEY (SecretText) |
---|---|---|---|---|
3.3.2 | HTTP/1.1 403 Forbidden | ![]() |
![]() |
N/A (not able to select in global plugin server config) |
3.3.x-SNAPSHOT (private-4272760c-reenberg) aka HAP-1219 |
HTTP/1.1 403 Forbidden | ![]() |
![]() |
N/A |
3.4.1 | HTTP/1.1 403 Forbidden | ![]() |
![]() |
N/A |
3.4.x-SNAPSHOT (private-879ba6cb-jenkins) by Yahavi Enables SecretText |
HTTP/1.1 403 Forbidden Logs access.log: 2019-12-17 23:31:14,749 [ACCEPTED LOGIN] for client : anonymous / 172.22.0.3. request.log: 20191217233114|23|REQUEST|172.22.0.3|anonymous|POST|/api/search/aql|HTTP/1.1|403|192 Note: I assume that the anonymous user fails (even though enonymous has READ access and can download through the web interface), as AQL is used here, and thus an actual user is needed? |
![]() |
![]() |
![]() The SecretText credentials are shown, but it fails: HTTP/1.1 401 Unauthorized Logs access.log: 2019-12-17 23:26:16,665 [DENIED LOGIN] for client : NA / 172.22.0.3. request.log: 20191217232616|36|REQUEST|172.22.0.3|non_authenticated_user|POST|/api/search/aql|HTTP/1.1|401|192 NOTE: I have never before seen the 'non_authenticated_user', but maybe this is somewhat new? Anyways its seems that my API_KEY which was pasted as the SecretText credentials are not properly recognized. I have double checked that the API_KEY is pasted correctly. |
Obviously I need to retest this with 6.12 version of Artifactory, but I doubt that is where the issue lies.
ethorsa, I would suggest that you atleast look in the 'access.log' and 'request.log'. What I finds easiest, is to just tail -f ARTI_HOME/logs/*.log. Then add a few newlines to your terminal by pressing enter a few times, and then try your build. Assuming you don't have any other traffic to your Artifactory instance, then you will easily see anything that gets added to any of the log files.
If that is not possible for you, then you would need to fetch them from the web interface, as previously mentioned.
yahaviz, note that I tested your build, but couldn't get it to work. Maybee i derbed something?
I added a new secret, in the global scope:
- Secret: <pasted API_KEY>
- ID: 'api-user_secret'
- Description: 'api-user using API_KEY as secret text'
yahaviz, not to sidetrack this bug report, but I managed to do a TCP dump of the requests, just in case it may clarify why the SecretText failed for me.
I tried to repaste the API_KEY into the SecretText credential, just to make tripple sure, that I didn't make anything wrong.
Using the SecretText:
POST /artifactory/api/search/aql HTTP/1.1 Authorization: Bearer AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property") --- HTTP/1.1 401 Unauthorized Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 WWW-Authenticate: Basic realm="Artifactory Realm" Content-Type: application/json;charset=ISO-8859-1 Content-Length: 171 Date: Wed, 18 Dec 2019 00:42:20 GMT { "errors" : [ { "status" : 401, "message" : "Bad props auth token: basictoken=AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn" } ] }
Using API_KEY as user/pass:
POST /artifactory/api/search/aql HTTP/1.1 Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate Authorization: Basic YXBpLXVzZXI6QUtDcDVlM1ZFd0xYR0tzalB2Qlg2bnY0Q2dQdEdlS0RzVjU3NmM1eEgyOXZpWng3RE5LWnB2cWZXaXcxblVVaXVFS2FNdVlCbg== items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property") --- HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Wed, 18 Dec 2019 00:42:31 GMT ...
Obviously, using user/pass, is the same as user/API_KEY, using the basic auth, except my super secret password
POST /artifactory/api/search/aql HTTP/1.1 Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate Authorization: Basic dXNlcjpwYXNzd29yZA== items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property") --- HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Wed, 18 Dec 2019 00:42:53 GMT ...
I have test the latest v3.5.0 – which should include the snapshot changes – using the three credential types:
- Username / API key
- API key (= secret text)
- Username / encryped password
All remain failing with HTTP/1.1 403 and produce the same log messages:
access.log
2020-01-08 08:22:39,027 [ACCEPTED LOGIN] for client : anonymous / <IP>.
request.log
20200108082239|8|REQUEST|<IP>|anonymous|POST|/api/search/aql|HTTP/1.1|403|203
The test with username / password showed a |11| instead of the |8|, but i don't know what this number stands for.
The overall setup is the same as used for the previous tests. For completeness, here are the used versions:
- Artifactory 6.12.3
- Jenkins 2.190.3
- Artifactory Jenkins Plugin 3.5.0
The curl call mentioned earlier still works when executed from the agent.
Are there any reasonable places to add some logging? I can build the plugin from source, so maybe that's a way to gain more insight.
ethorsa,
I suggest setting an http proxy (such as Postman or Charles) between Jenkins and Artifactory to see what's being sent. You can then compare the payload sent by cUrl vs. Jenkins. This should revel the root cause.
I have probably found a hint to the root cause: The credentials aren't resolved from the it's id.
This came up when I added some additional logging to GenericDownloadExecutor execute() method which logs:
- preferredResolver
- CredentialsId
- Username
- resolverCredentials
- Username
- Password
- AccessToken
A new logger was registered for all Log-Levels of org.jfrog.
After running my test pipeline only the CredentialsId contained the correct value; all others (username, password, access token) were empty.
I can reproduce the same issue on a Linux based Master (v2.204.2, running on OpenJDK11).
There's a Github Issue reporting issues regarding anonymous access when the authorize-project plugin is installed. I have the plugin installed on all instances too (v1.3.0).
Update: I have removed the plugin and restarted the master – still getting 403 error.
Since you mention that you have the authorize-project plugin installed, chances are that you are using some other plugins that might break things?
I would suggest to verify that it fails for you on a clean install. If not, then start adding plugins until you find out which one that might interfere.
I still can't reproduce your results, and I'm doing it in a clean install with only this plugin installed and all its dependencies.
Also note that we are running this in production with god knows which other plugins (though not the authorize-project), and we don't see any issue using credentials.
I have setup a clean Jenkins v2.204.2 without any plugins (not even suggested ones). Next adding necessary plugins (pipeline, git and artifactory support) and start testing using the
pipeline mentioned in earlier. The job succeeded. Gradually adding plugin by plugin with a test after each – still working. Finally this brought up the cause: After installation of the Authorize Project plugin it kept working, but after setting it's strategy to "Run as Specific User" it started failing with the well known error. Changing strategy to "Run as User who Triggered Build" it worked again (however, this breaks functionality with pipelines). See plugins-of-clean-test.txt for a complete list of installed plugins and their versions.
There are already related issues: JENKINS-58902, JENKINS-55624
Could this be a bug in the "authorize-project" plugin? See https://github.com/jfrog/jenkins-artifactory-plugin/issues/247
We ran in the same problem with v3.4.1 for Maven projects in Jenkins 2.190.2 (LTS):
Downgrading to v3.3.x worked around the issue for us too.
I kind of feel this is related to picking up the right credentials.
Our jobs do not 'Override default deployer credentials' and rely on the ~/.m2/settings.xml file being populated with those.
But so far, I've not managed to work around this issue by changing the credential settings of the job.
Looking at the recent commits, I wonder if this one could be the culprit:
[HAP-1219] folder scoped credentials for deploy step (#171)