Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59973

rtUpload / rtDownload fail with HTTP 401 / 403

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • artifactory-plugin
    • None
    • - Jenkins v2.190.1 / Windows
      - Artifactory Plugin v3.4.x
      - Artifactory 6.12.2

      After Updating to the artifactory plugin v3.4.1 all up- or downloads using the declarative steps fail by 401 (upload) / 403 (download) HTTP errors.

      Both are working with v3.3.x and broken with 3.4.x.

      According the Artifactory logs the steps do not authenticate. 

       

      Upload

      rtUpload(serverId: 'artifactory-test', specPath: 'spec-ul.json')
      rtPublishBuildInfo(serverId: 'artifactory-test')
      

      Fails with error 401:

      Unauthorized Status code: 401
          at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecDeploymentConsumer.consumerRun(SpecDeploymentConsumer.java:44)
      [...]
      

       

      Download

      rtDownload(serverId: 'artifactory-test', specPath: 'spec-dl.json')  

      Fails with error 403:

      java.io.IOException: Failed to search artifact by the aql 'items.find({"repo": "msys2","path": {"$ne": "."}, "$or": [{"$and": [{"path": { "$match": "distrib"},"name": { "$match": "msys2-x86_64-latest.tar.xz"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")': HTTP/1.1 403 
          at org.jfrog.build.extractor.clientConfiguration.client.ArtifactoryDependenciesClient.getResponseStream(ArtifactoryDependenciesClient.java:136)
      [...]
      

      Configuration

      Artifactory is configured in the Jenkins settings (id artifactory-test in the examples) and useses the Credentials Plugin to access credentials.

      The Test Connection button returns the correct Artifactory version.

       

      Tested version of the plugin

       

      Version Result
      3.3.0
      3.3.1
      3.3.2
      3.4.0
      3.4.1

       

      Workaround

      Downgrade to a v3.3.x release.

          [JENKINS-59973] rtUpload / rtDownload fail with HTTP 401 / 403

          We ran in the same problem with v3.4.1 for Maven projects in Jenkins 2.190.2 (LTS):

          10:05:03 [INFO] Deploying artifact: https://xxx.jar
          10:05:06 [ERROR] org.jfrog.build.extractor.maven.BuildInfoRecorder.sessionEnded() listener has failed: 
          10:05:06 java.lang.RuntimeException: Error occurred while publishing artifact to Artifactory: /var/lib/jenkins/workspace/Card Node Snapshot/xxx.jar.
          10:05:06  Skipping deployment of remaining artifacts (if any) and build info.
          10:05:06     at org.jfrog.build.extractor.maven.BuildDeploymentHelper.deployArtifacts (BuildDeploymentHelper.java:303)
          10:05:06     at org.jfrog.build.extractor.maven.BuildDeploymentHelper.deploy (BuildDeploymentHelper.java:103)
          10:05:06     at org.jfrog.build.extractor.maven.BuildInfoRecorder.sessionEnded (BuildInfoRecorder.java:173)
          10:05:06     at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire (DefaultExecutionEventCatapult.java:64)
          10:05:06     at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire (DefaultExecutionEventCatapult.java:42)
          10:05:06     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:137)
          10:05:06     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
          10:05:06     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
          10:05:06     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
          10:05:06     at org.jvnet.hudson.maven3.launcher.Maven35Launcher.main (Maven35Launcher.java:130)
          ...
          10:05:06 Caused by: java.io.IOException: Failed to deploy file. Status code: 401 Response message: Artifactory returned the following errors: 
          10:05:06 Unauthorized Status code: 401
          

          Downgrading to v3.3.x worked around the issue for us too.

          I kind of feel this is related to picking up the right credentials.
          Our jobs do not 'Override default deployer credentials' and rely on the ~/.m2/settings.xml file being populated with those.
          But so far, I've not managed to work around this issue by changing the credential settings of the job.

          Looking at the recent commits, I wonder if this one could be the culprit:

          [HAP-1219] folder scoped credentials for deploy step (#171)

          Benoit Donneaux added a comment - We ran in the same problem with v3.4.1 for Maven projects in Jenkins 2.190.2 (LTS): 10:05:03 [INFO] Deploying artifact: https://xxx.jar 10:05:06 [ERROR] org.jfrog.build.extractor.maven.BuildInfoRecorder.sessionEnded() listener has failed: 10:05:06 java.lang.RuntimeException: Error occurred while publishing artifact to Artifactory: /var/lib/jenkins/workspace/Card Node Snapshot/xxx.jar. 10:05:06 Skipping deployment of remaining artifacts (if any) and build info. 10:05:06 at org.jfrog.build.extractor.maven.BuildDeploymentHelper.deployArtifacts (BuildDeploymentHelper.java:303) 10:05:06 at org.jfrog.build.extractor.maven.BuildDeploymentHelper.deploy (BuildDeploymentHelper.java:103) 10:05:06 at org.jfrog.build.extractor.maven.BuildInfoRecorder.sessionEnded (BuildInfoRecorder.java:173) 10:05:06 at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire (DefaultExecutionEventCatapult.java:64) 10:05:06 at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire (DefaultExecutionEventCatapult.java:42) 10:05:06 at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:137) 10:05:06 at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) 10:05:06 at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) 10:05:06 at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) 10:05:06 at org.jvnet.hudson.maven3.launcher.Maven35Launcher.main (Maven35Launcher.java:130) ... 10:05:06 Caused by: java.io.IOException: Failed to deploy file. Status code: 401 Response message: Artifactory returned the following errors: 10:05:06 Unauthorized Status code: 401 Downgrading to v3.3.x worked around the issue for us too. I kind of feel this is related to picking up the right credentials. Our jobs do not 'Override default deployer credentials' and rely on the ~/.m2/settings.xml file being populated with those. But so far, I've not managed to work around this issue by changing the credential settings of the job. Looking at the recent commits, I wonder if this one could be the culprit: [HAP-1219] folder scoped credentials for deploy step (#171)

          ethorsa added a comment -

          I have manged to git bisect the causing commit: f2ca1492f64deb34b4cbb1ff286bd94411829bed

          ethorsa added a comment - I have manged to git bisect the causing commit: f2ca1492f64deb34b4cbb1ff286bd94411829bed

          ethorsa, could you perhaps elaborate a bit about the credentials used on the repository you are trying to upload to?

          I'm the author of that patch, and I have tried to reproduce it, without any luck.  However offa seems to be able to reproduce it, so there might be something missing in the example I used to try and reproduce it.

          See the link you provided for more information about what I have tried so far.  In short, I have tried the maven-example from the jfrog/jenkins-examples repo, against a brand new installation of Artifactory-oss (version 6.16.0 rev 61600900), running in a container.  I have configured a repository named 'libs-snapshot-local' (as this is what the example upload spec uses), where anonymous have manage permissions for both repo action and build action.

          Jesper Reenberg added a comment - ethorsa , could you perhaps elaborate a bit about the credentials used on the repository you are trying to upload to? I'm the author of that patch, and I have tried to reproduce it, without any luck.  However offa seems to be able to reproduce it, so there might be something missing in the example I used to try and reproduce it. See the link you provided for more information about what I have tried so far.  In short, I have tried the maven-example from the jfrog/jenkins-examples repo, against a brand new installation of Artifactory-oss (version 6.16.0 rev 61600900), running in a container.  I have configured a repository named 'libs-snapshot-local' (as this is what the example upload spec uses), where anonymous have manage permissions for both repo action and build action.

          Yahav Itzhak added a comment -

          Yahav Itzhak added a comment - ethorsa bdonneaux Thanks for reporting this issue. We fix this issue in this commit: https://github.com/jfrog/jenkins-artifactory-plugin/commit/879ba6cb7b5ba14c7ff3ffb4184e4fa249744eb5 Here is a snapshot with the fix: https://oss.jfrog.org/oss-snapshot-local/org/jenkins-ci/plugins/artifactory/3.4.x-SNAPSHOT/artifactory-3.4.x-20191208.120519-26.hpi We'll appreciate your feedback for that!

          ethorsa added a comment - - edited

          Unfortunately I still run into the error using the attached snapshot version.

          I have setup a minimal example like this:

          Pipeline Code

          The pipeline code is integrated via multibranch pipeline (fetched from git repository). Beside the Git connection there's nothing configured in the multibranch pipeline project itself.

           

          pipeline {
              agent any
          
              stages {
                  stage("Test") {
                      steps {
                          rtDownload(serverId: 'artifactory-test', specPath: 'spec-dl.json')
                      }
                  }
          
              }
          }
          

           

          spec-dl.json:

          {
              "files": [{
                  "pattern": "jenkins-artifactory-test/*",
                  "target": "jenkins-artifactory-test/",
                  "flat": "true"
              }]
          }

           

          The git repository contains the Jenkinsfile and specfile. The job does not use folders – however, a multibranch pipeline itself is some kind of folder.

           

          Artifactory repository

          The related artifactory repository jenkins-artifactory-test is of generic type and has permissions set, so all user can do anything. Anonymous user don't have access though.

          For this test it contains only a simple example.md file with some chars in it.

           

          Jenkins Configuration

          The Artifactory connection is specified in the Jenkins global configuration (Artifactory section):

          • Use the Credentials Plugin: yes
          • Server ID: artifactory-test (as used in the pipeline)
          • Default Deployer Credentials:
            • Credentials: A credential stored by the credentials Plugin (Scope: Global, contains username and associated Artifactory API key)
          • Use Different Resolver Credentials: no

          Pressing the Test Connection button shows "Found Artifactory 6.12.2".

          Screenshot of the configuration attached.

          Running the Pipeline

          Executing the pipeline fails with following error (Blue Ocean, full log attached):

           

          Using spec file: <Workspach Path>\spec-dl.json
          
          Searching for artifacts...
          
          Failed to search artifact by the aql 'items.find({"repo": "jenkins-artifactory-test","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")': HTTP/1.1 403
          

           

          There are no related entries in the global Jenkins All Log. Are there any logs that could be helpful?

           

          Versions

          • Jenkins: v2.190.3
            • Artifactory Plugin: Snapshot by yahaviz
            • Credentials Plugin: 2.3.0
          • Artifactory: 6.12.2

           

          ethorsa added a comment - - edited Unfortunately I still run into the error using the attached snapshot version. I have setup a minimal example like this: Pipeline Code The pipeline code is integrated via multibranch pipeline (fetched from git repository). Beside the Git connection there's nothing configured in the multibranch pipeline project itself.   pipeline { agent any stages { stage( "Test" ) { steps { rtDownload(serverId: 'artifactory-test' , specPath: 'spec-dl.json' ) } } } }   spec-dl.json: { "files" : [{ "pattern" : "jenkins-artifactory-test/*" , "target" : "jenkins-artifactory-test/" , "flat" : " true " }] }   The git repository contains the Jenkinsfile and specfile. The job does not use folders – however, a multibranch pipeline itself is some kind of folder.   Artifactory repository The related artifactory repository jenkins-artifactory-test is of generic type and has permissions set, so all user can do anything. Anonymous user don't have access though. For this test it contains only a simple example.md file with some chars in it.   Jenkins Configuration The Artifactory connection is specified in the Jenkins global configuration (Artifactory section): Use the Credentials Plugin: yes Server ID: artifactory-test (as used in the pipeline) Default Deployer Credentials: Credentials: A credential stored by the credentials Plugin (Scope: Global, contains username and associated Artifactory API key) Use Different Resolver Credentials: no Pressing the Test Connection button shows "Found Artifactory 6.12.2" . Screenshot of the configuration attached. Running the Pipeline Executing the pipeline fails with following error (Blue Ocean, full log attached):   Using spec file: <Workspach Path>\spec-dl.json Searching for artifacts... Failed to search artifact by the aql 'items.find({"repo": "jenkins-artifactory-test","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")': HTTP/1.1 403   There are no related entries in the global Jenkins All Log. Are there any logs that could be helpful?   Versions Jenkins: v2.190.3 Artifactory Plugin: Snapshot by yahaviz Credentials Plugin: 2.3.0 Artifactory: 6.12.2  

          Yahav Itzhak added a comment -

          ethorsa,

          Unfortunately, I didn't manage to reproduce your issue.

          I ran this project https://github.com/yahavi/jenkins-reproducer in multibranch pipeline and it worked.

          Credentials plugin and Artifactory version as yours. "Allow Anonymous Access" is not checked.

          I used this docker image for artifactory: https://bintray.com/jfrog/reg2/jfrog%3Aartifactory-oss/6.12.2

          Repository: generic-local

           

          Please look at Artifactory logs, specifically you need the access.log.

          You can access Artifactory logs from the UI: https://www.jfrog.com/confluence/display/RTF/Artifactory+Log+Files#ArtifactoryLogFiles-ViewingLogFilesfromtheUI

          Looking forward for your feedback!

          Yahav Itzhak added a comment - ethorsa , Unfortunately, I didn't manage to reproduce your issue. I ran this project https://github.com/yahavi/jenkins-reproducer in multibranch pipeline and it worked. Credentials plugin and Artifactory version as yours. "Allow Anonymous Access" is not checked. I used this docker image for artifactory: https://bintray.com/jfrog/reg2/jfrog%3Aartifactory-oss/6.12.2 Repository: generic-local   Please look at Artifactory logs, specifically you need the access.log. You can access Artifactory logs from the UI: https://www.jfrog.com/confluence/display/RTF/Artifactory+Log+Files#ArtifactoryLogFiles-ViewingLogFilesfromtheUI Looking forward for your feedback!

          To narrow down the possible root cause of the issue, can you please try sending this AQL to Artifactory using curl? If you get a 403 response using curl as well, we can take Jenkins out of the equation, and focus on why Artifactory rejects this request. 

          Eyal Ben Moshe added a comment - To narrow down the possible root cause of the issue, can you please try sending this AQL to Artifactory using curl? If you get a 403 response using curl as well, we can take Jenkins out of the equation, and focus on why Artifactory rejects this request. 

          ethorsa added a comment -

          This is really wired ... 

           

          I have tested with Username + API key (Username Password) and API Key only (Secret Text) Credentials.

          Interestingly the access.log shows only anonymous access when running the build:

          [ACCEPTED LOGIN]   for client : anonymous / <IP OF AGENT>. 

          The connection test in the global config uses the correct user though.

           

          Sending the AQL shown in the error via curl responds the correct Json result. Here's my call:

          curl --insecure \
              -H 'X-JFrog-Art-Api:<API KEY>' \
              -i "https://<HOST>/artifactory/api/search/aql" \
              -X POST \
              -H 'Content-Type:text/plain' \
              --data 'items.find({"repo": "jenkins-artifactory-test","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")'

          The API key is the same as configured in the jenkins credentials.

          ethorsa added a comment - This is really wired ...    I have tested with Username + API key ( Username Password ) and API Key only ( Secret Text ) Credentials. Interestingly the access.log shows only anonymous access when running the build: [ACCEPTED LOGIN] for client : anonymous / <IP OF AGENT>. The connection test in the global config uses the correct user though.   Sending the AQL shown in the error via curl responds the correct Json result. Here's my call: curl --insecure \ -H 'X-JFrog-Art-Api:<API KEY>' \ -i "https://<HOST>/artifactory/api/search/aql" \ -X POST \ -H 'Content-Type:text/plain' \ --data 'items.find({"repo": "jenkins-artifactory-test","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")' The API key is the same as configured in the jenkins credentials.

          ethorsa added a comment -

          Using the rtServer() yields the same 403 error:

           

          rtServer(
              id: 'Artifactory-1',
              url: 'https://<HOST>/artifactory',
              credentialsId: 'artifactory-test' // Username and api key configured by the credentials plugin
          )
          
          rtDownload(serverId: 'Artifactory-1', specPath: 'spec-dl.json')
          

          ethorsa added a comment - Using the rtServer() yields the same 403 error:   rtServer( id: 'Artifactory-1' , url: 'https: //<HOST>/artifactory' , credentialsId: 'artifactory-test' // Username and api key configured by the credentials plugin ) rtDownload(serverId: 'Artifactory-1' , specPath: 'spec-dl.json' )

          Ah, there might be the mismatch between my setup and yours.  I wasn't aware that you were using API keys. 

          Hopefully I will be able to test that tonight.

          Jesper Reenberg added a comment - Ah, there might be the mismatch between my setup and yours.  I wasn't aware that you were using API keys.  Hopefully I will be able to test that tonight.

          Jesper Reenberg added a comment - - edited

          My test has also been conducted with yahaviz's example repo: https://github.com/yahavi/jenkins-reproducer

          To verify my simple setup outside of Jenkins, using my newly created API_KEY:

          $ curl --insecure \
          >     -H 'X-JFrog-Art-Api:AKCp5e3VEwLXG..vqfWiw1nUUiuEKaMuYBn' \
          >     -i "http://localhost/artifactory/api/search/aql" \
          >     -X POST \
          >     -H 'Content-Type:text/plain' \
          >     --data 'items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")'
          HTTP/1.1 200 OK
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          Content-Type: application/json
          Transfer-Encoding: chunked
          Date: Tue, 17 Dec 2019 22:16:13 GMT
          
          
          {
          "results" : [ {
            "repo" : "generic-local",
            "path" : ".",
            "name" : "hello",
            "type" : "file",
            "size" : 14,
            "actual_md5" : "bea8252ff4e80f41719ea13cdf007273",
            "actual_sha1" : "60fde9c2310b0d4cad4dab8d126b04387efba289"
          } ],
          "range" : {
            "start_pos" : 0,
            "end_pos" : 1,
            "total" : 1
          }
          }
          

           

          Used Versions

          Jenkins ver. 2.190.3

          Artifactory OSS 6.16.0 rev 61600900

          Test matrix

          artifactory-jenkins-plugin Anonymous (Credentials set to '- none -') User/password User/API_KEY API_KEY (SecretText)
          3.3.2 HTTP/1.1 403 Forbidden N/A (not able to select in global plugin server config)
          3.3.x-SNAPSHOT
          (private-4272760c-reenberg)
          aka HAP-1219
          HTTP/1.1 403 Forbidden  N/A
          3.4.1 HTTP/1.1 403 Forbidden N/A
          3.4.x-SNAPSHOT (private-879ba6cb-jenkins) by Yahavi
          Enables SecretText
          HTTP/1.1 403 Forbidden
           
          Logs
           
          access.log:
          2019-12-17 23:31:14,749 [ACCEPTED LOGIN]   for client : anonymous / 172.22.0.3.
           
          request.log:
          20191217233114|23|REQUEST|172.22.0.3|anonymous|POST|/api/search/aql|HTTP/1.1|403|192
           
          Note: I assume that the anonymous user fails (even though enonymous has READ access and can download through the web interface), as AQL is used here, and thus an actual user is needed?

          The SecretText credentials are shown, but it fails: HTTP/1.1 401 Unauthorized
           
          Logs 
           
          access.log:
          2019-12-17 23:26:16,665 [DENIED LOGIN]   for client : NA / 172.22.0.3.
           
          request.log:
          20191217232616|36|REQUEST|172.22.0.3|non_authenticated_user|POST|/api/search/aql|HTTP/1.1|401|192
           
          NOTE: I have never before seen the 'non_authenticated_user', but maybe this is somewhat new?  Anyways its seems that my API_KEY which was pasted as the SecretText credentials are not properly recognized.  I have double checked that the API_KEY is pasted correctly.

           

           Obviously I need to retest this with 6.12 version of Artifactory, but I doubt that is where the issue lies.

           

          ethorsa, I would suggest that you atleast look in the 'access.log' and 'request.log'.  What I finds easiest, is to just tail -f ARTI_HOME/logs/*.log. Then add a few newlines to your terminal by pressing enter a few times, and then try your build. Assuming you don't have any other traffic to your Artifactory instance, then you will easily see anything that gets added to any of the log files.
          If that is not possible for you, then you would need to fetch them from the web interface, as previously mentioned.

           

          yahaviz, note that I tested your build, but couldn't get it to work.  Maybee i derbed something?

          I added a new secret, in the global scope:

          • Secret: <pasted API_KEY>
          • ID: 'api-user_secret'
          • Description: 'api-user using API_KEY as secret text'

          Jesper Reenberg added a comment - - edited My test has also been conducted with yahaviz 's example repo:  https://github.com/yahavi/jenkins-reproducer To verify my simple setup outside of Jenkins, using my newly created API_KEY: $ curl --insecure \ > -H 'X-JFrog-Art-Api:AKCp5e3VEwLXG..vqfWiw1nUUiuEKaMuYBn' \ > -i "http: //localhost/artifactory/api/search/aql" \ > -X POST \ > -H 'Content-Type:text/plain' \ > --data 'items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" )' HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Tue, 17 Dec 2019 22:16:13 GMT { "results" : [ { "repo" : " generic -local" , "path" : "." , "name" : "hello" , "type" : "file" , "size" : 14, "actual_md5" : "bea8252ff4e80f41719ea13cdf007273" , "actual_sha1" : "60fde9c2310b0d4cad4dab8d126b04387efba289" } ], "range" : { "start_pos" : 0, "end_pos" : 1, "total" : 1 } }   Used Versions Jenkins ver. 2.190.3 Artifactory OSS 6.16.0 rev 61600900 Test matrix artifactory-jenkins-plugin Anonymous (Credentials set to '- none -') User/password User/API_KEY API_KEY (SecretText) 3.3.2 HTTP/1.1 403 Forbidden N/A (not able to select in global plugin server config) 3.3.x-SNAPSHOT (private-4272760c-reenberg) aka HAP-1219 HTTP/1.1 403 Forbidden  N/A 3.4.1 HTTP/1.1 403 Forbidden N/A 3.4.x-SNAPSHOT (private-879ba6cb-jenkins) by Yahavi Enables SecretText HTTP/1.1 403 Forbidden   Logs   access.log: 2019-12-17 23:31:14,749 [ACCEPTED LOGIN]   for client : anonymous / 172.22.0.3.   request.log: 20191217233114|23|REQUEST|172.22.0.3|anonymous|POST|/api/search/aql|HTTP/1.1|403|192   Note: I assume that the anonymous user fails (even though enonymous has READ access and can download through the web interface), as AQL is used here, and thus an actual user is needed? The SecretText credentials are shown, but it fails: HTTP/1.1 401 Unauthorized   Logs     access.log: 2019-12-17 23:26:16,665 [DENIED LOGIN]   for client : NA / 172.22.0.3.   request.log: 20191217232616|36|REQUEST|172.22.0.3|non_authenticated_user|POST|/api/search/aql|HTTP/1.1|401|192   NOTE: I have never before seen the 'non_authenticated_user', but maybe this is somewhat new?  Anyways its seems that my API_KEY which was pasted as the SecretText credentials are not properly recognized.  I have double checked that the API_KEY is pasted correctly.    Obviously I need to retest this with 6.12 version of Artifactory, but I doubt that is where the issue lies.   ethorsa , I would suggest that you atleast look in the 'access.log' and 'request.log'.  What I finds easiest, is to just tail -f ARTI_HOME/logs/*.log . Then add a few newlines to your terminal by pressing enter a few times, and then try your build. Assuming you don't have any other traffic to your Artifactory instance, then you will easily see anything that gets added to any of the log files. If that is not possible for you, then you would need to fetch them from the web interface, as previously mentioned.   yahaviz , note that I tested your build, but couldn't get it to work.  Maybee i derbed something? I added a new secret, in the global scope: Secret: <pasted API_KEY> ID: 'api-user_secret' Description: 'api-user using API_KEY as secret text'

          Jesper Reenberg added a comment - - edited

          yahaviz, not to sidetrack this bug report, but I managed to do a TCP dump of the requests, just in case it may clarify why the SecretText failed for me. 

          I tried to repaste the API_KEY into the SecretText credential, just to make tripple sure, that I didn't make anything wrong.

           

          Using the SecretText:

          POST /artifactory/api/search/aql HTTP/1.1
          Authorization: Bearer AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn
          Content-Length: 192
          Content-Type: text/plain; charset=ISO-8859-1
          Host: artifactory:8081
          Connection: Keep-Alive
          User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT
          Accept-Encoding: gzip,deflate
          
          
          items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")
          
          ---
          
          HTTP/1.1 401 Unauthorized
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          WWW-Authenticate: Basic realm="Artifactory Realm"
          Content-Type: application/json;charset=ISO-8859-1
          Content-Length: 171
          Date: Wed, 18 Dec 2019 00:42:20 GMT
          
          
          {
            "errors" : [ {
              "status" : 401,
              "message" : "Bad props auth token: basictoken=AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn"
            } ]
          } 

           

          Using API_KEY as user/pass:

          POST /artifactory/api/search/aql HTTP/1.1
          Content-Length: 192
          Content-Type: text/plain; charset=ISO-8859-1
          Host: artifactory:8081
          Connection: Keep-Alive
          User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT
          Accept-Encoding: gzip,deflate
          Authorization: Basic YXBpLXVzZXI6QUtDcDVlM1ZFd0xYR0tzalB2Qlg2bnY0Q2dQdEdlS0RzVjU3NmM1eEgyOXZpWng3RE5LWnB2cWZXaXcxblVVaXVFS2FNdVlCbg==
          
          
          items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")
          
          ---
          
          HTTP/1.1 200 OK
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          Content-Type: application/json
          Transfer-Encoding: chunked
          Date: Wed, 18 Dec 2019 00:42:31 GMT
          
          ...

           

          Obviously, using user/pass, is the same as user/API_KEY, using the basic auth, except my super secret password

          POST /artifactory/api/search/aql HTTP/1.1
          Content-Length: 192
          Content-Type: text/plain; charset=ISO-8859-1
          Host: artifactory:8081
          Connection: Keep-Alive
          User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT
          Accept-Encoding: gzip,deflate
          Authorization: Basic dXNlcjpwYXNzd29yZA==
          
          
          items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")
          
          ---
          
          HTTP/1.1 200 OK
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          Content-Type: application/json
          Transfer-Encoding: chunked
          Date: Wed, 18 Dec 2019 00:42:53 GMT
          
          ... 

           

          Jesper Reenberg added a comment - - edited yahaviz , not to sidetrack this bug report, but I managed to do a TCP dump of the requests, just in case it may clarify why the SecretText failed for me.  I tried to repaste the API_KEY into the SecretText credential, just to make tripple sure, that I didn't make anything wrong.   Using the SecretText: POST /artifactory/api/search/aql HTTP/1.1 Authorization: Bearer AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" ) --- HTTP/1.1 401 Unauthorized Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 WWW-Authenticate: Basic realm= "Artifactory Realm" Content-Type: application/json;charset=ISO-8859-1 Content-Length: 171 Date: Wed, 18 Dec 2019 00:42:20 GMT { "errors" : [ { "status" : 401, "message" : "Bad props auth token: basictoken=AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn" } ] }   Using API_KEY as user/pass: POST /artifactory/api/search/aql HTTP/1.1 Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate Authorization: Basic YXBpLXVzZXI6QUtDcDVlM1ZFd0xYR0tzalB2Qlg2bnY0Q2dQdEdlS0RzVjU3NmM1eEgyOXZpWng3RE5LWnB2cWZXaXcxblVVaXVFS2FNdVlCbg== items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" ) --- HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Wed, 18 Dec 2019 00:42:31 GMT ...   Obviously, using user/pass, is the same as user/API_KEY, using the basic auth, except my super secret password POST /artifactory/api/search/aql HTTP/1.1 Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate Authorization: Basic dXNlcjpwYXNzd29yZA== items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" ) --- HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Wed, 18 Dec 2019 00:42:53 GMT ...  

          ethorsa added a comment -

          I have test the latest v3.5.0 – which should include the snapshot changes – using the three credential types:

          • Username / API key
          • API key (= secret text)
          • Username / encryped password

           

          All remain failing with HTTP/1.1 403 and produce the same log messages:

          access.log

          2020-01-08 08:22:39,027 [ACCEPTED LOGIN]   for client : anonymous / <IP>.

          request.log

          20200108082239|8|REQUEST|<IP>|anonymous|POST|/api/search/aql|HTTP/1.1|403|203 

          The test with username / password showed a |11| instead of the |8|, but i don't know what this number stands for.

           

          The overall setup is the same as used for the previous tests. For completeness, here are the used versions:

          • Artifactory 6.12.3
          • Jenkins 2.190.3
          • Artifactory Jenkins Plugin 3.5.0

           

          The curl call mentioned earlier still works when executed from the agent.

           

          Are there any reasonable places to add some logging? I can build the plugin from source, so maybe that's a way to gain more insight.

          ethorsa added a comment - I have test the latest v3.5.0 – which should include the snapshot changes – using the three credential types: Username / API key API key (= secret text) Username / encryped password   All remain failing with HTTP/1.1 403 and produce the same log messages: access.log 2020-01-08 08:22:39,027 [ACCEPTED LOGIN]   for client : anonymous / <IP>. request.log 20200108082239|8|REQUEST|<IP>|anonymous|POST|/api/search/aql|HTTP/1.1|403|203  The test with username / password showed a |11| instead of the |8| , but i don't know what this number stands for.   The overall setup is the same as used for the previous tests. For completeness, here are the used versions: Artifactory 6.12.3 Jenkins 2.190.3 Artifactory Jenkins Plugin 3.5.0   The curl call mentioned earlier still works when executed from the agent.   Are there any reasonable places to add some logging? I can build the plugin from source, so maybe that's a way to gain more insight.

          ethorsa,

          I suggest setting an http proxy (such as Postman or Charles) between Jenkins and Artifactory to see what's being sent. You can then compare the payload sent by cUrl vs. Jenkins. This should revel the root cause.

          Eyal Ben Moshe added a comment - ethorsa , I suggest setting an http proxy (such as Postman or Charles) between Jenkins and Artifactory to see what's being sent. You can then compare the payload sent by cUrl vs. Jenkins. This should revel the root cause.

          ethorsa added a comment - - edited

          I have probably found a hint to the root cause: The credentials aren't resolved from the it's id.

          This came up when I added some additional logging to GenericDownloadExecutor execute() method which logs:

          • preferredResolver
            • CredentialsId
            • Username
          • resolverCredentials
            • Username
            • Password
            • AccessToken

          A new logger was registered for all Log-Levels of org.jfrog.

          After running my test pipeline only the CredentialsId contained the correct value; all others (username, password, access token) were empty.

           

          ethorsa added a comment - - edited I have probably found a hint to the root cause: The credentials aren't resolved from the it's id. This came up when I added some additional logging to  GenericDownloadExecutor execute() method which logs: preferredResolver CredentialsId Username resolverCredentials Username Password AccessToken A new logger was registered for all Log-Levels of  org.jfrog . After running my test pipeline only the  CredentialsId contained the correct value; all others (username, password, access token) were empty.  

          ethorsa added a comment - - edited

          I can reproduce the same issue on a Linux based Master (v2.204.2, running on OpenJDK11).

          There's a Github Issue reporting issues regarding anonymous access when the authorize-project plugin is installed. I have the plugin installed on all instances too (v1.3.0).

          Update: I have removed the plugin and restarted the master – still getting 403 error.

          ethorsa added a comment - - edited I can reproduce the same issue on a Linux based Master (v2.204.2, running on OpenJDK11). There's a  Github Issue reporting issues regarding anonymous access when the authorize-project plugin is installed. I have the plugin installed on all instances too (v1.3.0). Update: I have removed the plugin and restarted the master – still getting 403 error.

           Since you mention that you have the authorize-project plugin installed, chances are that you are using some other plugins that might break things?

          I would suggest to verify that it fails for you on a clean install.  If not, then start adding plugins until you find out which one that might interfere.

          I still can't reproduce your results, and I'm doing it in a clean install with only this plugin installed and all its dependencies.  
          Also note that we are running this in production with god knows which other plugins (though not the authorize-project), and we don't see any issue using credentials.

          Jesper Reenberg added a comment -  Since you mention that you have the authorize-project plugin installed, chances are that you are using some other plugins that might break things? I would suggest to verify that it fails for you on a clean install.  If not, then start adding plugins until you find out which one that might interfere. I still can't reproduce your results, and I'm doing it in a clean install with only this plugin installed and all its dependencies.   Also note that we are running this in production with god knows which other plugins (though not the authorize-project), and we don't see any issue using credentials.

          ethorsa added a comment - - edited

          I have setup a clean Jenkins v2.204.2 without any plugins (not even suggested ones). Next adding necessary plugins (pipeline, git and artifactory support) and start testing using the
          pipeline mentioned in earlier. The job succeeded. Gradually adding plugin by plugin with a test after each – still working. Finally this brought up the cause: After installation of the Authorize Project plugin it kept working, but after setting it's strategy to "Run as Specific User" it started failing with the well known error. Changing strategy to "Run as User who Triggered Build" it worked again (however, this breaks functionality with pipelines). See plugins-of-clean-test.txt for a complete list of installed plugins and their versions.

          There are already related issues: JENKINS-58902JENKINS-55624

          ethorsa added a comment - - edited I have setup a clean Jenkins v2.204.2 without any plugins (not even suggested ones). Next adding necessary plugins (pipeline, git and artifactory support) and start testing using the pipeline mentioned in earlier. The job succeeded. Gradually adding plugin by plugin with a test after each – still working. Finally this brought up the cause: After installation of the Authorize Project plugin it kept working, but after setting it's strategy to "Run as Specific User" it started failing with the well known error. Changing strategy to "Run as User who Triggered Build" it worked again (however, this breaks functionality with pipelines). See  plugins-of-clean-test.txt for a complete list of installed plugins and their versions. There are already related issues:  JENKINS-58902 ,  JENKINS-55624

          ethorsa added a comment -

          JENKINS-58902 gives some implementation hints.

          ethorsa added a comment - JENKINS-58902 gives some implementation hints.

          Eyal Ben Moshe added a comment - - edited

          Could this be a bug in the "authorize-project" plugin? See https://github.com/jfrog/jenkins-artifactory-plugin/issues/247

          Eyal Ben Moshe added a comment - - edited Could this be a bug in the "authorize-project" plugin? See  https://github.com/jfrog/jenkins-artifactory-plugin/issues/247

          ethorsa added a comment -

          A quick update: Still broken with recent Artifactory Plugin v3.6.1 (Jenkins v2.204.5, Authorize Project v1.3.0).

          ethorsa added a comment - A quick update: Still broken with recent Artifactory Plugin v3.6.1 (Jenkins v2.204.5, Authorize Project v1.3.0).

            Unassigned Unassigned
            ethorsa ethorsa
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: