Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59973

rtUpload / rtDownload fail with HTTP 401 / 403

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • artifactory-plugin
    • None
    • - Jenkins v2.190.1 / Windows
      - Artifactory Plugin v3.4.x
      - Artifactory 6.12.2

      After Updating to the artifactory plugin v3.4.1 all up- or downloads using the declarative steps fail by 401 (upload) / 403 (download) HTTP errors.

      Both are working with v3.3.x and broken with 3.4.x.

      According the Artifactory logs the steps do not authenticate. 

       

      Upload

      rtUpload(serverId: 'artifactory-test', specPath: 'spec-ul.json')
      rtPublishBuildInfo(serverId: 'artifactory-test')
      

      Fails with error 401:

      Unauthorized Status code: 401
          at org.jfrog.build.extractor.clientConfiguration.util.spec.SpecDeploymentConsumer.consumerRun(SpecDeploymentConsumer.java:44)
      [...]
      

       

      Download

      rtDownload(serverId: 'artifactory-test', specPath: 'spec-dl.json')  

      Fails with error 403:

      java.io.IOException: Failed to search artifact by the aql 'items.find({"repo": "msys2","path": {"$ne": "."}, "$or": [{"$and": [{"path": { "$match": "distrib"},"name": { "$match": "msys2-x86_64-latest.tar.xz"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")': HTTP/1.1 403 
          at org.jfrog.build.extractor.clientConfiguration.client.ArtifactoryDependenciesClient.getResponseStream(ArtifactoryDependenciesClient.java:136)
      [...]
      

      Configuration

      Artifactory is configured in the Jenkins settings (id artifactory-test in the examples) and useses the Credentials Plugin to access credentials.

      The Test Connection button returns the correct Artifactory version.

       

      Tested version of the plugin

       

      Version Result
      3.3.0
      3.3.1
      3.3.2
      3.4.0
      3.4.1

       

      Workaround

      Downgrade to a v3.3.x release.

          [JENKINS-59973] rtUpload / rtDownload fail with HTTP 401 / 403

          Jesper Reenberg added a comment - - edited

          yahaviz, not to sidetrack this bug report, but I managed to do a TCP dump of the requests, just in case it may clarify why the SecretText failed for me. 

          I tried to repaste the API_KEY into the SecretText credential, just to make tripple sure, that I didn't make anything wrong.

           

          Using the SecretText:

          POST /artifactory/api/search/aql HTTP/1.1
          Authorization: Bearer AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn
          Content-Length: 192
          Content-Type: text/plain; charset=ISO-8859-1
          Host: artifactory:8081
          Connection: Keep-Alive
          User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT
          Accept-Encoding: gzip,deflate
          
          
          items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")
          
          ---
          
          HTTP/1.1 401 Unauthorized
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          WWW-Authenticate: Basic realm="Artifactory Realm"
          Content-Type: application/json;charset=ISO-8859-1
          Content-Length: 171
          Date: Wed, 18 Dec 2019 00:42:20 GMT
          
          
          {
            "errors" : [ {
              "status" : 401,
              "message" : "Bad props auth token: basictoken=AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn"
            } ]
          } 

           

          Using API_KEY as user/pass:

          POST /artifactory/api/search/aql HTTP/1.1
          Content-Length: 192
          Content-Type: text/plain; charset=ISO-8859-1
          Host: artifactory:8081
          Connection: Keep-Alive
          User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT
          Accept-Encoding: gzip,deflate
          Authorization: Basic YXBpLXVzZXI6QUtDcDVlM1ZFd0xYR0tzalB2Qlg2bnY0Q2dQdEdlS0RzVjU3NmM1eEgyOXZpWng3RE5LWnB2cWZXaXcxblVVaXVFS2FNdVlCbg==
          
          
          items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")
          
          ---
          
          HTTP/1.1 200 OK
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          Content-Type: application/json
          Transfer-Encoding: chunked
          Date: Wed, 18 Dec 2019 00:42:31 GMT
          
          ...

           

          Obviously, using user/pass, is the same as user/API_KEY, using the basic auth, except my super secret password

          POST /artifactory/api/search/aql HTTP/1.1
          Content-Length: 192
          Content-Type: text/plain; charset=ISO-8859-1
          Host: artifactory:8081
          Connection: Keep-Alive
          User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT
          Accept-Encoding: gzip,deflate
          Authorization: Basic dXNlcjpwYXNzd29yZA==
          
          
          items.find({"repo": "generic-local","$or": [{"$and": [{"path": { "$match": "*"},"name": { "$match": "*"}}]}]}).include("name","repo","path","actual_md5","actual_sha1","size","type","property")
          
          ---
          
          HTTP/1.1 200 OK
          Server: Artifactory/6.16.0
          X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000
          Content-Type: application/json
          Transfer-Encoding: chunked
          Date: Wed, 18 Dec 2019 00:42:53 GMT
          
          ... 

           

          Jesper Reenberg added a comment - - edited yahaviz , not to sidetrack this bug report, but I managed to do a TCP dump of the requests, just in case it may clarify why the SecretText failed for me.  I tried to repaste the API_KEY into the SecretText credential, just to make tripple sure, that I didn't make anything wrong.   Using the SecretText: POST /artifactory/api/search/aql HTTP/1.1 Authorization: Bearer AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" ) --- HTTP/1.1 401 Unauthorized Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 WWW-Authenticate: Basic realm= "Artifactory Realm" Content-Type: application/json;charset=ISO-8859-1 Content-Length: 171 Date: Wed, 18 Dec 2019 00:42:20 GMT { "errors" : [ { "status" : 401, "message" : "Bad props auth token: basictoken=AKCp5e3VEwLXGKsjPvBX6nv4CgPtGeKDsV576c5xH29viZx7DNKZpvqfWiw1nUUiuEKaMuYBn" } ] }   Using API_KEY as user/pass: POST /artifactory/api/search/aql HTTP/1.1 Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate Authorization: Basic YXBpLXVzZXI6QUtDcDVlM1ZFd0xYR0tzalB2Qlg2bnY0Q2dQdEdlS0RzVjU3NmM1eEgyOXZpWng3RE5LWnB2cWZXaXcxblVVaXVFS2FNdVlCbg== items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" ) --- HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Wed, 18 Dec 2019 00:42:31 GMT ...   Obviously, using user/pass, is the same as user/API_KEY, using the basic auth, except my super secret password POST /artifactory/api/search/aql HTTP/1.1 Content-Length: 192 Content-Type: text/plain; charset=ISO-8859-1 Host: artifactory:8081 Connection: Keep-Alive User-Agent: ArtifactoryBuildClient/2.13.x-SNAPSHOT Accept-Encoding: gzip,deflate Authorization: Basic dXNlcjpwYXNzd29yZA== items.find({ "repo" : " generic -local" , "$or" : [{ "$and" : [{ "path" : { "$match" : "*" }, "name" : { "$match" : "*" }}]}]}).include( "name" , "repo" , "path" , "actual_md5" , "actual_sha1" , "size" , "type" , "property" ) --- HTTP/1.1 200 OK Server: Artifactory/6.16.0 X-Artifactory-Id: b2b51933cd264b43:10d5af30:16f15b57d38:-8000 Content-Type: application/json Transfer-Encoding: chunked Date: Wed, 18 Dec 2019 00:42:53 GMT ...  

          ethorsa added a comment -

          I have test the latest v3.5.0 – which should include the snapshot changes – using the three credential types:

          • Username / API key
          • API key (= secret text)
          • Username / encryped password

           

          All remain failing with HTTP/1.1 403 and produce the same log messages:

          access.log

          2020-01-08 08:22:39,027 [ACCEPTED LOGIN]   for client : anonymous / <IP>.

          request.log

          20200108082239|8|REQUEST|<IP>|anonymous|POST|/api/search/aql|HTTP/1.1|403|203 

          The test with username / password showed a |11| instead of the |8|, but i don't know what this number stands for.

           

          The overall setup is the same as used for the previous tests. For completeness, here are the used versions:

          • Artifactory 6.12.3
          • Jenkins 2.190.3
          • Artifactory Jenkins Plugin 3.5.0

           

          The curl call mentioned earlier still works when executed from the agent.

           

          Are there any reasonable places to add some logging? I can build the plugin from source, so maybe that's a way to gain more insight.

          ethorsa added a comment - I have test the latest v3.5.0 – which should include the snapshot changes – using the three credential types: Username / API key API key (= secret text) Username / encryped password   All remain failing with HTTP/1.1 403 and produce the same log messages: access.log 2020-01-08 08:22:39,027 [ACCEPTED LOGIN]   for client : anonymous / <IP>. request.log 20200108082239|8|REQUEST|<IP>|anonymous|POST|/api/search/aql|HTTP/1.1|403|203  The test with username / password showed a |11| instead of the |8| , but i don't know what this number stands for.   The overall setup is the same as used for the previous tests. For completeness, here are the used versions: Artifactory 6.12.3 Jenkins 2.190.3 Artifactory Jenkins Plugin 3.5.0   The curl call mentioned earlier still works when executed from the agent.   Are there any reasonable places to add some logging? I can build the plugin from source, so maybe that's a way to gain more insight.

          ethorsa,

          I suggest setting an http proxy (such as Postman or Charles) between Jenkins and Artifactory to see what's being sent. You can then compare the payload sent by cUrl vs. Jenkins. This should revel the root cause.

          Eyal Ben Moshe added a comment - ethorsa , I suggest setting an http proxy (such as Postman or Charles) between Jenkins and Artifactory to see what's being sent. You can then compare the payload sent by cUrl vs. Jenkins. This should revel the root cause.

          ethorsa added a comment - - edited

          I have probably found a hint to the root cause: The credentials aren't resolved from the it's id.

          This came up when I added some additional logging to GenericDownloadExecutor execute() method which logs:

          • preferredResolver
            • CredentialsId
            • Username
          • resolverCredentials
            • Username
            • Password
            • AccessToken

          A new logger was registered for all Log-Levels of org.jfrog.

          After running my test pipeline only the CredentialsId contained the correct value; all others (username, password, access token) were empty.

           

          ethorsa added a comment - - edited I have probably found a hint to the root cause: The credentials aren't resolved from the it's id. This came up when I added some additional logging to  GenericDownloadExecutor execute() method which logs: preferredResolver CredentialsId Username resolverCredentials Username Password AccessToken A new logger was registered for all Log-Levels of  org.jfrog . After running my test pipeline only the  CredentialsId contained the correct value; all others (username, password, access token) were empty.  

          ethorsa added a comment - - edited

          I can reproduce the same issue on a Linux based Master (v2.204.2, running on OpenJDK11).

          There's a Github Issue reporting issues regarding anonymous access when the authorize-project plugin is installed. I have the plugin installed on all instances too (v1.3.0).

          Update: I have removed the plugin and restarted the master – still getting 403 error.

          ethorsa added a comment - - edited I can reproduce the same issue on a Linux based Master (v2.204.2, running on OpenJDK11). There's a  Github Issue reporting issues regarding anonymous access when the authorize-project plugin is installed. I have the plugin installed on all instances too (v1.3.0). Update: I have removed the plugin and restarted the master – still getting 403 error.

           Since you mention that you have the authorize-project plugin installed, chances are that you are using some other plugins that might break things?

          I would suggest to verify that it fails for you on a clean install.  If not, then start adding plugins until you find out which one that might interfere.

          I still can't reproduce your results, and I'm doing it in a clean install with only this plugin installed and all its dependencies.  
          Also note that we are running this in production with god knows which other plugins (though not the authorize-project), and we don't see any issue using credentials.

          Jesper Reenberg added a comment -  Since you mention that you have the authorize-project plugin installed, chances are that you are using some other plugins that might break things? I would suggest to verify that it fails for you on a clean install.  If not, then start adding plugins until you find out which one that might interfere. I still can't reproduce your results, and I'm doing it in a clean install with only this plugin installed and all its dependencies.   Also note that we are running this in production with god knows which other plugins (though not the authorize-project), and we don't see any issue using credentials.

          ethorsa added a comment - - edited

          I have setup a clean Jenkins v2.204.2 without any plugins (not even suggested ones). Next adding necessary plugins (pipeline, git and artifactory support) and start testing using the
          pipeline mentioned in earlier. The job succeeded. Gradually adding plugin by plugin with a test after each – still working. Finally this brought up the cause: After installation of the Authorize Project plugin it kept working, but after setting it's strategy to "Run as Specific User" it started failing with the well known error. Changing strategy to "Run as User who Triggered Build" it worked again (however, this breaks functionality with pipelines). See plugins-of-clean-test.txt for a complete list of installed plugins and their versions.

          There are already related issues: JENKINS-58902JENKINS-55624

          ethorsa added a comment - - edited I have setup a clean Jenkins v2.204.2 without any plugins (not even suggested ones). Next adding necessary plugins (pipeline, git and artifactory support) and start testing using the pipeline mentioned in earlier. The job succeeded. Gradually adding plugin by plugin with a test after each – still working. Finally this brought up the cause: After installation of the Authorize Project plugin it kept working, but after setting it's strategy to "Run as Specific User" it started failing with the well known error. Changing strategy to "Run as User who Triggered Build" it worked again (however, this breaks functionality with pipelines). See  plugins-of-clean-test.txt for a complete list of installed plugins and their versions. There are already related issues:  JENKINS-58902 ,  JENKINS-55624

          ethorsa added a comment -

          JENKINS-58902 gives some implementation hints.

          ethorsa added a comment - JENKINS-58902 gives some implementation hints.

          Eyal Ben Moshe added a comment - - edited

          Could this be a bug in the "authorize-project" plugin? See https://github.com/jfrog/jenkins-artifactory-plugin/issues/247

          Eyal Ben Moshe added a comment - - edited Could this be a bug in the "authorize-project" plugin? See  https://github.com/jfrog/jenkins-artifactory-plugin/issues/247

          ethorsa added a comment -

          A quick update: Still broken with recent Artifactory Plugin v3.6.1 (Jenkins v2.204.5, Authorize Project v1.3.0).

          ethorsa added a comment - A quick update: Still broken with recent Artifactory Plugin v3.6.1 (Jenkins v2.204.5, Authorize Project v1.3.0).

            Unassigned Unassigned
            ethorsa ethorsa
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: