Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42931

Upload spec does not resolve placeholders in target path

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • artifactory-plugin
    • None
    • Ubuntu 16.04.1 LTS
      Jenkins version 2.32.1
      artifactory-plugin version 2.9.2

      This works with artifactory-plugin version 2.9.1.

       

      def uploadSpec = """{
       "files": [
         {
           "pattern": "local/${organization}/${moduleName}/(.*?)/(.*?)/(.*?)-SNAPSHOT/(.*?)/(.*?).jar",
           "regexp" : true,
           "target": "${repositoryName}/${organization}/${moduleName}/{1}/{2}/{3}/{4}/{5}.jar"
         }
       ]
      }"""

      with log output:

      For pattern: local/<orga>/<module>/(*)/(*)/(*)-SNAPSHOT/(*)/(*).jar 3 artifacts were found.
      Deploying artifact: http://<artifactory-server>/<repository>/<orga>/<module>/scala_2.10/sbt_0.13/1.0.2/jars/<module>.jar
      Deploying artifact: http://<artifactory-server>/<repository>/<orga>/<module>/scala_2.10/sbt_0.13/1.0.2/jars/<module>-sources.jar
      Deploying artifact: http://<artifactory-server>/<repository>/<orga>/<module>/scala_2.10/sbt_0.13/1.0.2/jars/<module>-javadoc.jar
      

       

      After an upgrade to 2.9.2, the placeholders are no longer resolved. Log shows:

      For pattern: local/<orga>/<module>/(*)/(*)/(*)-SNAPSHOT/(*)/(*).jar 3 artifacts were found.
      Deploying artifact: http://<artifactory-server>/<repository>/<orga>/<module>/{1}/{2}/{3}/{4}/{5}.jar

       

          [JENKINS-42931] Upload spec does not resolve placeholders in target path

          Thanks for reporting this issue.

          We're looking into this, while enhancing the File Specs integrations tests.

          We'll update soon.

          Eyal Ben Moshe added a comment - Thanks for reporting this issue. We're looking into this, while enhancing the File Specs integrations tests. We'll update soon.

          Dima Nevelev added a comment - - edited

          dg_skolberg Unfortunately we were unable to reproduce this issue.
          Please provide us through Github an example of a Jenkinsfile which reproduces the problem.
          The script we used:

          node("remote") {
          
          def server = Artifactory.server 'localhost-8081-admin-password'
          
          def downloadSpec = """{
          "files": [
          {
          "pattern": "CliTestsSpec/*/gerrit-acceptance-framework-2.12.1.jar",
          "target": "local/com/google/gerrit/gerrit-acceptance-framework/2.12.1-SNAPSHOT/gerrit-acceptance/gerrit-acceptance-framework-2.12.1.jar",
          "recursive": "true",
          "flat": "true"
          }
          ]
          }"""
          
          def uploadSpec = """{
          "files": [
          {
          "pattern": "local/${organization}/${moduleName}/(.*?)/(.*?)/(.*?)-SNAPSHOT/(.*?)/(.*?).jar",
          "regexp" : true,
          "target": "${repositoryName}/${organization}/${moduleName}/{1}/{2}/{3}/{4}/{5}.jar"
          }
          ]
          }"""
          
          def buildInfo2 = server.download spec: downloadSpec
          
          def buildInfo1 = server.upload spec: uploadSpec
          
          buildInfo1.append buildInfo2
          
          server.publishBuildInfo buildInfo1
          }
          

          The log output:

          Started by user admin
          [Pipeline] node
          Running on a in /Users/diman/sandbox/jenkins-agent-home-service/workspace/JENKINS-42931
          [Pipeline] {
          [Pipeline] getArtifactoryServer
          [Pipeline] artifactoryDownload
          Beginning to resolve Build Info published dependencies.
          The file '/Users/diman/sandbox/jenkins-agent-home-service/workspace/JENKINS-42931/local/com/google/gerrit/gerrit-acceptance-framework/2.12.1-SNAPSHOT/gerrit-acceptance/gerrit-acceptance-framework-2.12.1.jar' exists locally.
          Finished resolving Build Info published dependencies.
          [Pipeline] artifactoryUpload
          For pattern: local/com/google/(.*?)/(.*?)/(.*?)-SNAPSHOT/(.*?)/(.*?).jar 1 artifacts were found.
          Deploying artifact: http://localhost:8081/artifactory/aql-a1/com/google/gerrit/gerrit-acceptance-framework/2.12.1/gerrit-acceptance/gerrit-acceptance-framework-2.12.1.jar
          [Pipeline] publishBuildInfo
          Deploying build info to: http://localhost:8081/artifactory/api/build
          Build successfully deployed. Browse it in Artifactory under http://localhost:8081/artifactory/webapp/builds/JENKINS-42931/4
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          

           

           

          Dima Nevelev added a comment - - edited dg_skolberg Unfortunately we were unable to reproduce this issue. Please provide us through Github an example of a Jenkinsfile which reproduces the problem. The script we used: node( "remote" ) { def server = Artifactory.server 'localhost-8081-admin-password' def downloadSpec = """{ "files" : [ { "pattern" : "CliTestsSpec/*/gerrit-acceptance-framework-2.12.1.jar" , "target" : "local/com/google/gerrit/gerrit-acceptance-framework/2.12.1-SNAPSHOT/gerrit-acceptance/gerrit-acceptance-framework-2.12.1.jar" , "recursive" : " true " , "flat" : " true " } ] }""" def uploadSpec = """{ "files" : [ { "pattern" : "local/${organization}/${moduleName}/(.*?)/(.*?)/(.*?)-SNAPSHOT/(.*?)/(.*?).jar" , "regexp" : true , "target" : "${repositoryName}/${organization}/${moduleName}/{1}/{2}/{3}/{4}/{5}.jar" } ] }""" def buildInfo2 = server.download spec: downloadSpec def buildInfo1 = server.upload spec: uploadSpec buildInfo1.append buildInfo2 server.publishBuildInfo buildInfo1 } The log output: Started by user admin [Pipeline] node Running on a in /Users/diman/sandbox/jenkins-agent-home-service/workspace/JENKINS-42931 [Pipeline] { [Pipeline] getArtifactoryServer [Pipeline] artifactoryDownload Beginning to resolve Build Info published dependencies. The file '/Users/diman/sandbox/jenkins-agent-home-service/workspace/JENKINS-42931/local/com/google/gerrit/gerrit-acceptance-framework/2.12.1-SNAPSHOT/gerrit-acceptance/gerrit-acceptance-framework-2.12.1.jar' exists locally. Finished resolving Build Info published dependencies. [Pipeline] artifactoryUpload For pattern: local/com/google/(.*?)/(.*?)/(.*?)-SNAPSHOT/(.*?)/(.*?).jar 1 artifacts were found. Deploying artifact: http: //localhost:8081/artifactory/aql-a1/com/google/gerrit/gerrit-acceptance-framework/2.12.1/gerrit-acceptance/gerrit-acceptance-framework-2.12.1.jar [Pipeline] publishBuildInfo Deploying build info to: http: //localhost:8081/artifactory/api/build Build successfully deployed. Browse it in Artifactory under http: //localhost:8081/artifactory/webapp/builds/JENKINS-42931/4 [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS    

          Josh Hasner added a comment - - edited

          I am also having the same problem. In my case, I am not using a jenkinsfile, but a Freestyle project, using the "Generic-Artifactory Integration" option. I attached a pic of the config.

          Log output for 2.9.1(anything between <> was sensitive info and omitted):

          Jenkins Artifactory Plugin version: 2.9.1
          For pattern: C:/Jenkins/workspace/<project>/jobs/build/appcodebuild/BuildArtifacts//(*)_(*)-(*).zip 1 artifacts were found.
          Deploying artifact: http://<art-url>/generic-test/<project>/1.3/<project>_1.3-183.zip
          

          Log output for 2.9.2(anything between <> was sensitive info and omitted):

          Jenkins Artifactory Plugin version: 2.9.2
          For pattern: C:/Jenkins/workspace/<project>/jobs/build/appcodebuild/BuildArtifacts//(*)_(*)-(*).zip 1 artifacts were found.
          Deploying artifact: http://<art-url>/generic-test/{1}/{2}/{1}_{2}-{3}.zip
          

           

          EDIT: Jenkins version I am using is 2.50

          Josh Hasner added a comment - - edited I am also having the same problem. In my case, I am not using a jenkinsfile, but a Freestyle project, using the "Generic-Artifactory Integration" option. I attached a pic of the config. Log output for 2.9.1(anything between <> was sensitive info and omitted): Jenkins Artifactory Plugin version: 2.9.1 For pattern: C:/Jenkins/workspace/<project>/jobs/build/appcodebuild/BuildArtifacts //(*)_(*)-(*).zip 1 artifacts were found. Deploying artifact: http: //<art-url>/ generic -test/<project>/1.3/<project>_1.3-183.zip Log output for 2.9.2(anything between <> was sensitive info and omitted): Jenkins Artifactory Plugin version: 2.9.2 For pattern: C:/Jenkins/workspace/<project>/jobs/build/appcodebuild/BuildArtifacts //(*)_(*)-(*).zip 1 artifacts were found. Deploying artifact: http: //<art-url>/ generic -test/{1}/{2}/{1}_{2}-{3}.zip   EDIT: Jenkins version I am using is 2.50

          Dima Nevelev added a comment -

          jth08527 you right, there is a problem with the placeholders when using a full path.
          We are working on this and will update any progress in the following Jira issue: https://www.jfrog.com/jira/browse/HAP-897
          dg_skolberg we still unable to reproduce your case (case of relative to workspace pattern). Please provide us the content of your working directory. Feel free to mask sensitive data.

          Dima Nevelev added a comment - jth08527  you right, there is a problem with the placeholders when using a full path. We are working on this and will update any progress in the following Jira issue: https://www.jfrog.com/jira/browse/HAP-897 dg_skolberg  we still unable to reproduce your case (case of relative to workspace pattern). Please provide us the content of your working directory. Feel free to mask sensitive data.

          Sorry for the late answer.

          I am unable to reproduce the issue with the latest version (2.10.3) with the exact same script originally used.

          So, my issue seems to be fixed.

          Sascha Kolberg added a comment - Sorry for the late answer. I am unable to reproduce the issue with the latest version (2.10.3) with the exact same script originally used. So, my issue seems to be fixed.

          Josh Hasner added a comment -

          This issue seems to have returned in plugin version 2.11.

          The issue I originally reported was fixed as a part of https://www.jfrog.com/jira/browse/HAP-897

          I am not able to try and upgrade to 2.12 to see if the issue persists, and there is no option to downgrade in the Jenkins UI either.

          Can it be confirmed that this is also happening in 2.12?

          Josh Hasner added a comment - This issue seems to have returned in plugin version 2.11. The issue I originally reported was fixed as a part of https://www.jfrog.com/jira/browse/HAP-897 I am not able to try and upgrade to 2.12 to see if the issue persists, and there is no option to downgrade in the Jenkins UI either. Can it be confirmed that this is also happening in 2.12?

          Jan Krag added a comment -

          I have not finished diagnosing, but I seem to have the same problem on

          Jenkins Artifactory Plugin version: 2.12.2.

           

          My spec file (configured inline in a freestyle sandbox job):

          {
           "files": [
               {
                  "pattern": "release_(*)-(*)-(*)-(*)-(*)-(*)-(*).tar.gz", 
                  "target": "Sandbox/fpga/1.0.0-${BUILD_NUMBER}/${3}--${4}.tgz", 
                  "flat": "true"
               }
           ]
          }

           

          It finds the artifact but doesn't replace the placeholders.

          While writing this, my co-worker has found out that it works correctly if we switch to regex mode and rewrite the pattern to:

          {
          "files": [
              {
                "pattern": "release_(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+).tar.gz",
                "target": "Sandbox/fpga/1.0.0-${BUILD_NUMBER}/{1}--{2}--{3}.tgz",
                "flat": "true",
                "regexp": "true"
              }
            ]
          }

           

          Jan Krag added a comment - I have not finished diagnosing, but I seem to have the same problem on Jenkins Artifactory Plugin version: 2.12.2.   My spec file (configured inline in a freestyle sandbox job): { "files" : [ { "pattern" : "release_(*)-(*)-(*)-(*)-(*)-(*)-(*).tar.gz" , "target" : "Sandbox/fpga/1.0.0-${BUILD_NUMBER}/${3}--${4}.tgz" , "flat" : " true " } ] }   It finds the artifact but doesn't replace the placeholders. While writing this, my co-worker has found out that it works correctly if we switch to regex mode and rewrite the pattern to: { "files" : [ { "pattern" : "release_(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+).tar.gz" , "target" : "Sandbox/fpga/1.0.0-${BUILD_NUMBER}/{1}--{2}--{3}.tgz" , "flat" : " true " , "regexp" : " true " } ] }  

          jth08527, Sorry for the late response. 

          We have solved the issues above and added tests to prevent this from happening again.

          Our latest release version 2.12.2 supports full path as well as relative paths placeholders. 

          jkrag, Could you try and remove the '$' sign from your first attempt without the regexp?

          Roman Gurevitch added a comment - jth08527 , Sorry for the late response.  We have solved the issues above and added tests to prevent this from happening again. Our latest release version 2.12.2 supports full path as well as relative paths placeholders.  jkrag , Could you try and remove the '$' sign from your first attempt without the regexp?

          Josh Hasner added a comment -

          romang, the issue is not fixed with 2.12.2. The plugin was updated, and Jenkins restarted, here is the error:

          Jenkins Artifactory Plugin version: 2.12.2
          For pattern: /home/centos/ap/jenkins/workspace/silkhp/jobs/build/appcodebuild/BuildArtifacts//_-.zip 1 artifacts were found.
          Deploying artifact: http://<art_url>/generic-test/{1}/{2}/{1}_{2}-{3}.zip
          ERROR: remote file operation failed: /home/centos/ap/jenkins/workspace/silkhp/jobs/deploy/copy to artifactory at hudson.remoting.Channel@37bbdde7:unixaws: java.io.IOException: Failed to deploy file. Status code: 400

          Josh Hasner added a comment - romang , the issue is not fixed with 2.12.2. The plugin was updated, and Jenkins restarted, here is the error: Jenkins Artifactory Plugin version: 2.12.2 For pattern: /home/centos/ap/jenkins/workspace/silkhp/jobs/build/appcodebuild/BuildArtifacts// _ - .zip 1 artifacts were found. Deploying artifact: http://<art_url>/generic-test/{1}/{2}/{1}_{2}-{3}.zip ERROR: remote file operation failed: /home/centos/ap/jenkins/workspace/silkhp/jobs/deploy/copy to artifactory at hudson.remoting.Channel@37bbdde7:unixaws: java.io.IOException: Failed to deploy file. Status code: 400

          Josh,

          I created the below File Specs and local files and got the expected result. It could be the issue you got can be solved by using the below File Specs.

          I created the following 3 files on my file-system:

          eyalb-mac:spec-test eyalb$ ll
          total 0
          drwxr-xr-x   5 eyalb  staff  170 Aug  9 20:18 .
          drwxr-xr-x  29 eyalb  staff  986 Aug  9 20:13 ..
          -rw-r--r--   1 eyalb  staff    0 Aug  9 20:16 release_1-2-3-4-5-6-7.tar.gz
          -rw-r--r--   1 eyalb  staff    0 Aug  9 20:18 release_11-22-33-44-55-66-77.tar.gz
          -rw-r--r--   1 eyalb  staff    0 Aug  9 20:18 release_111-222-333-444-555-668-777.tar.gz

          And used the following File Spec:

          {
          "files": [
          {
          "pattern": "spec-test/release_(*)-(*)-(*)-(*)-(*)-(*)-(*).tar.gz",
          "target": "tests/fpga/1.0.0-${BUILD_NUMBER}/{1}--{2}--{3}.tgz",
          "flat": "true"
          }
          ]
          }

          I got the 3 files uploaded as the following build log shows:

          Building in workspace /Users/eyalb/.jenkins/workspace/generic-3
          Jenkins Artifactory Plugin version: 2.12.2-SNAPSHOT (private-08/09/2017 15:15-eyalb)
          For pattern: spec-test/release_(*)-(*)-(*)-(*)-(*)-(*)-(*).tar.gz 3 artifacts were found.
          Deploying artifact: http://localhost:8081/artifactory/tests/fpga/1.0.0-2/1--2--3.tgz
          Deploying artifact: http://localhost:8081/artifactory/tests/fpga/1.0.0-2/11--22--33.tgz
          Deploying artifact: http://localhost:8081/artifactory/tests/fpga/1.0.0-2/111--222--333.tgz
          Deploying build info to: http://localhost:8081/artifactory/api/build
          Finished: SUCCESS

          I then tried the same with a regular expression with the same 3 files in my file-system:

          {
          "files": [
          {
          "pattern": "spec-test/release_(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+).tar.gz",
          "target": "tests/fpga/1.0.0-${BUILD_NUMBER}/{1}--{2}--{3}.tgz",
          "flat": "true",
          "regexp": "true"
          }
          ]
          }

          and got the expected result again:

          Started by user admin
          Building in workspace /Users/eyalb/.jenkins/workspace/generic-2
          Jenkins Artifactory Plugin version: 2.12.2-SNAPSHOT (private-08/09/2017 15:15-eyalb)
          For pattern: spec-test/release_(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+).tar.gz 3 artifacts were found.
          Deploying artifact: http://localhost:8081/artifactory/tests/fpga/1.0.0-3/111--222--333.tgz
          Deploying artifact: http://localhost:8081/artifactory/tests/fpga/1.0.0-3/1--2--3.tgz
          Deploying artifact: http://localhost:8081/artifactory/tests/fpga/1.0.0-3/11--22--33.tgz
          Deploying build info to: http://localhost:8081/artifactory/api/build
          Finished: SUCCESS

           

          Eyal Ben Moshe added a comment - Josh, I created the below File Specs and local files and got the expected result. It could be the issue you got can be solved by using the below File Specs. I created the following 3 files on my file-system: eyalb-mac:spec-test eyalb$ ll total 0 drwxr-xr-x   5 eyalb  staff  170 Aug  9 20:18 . drwxr-xr-x  29 eyalb  staff  986 Aug  9 20:13 .. -rw-r--r--   1 eyalb  staff    0 Aug  9 20:16 release_1-2-3-4-5-6-7.tar.gz -rw-r--r--   1 eyalb  staff    0 Aug  9 20:18 release_11-22-33-44-55-66-77.tar.gz -rw-r--r--   1 eyalb  staff    0 Aug  9 20:18 release_111-222-333-444-555-668-777.tar.gz And used the following File Spec: { "files" : [ { "pattern" : "spec-test/release_(*)-(*)-(*)-(*)-(*)-(*)-(*).tar.gz" , "target" : "tests/fpga/1.0.0-${BUILD_NUMBER}/{1}--{2}--{3}.tgz" , "flat" : " true " } ] } I got the 3 files uploaded as the following build log shows: Building in workspace /Users/eyalb/.jenkins/workspace/ generic -3 Jenkins Artifactory Plugin version: 2.12.2-SNAPSHOT ( private -08/09/2017 15:15-eyalb) For pattern: spec-test/release_(*)-(*)-(*)-(*)-(*)-(*)-(*).tar.gz 3 artifacts were found. Deploying artifact: http: //localhost:8081/artifactory/tests/fpga/1.0.0-2/1--2--3.tgz Deploying artifact: http: //localhost:8081/artifactory/tests/fpga/1.0.0-2/11--22--33.tgz Deploying artifact: http: //localhost:8081/artifactory/tests/fpga/1.0.0-2/111--222--333.tgz Deploying build info to: http: //localhost:8081/artifactory/api/build Finished: SUCCESS I then tried the same with a regular expression with the same 3 files in my file-system: { "files" : [ { "pattern" : "spec-test/release_(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+).tar.gz" , "target" : "tests/fpga/1.0.0-${BUILD_NUMBER}/{1}--{2}--{3}.tgz" , "flat" : " true " , "regexp" : " true " } ] } and got the expected result again: Started by user admin Building in workspace /Users/eyalb/.jenkins/workspace/ generic -2 Jenkins Artifactory Plugin version: 2.12.2-SNAPSHOT ( private -08/09/2017 15:15-eyalb) For pattern: spec-test/release_(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+)-(\d+).tar.gz 3 artifacts were found. Deploying artifact: http: //localhost:8081/artifactory/tests/fpga/1.0.0-3/111--222--333.tgz Deploying artifact: http: //localhost:8081/artifactory/tests/fpga/1.0.0-3/1--2--3.tgz Deploying artifact: http: //localhost:8081/artifactory/tests/fpga/1.0.0-3/11--22--33.tgz Deploying build info to: http: //localhost:8081/artifactory/api/build Finished: SUCCESS  

          Josh Hasner added a comment -

          eyalbe, you seem to be using a different build than me? My build output is 2.12.2, while yours is 2.12.2-SNAPSHOT (private-08/09/2017 15:15-eyalb). Would this make a difference? When I updated the plugin, I updated it from the plugin manager UI. Is my issue fixed in this snapshot version you are using?

          Josh Hasner added a comment - eyalbe , you seem to be using a different build than me? My build output is 2.12.2, while yours is 2.12.2-SNAPSHOT (private-08/09/2017 15:15-eyalb). Would this make a difference? When I updated the plugin, I updated it from the plugin manager UI. Is my issue fixed in this snapshot version you are using?

          The plugin version I tested this with is almost similar to 2.12.2. I'll soon run the same test with 2.12.2 and let you know if I get different results.

          Eyal Ben Moshe added a comment - The plugin version I tested this with is almost similar to 2.12.2. I'll soon run the same test with 2.12.2 and let you know if I get different results.

          I've just tested the 2 File Specs with 2.12.2. All deployed as expected.

          Please copy my Specs. I see no reason why it shouldn't work for you as well.

          Eyal Ben Moshe added a comment - I've just tested the 2 File Specs with 2.12.2. All deployed as expected. Please copy my Specs. I see no reason why it shouldn't work for you as well.

          Josh Hasner added a comment -

          eyalbe, I think I may have found my issue. The path for the pattern had a double slash just before the file name. In version 2.9.1, this worked fine. Not sure why it matters now. Regardless, I should be passing a proper path. I fixed this, and now it seems to work. Apologies for making you run around, and thank you for your testing.

          Josh Hasner added a comment - eyalbe , I think I may have found my issue. The path for the pattern had a double slash just before the file name. In version 2.9.1, this worked fine. Not sure why it matters now. Regardless, I should be passing a proper path. I fixed this, and now it seems to work. Apologies for making you run around, and thank you for your testing.

          Great to hear the issue is resolved and I'm happy I could help.

          Eyal Ben Moshe added a comment - Great to hear the issue is resolved and I'm happy I could help.

            eyalbe Eyal Ben Moshe
            dg_skolberg Sascha Kolberg
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: