• Icon: New Feature New Feature
    • Resolution: Postponed
    • Icon: Major Major
    • multijob-plugin
    • None

      We have MultiJob Projects setup in which we do a build in the first phase and test in the second phase. We want to grab the artifact from the first phase as part of the current MultiJob Build in the second phase as well as at the end of the MultiJob itself. It would be great if MultiJob Plugin could add support for this as Copy Artifact Plugin prefers this code to live in MultiJob Plugin as its specific for it.

          [JENKINS-25111] Ability to Copy Artifacts from a Phase Job

          You may copy the files on the good workspace or in a shared. If the jobs run on the same server, you can use ${WORKSPACE} for determine the workspace of the MultiJob Build in sub build phase.

          Mathieu Cantin added a comment - You may copy the files on the good workspace or in a shared. If the jobs run on the same server, you can use ${WORKSPACE} for determine the workspace of the MultiJob Build in sub build phase.

          Ray Sennewald added a comment -

          Yes but that isn't the case. We have phase jobs which run on different servers, so this solution won't work. I have an open Pull Request for implementing this functionality here: https://github.com/jenkinsci/tikal-multijob-plugin/pull/54, I just need to get the tests passing and then hopefully they will merge it in .

          Ray Sennewald added a comment - Yes but that isn't the case. We have phase jobs which run on different servers, so this solution won't work. I have an open Pull Request for implementing this functionality here: https://github.com/jenkinsci/tikal-multijob-plugin/pull/54 , I just need to get the tests passing and then hopefully they will merge it in .

          One way to address this would be by implementing JENKINS-25675. Then we could get the build numbers of the triggered phase jobs and pass these as environment variables to the Copy Artifacts's "Specific build" selector.

          Sebastian Schuberth added a comment - One way to address this would be by implementing JENKINS-25675 . Then we could get the build numbers of the triggered phase jobs and pass these as environment variables to the Copy Artifacts's "Specific build" selector.

          Code changed in jenkins
          User: Ray Sennewald
          Path:
          pom.xml
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java
          src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties
          src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly
          src/main/webapp/help-copyArtifact.html
          src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/4c1af3d2ae9eac89a03cf90be7455baca04c932e
          Log:
          JENKINS-25111 Copy Artifacts from Phase Job

          THIS IS AN INTEGRATION WITH Copy Artifact Plugin.
          -Added a new Build Selector called MultiJobBuildSelector
          to CopyArtifact Build Selector.
          -If selected, this will make the Copy Artifact Plugin look
          for an artifact from that job's build which was part of the
          current MultiJobBuild.
          -This selector will work properly if you configure it with
          a MultiJobProject which has a PhaseJob that archives an
          artifact that you want to retrieve as part of the MultiJob.
          -This selector will also work if you configure it to Copy an
          Artifact from another Phase Job which was part of the
          current MultiJob Build.
          -Adds CopyArtifact Plugin as an optional dependency.
          -Adds Matrix Plugin as an optional dependency.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ray Sennewald Path: pom.xml src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly src/main/webapp/help-copyArtifact.html src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java http://jenkins-ci.org/commit/tikal-multijob-plugin/4c1af3d2ae9eac89a03cf90be7455baca04c932e Log: JENKINS-25111 Copy Artifacts from Phase Job THIS IS AN INTEGRATION WITH Copy Artifact Plugin. -Added a new Build Selector called MultiJobBuildSelector to CopyArtifact Build Selector. -If selected, this will make the Copy Artifact Plugin look for an artifact from that job's build which was part of the current MultiJobBuild. -This selector will work properly if you configure it with a MultiJobProject which has a PhaseJob that archives an artifact that you want to retrieve as part of the MultiJob. -This selector will also work if you configure it to Copy an Artifact from another Phase Job which was part of the current MultiJob Build. -Adds CopyArtifact Plugin as an optional dependency. -Adds Matrix Plugin as an optional dependency.

          Code changed in jenkins
          User: itaior
          Path:
          pom.xml
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java
          src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties
          src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly
          src/main/webapp/help-copyArtifact.html
          src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/9061bb42b52507d3d3624afb6890a3171315c4bc
          Log:
          Merge pull request #54 from rsennewald/JENKINS-25111

          JENKINS-25111 Copy Artifacts from Phase Job

          Compare: https://github.com/jenkinsci/tikal-multijob-plugin/compare/0cd86c543fa3...9061bb42b525

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: itaior Path: pom.xml src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly src/main/webapp/help-copyArtifact.html src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java http://jenkins-ci.org/commit/tikal-multijob-plugin/9061bb42b52507d3d3624afb6890a3171315c4bc Log: Merge pull request #54 from rsennewald/ JENKINS-25111 JENKINS-25111 Copy Artifacts from Phase Job Compare: https://github.com/jenkinsci/tikal-multijob-plugin/compare/0cd86c543fa3...9061bb42b525

          Paul Holmberg added a comment -

          We pulled the change above and rebuilt the plugin but still got 'unable to find job to copy artifact from' errors. We are trying to copy junit test result files from child phases up to the parent job so as to collate build results in one place.

          In case someone else is currently blocked by this there is a workaround using the EnvInject plugin. After your build phases have run execute a python build step with the following code. It uses the Jenkins JSON API to create a new file called propsfile containing the build numbers and success/failures of all the child phases that have executed so far as JOB_NAME_BUILD_NUMBER and JOB_NAME_BUILD_SUCCESS (note it replaces hyphens with underscores to avoid invalid environment variables).

          #!/usr/bin/env python
          import os
          import json
          import urllib2
          
          url = os.environ['BUILD_URL']
          
          req = urllib2.Request('%s/api/json' % url, headers = {'Accept' : 'application/json'})
          res = urllib2.urlopen(req)
          res = json.loads(res.read())
          
          f = open('propsfile', 'w')
          try:
              buildNumbers = {}
              for subbuild in res['subBuilds']:
                  name = subbuild['jobName'].upper()
                  name = name.replace('-', '_')
                  f.write('%s_BUILD_NUMBER=%s\n' % (name, subbuild['buildNumber']))
                  f.write('%s_BUILD_SUCCESS=%d\n' % (name, subbuild['result'].lower() == 'success'))
          finally:
              f.close()
          

          Then add a subsequent envinject step to pull the contents of that propsfile in and you can use the resulting environment variables to copy artifacts from the correct triggered child builds. You can also use the build success environment variables and the ConditionalJob plugin to optionally run steps (in the case where you have an early smoke test phase that prevents the lengthy subsequent phases from executing if it fails).

          Paul Holmberg added a comment - We pulled the change above and rebuilt the plugin but still got 'unable to find job to copy artifact from' errors. We are trying to copy junit test result files from child phases up to the parent job so as to collate build results in one place. In case someone else is currently blocked by this there is a workaround using the EnvInject plugin. After your build phases have run execute a python build step with the following code. It uses the Jenkins JSON API to create a new file called propsfile containing the build numbers and success/failures of all the child phases that have executed so far as JOB_NAME_BUILD_NUMBER and JOB_NAME_BUILD_SUCCESS (note it replaces hyphens with underscores to avoid invalid environment variables). #!/usr/bin/env python import os import json import urllib2 url = os.environ[ 'BUILD_URL' ] req = urllib2.Request( '%s/api/json' % url, headers = { 'Accept' : 'application/json' }) res = urllib2.urlopen(req) res = json.loads(res.read()) f = open( 'propsfile' , 'w' ) try : buildNumbers = {} for subbuild in res[ 'subBuilds' ]: name = subbuild[ 'jobName' ].upper() name = name.replace( '-' , '_' ) f.write( '%s_BUILD_NUMBER=%s\n' % (name, subbuild[ 'buildNumber' ])) f.write( '%s_BUILD_SUCCESS=%d\n' % (name, subbuild[ 'result' ].lower() == 'success' )) finally : f.close() Then add a subsequent envinject step to pull the contents of that propsfile in and you can use the resulting environment variables to copy artifacts from the correct triggered child builds. You can also use the build success environment variables and the ConditionalJob plugin to optionally run steps (in the case where you have an early smoke test phase that prevents the lengthy subsequent phases from executing if it fails).

          FYI, for me the patch is working (except that it did not support build filters until [1] was merged just today). You can look at the job config at [2] to see how we're using it.

          [1] https://github.com/jenkinsci/tikal-multijob-plugin/pull/64
          [2] https://dscho.cloudapp.net/job/gfw-msys1-test-git-multi-node/configure

          Sebastian Schuberth added a comment - FYI, for me the patch is working (except that it did not support build filters until [1] was merged just today). You can look at the job config at [2] to see how we're using it. [1] https://github.com/jenkinsci/tikal-multijob-plugin/pull/64 [2] https://dscho.cloudapp.net/job/gfw-msys1-test-git-multi-node/configure

          This patch also works for me for a simple 1 phase multijob (tried revision 17c94362f0111b12b7dea7d1a9728666f9949a39) and provides a very neat solution to the problem of triggering multiple jobs and gathering results. Is there a plan to release a new version of the plugin with this functionality soon?

          Russell Gallop added a comment - This patch also works for me for a simple 1 phase multijob (tried revision 17c94362f0111b12b7dea7d1a9728666f9949a39) and provides a very neat solution to the problem of triggering multiple jobs and gathering results. Is there a plan to release a new version of the plugin with this functionality soon?

          It appears that the patch does not work with folders (Using revision 17c94362 of multi-job-plugin). I have configured something like:
          Copy artifacts from another project
          Project name: folder/job
          Which build: Build triggered by current MultiJob build

          And in the console log I get:
          ERROR: Unable to find a build for artifact copy from: folder/job

          It works if I use "Last successful build" and it works if I use a project name without the folder, just not folders in combination with "Build triggered by current MultiJob build".

          Russell Gallop added a comment - It appears that the patch does not work with folders (Using revision 17c94362 of multi-job-plugin). I have configured something like: Copy artifacts from another project Project name: folder/job Which build: Build triggered by current MultiJob build And in the console log I get: ERROR: Unable to find a build for artifact copy from: folder/job It works if I use "Last successful build" and it works if I use a project name without the folder, just not folders in combination with "Build triggered by current MultiJob build".

          I created a Pull Request to fix the problem with folders: https://github.com/jenkinsci/tikal-multijob-plugin/pull/74

          Olivier Sechet added a comment - I created a Pull Request to fix the problem with folders: https://github.com/jenkinsci/tikal-multijob-plugin/pull/74

          Code changed in jenkins
          User: osechet
          Path:
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/f326c2f74b232e07c891dc2a9e5382c6c0befebb
          Log:
          Fix problem when using folders

          As mentioned in a comment of the JENKINS-25111 issue, the Multijob build selector does not work when using folders. The problem is due to the use of getFullDisplayName() instead of getDisplayName() when looking for a sub build. getFullDisplayName() includes the parent names whereas sub build's job name contains only the job name. It's the reason the check couldn't find a match.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: osechet Path: src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java http://jenkins-ci.org/commit/tikal-multijob-plugin/f326c2f74b232e07c891dc2a9e5382c6c0befebb Log: Fix problem when using folders As mentioned in a comment of the JENKINS-25111 issue, the Multijob build selector does not work when using folders. The problem is due to the use of getFullDisplayName() instead of getDisplayName() when looking for a sub build. getFullDisplayName() includes the parent names whereas sub build's job name contains only the job name. It's the reason the check couldn't find a match.

          Code changed in jenkins
          User: Shachar Ben-Zeev
          Path:
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/93a4453f1c253ddf602a695b992a2a3886e5a0ff
          Log:
          Merge pull request #74 from osechet/JENKINS-25111-folders

          JENKINS-25111 Fix problem in MultijobBuildSelector when using folders

          Compare: https://github.com/jenkinsci/tikal-multijob-plugin/compare/bf52cc2b1f62...93a4453f1c25

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Shachar Ben-Zeev Path: src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java http://jenkins-ci.org/commit/tikal-multijob-plugin/93a4453f1c253ddf602a695b992a2a3886e5a0ff Log: Merge pull request #74 from osechet/ JENKINS-25111 -folders JENKINS-25111 Fix problem in MultijobBuildSelector when using folders Compare: https://github.com/jenkinsci/tikal-multijob-plugin/compare/bf52cc2b1f62...93a4453f1c25

          Geoffroy Jabouley added a comment - - edited

          Hi

          will this feature support multijob of multijob?

          Lets say i have the following job organization:

          multiJobA
          |--> PhaseA1
          |------> jobA1
          |--> PhaseA2 
          |------> multijobB
                   |--> PhaseB1
                   |--------> jobB1
          

          I need to retrieve an artefact from jobA1 in jobB1.
          We currently pass the JOBA1_BUILD_NUMBER variable as parameter of MultijobB.

          Any change to use this new functionality with our use case?

          Geoffroy Jabouley added a comment - - edited Hi will this feature support multijob of multijob? Lets say i have the following job organization: multiJobA |--> PhaseA1 |------> jobA1 |--> PhaseA2 |------> multijobB |--> PhaseB1 |--------> jobB1 I need to retrieve an artefact from jobA1 in jobB1. We currently pass the JOBA1_BUILD_NUMBER variable as parameter of MultijobB. Any change to use this new functionality with our use case?

          Closing issue as part of tikal-multijob-plugin issues cleanup.
          If still relevant, please open a matching issue in https://github.com/jenkinsci/tikal-multijob-plugin/issues (you can refer to this issue in its description)

          Yoram Michaeli added a comment - Closing issue as part of tikal-multijob-plugin issues cleanup. If still relevant, please open a matching issue in https://github.com/jenkinsci/tikal-multijob-plugin/issues (you can refer to this issue in its description)

            rsennewald Ray Sennewald
            rsennewald Ray Sennewald
            Votes:
            5 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: