• Icon: New Feature New Feature
    • Resolution: Postponed
    • Icon: Major Major
    • multijob-plugin
    • None

      We have MultiJob Projects setup in which we do a build in the first phase and test in the second phase. We want to grab the artifact from the first phase as part of the current MultiJob Build in the second phase as well as at the end of the MultiJob itself. It would be great if MultiJob Plugin could add support for this as Copy Artifact Plugin prefers this code to live in MultiJob Plugin as its specific for it.

          [JENKINS-25111] Ability to Copy Artifacts from a Phase Job

          Ray Sennewald created issue -

          You may copy the files on the good workspace or in a shared. If the jobs run on the same server, you can use ${WORKSPACE} for determine the workspace of the MultiJob Build in sub build phase.

          Mathieu Cantin added a comment - You may copy the files on the good workspace or in a shared. If the jobs run on the same server, you can use ${WORKSPACE} for determine the workspace of the MultiJob Build in sub build phase.

          Ray Sennewald added a comment -

          Yes but that isn't the case. We have phase jobs which run on different servers, so this solution won't work. I have an open Pull Request for implementing this functionality here: https://github.com/jenkinsci/tikal-multijob-plugin/pull/54, I just need to get the tests passing and then hopefully they will merge it in .

          Ray Sennewald added a comment - Yes but that isn't the case. We have phase jobs which run on different servers, so this solution won't work. I have an open Pull Request for implementing this functionality here: https://github.com/jenkinsci/tikal-multijob-plugin/pull/54 , I just need to get the tests passing and then hopefully they will merge it in .
          Sebastian Schuberth made changes -
          Link New: This issue is related to JENKINS-16847 [ JENKINS-16847 ]

          One way to address this would be by implementing JENKINS-25675. Then we could get the build numbers of the triggered phase jobs and pass these as environment variables to the Copy Artifacts's "Specific build" selector.

          Sebastian Schuberth added a comment - One way to address this would be by implementing JENKINS-25675 . Then we could get the build numbers of the triggered phase jobs and pass these as environment variables to the Copy Artifacts's "Specific build" selector.

          Code changed in jenkins
          User: Ray Sennewald
          Path:
          pom.xml
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java
          src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties
          src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly
          src/main/webapp/help-copyArtifact.html
          src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/4c1af3d2ae9eac89a03cf90be7455baca04c932e
          Log:
          JENKINS-25111 Copy Artifacts from Phase Job

          THIS IS AN INTEGRATION WITH Copy Artifact Plugin.
          -Added a new Build Selector called MultiJobBuildSelector
          to CopyArtifact Build Selector.
          -If selected, this will make the Copy Artifact Plugin look
          for an artifact from that job's build which was part of the
          current MultiJobBuild.
          -This selector will work properly if you configure it with
          a MultiJobProject which has a PhaseJob that archives an
          artifact that you want to retrieve as part of the MultiJob.
          -This selector will also work if you configure it to Copy an
          Artifact from another Phase Job which was part of the
          current MultiJob Build.
          -Adds CopyArtifact Plugin as an optional dependency.
          -Adds Matrix Plugin as an optional dependency.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ray Sennewald Path: pom.xml src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly src/main/webapp/help-copyArtifact.html src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java http://jenkins-ci.org/commit/tikal-multijob-plugin/4c1af3d2ae9eac89a03cf90be7455baca04c932e Log: JENKINS-25111 Copy Artifacts from Phase Job THIS IS AN INTEGRATION WITH Copy Artifact Plugin. -Added a new Build Selector called MultiJobBuildSelector to CopyArtifact Build Selector. -If selected, this will make the Copy Artifact Plugin look for an artifact from that job's build which was part of the current MultiJobBuild. -This selector will work properly if you configure it with a MultiJobProject which has a PhaseJob that archives an artifact that you want to retrieve as part of the MultiJob. -This selector will also work if you configure it to Copy an Artifact from another Phase Job which was part of the current MultiJob Build. -Adds CopyArtifact Plugin as an optional dependency. -Adds Matrix Plugin as an optional dependency.

          Code changed in jenkins
          User: itaior
          Path:
          pom.xml
          src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java
          src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties
          src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly
          src/main/webapp/help-copyArtifact.html
          src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java
          http://jenkins-ci.org/commit/tikal-multijob-plugin/9061bb42b52507d3d3624afb6890a3171315c4bc
          Log:
          Merge pull request #54 from rsennewald/JENKINS-25111

          JENKINS-25111 Copy Artifacts from Phase Job

          Compare: https://github.com/jenkinsci/tikal-multijob-plugin/compare/0cd86c543fa3...9061bb42b525

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: itaior Path: pom.xml src/main/java/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector.java src/main/resources/com/tikal/jenkins/plugins/multijob/Messages.properties src/main/resources/com/tikal/jenkins/plugins/multijob/MultiJobBuildSelector/config.jelly src/main/webapp/help-copyArtifact.html src/test/java/com/tikal/jenkins/plugins/multijob/test/testutils/FileWriteBuilder.java http://jenkins-ci.org/commit/tikal-multijob-plugin/9061bb42b52507d3d3624afb6890a3171315c4bc Log: Merge pull request #54 from rsennewald/ JENKINS-25111 JENKINS-25111 Copy Artifacts from Phase Job Compare: https://github.com/jenkinsci/tikal-multijob-plugin/compare/0cd86c543fa3...9061bb42b525

          Paul Holmberg added a comment -

          We pulled the change above and rebuilt the plugin but still got 'unable to find job to copy artifact from' errors. We are trying to copy junit test result files from child phases up to the parent job so as to collate build results in one place.

          In case someone else is currently blocked by this there is a workaround using the EnvInject plugin. After your build phases have run execute a python build step with the following code. It uses the Jenkins JSON API to create a new file called propsfile containing the build numbers and success/failures of all the child phases that have executed so far as JOB_NAME_BUILD_NUMBER and JOB_NAME_BUILD_SUCCESS (note it replaces hyphens with underscores to avoid invalid environment variables).

          #!/usr/bin/env python
          import os
          import json
          import urllib2
          
          url = os.environ['BUILD_URL']
          
          req = urllib2.Request('%s/api/json' % url, headers = {'Accept' : 'application/json'})
          res = urllib2.urlopen(req)
          res = json.loads(res.read())
          
          f = open('propsfile', 'w')
          try:
              buildNumbers = {}
              for subbuild in res['subBuilds']:
                  name = subbuild['jobName'].upper()
                  name = name.replace('-', '_')
                  f.write('%s_BUILD_NUMBER=%s\n' % (name, subbuild['buildNumber']))
                  f.write('%s_BUILD_SUCCESS=%d\n' % (name, subbuild['result'].lower() == 'success'))
          finally:
              f.close()
          

          Then add a subsequent envinject step to pull the contents of that propsfile in and you can use the resulting environment variables to copy artifacts from the correct triggered child builds. You can also use the build success environment variables and the ConditionalJob plugin to optionally run steps (in the case where you have an early smoke test phase that prevents the lengthy subsequent phases from executing if it fails).

          Paul Holmberg added a comment - We pulled the change above and rebuilt the plugin but still got 'unable to find job to copy artifact from' errors. We are trying to copy junit test result files from child phases up to the parent job so as to collate build results in one place. In case someone else is currently blocked by this there is a workaround using the EnvInject plugin. After your build phases have run execute a python build step with the following code. It uses the Jenkins JSON API to create a new file called propsfile containing the build numbers and success/failures of all the child phases that have executed so far as JOB_NAME_BUILD_NUMBER and JOB_NAME_BUILD_SUCCESS (note it replaces hyphens with underscores to avoid invalid environment variables). #!/usr/bin/env python import os import json import urllib2 url = os.environ[ 'BUILD_URL' ] req = urllib2.Request( '%s/api/json' % url, headers = { 'Accept' : 'application/json' }) res = urllib2.urlopen(req) res = json.loads(res.read()) f = open( 'propsfile' , 'w' ) try : buildNumbers = {} for subbuild in res[ 'subBuilds' ]: name = subbuild[ 'jobName' ].upper() name = name.replace( '-' , '_' ) f.write( '%s_BUILD_NUMBER=%s\n' % (name, subbuild[ 'buildNumber' ])) f.write( '%s_BUILD_SUCCESS=%d\n' % (name, subbuild[ 'result' ].lower() == 'success' )) finally : f.close() Then add a subsequent envinject step to pull the contents of that propsfile in and you can use the resulting environment variables to copy artifacts from the correct triggered child builds. You can also use the build success environment variables and the ConditionalJob plugin to optionally run steps (in the case where you have an early smoke test phase that prevents the lengthy subsequent phases from executing if it fails).

          FYI, for me the patch is working (except that it did not support build filters until [1] was merged just today). You can look at the job config at [2] to see how we're using it.

          [1] https://github.com/jenkinsci/tikal-multijob-plugin/pull/64
          [2] https://dscho.cloudapp.net/job/gfw-msys1-test-git-multi-node/configure

          Sebastian Schuberth added a comment - FYI, for me the patch is working (except that it did not support build filters until [1] was merged just today). You can look at the job config at [2] to see how we're using it. [1] https://github.com/jenkinsci/tikal-multijob-plugin/pull/64 [2] https://dscho.cloudapp.net/job/gfw-msys1-test-git-multi-node/configure

          This patch also works for me for a simple 1 phase multijob (tried revision 17c94362f0111b12b7dea7d1a9728666f9949a39) and provides a very neat solution to the problem of triggering multiple jobs and gathering results. Is there a plan to release a new version of the plugin with this functionality soon?

          Russell Gallop added a comment - This patch also works for me for a simple 1 phase multijob (tried revision 17c94362f0111b12b7dea7d1a9728666f9949a39) and provides a very neat solution to the problem of triggering multiple jobs and gathering results. Is there a plan to release a new version of the plugin with this functionality soon?

            rsennewald Ray Sennewald
            rsennewald Ray Sennewald
            Votes:
            5 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: