Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17680

Upgrade to new Copy Artifacts version erases all job names to copy from

      After an update to the new copy artifacts version 1.26 existing configurations are broken because all job names to copy artifacts from are empty. The configuration files still contain the correct names, e.g.:

      <hudson.plugins.copyartifact.CopyArtifact plugin="copyartifact@1.25">
      <projectName>rsc-trunk/label=$label</projectName>
      <filter>*/.tar.gz</filter>
      <target>upstream</target>
      <selector class="hudson.plugins.copyartifact.StatusBuildSelector"/>
      <flatten>true</flatten>
      </hudson.plugins.copyartifact.CopyArtifact>

      As you can see, most of our jobs use variables in the projectName tag. Maybe this is the reason for the problem. I actually cannot find a job without this, so I cannot verify this idea.

      With 1.25 everything is ok.

          [JENKINS-17680] Upgrade to new Copy Artifacts version erases all job names to copy from

          Max Wahler added a comment -

          I've also seen this issue. In 1.26, the tag name changed from projectName to project. I edited all our config.xml files manually and it does the trick. Bad news is, that saving the configuration in Jenkins throws an exception.

          Max Wahler added a comment - I've also seen this issue. In 1.26, the tag name changed from projectName to project. I edited all our config.xml files manually and it does the trick. Bad news is, that saving the configuration in Jenkins throws an exception.

          I'd really like to see an automatic migration path for this.

          Johannes Wienke added a comment - I'd really like to see an automatic migration path for this.

          Marvi Benedet added a comment - - edited

          if you have a lot of projects to correct:

          cd <JENKINS-HOME>/jobs
          tar cfz config_bak.tgz */config.xml
          sed 's/projectName>/project>/g' -i */config.xml

          (if you use it in promotions also: )

          tar cfz config_bak_promotions.tgz */promotions/*/config.xml
          sed 's/projectName>/project>/g' -i */promotions/*/config.xml

          in web interface:
          manage jenkins/reload configurations from disk

          Marvi Benedet added a comment - - edited if you have a lot of projects to correct: cd <JENKINS-HOME>/jobs tar cfz config_bak.tgz */config.xml sed 's/projectName>/project>/g' -i */config.xml (if you use it in promotions also: ) tar cfz config_bak_promotions.tgz */promotions/*/config.xml sed 's/projectName>/project>/g' -i */promotions/*/config.xml in web interface: manage jenkins/reload configurations from disk

          Alan Birtles added a comment -

          I've upgraded from 1.25 to 1.27 and it seems to automatically perform the upgrade when a project is built. Unfortunately I modified a few jobs before they were built so encountered this bug

          Alan Birtles added a comment - I've upgraded from 1.25 to 1.27 and it seems to automatically perform the upgrade when a project is built. Unfortunately I modified a few jobs before they were built so encountered this bug

          ikedam added a comment -

          This seems caused by 6243f6bcd5, splitting projectName into project and parameters.
          This can be fixed with readResolve(), as described in https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility

          ikedam added a comment - This seems caused by 6243f6bcd5 , splitting projectName into project and parameters . This can be fixed with readResolve() , as described in  https://wiki.jenkins-ci.org/display/JENKINS/Hint+on+retaining+backward+compatibility

          ikedam added a comment -

          There seems a little complex problem:

          • project names in Copy Artifact < 1.26 accept axis parameters and build parameters by splitting with a slash.
          • project names in Copy Artifact >= 1.26 accept only axis parameters, no longer build parameters.
            • This is the same bahavior to Jenkins core.

          So readResolve() cannot simply split a old project name into a project name and parameters. It must consider whether values after a slash are axis parameters or build parameters.

          ikedam added a comment - There seems a little complex problem: project names in Copy Artifact < 1.26 accept axis parameters and build parameters by splitting with a slash. project names in Copy Artifact >= 1.26 accept only axis parameters, no longer build parameters. This is the same bahavior to Jenkins core. So readResolve() cannot simply split a old project name into a project name and parameters. It must consider whether values after a slash are axis parameters or build parameters.

          ikedam added a comment -

          I posted a pull request.
          For it is difficult to fix the configuration automatically, a warning message will be displayed.
          https://github.com/jenkinsci/copyartifact-plugin/pull/21

          But it is not reviewed for a month...
          There also left some other reviews, and no maintainer seems available...

          ikedam added a comment - I posted a pull request. For it is difficult to fix the configuration automatically, a warning message will be displayed. https://github.com/jenkinsci/copyartifact-plugin/pull/21 But it is not reviewed for a month... There also left some other reviews, and no maintainer seems available...

          Jesse Glick added a comment -

          The upgrade is automatic if you simply build your project after updating the plugin. You must not configure the project before building it. I closed PR #21 since it does not really solve the problem; there are various ways the storage could be upgraded automatically without requiring a build.

          Jesse Glick added a comment - The upgrade is automatic if you simply build your project after updating the plugin. You must not configure the project before building it. I closed PR #21 since it does not really solve the problem; there are various ways the storage could be upgraded automatically without requiring a build.

          ikedam added a comment -

          > The upgrade is automatic if you simply build your project after updating the plugin.

          I know that and that cannot resolve the problem at all.
          The problem is that, modifying the configuration before building breaks the configuration, and there is a case that a project administrator and a Jenkins administrator are different.

          > there are various ways the storage could be upgraded automatically without requiring a build.

          Do you plan to work that?
          Or please let me know the outline of those ways, and I work it.

          I think the difficulty is the case a project is not contained in ItemGroup other than Jenkins, and cannot be enumerated with Jenkins#getItems.
          I have no idea about the proper way to visit all projects.

          ikedam added a comment - > The upgrade is automatic if you simply build your project after updating the plugin. I know that and that cannot resolve the problem at all. The problem is that, modifying the configuration before building breaks the configuration, and there is a case that a project administrator and a Jenkins administrator are different. > there are various ways the storage could be upgraded automatically without requiring a build. Do you plan to work that? Or please let me know the outline of those ways, and I work it. I think the difficulty is the case a project is not contained in ItemGroup other than Jenkins , and cannot be enumerated with Jenkins#getItems . I have no idea about the proper way to visit all projects.

          Jesse Glick added a comment -

          Do you plan to work [on] that?

          Not currently.

          please let me know the outline of those ways

          Probably something like

          @Initializer(after=JOB_LOADED)
          public static void upgradeAll() {
              for (AbstractProject p : Jenkins.getInstance().getAllItems(AbstractProject.class)) {
                  try {
                      ListenerImpl.getCopiers(p); // calls upgradeIfNecessary as a side effect
                  } catch (IOException x) {
                      Logger.getLogger(CopyArtifact.class.getName()).log(Level.WARNING, "could not upgrade configuration of " + p, x);
                  }
              }
          }
          

          except you would also want to set a persisted flag in CopyArtifactPlugin so that this work is only done once, not after every startup.

          Jesse Glick added a comment - Do you plan to work [on] that? Not currently. please let me know the outline of those ways Probably something like @Initializer(after=JOB_LOADED) public static void upgradeAll() { for (AbstractProject p : Jenkins.getInstance().getAllItems(AbstractProject.class)) { try { ListenerImpl.getCopiers(p); // calls upgradeIfNecessary as a side effect } catch (IOException x) { Logger.getLogger(CopyArtifact. class. getName()).log(Level.WARNING, "could not upgrade configuration of " + p, x); } } } except you would also want to set a persisted flag in CopyArtifactPlugin so that this work is only done once, not after every startup.

          ikedam added a comment -

          I verified Jenkins#getAlItems() visits all projects even they are contained in ItemGroup other than Jenkins.
          I work with that. Thanks!

          ikedam added a comment - I verified Jenkins#getAlItems() visits all projects even they are contained in ItemGroup other than Jenkins . I work with that. Thanks!

          ikedam added a comment -

          ikedam added a comment - Send a pull request: https://github.com/jenkinsci/copyartifact-plugin/pull/23

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          src/main/java/hudson/plugins/copyartifact/CopyArtifactUpgradeListener.java
          http://jenkins-ci.org/commit/copyartifact-plugin/addc9530dc24d04ae21002fed8f6ef86a27e32a3
          Log:
          JENKINS-17680 Upgrade the configuration < 1.26 automatically.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/main/java/hudson/plugins/copyartifact/CopyArtifactUpgradeListener.java http://jenkins-ci.org/commit/copyartifact-plugin/addc9530dc24d04ae21002fed8f6ef86a27e32a3 Log: JENKINS-17680 Upgrade the configuration < 1.26 automatically.

          Code changed in jenkins
          User: ikedam
          Path:
          src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java
          http://jenkins-ci.org/commit/copyartifact-plugin/7214a147cd8a5ebb88bdfce6ae3e5b03b341e8a2
          Log:
          [FIXED JENKINS-17680] Update a test for configuration-upgrading.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java http://jenkins-ci.org/commit/copyartifact-plugin/7214a147cd8a5ebb88bdfce6ae3e5b03b341e8a2 Log: [FIXED JENKINS-17680] Update a test for configuration-upgrading.

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          http://jenkins-ci.org/commit/copyartifact-plugin/bc9cc1894ac32934b912b004784c8ad92fa5ae19
          Log:
          JENKINS-17680 Make minimized diffs.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java http://jenkins-ci.org/commit/copyartifact-plugin/bc9cc1894ac32934b912b004784c8ad92fa5ae19 Log: JENKINS-17680 Make minimized diffs.

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          src/main/java/hudson/plugins/copyartifact/CopyArtifactUpgradeListener.java
          http://jenkins-ci.org/commit/copyartifact-plugin/854deb121eace1d0f4a90664a6d2d2ad733e0d61
          Log:
          JENKINS-17680 use @Initializer instead of ItemListener.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/main/java/hudson/plugins/copyartifact/CopyArtifactUpgradeListener.java http://jenkins-ci.org/commit/copyartifact-plugin/854deb121eace1d0f4a90664a6d2d2ad733e0d61 Log: JENKINS-17680 use @Initializer instead of ItemListener.

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java
          src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testProjectNameSplit.zip
          http://jenkins-ci.org/commit/copyartifact-plugin/b42c2158119f8c8b06d0562bbd1d9bf7b7ee5f49
          Log:
          JENKINS-17680 Upgrade is applied also to MatrixProject.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testProjectNameSplit.zip http://jenkins-ci.org/commit/copyartifact-plugin/b42c2158119f8c8b06d0562bbd1d9bf7b7ee5f49 Log: JENKINS-17680 Upgrade is applied also to MatrixProject.

          Code changed in jenkins
          User: ikedam
          Path:
          src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java
          src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testWrappedCopierProjectNameSplit.zip
          http://jenkins-ci.org/commit/copyartifact-plugin/3c6babe659e45386de6db676054d4eb0b9b9de4a
          Log:
          JENKINS-17680 Added a test for verifying upgrading on building works.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testWrappedCopierProjectNameSplit.zip http://jenkins-ci.org/commit/copyartifact-plugin/3c6babe659e45386de6db676054d4eb0b9b9de4a Log: JENKINS-17680 Added a test for verifying upgrading on building works.

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          http://jenkins-ci.org/commit/copyartifact-plugin/355aedb310d61d2f26e235630b4fc99c7c886e73
          Log:
          JENKINS-17680 Changed a confusiong method name.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java http://jenkins-ci.org/commit/copyartifact-plugin/355aedb310d61d2f26e235630b4fc99c7c886e73 Log: JENKINS-17680 Changed a confusiong method name.

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java
          src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testProjectNameSplit.zip
          src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testWrappedCopierProjectNameSplit.zip
          http://jenkins-ci.org/commit/copyartifact-plugin/f13ba75afffcd971475875873fefc398600a084b
          Log:
          Merge pull request #23 from ikedam/feature/JENKINS-17680-3

          JENKINS-17680 Upgrade configurations from version 1.25

          Compare: https://github.com/jenkinsci/copyartifact-plugin/compare/428d32a6be6b...f13ba75afffc

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testProjectNameSplit.zip src/test/resources/hudson/plugins/copyartifact/CopyArtifactTest/testWrappedCopierProjectNameSplit.zip http://jenkins-ci.org/commit/copyartifact-plugin/f13ba75afffcd971475875873fefc398600a084b Log: Merge pull request #23 from ikedam/feature/ JENKINS-17680 -3 JENKINS-17680 Upgrade configurations from version 1.25 Compare: https://github.com/jenkinsci/copyartifact-plugin/compare/428d32a6be6b...f13ba75afffc

          Code changed in jenkins
          User: ikedam
          Path:
          src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
          src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java
          http://jenkins-ci.org/commit/copyartifact-plugin/f468296f905b23eebf0a36790612b100c8d7a8cd
          Log:
          Fixed unnecessary codes added by changes for JENKINS-17680.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java http://jenkins-ci.org/commit/copyartifact-plugin/f468296f905b23eebf0a36790612b100c8d7a8cd Log: Fixed unnecessary codes added by changes for JENKINS-17680 .

          ikedam added a comment -

          Fixed in Copy Artifact 1.28.

          ikedam added a comment - Fixed in Copy Artifact 1.28.

          Michael Prokop added a comment - - edited

          If the config.xml of a job includes both the project and projectName settings, being identical like:

          <project>demojob/architecture=$architecture</project>
          <projectName>demojob/architecture=$architecture</projectName>

          then it fails to run with current version(s) of the Copy Artifacts plugin (1.28) because no artifacts can be copied from axis architecture of demojob ("Unable to find a build for artifact copy from: demojob") even though the artifacts are there.

          The problem about it is that the web interface displays the correct configuration (Project Name: "demojob/architecture=$architecture" and "Parameter filters" being empty) but the wrong configuration is actually used (Project Name: "demojob" and "Parameter filters: "architecture=$architecture"). When you execute the job the Copy Artifacts plugin automatically converts the configuration to:

          <project>demojob</project>
          <parameters>architecture=$architecture</parameters>

          which is something different and doesn't work. When you manually select "Configure" on the original job (the one with project + projectName settings being identical) in the web interface and then click on "Save" it's fine though.

          This is a problem for folks generating config.xml in an automated way (where you never manually click "Configure" but generate the config.xml from scratch). The mentioned configuration is used for example in jenkins-job-builder:

          https://github.com/openstack-infra/jenkins-job-builder/blob/master/jenkins_jobs/modules/builders.py#L122

          I can share configurations for jenkins-job-builder to easily reproduce the behavior if needed.

          Any idea how a smoother upgrade path could be possible? I'm pretty sure this is breaking quite some setups.

          Michael Prokop added a comment - - edited If the config.xml of a job includes both the project and projectName settings, being identical like: <project>demojob/architecture=$architecture</project> <projectName>demojob/architecture=$architecture</projectName> then it fails to run with current version(s) of the Copy Artifacts plugin (1.28) because no artifacts can be copied from axis architecture of demojob ("Unable to find a build for artifact copy from: demojob") even though the artifacts are there. The problem about it is that the web interface displays the correct configuration (Project Name: "demojob/architecture=$architecture" and "Parameter filters" being empty) but the wrong configuration is actually used (Project Name: "demojob" and "Parameter filters: "architecture=$architecture"). When you execute the job the Copy Artifacts plugin automatically converts the configuration to: <project>demojob</project> <parameters>architecture=$architecture</parameters> which is something different and doesn't work. When you manually select "Configure" on the original job (the one with project + projectName settings being identical) in the web interface and then click on "Save" it's fine though. This is a problem for folks generating config.xml in an automated way (where you never manually click "Configure" but generate the config.xml from scratch). The mentioned configuration is used for example in jenkins-job-builder: https://github.com/openstack-infra/jenkins-job-builder/blob/master/jenkins_jobs/modules/builders.py#L122 I can share configurations for jenkins-job-builder to easily reproduce the behavior if needed. Any idea how a smoother upgrade path could be possible? I'm pretty sure this is breaking quite some setups.

          Jesse Glick added a comment -

          The projectName field is deprecated, and may not be used together with project; there is a one-way conversion of projectName (which gets deleted) to project plus parameters. If jenkins-job-builder writes out a configuration including both fields then it is simply buggy.

          Jesse Glick added a comment - The projectName field is deprecated, and may not be used together with project ; there is a one-way conversion of projectName (which gets deleted) to project plus parameters . If jenkins-job-builder writes out a configuration including both fields then it is simply buggy.

          Alright, sad but fair enough - thanks for clarification. I've filed a patch upstream against jenkins-job-builder (FTR: https://review.openstack.org/#/c/56412/).

          Michael Prokop added a comment - Alright, sad but fair enough - thanks for clarification. I've filed a patch upstream against jenkins-job-builder (FTR: https://review.openstack.org/#/c/56412/ ).

            ikedam ikedam
            languitar Johannes Wienke
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: