Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-4960

Provide a SCMCheckoutStrategy to checkout only once for all matrix entries

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Critical Critical
    • matrix-project-plugin
    • None
    • Platform: All, OS: All

      When a project got a configuration matrix, the source code is checked out multiple times:

      • Once for reference incase of poll scm on schedule.
      • Once per matrix item, increasing proportionaly to the complexity.

      It would be great to have a strategy, which runs checkout only once in the master build.

      A patch proposal from @asolsson (initial implementation before creation of SCMCheckoutStrategy )

      I've implemented an enhancement to the matrix project to avoid a SCM checkout
      for each target in the matrix. This can be useful if checkout takes a long time
      and if the build step can build concurrent builds from the same SCM area.
      It's implemented as a option when setting up the Matrix project, if not
      used/checked the Matrix project will work as usual.

          [JENKINS-4960] Provide a SCMCheckoutStrategy to checkout only once for all matrix entries

          asolsson added a comment -

          Created an attachment (id=1043)
          Diff file

          asolsson added a comment - Created an attachment (id=1043) Diff file

          As we discussed in the IRC, some minimal error check to make sure configuration builds are running on the same node as the parent build would be great, to make sure the users won't be surprised when they use the distributed builds.

          In the future, the error check can be replaced by the copy from the parent workspace to child workspace, which is another often requested feature.

          Kohsuke Kawaguchi added a comment - As we discussed in the IRC, some minimal error check to make sure configuration builds are running on the same node as the parent build would be great, to make sure the users won't be surprised when they use the distributed builds. In the future, the error check can be replaced by the copy from the parent workspace to child workspace, which is another often requested feature.

          I just realized that those two behaviors aren't exactly the same thing — the latter user might expect Hudson to create a copy of the workspace if two configurations happen to run on the same host, while the original requestor wouldn't like that.

          Hmm.

          Kohsuke Kawaguchi added a comment - I just realized that those two behaviors aren't exactly the same thing — the latter user might expect Hudson to create a copy of the workspace if two configurations happen to run on the same host, while the original requestor wouldn't like that. Hmm.

          asolsson added a comment -

          Diff file with discussed changes.
          The current node is compared to the parentBuild node, and if different an exception is thrown causing the build to fail.

          asolsson added a comment - Diff file with discussed changes. The current node is compared to the parentBuild node, and if different an exception is thrown causing the build to fail.

          asolsson added a comment - - edited

          Updated patch (patch.diff):
          useSameScmCheckout will only work on one node.

          Added an option to skip checkout on the master, if a user only would like to do the checkouts per-configuration, see duplicated issues.

          asolsson added a comment - - edited Updated patch (patch.diff): useSameScmCheckout will only work on one node. Added an option to skip checkout on the master, if a user only would like to do the checkouts per-configuration, see duplicated issues.

          Thanks. Started looking at the patch now.

          Kohsuke Kawaguchi added a comment - Thanks. Started looking at the patch now.

          miikka added a comment -

          Hi, any updates regarding this one?-)

          miikka added a comment - Hi, any updates regarding this one?-)

          I would like to add a use-case to this particular issue since it's so very close to what I am looking for.

          I want the effect of each build slave doing it's own checkout in it's own workspace but with the coherency (i.e. each slave has a guaranteed exact copy of the SCM checkout) and bandwidth (and load on the SCM server) savings of only having a single checkout that each slave uses.

          Rather than having each child job do it's own checkout, it would be nice for the child to just get a copy of the parent's SCM checkout.

          Brian J Murrell added a comment - I would like to add a use-case to this particular issue since it's so very close to what I am looking for. I want the effect of each build slave doing it's own checkout in it's own workspace but with the coherency (i.e. each slave has a guaranteed exact copy of the SCM checkout) and bandwidth (and load on the SCM server) savings of only having a single checkout that each slave uses. Rather than having each child job do it's own checkout, it would be nice for the child to just get a copy of the parent's SCM checkout.

          Sagar Khushalani added a comment - - edited

          "Added an option to skip checkout on the master, if a user only would like to do the checkouts per-configuration, see duplicated issues."

          Would like to see that implemented. It would make it really nice to not have to check out to three machines, when only two actually do any building...

          Sagar Khushalani added a comment - - edited "Added an option to skip checkout on the master, if a user only would like to do the checkouts per-configuration, see duplicated issues." Would like to see that implemented. It would make it really nice to not have to check out to three machines, when only two actually do any building...

          Garen Parham added a comment -

          If this capability were available, JENKINS-6902 would be a non-issue.

          Garen Parham added a comment - If this capability were available, JENKINS-6902 would be a non-issue.

          I updated the patch (patch.diff from January 11th 2010) to current version of Jenkins and created a pull request accordingly:

          https://github.com/jenkinsci/jenkins/pull/375/commits

          I need this feature badly, since I have to support a huge repository with a rapidly changing set of branches for a fast growing number of platforms.

          Thorsten Möllers added a comment - I updated the patch (patch.diff from January 11th 2010) to current version of Jenkins and created a pull request accordingly: https://github.com/jenkinsci/jenkins/pull/375/commits I need this feature badly, since I have to support a huge repository with a rapidly changing set of branches for a fast growing number of platforms.

          Code changed in jenkins
          User: Thorsten Möllers
          Path:
          core/src/main/java/hudson/matrix/MatrixProject.java
          core/src/main/java/hudson/matrix/MatrixRun.java
          core/src/main/java/hudson/model/AbstractBuild.java
          core/src/main/resources/hudson/matrix/MatrixProject/configure-entries.jelly
          http://jenkins-ci.org/commit/jenkins/3c30e139dec705edfbf51d026a5c5a465e0a379f
          Log:
          JENKINS-4960: Adapt changes to release jenkins-1.458

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Möllers Path: core/src/main/java/hudson/matrix/MatrixProject.java core/src/main/java/hudson/matrix/MatrixRun.java core/src/main/java/hudson/model/AbstractBuild.java core/src/main/resources/hudson/matrix/MatrixProject/configure-entries.jelly http://jenkins-ci.org/commit/jenkins/3c30e139dec705edfbf51d026a5c5a465e0a379f Log: JENKINS-4960 : Adapt changes to release jenkins-1.458

          Rune Darrud added a comment -

          Sponsored feature request

          Rune Darrud added a comment - Sponsored feature request

          This was implemented as an SCMCheckoutStrategy extension point, but we still need someone to deliver the actual implementation of it.

          Kohsuke Kawaguchi added a comment - This was implemented as an SCMCheckoutStrategy extension point, but we still need someone to deliver the actual implementation of it.

          Oleg Nenashev added a comment -

          BTW, the real issue has not been fixed (duplicated by https://issues.jenkins-ci.org/browse/JENKINS-13042)
          The issue's title and contents are quite different and confusing. I've refactored descriptions in order to represent the current state.

          BTW, the strategy should be available in previous LTS releases (1.509.x is mandatory), so it should be implemented as a plugin.

          Oleg Nenashev added a comment - BTW, the real issue has not been fixed (duplicated by https://issues.jenkins-ci.org/browse/JENKINS-13042 ) The issue's title and contents are quite different and confusing. I've refactored descriptions in order to represent the current state. BTW, the strategy should be available in previous LTS releases (1.509.x is mandatory), so it should be implemented as a plugin.

          Roman80 added a comment -

          What is the current status of this change, because we need this change too?

          Roman80 added a comment - What is the current status of this change, because we need this change too?

          Oleg Nenashev added a comment -

          roman80 Jenkins core side is done. https://wiki.jenkins-ci.org/display/JENKINS/Extension+points#Extensionpoints-jenkins.scm.SCMCheckoutStrategy
          Now somebody should implement a short plugin or maybe just patch matrix project plugin since it's decoupled now.

          Oleg Nenashev added a comment - roman80 Jenkins core side is done. https://wiki.jenkins-ci.org/display/JENKINS/Extension+points#Extensionpoints-jenkins.scm.SCMCheckoutStrategy Now somebody should implement a short plugin or maybe just patch matrix project plugin since it's decoupled now.

          Roman80 added a comment - - edited

          I'm new to jenkins development, so what can I do to get this done?
          What's is so difficult to fix this?
          Aren't there some interfaces to implement?

          Roman80 added a comment - - edited I'm new to jenkins development, so what can I do to get this done? What's is so difficult to fix this? Aren't there some interfaces to implement?

          Elso Andras added a comment -

          This is a tiny class, add to an own plugin. You can select this checkout strategy, this will skip the axis scm checkouts:

          public class NoAxisCheckoutSCMCheckoutStrategyImpl extends SCMCheckoutStrategy {
              @DataBoundConstructor
              public NoAxisCheckoutSCMCheckoutStrategyImpl() {}
              @Extension
              public static class DescriptorImpl extends SCMCheckoutStrategyDescriptor {
                  @Override
                  public String getDisplayName() {
                      return "No axis checkout";
                  }
                  @Override
                  public boolean isApplicable(AbstractProject project) {
                      return (project instanceof MatrixProject);
                  }
              }
              @Override
              public void checkout(AbstractBuild.AbstractBuildExecution execution) throws IOException, InterruptedException {
                  if (execution instanceof MatrixBuild.MatrixBuildExecution) {
                      super.checkout(execution);
                  }
              }
          }
          
          

          Elso Andras added a comment - This is a tiny class, add to an own plugin. You can select this checkout strategy, this will skip the axis scm checkouts: public class NoAxisCheckoutSCMCheckoutStrategyImpl extends SCMCheckoutStrategy { @DataBoundConstructor public NoAxisCheckoutSCMCheckoutStrategyImpl() {} @Extension public static class DescriptorImpl extends SCMCheckoutStrategyDescriptor { @Override public String getDisplayName() { return "No axis checkout" ; } @Override public boolean isApplicable(AbstractProject project) { return (project instanceof MatrixProject); } } @Override public void checkout(AbstractBuild.AbstractBuildExecution execution) throws IOException, InterruptedException { if (execution instanceof MatrixBuild.MatrixBuildExecution) { super .checkout(execution); } } }

            Unassigned Unassigned
            asolsson asolsson
            Votes:
            33 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated: