Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43507

Allow SCMSource and SCMNavigator subtypes to share common traits

      Examples of traits from the JENKINS-43426 epic:

      • There will be a strategy to use switch the checkout to over SSH and this will have a option to select the SSH Credentials for checkout.
      • There will be at least one strategy to subset the branches by name (include / exclude)
        • There will one implementation that uses the existing wildcard style
        • Other strategies may be added later to support different branch name matching rules, e.g. Regex, etc. These secondary strategies do not form part of the MVP but it may prove assist extension point API validation to proof-of-concept implement a second one.
      • There will be three strategies to subset the branches by type:
        • There will be a strategy to select origin branches. This strategy will have a drop-down mode selection:
          • Only origin branches that are not filed as PRs
          • Only origin branches that are filed as PRs
          • All origin branches irrespective of whether filed as PR or not
        • There will be a strategy to select origin PRs. This strategy will have a drop-down mode selection:
          • Merge commit PRs
          • Head commit PRs
          • Both merge commit and head commit PRs
        • There will be a strategy to select fork PRs. This strategy will have a drop-down mode selection:
          • Merge commit PRs
          • Head commit PRs
          • Both merge commit and head commit PRs
      • There will also be a drop-down for trust selection:
          • Show all fork PRs but only trust fork PRs from repository contributors
          • Only show PRs from repository contributors
          • Show and trust all fork PRs
             TBD determine if we need to expose suppression of automatic builds of untrusted PRs here.
        • There will be a proof-of-concept implementation of a tag support branch selector. This is not a feature experienced by users today, but there are api's & code which is ready to provide it. We won't ship this enabled by default in the release.
      • There will be the ability to control a subset of the Git plugin’s additional behaviours for the generated SCM of branches. This will be an applies to all setting. The available options will be subject to a whitelisting extension point (so that plugins can define additional Git behaviours and whitelist them in at the same time). The default whitelist will be:
        • Advanced Checkout behaviours
        • Advanced Clone behaviours
        • Advanced Submodule behaviours
        • Clean after checkout
        • Clean before checkout
        • Custom user name / email address
        • Git LFS pull after checkout
        • Use commit author in changelog (says it requires workspace polling, but really does not / should not require workspace polling)
        • Wipe out repository & force clone

      Acceptance criteria

      • The SCM API provides the concept of a trait that applies to SCMSource and SCMNavigator
      • Traits will be opt-in, i.e. the SCMSource implementation must be written to use traits
      • Common trait implementations can be shared by multiple implementations.
      • Trait implementations can be specific to a single SCMSource or SCMNavigator
      • The SCM API will provide a trait implementation that allows for filtering branches based on include/exclude wildcard name matching
      • The SCM API MockSCM implementation will support traits
      • The common trait implementations in SCM API will have tests
      • The implementers guide will be updated to highlight the trait style of implementation and encourage following that path. 

          [JENKINS-43507] Allow SCMSource and SCMNavigator subtypes to share common traits

          Greg Smith added a comment - - edited

          OK – Sorry, i was able to find that UI now, via "Add" under "Behaviors" on the github organization UI.

          We added that to our Github organization configuration.  Should that automatically fix all builds using a simple "checkout scm" in their pipeline code?  As we have seen no change in behavior or resolution to the problem yet after adding that extra configuration.

          Under what conditions are those additional behaviors used?

          Greg Smith added a comment - - edited OK – Sorry, i was able to find that UI now, via "Add" under "Behaviors" on the github organization UI. We added that to our Github organization configuration.  Should that automatically fix all builds using a simple "checkout scm" in their pipeline code?  As we have seen no change in behavior or resolution to the problem yet after adding that extra configuration. Under what conditions are those additional behaviors used?

          gregcovertsmith We're using the Bitbucket equivalent; we had to issue a rescan operation for the behaviors to be added on individual sub-jobs.

          Thibault Nélis added a comment - gregcovertsmith We're using the Bitbucket equivalent; we had to issue a rescan operation for the behaviors to be added on individual sub-jobs.

          rikoe We had called out the change in the way the Git SCM Source checks out:

          The GitHub and Bitbucket plugins both inherit their per-branch checkout behaviour from the Git plugin. I am sorry that it was not obvious that you should also pay attention to the class structure changes involved and that the exposure of the ability to configure all of the Git Branch Source's behaviours would now mean that you would get the new Git Branch Source defaults for checking out. Is there some text you think would be good to add to the GitHub Branch Source wiki page that would make this change more obvious?

          Stephen Connolly added a comment - rikoe We had called out the change in the way the Git SCM Source checks out: The GitHub and Bitbucket plugins both inherit their per-branch checkout behaviour from the Git plugin. I am sorry that it was not obvious that you should also pay attention to the class structure changes involved and that the exposure of the ability to configure all of the Git Branch Source's behaviours would now mean that you would get the new Git Branch Source defaults for checking out. Is there some text you think would be good to add to the GitHub Branch Source wiki page that would make this change more obvious?

          Riko Eksteen added a comment -

          stephenconnolly, thanks for your reply. Our process is usually as follows: Jenkins warns us about out-of-date plugins, we hit upgrade for the out-of-date plugins. But not before I go onto the release notes, as far as possible, and check what changed.

          In this instance I think we upgraded from 3.3.2 to 3.4.1. Now that you have pointed out to me that you did specify the changes in the release notes, my take-away is this:

          • I should have read the release notes more carefully to make sure I understand what exactly had changed.
          • From my perspective, changing the plugin to to fetch without tags instead of with tags is a breaking change, it regresses the behaviour that myself - and other people, going by this thread - relied on.
          • If you agree with the former point (which you may not), in semver terms, a breaking change is a new major version, i.e. 4.0 as opposed to 3.x.
          • Irrespective of the above, in terms of the release notes, it would have been great if their was a clearly visible warning in the first line of the release notes indicating that some people may see unexpected behaviour, instead of that information being hidden away in a long paragraph towards the end. (I appreciate that this may only be evident in hindsight.)
          • Even now, when clicking through to the JIRA from the release notes, it is listed in the JIRA properties as a "minor" change, and appears to be quite innocuous, while being anything but - from my (extremely) limited understanding it is quite a major change that affect multiple interlocking plugins.

          Thanks for taking the time to respond and reading my reply!

          Riko Eksteen added a comment - stephenconnolly , thanks for your reply. Our process is usually as follows: Jenkins warns us about out-of-date plugins, we hit upgrade for the out-of-date plugins. But not before I go onto the release notes, as far as possible, and check what changed. In this instance I think we upgraded from 3.3.2 to 3.4.1. Now that you have pointed out to me that you did specify the changes in the release notes, my take-away is this: I should have read the release notes more carefully to make sure I understand what exactly had changed. From my perspective, changing the plugin to to fetch without tags instead of with tags is a breaking change, it regresses the behaviour that myself - and other people, going by this thread - relied on. If you agree with the former point (which you may not), in semver terms, a breaking change is a new major version, i.e. 4.0 as opposed to 3.x. Irrespective of the above, in terms of the release notes, it would have been great if their was a clearly visible warning in the first line of the release notes indicating that some people may see unexpected behaviour, instead of that information being hidden away in a long paragraph towards the end. (I appreciate that this may only be evident in hindsight.) Even now, when clicking through to the JIRA from the release notes, it is listed in the JIRA properties as a "minor" change, and appears to be quite innocuous, while being anything but - from my (extremely) limited understanding it is quite a major change that affect multiple interlocking plugins. Thanks for taking the time to respond and reading my reply!

          This new behaviour is just a nightmare for us, we have upgraded from 3.3.2 to 3.5.0.

          About 2 days spent to try to repair our build based on tags push event and nothing is working as it should...

          "Because each branch job in a multibranch project will only ever build the one specific branch, the default behaviour for a Git Branch Source is now to use a minimal refspec corresponding to just the required branch. Tags will not be checked out by default. If you have a multibranch project that requires the full set of ref-specs (for example, you might have a pipeline that will use some analysis tool on the diff with some other branch) you can restore the previous behaviour by adding the "Advanced Clone Behaviours". Note: In some cases you may also need to add the "Specify ref specs" behaviour."

          This solution just solves nothing, i added the Advanced Clone Behaviours and the "Specify ref specs" and nothing happened the plugin still don't care about our tags. It would be really appreciate to have a clearly detailed sample about how to set your plugin to let it work with tags correctly.

          Ludovic Mercier added a comment - This new behaviour is just a nightmare for us, we have upgraded from 3.3.2 to 3.5.0. About 2 days spent to try to repair our build based on tags push event and nothing is working as it should... "Because each branch job in a multibranch project will only ever build the one specific branch, the default behaviour for a Git Branch Source is now to use a minimal refspec corresponding to just the required branch. Tags will not be checked out by default. If you have a multibranch project that requires the full set of ref-specs (for example, you might have a pipeline that will use some analysis tool on the diff with some other branch) you can restore the previous behaviour by adding the "Advanced Clone Behaviours". Note: In some cases you may also need to add the "Specify ref specs" behaviour." This solution just solves nothing, i added the Advanced Clone Behaviours and the "Specify ref specs" and nothing happened the plugin still don't care about our tags. It would be really appreciate to have a clearly detailed sample about how to set your plugin to let it work with tags correctly.

          ludovic_mercier are you trying to build tags or are you trying to compare against tags?

          If building tags, this is not yet supported (may have accidentally worked previously, but was not intended to work)

          If comparing against tags, have you done a reindex first to update all the job configurations?

          Stephen Connolly added a comment - ludovic_mercier are you trying to build tags or are you trying to compare against tags? If building tags, this is not yet supported (may have accidentally worked previously, but was not intended to work) If comparing against tags, have you done a reindex first to update all the job configurations?

          Hi stephenconnolly,

          Thanks for your time, We were building tags effectively and i must say it's not a good news to read that it shouldn't have worked.

          It seems that we will have to change our strategy and workflow according to this or maybe you have a potential roadmap about this feature.

          Ludovic Mercier added a comment - Hi stephenconnolly , Thanks for your time, We were building tags effectively and i must say it's not a good news to read that it shouldn't have worked. It seems that we will have to change our strategy and workflow according to this or maybe you have a potential roadmap about this feature.

          ludovic_mercier so there are issues with building tags that need to be addressed: 

          • We probably want the concept of interesting heads: JENKINS-45502
            • Old tags are not interesting for sure
            • New tags might be interesting if the user wants to configure it that way
          • Interesting heads is not strictly required for branch-api as there is the ability to define a BranchBuildStrategy and that would allow suppressing automatic builds of tags on project creation (which is really the big issue for tags)
          • Support for implementing tags in the GitHub plugin is being tracked in JENKINS-34395, likely we'll just consolidate the Bitbucket, Git and Gitea implementations of the same feature under this banner 

          HTH

          Stephen Connolly added a comment - ludovic_mercier so there are issues with building tags that need to be addressed:  We probably want the concept of interesting heads: JENKINS-45502 Old tags are not interesting for sure New tags might be interesting if the user wants to configure it that way Interesting heads is not strictly required for branch-api as there is the ability to define a BranchBuildStrategy  and that would allow suppressing automatic builds of tags on project creation (which is really the big issue for tags) Support for implementing tags in the GitHub plugin is being tracked in  JENKINS-34395 , likely we'll just consolidate the Bitbucket, Git and Gitea implementations of the same feature under this banner  HTH

          rikoe

           it is listed in the JIRA properties as a "minor" change

          Nope, it was listed as a PRIORITY of "minor", not a scope.

          The refactoring itself is binary compatible but adds new methods. There are extensive tests of data migration. I believe there is only one use case where we missed a potentially breaking change, namely if you construct the SCMSource instance in your pipeline directly (e.g. for use with resolveScm) in all other cases the existing configuration should have been migrated:

          If you have a migration data set that is not migrating correctly, then that is a bug which should be filed in a separate JIRA and we can add an additional test case for your configuration and then fix it

           

          Stephen Connolly added a comment - rikoe  it is listed in the JIRA properties as a "minor" change Nope, it was listed as a PRIORITY of "minor", not a scope. The refactoring itself is binary compatible but adds new methods. There are extensive tests of data migration. I believe there is only one use case where we missed a potentially breaking change, namely if you construct the SCMSource instance in your pipeline directly (e.g. for use with resolveScm ) in all other cases the existing configuration should have been migrated: https://github.com/jenkinsci/github-branch-source-plugin/blob/f2a9fdd871e218429cd1de198b130596e90ee167/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSourceTraitsTest.java#L107-L396  and https://github.com/jenkinsci/github-branch-source-plugin/blob/f2a9fdd871e218429cd1de198b130596e90ee167/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSourceTraitsTest.java#L800-L1997   https://github.com/jenkinsci/github-branch-source-plugin/blob/f2a9fdd871e218429cd1de198b130596e90ee167/src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigatorTraitsTest.java#L58-L1561 https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/test/java/com/cloudbees/jenkins/plugins/bitbucket/BitbucketSCMSourceTest.java#L57-L434 https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/e220b43404aca15574c7a9be3724a40587093414/src/test/java/com/cloudbees/jenkins/plugins/bitbucket/BitbucketSCMNavigatorTest.java#L53-L461 https://github.com/jenkinsci/git-plugin/blob/e08051a8e571f32887d463c72361123b6b37befb/src/test/java/jenkins/plugins/git/GitSCMSourceTraitsTest.java#L81-L211 If you have a migration data set that is not migrating correctly, then that is a bug which should be filed in a separate JIRA and we can add an additional test case for your configuration and then fix it  

          Greg Smith added a comment -

          Wanted to follow up, and say that once that once we completed a full rescan of our org, the additional behaviors were copied down to each project, and the original behaviors were restored.  This resolved all of our problems.

          Several of the other featured included in this change are very handy, so I wanted to say thanks again, I think this change overall will be be very useful in the long term. 

          Greg Smith added a comment - Wanted to follow up, and say that once that once we completed a full rescan of our org, the additional behaviors were copied down to each project, and the original behaviors were restored.  This resolved all of our problems. Several of the other featured included in this change are very handy, so I wanted to say thanks again, I think this change overall will be be very useful in the long term. 

            stephenconnolly Stephen Connolly
            stephenconnolly Stephen Connolly
            Votes:
            4 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated:
              Resolved: