Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35988

Equivalent to polling commit exclusions for branch indexing

    • Icon: New Feature New Feature
    • Resolution: Fixed
    • Icon: Major Major
    • branch-api-plugin
    • None
    • Windows

      I have an organizational project configured which scans repositories for jenkinsfile. One of the repository (https://github.com/VirtoCommerce/vc-module-jenkinssample) has a following Jenkinsfile defined:

          node 
          {
          	stage 'Checkout'
                  checkout([
                      $class: 'GitSCM', 
                      branches: [[name: '*/master']], 
                      extensions: [[
          			    $class: 'PathRestriction', 
          			    excludedRegions: 'CommonAssemblyInfo\\.cs', 
          			    includedRegions: ''
          		    ]], 
                      userRemoteConfigs: [[
                          url: 'git@github.com:VirtoCommerce/vc-module-jenkinssample.git']]])
          }
      

      As you might see I have an excluded region defined, so changes made to CommonAssemblyInfo.cs don't trigger any builds. However "Branch Indexing" still forces the job to build when I commit changes to "CommonAssemblyInfo.cs". How can I prevent this?

      Btw, the above script works fine in a separate/individual Pipeline Job.

        1. screenshot-2.png
          screenshot-2.png
          95 kB
        2. screenshot-1.png
          screenshot-1.png
          2 kB
        3. multibranch_indexing_options.png
          multibranch_indexing_options.png
          182 kB
        4. non_multibranch_pipeline_filters.png
          non_multibranch_pipeline_filters.png
          147 kB
        5. workaround.txt
          0.9 kB

          [JENKINS-35988] Equivalent to polling commit exclusions for branch indexing

          Jesse Glick added a comment -

          The Branch API does not support polling exclusions as such. It merely defines revisions. For a GitHub branch source to support these would be rather difficult, I think. Git polling generally does not even support this mode without using workspace-based polling, which is usually a bad idea.

          Jesse Glick added a comment - The Branch API does not support polling exclusions as such. It merely defines revisions. For a GitHub branch source to support these would be rather difficult, I think. Git polling generally does not even support this mode without using workspace-based polling, which is usually a bad idea.

          Can this functionality be moved to later stages?
          So the repository is fetched, but some steps can be skipped in case there are no appropriate changes as described in JENKINS-37978.

          Alexander Vorobiev added a comment - Can this functionality be moved to later stages? So the repository is fetched, but some steps can be skipped in case there are no appropriate changes as described in JENKINS-37978 .

          Jesse Glick added a comment -

          vorobievalex you can do that yourself already, using currentBuild.changeSets.

          Jesse Glick added a comment - vorobievalex you can do that yourself already, using currentBuild.changeSets .

          Jesse Glick added a comment - - edited

          The existing SCM.compareRemoteRevisionWith implementing commit exclusions for traditional polling cannot be used from multibranch projects since it relies on SCMRevisionAction whereas we need to deal with SCMRevision. So we would need a new API in scm-api to provide metadata, similar to SCMFileSystem.changesSince. Then it would need to implemented in branch source plugins; for example

          Jesse Glick added a comment - - edited The existing SCM.compareRemoteRevisionWith implementing commit exclusions for traditional polling cannot be used from multibranch projects since it relies on SCMRevisionAction whereas we need to deal with SCMRevision . So we would need a new API in scm-api to provide metadata, similar to SCMFileSystem.changesSince . Then it would need to implemented in branch source plugins; for example in git we could use the existing master repository cache to supply such metadata (no need for a workspace) in github-branch-source we could probably get that information from the GitHub API

          This is definitely a much needed feature for easier adoption. Being able to just not run a build at all if a commit message or particular user checked in helps resolve various builds that decide they need to check back into github (not the least being the pesky maven-release plugin) but likewise other more complex build flows that need to update meta-data in the source repo with build information.

          Workarounds are cumbersome and result in the build log being full of aborted builds, it also turns a lot of my users off because they could get this capability in existing job types so it stops them from using these awesome new Jenkins features.

          Maxfield Stewart added a comment - This is definitely a much needed feature for easier adoption. Being able to just not run a build at all if a commit message or particular user checked in helps resolve various builds that decide they need to check back into github (not the least being the pesky maven-release plugin) but likewise other more complex build flows that need to update meta-data in the source repo with build information. Workarounds are cumbersome and result in the build log being full of aborted builds, it also turns a lot of my users off because they could get this capability in existing job types so it stops them from using these awesome new Jenkins features.

          Wojciech Pituła added a comment - - edited

          workaround.txt- This is a workaround we have used.

          Wojciech Pituła added a comment - - edited workaround.txt - This is a workaround we have used.

          Gerard Krupa added a comment -

          I've been using a similar workaround but extended it to copy the result of the previous build (again, less than ideal since it's pretty clunky and requires some hairy APIs to be white-listed):

           

          currentBuild.setResult(currentBuild.rawBuild.getPreviousBuild()?.result?.toString())
          

          A slightly better workaround (though still not ideal), from my point of view, would be some way of causing the current build to be deleted as soon as it completes so that the UI doesn't get polluted with these alternating builds/non-builds.

           

          Gerard Krupa added a comment - I've been using a similar workaround but extended it to copy the result of the previous build (again, less than ideal since it's pretty clunky and requires some hairy APIs to be white-listed):   currentBuild.setResult(currentBuild.rawBuild.getPreviousBuild()?.result?.toString()) A slightly better workaround (though still not ideal), from my point of view, would be some way of causing the current build to be deleted as soon as it completes so that the UI doesn't get polluted with these alternating builds/non-builds.  

          Jesse Glick added a comment -

          The currently best workaround I know of is to inspect currentBuild.changeSets from your Jenkinsfile and exit early. JENKINS-27092 would make that a bit easier, as well as allowing visualizations such as Blue Ocean to automatically hide the build or fold it into an adjoining build with a normal completion status.

          Jesse Glick added a comment - The currently best workaround I know of is to inspect currentBuild.changeSets from your Jenkinsfile and exit early.  JENKINS-27092 would make that a bit easier, as well as allowing visualizations such as Blue Ocean to automatically hide the build or fold it into an adjoining build with a normal completion status.

          Jesse Glick added a comment -

          Some more discussion in git PR 573.

          Jesse Glick added a comment - Some more discussion in git PR 573.

          So, I just wanted to confirm a few things relating to this feature request.

          First, there seems to be 2 slightly different use cases relating to branch indexing as I see it. The first use case is when there is a multibranch pipeline job configured with branch indexing which then loads a Jenkinsfile containing a git() build step pointing to the same git repository as the Jenkinsfile itself but with customized polling options. The other use case is when there is a multibranch pipeline job configured with branch indexing but the polling options are configured directly in the multibranch pipeline job itself and not in the Jenkinsfiles being indexed. The second use case seems like a more focused and isolated (and perhaps simpler implement) variation of the first. This improvement seems to be focused on the former use case, but I think we need to consider the latter use case as well. Does this sound reasonable?

          In regards to the second use case I just mentioned, there currently doesn't seem to be any way to filter branch indexing triggers in any way even though similarly configured non-multibranch pipeline jobs provide several such options (ie: "Polling ignores commits with certain messages", "Polling ignores commits from certain users", etc.). If I am understanding correctly, this difference in behavior appears to be intentional due to some technical limitation in git. Could someone clarify what this technical limitation is exactly? Is it that the git toolset requires a workspace in order to perform this filtering? That doesn't seem to hold true for non-multibranch pipelines since the polling operation applies before the source repository is checked out (or at least it seems this way based on my crude understanding of the mechanics involved).

          Perhaps as a first step someone could look into adding support for basic branch indexing filters to multibranch pipeline jobs themselves and then perhaps look to extend that behavior to support further customizations from the indexed Jenkinsfiles.

          If you think these 2 slightly different use cases are independent enough to warrant their own feature requests I'd be happy to create a new one for the second use case and link it to this one instead.

          Kevin Phillips added a comment - So, I just wanted to confirm a few things relating to this feature request. First, there seems to be 2 slightly different use cases relating to branch indexing as I see it. The first use case is when there is a multibranch pipeline job configured with branch indexing which then loads a Jenkinsfile containing a git() build step pointing to the same git repository as the Jenkinsfile itself but with customized polling options. The other use case is when there is a multibranch pipeline job configured with branch indexing but the polling options are configured directly in the multibranch pipeline job itself and not in the Jenkinsfiles being indexed. The second use case seems like a more focused and isolated (and perhaps simpler implement) variation of the first. This improvement seems to be focused on the former use case, but I think we need to consider the latter use case as well. Does this sound reasonable? In regards to the second use case I just mentioned, there currently doesn't seem to be any way to filter branch indexing triggers in any way even though similarly configured non-multibranch pipeline jobs provide several such options (ie: "Polling ignores commits with certain messages", "Polling ignores commits from certain users", etc.). If I am understanding correctly, this difference in behavior appears to be intentional due to some technical limitation in git. Could someone clarify what this technical limitation is exactly? Is it that the git toolset requires a workspace in order to perform this filtering? That doesn't seem to hold true for non-multibranch pipelines since the polling operation applies before the source repository is checked out (or at least it seems this way based on my crude understanding of the mechanics involved). Perhaps as a first step someone could look into adding support for basic branch indexing filters to multibranch pipeline jobs themselves and then perhaps look to extend that behavior to support further customizations from the indexed Jenkinsfiles. If you think these 2 slightly different use cases are independent enough to warrant their own feature requests I'd be happy to create a new one for the second use case and link it to this one instead.

          Here are a couple of screen shots illustrating the discrepancy between the multibranch and non-multibranch pipeline jobs, which seems to have no obvious explanation:

          Kevin Phillips added a comment - Here are a couple of screen shots illustrating the discrepancy between the multibranch and non-multibranch pipeline jobs, which seems to have no obvious explanation:

          Kevin Phillips added a comment - - edited

          I probably should also clarify that the more limited and isolated use case (#2) I mentioned in my earlier comment should ONLY apply a given filter when performing the branch indexing operation, at least as a first pass. So this would allow builds to be excluded ONLY when commits are made to the repository being indexed, not to any other repositories that may be checked out from the Jenkinsfiles that are detected. My hope would be that this would be easier to implement as well since the complexities involved with the polling mechanics should be greatly simplified. Then it could be treated as a future improvement to possibly extract nested Git configurations from the Jenkinsfiles if so desired.

          Kevin Phillips added a comment - - edited I probably should also clarify that the more limited and isolated use case (#2) I mentioned in my earlier comment should ONLY apply a given filter when performing the branch indexing operation, at least as a first pass. So this would allow builds to be excluded ONLY when commits are made to the repository being indexed, not to any other repositories that may be checked out from the Jenkinsfiles that are detected. My hope would be that this would be easier to implement as well since the complexities involved with the polling mechanics should be greatly simplified. Then it could be treated as a future improvement to possibly extract nested Git configurations from the Jenkinsfiles if so desired.

          Joshua Noble added a comment -

          I'm also looking for a way to exclude/ignore commits from a specific Git author. For example, an automated user who bumps the package.json version of an npm module. If that automated user pushes back to the repository after a build, that push will trigger yet another build. Is there a way to ignore/filter out events from certain commit authors? From what I can tell, the webhook events from GitHub include commit author information when a push occurs.

          Joshua Noble added a comment - I'm also looking for a way to exclude/ignore commits from a specific Git author. For example, an automated user who bumps the package.json version of an npm module. If that automated user pushes back to the repository after a build, that push will trigger yet another build. Is there a way to ignore/filter out events from certain commit authors? From what I can tell, the webhook events from GitHub include commit author information when a push occurs.

          Maxime Lemanissier added a comment - - edited

          I'm having a similar issue using pipeline library.

          When i merge a change on the master branch of my jenkins pipeline library, all jobs using this library (more than 100...) are all rebuilt simultaneously, which fills Jenkins queue for hours everytime a change is made on the library...

          This default behavior is probably well intended, as if the pipeline changes, all jobs using should be rebuilt, but when having a lot of jobs, this becomes unpractical/unusable. I would prefer to wait for the next manual build or SCM change on the project repository (not on the lib one) to trigger a new build.

          Is there a way to not trigger builds due to changes on the pipeline library?

          If not, can i use a specific naming convention for the master branch of the pipeline library, and excluding it from polling as shared in previous comment? Will this exclusion also apply on pipeline library?

          Maxime Lemanissier added a comment - - edited I'm having a similar issue using pipeline library. When i merge a change on the master branch of my jenkins pipeline library, all jobs using this library (more than 100...) are all rebuilt simultaneously, which fills Jenkins queue for hours everytime a change is made on the library... This default behavior is probably well intended, as if the pipeline changes, all jobs using should be rebuilt, but when having a lot of jobs, this becomes unpractical/unusable. I would prefer to wait for the next manual build or SCM change on the project repository (not on the lib one) to trigger a new build. Is there a way to not trigger builds due to changes on the pipeline library? If not, can i use a specific naming convention for the master branch of the pipeline library, and excluding it from polling as shared in previous comment? Will this exclusion also apply on pipeline library?

          Nate Faerber added a comment -

          I believe this is the feature I've been looking for.  It makes sense with the introduction of custom Jenkinsfile path to allow us to ignore changes that occur outside of the Jenkinsfile path.  Even that might be too restrictive.  It would solve a lot of people's problems if we can just specific the subdirectory we want to poll.  This would allow folks to put their Jenkinsfiles in completely different paths.

          Nate Faerber added a comment - I believe this is the feature I've been looking for.  It makes sense with the introduction of custom Jenkinsfile path to allow us to ignore changes that occur outside of the Jenkinsfile path.  Even that might be too restrictive.  It would solve a lot of people's problems if we can just specific the subdirectory we want to poll.  This would allow folks to put their Jenkinsfiles in completely different paths.

          Nathan Lowe added a comment -

          We'd love to see this as well. The closest I can get now is a step that scripts the currentBuild.result to ABORTED and fail the build. This shows up as skipped in some places:

          But failed in others:

          And jenkins reports the build as failed to Bitbucket.

          Nathan Lowe added a comment - We'd love to see this as well. The closest I can get now is a step that scripts the currentBuild.result to ABORTED and fail the build. This shows up as skipped in some places: But failed in others: And jenkins reports the build as failed to Bitbucket.

          any update on this?

          Andreas Sieferlinger added a comment - any update on this?

          Torben Knerr added a comment -

          +1 to what leedega mentioned above. "Polling ignores commits in certain paths" is available for "normal" Pipelines, just not for Multibranch Pipelines.

          jglick any chance the extension to scm-api you mentioned in https://issues.jenkins-ci.org/browse/JENKINS-35988?focusedCommentId=295310&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-295310 are on the roadmap?

          (in my case, it would even be for the bitbucket-branch-source-plugin....)

          Torben Knerr added a comment - +1 to what leedega mentioned above. "Polling ignores commits in certain paths" is available for "normal" Pipelines, just not for Multibranch Pipelines. jglick any chance the extension to scm-api you mentioned in https://issues.jenkins-ci.org/browse/JENKINS-35988?focusedCommentId=295310&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-295310 are on the roadmap? (in my case, it would even be for the bitbucket-branch-source-plugin....)

          Liam Newman added a comment - - edited

          Liam Newman added a comment - - edited acejam Ignore by commiter: https://github.com/jenkinsci/ignore-committer-strategy-plugin tknerr webrat woland nlowe_hyland natefaerber leedega I think this does what is being requested here: https://wiki.jenkins.io/display/JENKINS/Multibranch+Build+Strategy+Extension+Plugin

          Liam Newman added a comment -

          Resolving for now based on existing plugin. Please comment if not done.

          Liam Newman added a comment - Resolving for now based on existing plugin. Please comment if not done.

          Bryce Schober added a comment -

          bitwiseman Note that the ignore-committer-strategy plugin doesn't support simple (non-email) subversion usernames, and at least superficially (at least in its UI wording) only supports Git. See also JENKINS-57474.

          Bryce Schober added a comment - bitwiseman Note that the ignore-committer-strategy plugin doesn't support simple (non-email) subversion usernames, and at least superficially (at least in its UI wording) only supports Git. See also JENKINS-57474 .

          Liam Newman added a comment -

          bryceman
          Thanks for filing a new JIRA for that issue.

          Liam Newman added a comment - bryceman Thanks for filing a new JIRA for that issue.

            bitwiseman Liam Newman
            woland Alexander Siniouguine
            Votes:
            32 Vote for this issue
            Watchers:
            40 Start watching this issue

              Created:
              Updated:
              Resolved: