Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50191

Multibranch Pipeline tends to keep rebuilding same commits

      We have a number of Pipeline jobs set up with automatic generation through Organization folders (both Stash and Github) and MultiBranch Pipelines generated from those (and finally PR/branch leaf jobs wherever Jenkinsfile's are found). Our default pipeline template includes an SCM polling setup (ranging from H/2 to H/15) because we don't have webhook ability due to corporate firewalling.

      Quite often I see builds of the same commit IDs running in parallel, or even the later build starting hours after an earlier one has finished (even if successfully). They often have different causes, such as "Started by an SCM change" vs. "Branch indexing".

      If a branch or PR has errors building (e.g. bad timing - new commit noticed in the original branch for a PR, but PR merged or closed during the half a minute or so while a build is preparing, so code can not be checked out - but MBP re-scanning did not yet happen to disable the job), then we can have hundreds of such failed rebuild attempts.

      During FOSDEM-2018 discussion with abayer and markewaite (I'm not sure if we logged a ticket at that time) we concluded that such behavior is a bug, and that Jenkins is supposed to track in some matter which source commits it had processed (or even began processing - has queued or is building) and should avoid such automated duplicate re-runs of the same workload, regardless of whether the workspace (on an agent or master) is persistent or removed after each build.

      This does not happen often (e.g. every time), but sufficiently frequently to be noticeable and annoying (and sometimes expensive - e.g. if this kicks off the avalanche of product build and re-testing that costs several hours of blocked resources here and there).

          [JENKINS-50191] Multibranch Pipeline tends to keep rebuilding same commits

          Daniel Beck added a comment -

          Checkout needs to have finished for a build to be able to make subsequent SCM operations consider the commit already handled. Is this the problem? Huge checkouts, or perhaps even checkouts that aren't happening at the start of the build?

          Daniel Beck added a comment - Checkout needs to have finished for a build to be able to make subsequent SCM operations consider the commit already handled. Is this the problem? Huge checkouts, or perhaps even checkouts that aren't happening at the start of the build?

          Andrew Gray added a comment - - edited

          Seeing the same behaviour on Windows and latest LTS release 2.138.2 (however have seen the same behaviour since at least 2.107.x).

          Our situation:

          Multibranch pipeline (via a pollSCM(* * * * *) in the Jenkinsfile) will detect an SCM change (size of commit is irrelevant) and kick off a build.  Within a minute another build will be kicked off under the auspices of "Branch Indexing" (via the "Scan Multibranch Pipeline Triggers" setting).

          This second duplicate "Branch Indexing build" is buggy behaviour.  The pollSCM renders the branch Indexing build redundant.  Messy workaround is to manually cancel second duplicate build.

          Andrew Gray added a comment - - edited Seeing the same behaviour on Windows and latest LTS release 2.138.2 (however have seen the same behaviour since at least 2.107.x). Our situation: Multibranch pipeline (via a pollSCM(* * * * *) in the Jenkinsfile) will detect an SCM change (size of commit is irrelevant) and kick off a build.  Within a minute another build will be kicked off under the auspices of "Branch Indexing" (via the "Scan Multibranch Pipeline Triggers" setting). This second duplicate "Branch Indexing build" is buggy behaviour.  The pollSCM renders the branch Indexing build redundant.  Messy workaround is to manually cancel second duplicate build.

          nikhil nanal added a comment -

          I am facing the same issue with a multibranch pipeline (Jenkins 2.164, git hub branch source plugin 2.5.3, git plugin 3.10.0, scm -api 2.4.1). I am using scm polling enabled via the job configuration. If a PR build fails then it gets restarted arbitrarily during a subsequent polling interval even though there was no new change or pull request opened. These restarted PR builds hog my executors and if  a subsequent new PR comes in it has to wait  for the existing builds to complete or I have to manually cancel them. This is a serious problem for us. 

          I am using "Merging the pull request with current target branch revision" option in my builds. could this be happening if a PR failed but other PRs passed which caused the target revision to be updated and hence the failed PR also gets retriggered?

          nikhil nanal added a comment - I am facing the same issue with a multibranch pipeline (Jenkins 2.164, git hub branch source plugin 2.5.3, git plugin 3.10.0, scm -api 2.4.1). I am using scm polling enabled via the job configuration. If a PR build fails then it gets restarted arbitrarily during a subsequent polling interval even though there was no new change or pull request opened. These restarted PR builds hog my executors and if  a subsequent new PR comes in it has to wait  for the existing builds to complete or I have to manually cancel them. This is a serious problem for us.  I am using "Merging the pull request with current target branch revision" option in my builds. could this be happening if a PR failed but other PRs passed which caused the target revision to be updated and hence the failed PR also gets retriggered?

          Jim Klimov added a comment -

          For the bit about PR-merge builds, yes this is expected to re-run (once per seen target branch update though!) and part of the design idea about such ephemeral merge branches.

          Jim Klimov added a comment - For the bit about PR-merge builds, yes this is expected to re-run (once per seen target branch update though!) and part of the design idea about such ephemeral merge branches.

          Jay George added a comment -

          we are also facing similar issues. where an old build is kicked off of a stale branch via branch indexing where no changes is made with a same commit. which means the pipeline is deploying old code which causes huge problems for the dev team.  which is close to what other people are experiencing. 

          Our versions 

          Jenkins 2.222.3

          git hub branch source plugin  2.6.0

          git hub api  1.111

          scm-api  2.6.3

          Jay George added a comment - we are also facing similar issues. where an old build is kicked off of a stale branch via branch indexing where no changes is made with a same commit. which means the pipeline is deploying old code which causes huge problems for the dev team.  which is close to what other people are experiencing.  Our versions  Jenkins 2.222.3 git hub branch source plugin  2.6.0 git hub api  1.111 scm-api  2.6.3

          Allan BURDAJEWICZ added a comment - - edited

          SCM Polling is based on the polling baseline / last built revision. Multibranch Pipeline as far as I know is based on the the last recorded / seen revision (the scm-revision-hash.xml and recently the scm-last-seen-revision-hash.xml: https://github.com/jenkinsci/branch-api-plugin/blob/2.1200.v4b_a_3da_2eb_db_4/src/main/java/jenkins/branch/BranchProjectFactory.java#L154-L210. I am not sure if SCM Polling integrates well with the Multibranch APIs.

          Note that a similar problem is observed with just SCM API / Branch API mechanism, see https://issues.jenkins.io/browse/JENKINS-74839. But since there is no SSCM Polling involved I created a different one.

          Allan BURDAJEWICZ added a comment - - edited SCM Polling is based on the polling baseline / last built revision. Multibranch Pipeline as far as I know is based on the the last recorded / seen revision (the scm-revision-hash.xml and recently the scm-last-seen-revision-hash.xml : https://github.com/jenkinsci/branch-api-plugin/blob/2.1200.v4b_a_3da_2eb_db_4/src/main/java/jenkins/branch/BranchProjectFactory.java#L154-L210 . I am not sure if SCM Polling integrates well with the Multibranch APIs. Note that a similar problem is observed with just SCM API / Branch API mechanism, see https://issues.jenkins.io/browse/JENKINS-74839 . But since there is no SSCM Polling involved I created a different one.

            Unassigned Unassigned
            jimklimov Jim Klimov
            Votes:
            7 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: