Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18588

Git polling builds same branch multiple times when 'Execute concurrent builds if necessary' turned on

      We have a job setup to build pull request branches based on a naming convention on the branches. The branch specifier is /pr/. We have 'Execute concurrent builds if necessary' turned on for this job and sometimes Jenkins will kick off 4-5 builds for the same branch at the same time and fill up the executors. I've tried adding a quiet period but that didn't seem to help.

          [JENKINS-18588] Git polling builds same branch multiple times when 'Execute concurrent builds if necessary' turned on

          Jeff Skjonsby added a comment -

          The hacky workaround I'm using that basically works (some side-effects but those are tolerable for us for now):

          Add this groovy script as a build pre-step:

          println "checking for duplicate builds"

          // get current thread / Executor
          def thr = Thread.currentThread()
          // get current build
          def currentBuild = thr?.executable
          def build = currentBuild.previousBuild
          for(i in 0..10) {
          if (build == null || !build.building)

          { return; }

          def gitRevision = build.environment['GIT_COMMIT']
          println "${build.state} - building: ${build.building}"
          if (build.number != currentBuild.number && gitRevision == currentBuild.environment['GIT_COMMIT'])

          { println "build for revision $gitRevision already in progress, ABORTING..." build.executor.interrupt() }

          }

          This basically will at the start of a build look for already in-progress builds for a given revision and cancel it. You could allow the older build to "win" but we needed the newer one to win because we're using Stash for pull requests and needed the newer build so that Stash knows the build was successful.

          Jeff Skjonsby added a comment - The hacky workaround I'm using that basically works (some side-effects but those are tolerable for us for now): Add this groovy script as a build pre-step: println "checking for duplicate builds" // get current thread / Executor def thr = Thread.currentThread() // get current build def currentBuild = thr?.executable def build = currentBuild.previousBuild for(i in 0..10) { if (build == null || !build.building) { return; } def gitRevision = build.environment ['GIT_COMMIT'] println "${build.state} - building: ${build.building}" if (build.number != currentBuild.number && gitRevision == currentBuild.environment ['GIT_COMMIT'] ) { println "build for revision $gitRevision already in progress, ABORTING..." build.executor.interrupt() } } This basically will at the start of a build look for already in-progress builds for a given revision and cancel it. You could allow the older build to "win" but we needed the newer one to win because we're using Stash for pull requests and needed the newer build so that Stash knows the build was successful.

          We have this issue too.
          Please change the priority to Blocker, because it basically blocks the usage of concurrent builds.

          Mike Friedrich added a comment - We have this issue too. Please change the priority to Blocker, because it basically blocks the usage of concurrent builds.

          See also JENKINS-21464

          Richard van der Hoff added a comment - See also JENKINS-21464

          I have the same problem in Jenkins 1.583 building Stash's refs/pull-requests/*/merge branches

          Ilya Vassilevsky added a comment - I have the same problem in Jenkins 1.583 building Stash's refs/pull-requests/*/merge branches

          I can't reproduce this issue with latest git-plugin.
          If this issue persists with git-plugin >= 2.3.5 please reopen

          Nicolas De Loof added a comment - I can't reproduce this issue with latest git-plugin. If this issue persists with git-plugin >= 2.3.5 please reopen

          Hi there. I use git-plugin 2.3.5 and this issue exists.
          Job starts a few times.
          I fetch changes from repo from branch with slashes(feature/test). And after this job execution job starts few times and doesn't find any changes.
          I know that branches with slashes it is another Jenkins issue but job starts a few times anyway.

          Oleksandr Chyrkov added a comment - Hi there. I use git-plugin 2.3.5 and this issue exists. Job starts a few times. I fetch changes from repo from branch with slashes(feature/test). And after this job execution job starts few times and doesn't find any changes. I know that branches with slashes it is another Jenkins issue but job starts a few times anyway.

          Colin Bennett added a comment -

          I confirm the existence of this bug. Apparently since the Git plugin is looking only at the last completed build's Git commit hash, concurrent builds are not identified.

          The solution is probably to implement something like the Groovy script jskjons posted above, except implement it in the Git plugin directly. If manually triggered, we might actually want to rebuild the same commit twice concurrently (different parameters? want to compare consistency of test runs?) but for polling we would only expect to fire up a single build for a particular commit.

          Colin Bennett added a comment - I confirm the existence of this bug. Apparently since the Git plugin is looking only at the last completed build's Git commit hash, concurrent builds are not identified. The solution is probably to implement something like the Groovy script jskjons posted above, except implement it in the Git plugin directly. If manually triggered, we might actually want to rebuild the same commit twice concurrently (different parameters? want to compare consistency of test runs?) but for polling we would only expect to fire up a single build for a particular commit.

          Colin Bennett added a comment -

          The bug still exists with Git plugin v2.4.4 and Jenkins v1.642.3.

          I can see why it exists, since if the last built commit is X, and then Y is committed, a build is triggered for commit Y and the last build data in Jenkins is for X. But if polling happens again (in my case due to a GitLab web hook posting to Jenkins) then a second concurrent build will be started for commit Y, since the last completed build was for X.

          Colin Bennett added a comment - The bug still exists with Git plugin v2.4.4 and Jenkins v1.642.3. I can see why it exists, since if the last built commit is X, and then Y is committed, a build is triggered for commit Y and the last build data in Jenkins is for X. But if polling happens again (in my case due to a GitLab web hook posting to Jenkins) then a second concurrent build will be started for commit Y, since the last completed build was for X.

          I don't know Java very well. Is this the place where a new build is scheduled to run?

          https://github.com/jenkinsci/git-plugin/blob/f6ace83fe36740a1ab35612d3b4c5b400647f6ea/src/main/java/hudson/plugins/git/GitSCM.java#L1001

          Is this method the right place to check for existing running builds?

          Ilya Vassilevsky added a comment - I don't know Java very well. Is this the place where a new build is scheduled to run? https://github.com/jenkinsci/git-plugin/blob/f6ace83fe36740a1ab35612d3b4c5b400647f6ea/src/main/java/hudson/plugins/git/GitSCM.java#L1001 Is this method the right place to check for existing running builds?

          Jesse Glick added a comment -

          Use multibranch projects and you will not suffer from this kind of problem.

          Jesse Glick added a comment - Use multibranch projects and you will not suffer from this kind of problem.

            Unassigned Unassigned
            jskjons Jeff Skjonsby
            Votes:
            10 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: