Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45280

duplicate build triggered by same github push notification when there is long scm polling

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • jenkins:2.46.3, github-plugin: 1.27.0, git-plugin: 3.3.1

      we have a github repo that contains close to 5000 git tags, the jenkins job is configured to be triggered by webhook push and recent upgrade of jenkins we hit this change https://github.com/jenkinsci/git-plugin/commit/31fedce9c41c9006c886835e03c9fe825d49aba6 which caused our scm polling time increases from 20 seconds to up to 5 minutes because we set some regex filter for branches matching and that change now extend that matching to tags too. and as a result I've seen duplicate build being triggered by the same github push notification.

      All duplicated build jobs has the same scm polling log (same webhook notification) which lead me believe that there are multiple polling threads responding to the same webhook notification and they are not acting in a thread safeway to decide if there are new changes to build. I am looking at this line: https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/com/cloudbees/jenkins/GitHubPushTrigger.java#L109 and suspect that this could be the potential cause, since multiple thread can enter the poll and trigger the new build before the lastBuild being updated (I am not super familiar with the code base so apologize if I am missing the target here)

      one more thing to mention is that after I drop all the branch matching in my jenkins job configuration to put the scm polling time back too less than 1 sec, there will no longer be any duplicated build.

          [JENKINS-45280] duplicate build triggered by same github push notification when there is long scm polling

          We are seeing the same behaviour. 

          Though for me it looks more like that the branch spec from the push event is not verified against the branch spec from the jenkins job itself.

          Björn Häuser added a comment - We are seeing the same behaviour.  Though for me it looks more like that the branch spec from the push event is not verified against the branch spec from the jenkins job itself.

          Matthew Hicks added a comment -

          I am also seeing this behavior. We are using jenkins for feature branch build with 8 workers. Whenever resources get free, up to 4 of the same branch will start building.

          Matthew Hicks added a comment - I am also seeing this behavior. We are using jenkins for feature branch build with 8 workers. Whenever resources get free, up to 4 of the same branch will start building.

          Same here.

           

          Jenkins: 2.147

          github plugin: 1.29.3

          git client plugin: 2.7.3

          git plugin: 3.9.1

          Maksym Sobolyev added a comment - Same here.   Jenkins: 2.147 github plugin: 1.29.3 git client plugin: 2.7.3 git plugin: 3.9.1

          David Lakatos added a comment - - edited

          Same here with 2.190.3. When push brings 2 new commits, 2 jobs start. Both of them points to the Git repo's HEAD.

          David Lakatos added a comment - - edited Same here with 2.190.3. When push brings 2 new commits, 2 jobs start. Both of them points to the Git repo's HEAD.

          Lenny added a comment -

          Same here with the latest Jenkins and all latest plugins as of Sept 21, 2021

          Lenny added a comment - Same here with the latest Jenkins and all latest plugins as of Sept 21, 2021

          Lenny added a comment -

          This maybe unrelated, but here is a coincidental execptino in the log:

          Sep 21, 2021 5:23:44 PM SEVERE io.jenkins.blueocean.autofavorite.FavoritingScmListener onCheckoutUnexpected error when retrieving changeset
          hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %ai%ncommitter %cN <%cE> %ci%n%n%w(0,4,4)%B" -n 1 1bad1afcb14a41a82963e987511d6536874709c4 in /Users/software/var/jenkins/workspace/owlogix-org-repo_flowlogix_PR-85
          	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1272)
          	at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159)
          	at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:155)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          	at java.base/java.lang.Thread.run(Thread.java:829)
          

          Lenny added a comment - This maybe unrelated, but here is a coincidental execptino in the log: Sep 21, 2021 5:23:44 PM SEVERE io.jenkins.blueocean.autofavorite.FavoritingScmListener onCheckoutUnexpected error when retrieving changeset hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %ai%ncommitter %cN <%cE> %ci%n%n%w(0,4,4)%B" -n 1 1bad1afcb14a41a82963e987511d6536874709c4 in /Users/software/ var /jenkins/workspace/owlogix-org-repo_flowlogix_PR-85 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1272) at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159) at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:155) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:829)

          Lenny added a comment -

          Relates to JENKINS-57303

          Lenny added a comment - Relates to  JENKINS-57303

          Lenny added a comment -

          FYI: This is still an issue as of latest Jenkins and plugins as of 1/13/2022

          I have developed a workaround to detect & abort the duplicate job:

          https://github.com/flowlogix/jenkins/blob/master/util-lib/vars/guardDuplicateBuilds.groovy

          Lenny added a comment - FYI: This is still an issue as of latest Jenkins and plugins as of 1/13/2022 I have developed a workaround to detect & abort the duplicate job: https://github.com/flowlogix/jenkins/blob/master/util-lib/vars/guardDuplicateBuilds.groovy

          Lenny added a comment -

          Still an issue as of March 4, 2022 and getting worse

          Lenny added a comment - Still an issue as of March 4, 2022 and getting worse

            Unassigned Unassigned
            shiqiyang Shiqi Yang
            Votes:
            8 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: