Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50886

Declarative pipeline with pollSCM triggers 2 builds

      I have a multibranch declarative pipeline with a pollSCM trigger to poll every 2 minutes. Whenever there is a change to the branch in BitBucket it triggers correctly but about 2 minutes later it triggers again. I'm not sure if this is a bug or some configuration error on my behalf.

       

      I've attached a screenshot of the duplicate builds and poll log for one of the duplicate builds. Below is the top of my declarative pipeline for reference to how I am setting it up.

      pipeline {
         agent none
         environment {
            APPVERSION = '0.1.0'
            NUGET_API_KEY = credentials('build-server-nuget-id-readonly')
         }
         triggers {
             // poll repo every 2 minute for changes
             pollSCM('*/2 * * * *')
         }
         options {
             // add timestamps to output
             timestamps()
             overrideIndexTriggers(false)
             buildDiscarder(logRotator(numToKeepStr: '10'))
             skipStagesAfterUnstable()
             durabilityHint('PERFORMANCE_OPTIMIZED')
         }
      ...
      }

       

          [JENKINS-50886] Declarative pipeline with pollSCM triggers 2 builds

          markewaite abayer the build does not complete in less then 2 minutes.

          Excuse me if I have misunderstood how this feature is meant to work but I am assuming that when a scan runs and it detects a new commit, it triggers a build for the commit and records the commit hash as being triggered/seen. 2 minutes later it scans again if no new commits nothing happens, however if there is a new commit is triggers for that commit.

          From Mark's comment it sounds like the build has to be finished before the next poll.

           

          If there is any other information I can provide to get to the bottom of this, please let me know.

          Jonathan Kuleff added a comment - markewaite abayer the build does not complete in less then 2 minutes. Excuse me if I have misunderstood how this feature is meant to work but I am assuming that when a scan runs and it detects a new commit, it triggers a build for the commit and records the commit hash as being triggered/seen. 2 minutes later it scans again if no new commits nothing happens, however if there is a new commit is triggers for that commit. From Mark's comment it sounds like the build has to be finished before the next poll.   If there is any other information I can provide to get to the bottom of this, please let me know.

          Mark Waite added a comment -

          jonathank I can't duplicate the problem you're reporting.

          Steps I took trying to duplicate the problem:

          1. Created a JENKINS-50886 branch in my jenkins-bugs repository so that the job will be created in my multi-branch Pipelines
          2. Define an ant script that sleeps for MINUTES_TO_WAIT=13 + 30 seconds
          3. Define a scripted Pipeline in the Jenkinsfile that defines polling interval as MINUTES_TO_WAIT
          4. Scan the repository to detect the new branch
          5. Confirm that a single build is run when a change is detected and that no additional builds are run until the next polling interval detects changes

          I assumed that the difference between scripted and declarative is not relevant in this case.

          I assumed that a multi-branch Pipeline was a sufficient approximation for your case. Are you using a Pipeline (without multi-branch) or are you using multi-branch Pipeline?

          I was able to cause two concurrent builds of the same commit if I used curl to the Jenkins notifyCommit URL shortly after the commit happened. Since you're using polling, I assume you're not using notifyCommit and I assume you're not using webhooks. If you are using notifyCommit or webhooks, then you don't need polling.

          Mark Waite added a comment - jonathank I can't duplicate the problem you're reporting. Steps I took trying to duplicate the problem: Created a JENKINS-50886 branch in my jenkins-bugs repository so that the job will be created in my multi-branch Pipelines Define an ant script that sleeps for MINUTES_TO_WAIT=13 + 30 seconds Define a scripted Pipeline in the Jenkinsfile that defines polling interval as MINUTES_TO_WAIT Scan the repository to detect the new branch Confirm that a single build is run when a change is detected and that no additional builds are run until the next polling interval detects changes I assumed that the difference between scripted and declarative is not relevant in this case. I assumed that a multi-branch Pipeline was a sufficient approximation for your case. Are you using a Pipeline (without multi-branch) or are you using multi-branch Pipeline? I was able to cause two concurrent builds of the same commit if I used curl to the Jenkins notifyCommit URL shortly after the commit happened. Since you're using polling, I assume you're not using notifyCommit and I assume you're not using webhooks. If you are using notifyCommit or webhooks, then you don't need polling.

          markewaite thanks for looking at that. I'll try to reproduce with a simplified jenkinsfile for you.

           

          Here is what I did to try reproduce it:

          1. Created Bitbucket as a private repo
          2. Created a `feature/JENKINS-50886` branch and added the Jenkinsfile (declarative pipeline) and README.md (nothing special)
          3. Created a new pipeline in BlueOcean
          4. Jumped out to classic and turned on `Scan Multibranch Pipeline Triggers -> Periodically if not otherwise run` with an interval of 5 minutes, so new branches are picked up automatically

           

          This is were I have hit a road block, for some reason polling is not picking up changes, even though there are some.

           

          Poll Log
          
          Started on May 12, 2018 3:48:31 PM
          Done. Took 0 ms
          No changes

           

           

          Scan Multibranch Pipeline Log
          
          Started by timer
          [Sat May 12 15:49:00 GMT 2018] Starting branch indexing...
          Connecting to https://bitbucket.org using builder/****** (Bitbucket server credentials)
          Repository type: Git
          Looking up repo/jenkins-bug for branches
          Checking branch feature/JENKINS-50886 from repo/jenkins-bug
                ‘Jenkinsfile’ found
              Met criteria
          Changes detected: feature/JENKINS-50886 (dc08795440200d86ee6dfe67b9f1bcefa44f2e40 → 5dbf835960f9b4a599fed8b4dd2903795e334194)
          Scheduled build for branch: feature/JENKINS-50886
          Checking branch master from repo/jenkins-bug
                ‘Jenkinsfile’ found
              Met criteria
          No changes detected: master (still at 5883611d9e07b7595051a9aca1227920b42e747d)
          
            2 branches were processed
          Looking up repo/jenkins-bug for pull requests
          
            0 pull requests were processed
          [Sat May 12 15:49:26 GMT 2018] Finished branch indexing. Indexing took 25 sec
          Finished: SUCCESS

           

          Any idea why polling is not working or recommendations on where to look? One thing I noticed is that this new pipeline has `Polling Log` while existing have `Git Polling Log`

          Jonathan Kuleff added a comment - markewaite thanks for looking at that. I'll try to reproduce with a simplified jenkinsfile for you.   Here is what I did to try reproduce it: Created Bitbucket as a private repo Created a `feature/ JENKINS-50886 ` branch and added the Jenkinsfile (declarative pipeline) and README.md (nothing special) Created a new pipeline in BlueOcean Jumped out to classic and turned on `Scan Multibranch Pipeline Triggers -> Periodically if not otherwise run` with an interval of 5 minutes, so new branches are picked up automatically   This is were I have hit a road block, for some reason polling is not picking up changes, even though there are some.   Poll Log Started on May 12, 2018 3:48:31 PM Done. Took 0 ms No changes     Scan Multibranch Pipeline Log Started by timer [Sat May 12 15:49:00 GMT 2018] Starting branch indexing... Connecting to https://bitbucket.org using builder/****** (Bitbucket server credentials) Repository type: Git Looking up repo/jenkins-bug for branches Checking branch feature/JENKINS-50886 from repo/jenkins-bug ‘Jenkinsfile’ found Met criteria Changes detected: feature/JENKINS-50886 (dc08795440200d86ee6dfe67b9f1bcefa44f2e40 → 5dbf835960f9b4a599fed8b4dd2903795e334194) Scheduled build for branch: feature/JENKINS-50886 Checking branch master from repo/jenkins-bug ‘Jenkinsfile’ found Met criteria No changes detected: master (still at 5883611d9e07b7595051a9aca1227920b42e747d) 2 branches were processed Looking up repo/jenkins-bug for pull requests 0 pull requests were processed [Sat May 12 15:49:26 GMT 2018] Finished branch indexing. Indexing took 25 sec Finished: SUCCESS   Any idea why polling is not working or recommendations on where to look? One thing I noticed is that this new pipeline has `Polling Log` while existing have `Git Polling Log`

          Mark Waite added a comment -

          jonathank you may be encountering a bug that polling may not always correctly detect branches of the form parent/child. The plugin has some awkward spots trying to handle the difference between origin/master and feature/new-thing. Try without the "feature", or try fully qualifying the branch name as origin/feature/JENKINS-50886.

          Mark Waite added a comment - jonathank you may be encountering a bug that polling may not always correctly detect branches of the form parent/child . The plugin has some awkward spots trying to handle the difference between origin/master and feature/new-thing . Try without the "feature", or try fully qualifying the branch name as origin/feature/ JENKINS-50886 .

          markewaite this has mostly fixed itself up. We are using AWS EFS to back our container cluster that Jenkins is running on and it turns out it didn't have enough IOPS credits, a recently feature that allows us to provision IOPS has seems to fix this up for us bar the occasional double build.

          Jonathan Kuleff added a comment - markewaite this has mostly fixed itself up. We are using AWS EFS to back our container cluster that Jenkins is running on and it turns out it didn't have enough IOPS credits, a recently feature that allows us to provision IOPS has seems to fix this up for us bar the occasional double build.

          Gerd Aschemann added a comment - - edited

          I still encounter this problem every now and then. On a public Jenkins you can see it occasionally here: https://dev.dukecon.org/jenkins/job/dukecon_pwa/job/develop/, the respective Jenkinsfile is here: https://github.com/dukecon/dukecon_pwa/blob/develop/Jenkinsfile with 

          options { 
            disableConcurrentBuilds() 
            buildDiscarder(logRotator( numToKeepStr: '5', daysToKeepStr: '5'))
          }
          
          triggers { 
            pollSCM('* * * * *')
          }
           

          Gerd Aschemann added a comment - - edited I still encounter this problem every now and then. On a public Jenkins you can see it occasionally here:  https://dev.dukecon.org/jenkins/job/dukecon_pwa/job/develop/ , the respective Jenkinsfile is here:  https://github.com/dukecon/dukecon_pwa/blob/develop/Jenkinsfile  with  options { disableConcurrentBuilds() buildDiscarder(logRotator( numToKeepStr: '5' , daysToKeepStr: '5' )) } triggers { pollSCM( '* * * * *' ) }

          I encounter this issue too. Do we have a solution for this yet? I'm using Multibranch Pipeline project and Github repostitory. We are polling every 5minutes but the build takes more than 30mins to complete.Duplicate Builds are being triggered with the same  commit ID.

          Please help me.

           

          Thank you.

          Priyatham Tadikonda added a comment - I encounter this issue too. Do we have a solution for this yet? I'm using Multibranch Pipeline project and Github repostitory. We are polling every 5minutes but the build takes more than 30mins to complete.Duplicate Builds are being triggered with the same  commit ID. Please help me.   Thank you.

          Mark Waite added a comment -

          priyatham no solution in progress and no solution proposed. We recommend that webhooks are better than polling (faster response, lighter load on the git server). See "polling must die" by Kohsuke Kawaguchi for more details. If polling is truly the only way to achieve your goal, then you might also consider setting the polling interval to be greater than the average build time.

          Mark Waite added a comment - priyatham no solution in progress and no solution proposed. We recommend that webhooks are better than polling (faster response, lighter load on the git server). See "polling must die" by Kohsuke Kawaguchi for more details. If polling is truly the only way to achieve your goal, then you might also consider setting the polling interval to be greater than the average build time.

          Mark Austin added a comment - - edited

          I'm seeing this happen with one of my teams using the multi-branch pipeline setup. Two different people are able to trigger the problem rather easily, but I can't for the life of me reproduce it, so this leads me to question if local browser configs/plugins are contributing to the problem.

          Now my use case is a little different since I don't even want polling, but I've noticed that PollSCM is switched on by default with any multibranch pipeline I setup.

          Mark Austin added a comment - - edited I'm seeing this happen with one of my teams using the multi-branch pipeline setup. Two different people are able to trigger the problem rather easily, but I can't for the life of me reproduce it, so this leads me to question if local browser configs/plugins are contributing to the problem. Now my use case is a little different since I don't even want polling, but I've noticed that PollSCM is switched on by default with any multibranch pipeline I setup.

          Matt Beary added a comment -

          we are having this issue as well - in a Bitbucket Team project (via the Bitbucket Branch Source plugin), Jenkins is queueing a fresh build when no build has completed for the given commit rather than when no build has started.  It doesn't seem to matter whether the other builds are queued vs currently executing.

          Matt Beary added a comment - we are having this issue as well - in a Bitbucket Team project (via the Bitbucket Branch Source plugin), Jenkins is queueing a fresh build when no build has completed for the given commit rather than when no build has started .  It doesn't seem to matter whether the other builds are queued vs currently executing.

            Unassigned Unassigned
            jonathank Jonathan Kuleff
            Votes:
            6 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: