Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52816

scm poll triggers even there is no change in repository(bitbucket)

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • we are using Jenkins LTS 2.176.3 on hosted on amazonlinux , job-dsl - 1.74 , git plugin - 4.0.0-rc

      step 1: create a seed job 
      ```
      String basepath = 'Common/Java/testapp'

      String giturl = 'git@bitbucket.org:abc/testapp'

      pipelineJob("$basepath/1.Build-Dev-Int") {
      logRotator {
      numToKeep(10)
      artifactNumToKeep(10)
      }
      parameters {
      stringParam('USE_GIT_BRANCH', 'master')
      }
      throttleConcurrentBuilds {
      maxTotal(1)
      }

      triggers {
      scm('H/15 * * * *')
      }

      definition {
      cpsScm {
      scm {
      git

      { remote \{ url(giturl) }

      branches('master')
      extensions { }
      }
      scriptPath('pipeline/pipeline_dev_int.groovy')
      }
      }
      }
      }```
      step 2 : create pipeline_dev_int.groovy

      ```@Library('sharedlibrary@master')

      node()

      remaining groovy script```
       

      step 3:     configured shared library in manage jenkins
      even though there is no change in repo it is triggering jobs 
       

       

      whenever a branch/ commit made to the shared library even though there is no change in testapp repo it triggering the job for every 15 min and found this in polling history

      Using strategy: Default
      [poll] Last Built Revision: Revision 6717866352d0d5a5e9b41da88a799993ab1d4191 (origin/master)
      No credentials specified
      > git --version # timeout=10
      > git ls-remote -h git@bitbucket.org:abc/testapp # timeout=10
      Found 3 remote heads on git@bitbucket.org:abc/testapp
      [poll] Latest remote head revision on refs/heads/master is: 6717866352d0d5a5e9b41da88a799993ab1d4191 - already built by 97
      Using strategy: Specific revision
      [poll] Last Built Revision: Revision a7cf4fd0c7484765c4a5166a36613c387aee8501 (surya/MCDCA-1412-ECR-NEW-ACCOUNT)
      No credentials specified
      > git --version # timeout=10
      > git ls-remote -h git@bitbucket.org:abc/sharedlibrary # timeout=10
      Found 3 remote heads on git@bitbucket.org:abc/sharedlibrary
      [poll] Latest remote head revision on refs/heads/surya/MCDCA-1412-ECR-NEW-ACCOUNT is: ab8eb071afb815bca04fd4e50a991ae54ea37e8c
      Using strategy: Default
      [poll] Last Built Revision: Revision 6717866352d0d5a5e9b41da88a799993ab1d4191 (origin/master)
      using credential 6e83116e-4fb6-491a-a2c9-9ac9d51d4765
      > git --version # timeout=10
      using GIT_SSH to set credentials
      > git ls-remote -h git@bitbucket.org:abc/testapp # timeout=10
      Found 3 remote heads on git@bitbucket.org:abc/testapp
      [poll] Latest remote head revision on refs/heads/master is: 6717866352d0d5a5e9b41da88a799993ab1d4191 - already built by 97
      Done. Took 8.6 sec
      Changes found

      Polling Log

      This page captures the polling log that triggered this build.
      Started on Jul 30, 2018 7:15:00 PM Using strategy: Default [poll] Last Built Revision: Revision 80ae1a7997f138af5faa0e594eca7e88eb174026 (origin/master) > git --version # timeout=10 > git ls-remote -h git@bitbucket.org:xxx/xxxxxxx # timeout=10 Found 1 remote heads on git@bitbucket.org:xx/xxxxxx[poll] Latest remote head revision on refs/heads/master is: 80ae1a7997f138af5faa0e594eca7e88eb174026 - already built by 185 no polling baseline in /var/lib/jenkins/workspace/Sandbox/xxxx/xxxx/1.Build-Dev-Int@libs/xxx on Using strategy: Default > git --version # timeout=10 using GIT_SSH to set credentials Bitbucket SSH Key > git ls-remote -h git@bitbucket.org:xxxxx.git # timeout=10 Found 1 remote heads on git@bitbucket.org:xx/xxxx.git [poll] Latest remote head revision on refs/heads/master is: 80ae1a7997f138af5faa0e594eca7e88eb174026 Done. Took 2.7 sec Changes found

          [JENKINS-52816] scm poll triggers even there is no change in repository(bitbucket)

          Ashish Sharma added a comment -

          Hi markewaite / All,

          Hope everyone is keeping safe.

          We are also facing similar issue where git just keeps polling even if no new commits in the repo, just checking whether to create a new bug for the same, or it is the similar scenario as this issue?

          1. Git Plugin: v4.3.0
          2. Bitbucket Server: v6.10.4
          3. Webhook from Bitbucket to jenkins is not enabled due to org-wise admin policy
          4. Git polling enabled to poll every 10 mins
          5. Pipeline (declarative) script is uploaded in SCM [note that pipeline script is also in the same scm repo where the repo code lies which is being polled]
          6. Lightweight checkpout option ticked for pipeline script
          7. GitSCM plugin used to define git checkout and identify repository to be polled
          8. We have 2 scenarios:
            • First one is to trigger build only if new tag with a matching pattern is added to a commit, using below snippet for this:

              checkout([$class: 'GitSCM', branches: [[name: '**/tags/release-*']],  doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],  userRemoteConfigs: [[refspec: 'refs/tags/*:refs/remotes/origin/tags/*',  url: 'ssh://git@<git-url>.git']]])

            • Second scenario is to poll a branch directly for any commits, using below for this:

              checkout([$class: 'GitSCM', branches: [[name: '*/develop']],  doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],  userRemoteConfigs: [[url: 'ssh://git@<git-url>.git']]])

          9. In both of the scenarios, once first successful trigger is done, polling keep finding false positives and triggering the build even though no new commit/tag is pushed
          10. Sharing git polling log sample for the second scenario (first scenario logs are too long due to multiple tags in repo, can share if required), which should have been "no changes" but results in "changes found". Note that even on subsequent runs, it keeps producing same output, with exact same values of <prev_commit_id> and <new_commit_id>. and hence keeps finding changes.

               Git Polling Log

            Started on Jul 6, 2020 11:54:00 AM
            Using strategy: Default
            [poll] Last Built Revision: Revision <prev_commit_id>(refs/remotes/origin/<branch_name>)
            No credentials specified
            > git --version # timeout=10
            > git ls-remote -h – ssh://git@<git_url>.git # timeout=10
            Found 59 remote heads on ssh://git@<git_url>.git
            [poll] Latest remote head revision on refs/heads/jenkins_polling_check is: <prev_commit_id> - already built by 58
            Using strategy: Default [poll] Last Built Revision: Revision <prev_commit_id> (refs/remotes/origin/<branch_name>)
            No credentials specified
            > git --version # timeout=10
            > git ls-remote -h – ssh://git@<git_url>.git # timeout=10
            Found 59 remote heads on ssh://git@<git_url>.git
            [poll] Latest remote head revision on refs/heads/develop is: <new_commit_id> Done. Took 0.86 sec
            Changes found

          11. Build time for the job is less than pipeline polling interval. Build time <1 min, polling interval 10 mins.
          12. Also note that if same pipeline script is used as "inline scipt" in pipeline console instead of using "pipeline script from scm", this works fine 9/10 times (no false polling), still get same issue but frequency is very less in case of inline script. 
          13. However while using pipeline script in scm, this fails almost everytime. Our projects requires pipeline code to be in scm.

          Kindly suggest? Is this known issue or new issue? Is this related to pipeline script and the actual code repo (to be polled) staying in the same git repository? Please let me know if something is not clear.

          Thanks. 

          Ashish Sharma added a comment - Hi markewaite / All, Hope everyone is keeping safe. We are also facing similar issue where git just keeps polling even if no new commits in the repo, just checking whether to create a new bug for the same, or it is the similar scenario as this issue? Git Plugin: v4.3.0 Bitbucket Server: v6.10.4 Webhook from Bitbucket to jenkins is not enabled due to org-wise admin policy Git polling enabled to poll every 10 mins Pipeline (declarative) script is uploaded in SCM [note that pipeline script is also in the same scm repo where the repo code lies which is being polled] Lightweight checkpout option ticked for pipeline script GitSCM plugin used to define git checkout and identify repository to be polled We have 2 scenarios: First one is to trigger build only if new tag with a matching pattern is added to a commit, using below snippet for this: checkout([$class: 'GitSCM', branches: [ [name: '**/tags/release-*'] ],  doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],  userRemoteConfigs: [ [refspec: 'refs/tags/*:refs/remotes/origin/tags/*',  url: 'ssh://git@<git-url>.git'] ]]) Second scenario is to poll a branch directly for any commits, using below for this: checkout([$class: 'GitSCM', branches: [ [name: '*/develop'] ],  doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [],  userRemoteConfigs: [ [url: 'ssh://git@<git-url>.git'] ]]) In both of the scenarios, once first successful trigger is done, polling keep finding false positives and triggering the build even though no new commit/tag is pushed Sharing git polling log sample for the second scenario (first scenario logs are too long due to multiple tags in repo, can share if required), which should have been "no changes" but results in "changes found". Note that even on subsequent runs, it keeps producing same output, with exact same values of <prev_commit_id> and <new_commit_id>. and hence keeps finding changes.     Git Polling Log Started on Jul 6, 2020 11:54:00 AM Using strategy: Default [poll] Last Built Revision: Revision < prev_commit_id >(refs/remotes/origin/<branch_name>) No credentials specified > git --version # timeout=10 > git ls-remote -h – ssh://git@<git_url>.git # timeout=10 Found 59 remote heads on ssh://git@<git_url>.git [poll] Latest remote head revision on refs/heads/jenkins_polling_check is: <prev_commit_id> - already built by 58 Using strategy: Default [poll] Last Built Revision: Revision < prev_commit_id > (refs/remotes/origin/<branch_name>) No credentials specified > git --version # timeout=10 > git ls-remote -h – ssh://git@<git_url>.git # timeout=10 Found 59 remote heads on ssh://git@<git_url>.git [poll] Latest remote head revision on refs/heads/develop is: < new_commit_id > Done. Took 0.86 sec Changes found Build time for the job is less than pipeline polling interval. Build time <1 min, polling interval 10 mins. Also note that if same pipeline script is used as "inline scipt" in pipeline console instead of using "pipeline script from scm", this works fine 9/10 times (no false polling), still get same issue but frequency is very less in case of inline script.  However while using pipeline script in scm, this fails almost everytime. Our projects requires pipeline code to be in scm. Kindly suggest? Is this known issue or new issue? Is this related to pipeline script and the actual code repo (to be polled) staying in the same git repository? Please let me know if something is not clear. Thanks. 

          Mark Waite added a comment -

          ashisharma888 I don't know if that is the same issue or a different issue.

          I assume there is something in the workspace that is causing the git plugin to believe that the new_commit_id has not been built, even though you indicate that it has been built. You might try evaluating https://github.com/jenkinsci/git-plugin/pull/579 in your environment to see if the changes proposed there are a help for the case that you're seeing.

          Mark Waite added a comment - ashisharma888 I don't know if that is the same issue or a different issue. I assume there is something in the workspace that is causing the git plugin to believe that the new_commit_id has not been built, even though you indicate that it has been built. You might try evaluating https://github.com/jenkinsci/git-plugin/pull/579 in your environment to see if the changes proposed there are a help for the case that you're seeing.

          Ashish Sharma added a comment -

          Thanks markewaite, so this means it is alright to keep the pipeline script in the same repo/branch as the build code? Issue might be something else.

          Also, can you please help to guide how to use the pull request and update my current plugin from the same? Sorry this would be the first time. Thanks.

          Ashish Sharma added a comment - Thanks markewaite , so this means it is alright to keep the pipeline script in the same repo/branch as the build code? Issue might be something else. Also, can you please help to guide how to use the pull request and update my current plugin from the same? Sorry this would be the first time. Thanks.

          Mark Waite added a comment - - edited

          Yes, it is good to keep the pipeline script in the same repo and branch as the build code. I do that with many repositories on many different source control providers, including Bitbucket.

          You'll need to build the git plugin locally by following the instructions in the CONTRIBUTING file:

          $ git clone https://github.com/jenkinsci/git-plugin
          $ cd git-plugin
          $ mvn clean install
          

          After you've compiled it, you'll need to include the proposed changes from the pull request:

          $ git remote add jacob-keller https://github.com/jacob-keller/git-plugin
          $ git fetch jacob-keller
          $ git checkout -b testing-jk-fix-polling-local-branches
          $ git merge jacob-keller/jk-fix-polling-local-branches
          $ # Resolve the merge conflicts
          $ mvn clean install
          

          I did those those steps up through the install.

          You can download the file I generated as git.hpi

          Mark Waite added a comment - - edited Yes, it is good to keep the pipeline script in the same repo and branch as the build code. I do that with many repositories on many different source control providers, including Bitbucket. You'll need to build the git plugin locally by following the instructions in the CONTRIBUTING file: $ git clone https://github.com/jenkinsci/git-plugin $ cd git-plugin $ mvn clean install After you've compiled it, you'll need to include the proposed changes from the pull request: $ git remote add jacob-keller https://github.com/jacob-keller/git-plugin $ git fetch jacob-keller $ git checkout -b testing-jk-fix-polling-local-branches $ git merge jacob-keller/jk-fix-polling-local-branches $ # Resolve the merge conflicts $ mvn clean install I did those those steps up through the install. You can download the file I generated as git.hpi

          Ashish Sharma added a comment - - edited

          Thank you very much markewaite. After trying out the new hpi, and a various scenarios for a while here are my additional observations:

          • All of the behaviors mentioned below are same for latest official release of plugin and the version shared using the pull request mentioned above.
          • It looks like that the plugin is marking branch for polling which is specified in branch specifier field to read pipeline script from scm path
          • If I need to actually poll for changes in other branch or a tag pattern, it keeps failing
          • Our intention is to use branch / tag provided in GitSCM class of pipeline script to be marked for polling.
          • Also again sharing the same Git polling log I shared above, I think I also misread it last time:

            Started on Jul 6, 2020 11:54:00 AM
            Using strategy: Default
            [poll] Last Built Revision: Revision <prev_commit_id>(refs/remotes/origin/<branch_name>)
            No credentials specified
            > git --version # timeout=10
            > git ls-remote -h – ssh://git@<git_url>.git # timeout=10
            Found 59 remote heads on ssh://git@<git_url>.git
            [poll] Latest remote head revision on refs/heads/jenkins_polling_check is: <prev_commit_id> - already built by 58
            Using strategy: Default [poll] Last Built Revision: Revision <prev_commit_id> (refs/remotes/origin/<branch_name>)
            No credentials specified
            > git --version # timeout=10
            > git ls-remote -h – ssh://git@<git_url>.git # timeout=10
            Found 59 remote heads on ssh://git@<git_url>.git
            [poll] Latest remote head revision on refs/heads/develop is: <new_commit_id> Done. Took 0.86 sec
            Changes found

          • In this example, polling is checking against last built commit of branch refs/remotes/origin/<branch_name> where pipeline script lies, however it is comparing with the latest commit of branch refs/heads/develop which is present in the branch field of GitCSM class of the pipeline script. It keeps finding differences and hence keep marking polling as changes found.**

            So I think the only question is how can I remove my pipeline script branch from git polling, so that it only polls for changes based on branch specifier provided in GitSCM of the pipeline script?

          Apologies if I was not able to explain this clearly last time, it only became clear to me post trying and testing the scenarios, and trying to read polling log and build console out in more detail.

          Ashish Sharma added a comment - - edited Thank you very much markewaite . After trying out the new hpi, and a various scenarios for a while here are my additional observations: All of the behaviors mentioned below are same for latest official release of plugin and the version shared using the pull request mentioned above. It looks like that the plugin is marking branch for polling which is specified in branch specifier field to read pipeline script from scm path If I need to actually poll for changes in other branch or a tag pattern, it keeps failing Our intention is to use branch / tag provided in GitSCM class of pipeline script to be marked for polling. Also again sharing the same Git polling log I shared above, I think I also misread it last time: Started on Jul 6, 2020 11:54:00 AM Using strategy: Default [poll]  Last Built Revision: Revision < prev_commit_id >( refs/remotes/origin/<branch_name> ) No credentials specified > git --version # timeout=10 > git ls-remote -h – ssh://git@<git_url>.git # timeout=10 Found 59 remote heads on ssh://git@<git_url>.git [poll]  Latest remote head revision on refs/heads/jenkins_polling_check is: <prev_commit_id> - already built by 58 Using strategy: Default  [poll]  Last Built Revision: Revision < prev_commit_id > ( refs/remotes/origin/<branch_name> ) No credentials specified > git --version # timeout=10 > git ls-remote -h – ssh://git@<git_url>.git # timeout=10 Found 59 remote heads on ssh://git@<git_url>.git [poll]  Latest remote head revision on refs/heads/develop is: < new_commit_id > Done. Took 0.86 sec Changes found In this example, polling is checking against last built commit of branch  refs/remotes/origin/<branch_name> where pipeline script lies , however it is comparing with the latest commit of branch  refs/heads/develop which is present in the branch field of GitCSM class of the pipeline script. It keeps finding differences and hence keep marking polling as changes found.** So I think the only question is how can I remove my pipeline script branch from git polling, so that it only polls for changes based on branch specifier provided in GitSCM of the pipeline script ? Apologies if I was not able to explain this clearly last time, it only became clear to me post trying and testing the scenarios, and trying to read polling log and build console out in more detail.

          Mark Waite added a comment -

          ashisharma888 there is not a way to do what you're proposing with a pipeline job that reads the Pipeline from SCM (in other words, an SCM pipeline that is not a multibranch pipeline). It might be possible from a multibranch pipeline job that is defined to select only a subset of branches.

          Mark Waite added a comment - ashisharma888 there is not a way to do what you're proposing with a pipeline job that reads the Pipeline from SCM (in other words, an SCM pipeline that is not a multibranch pipeline). It might be possible from a multibranch pipeline job that is defined to select only a subset of branches.

          Ashish Sharma added a comment - - edited

          Ohh really, I thought it might be a common use case. Let me do similar POC around multi branch plugin.

          Thank you very much markewaite. You are awesome. Stay safe. Cheers.

          Ashish Sharma added a comment - - edited Ohh really, I thought it might be a common use case. Let me do similar POC around multi branch plugin. Thank you very much markewaite . You are awesome. Stay safe. Cheers.

          Ashish Sharma added a comment -

          markewaite Just to close the loop here, the use case we have is not resolved using multi branch pipeline as well as it can help to select which branch to run pipeline in but again can't solve the polling challenge I have. So to be able to get positive polling results for our case (Poll for a specific tag on new commit, and trigger build from the tag once the new commit comes), I have just pasted pipeline script directly in the jenkins pipeline script console.

          Ashish Sharma added a comment - markewaite Just to close the loop here, the use case we have is not resolved using multi branch pipeline as well as it can help to select which branch to run pipeline in but again can't solve the polling challenge I have. So to be able to get positive polling results for our case (Poll for a specific tag on new commit, and trigger build from the tag once the new commit comes), I have just pasted pipeline script directly in the jenkins pipeline script console.

          This issue does not seem to be related to Job DSL. Can we close the issue?

          Daniel Spilker added a comment - This issue does not seem to be related to Job DSL. Can we close the issue?

          Stanisław added a comment -

          daspilker the bug is not solved at all, why you want to close it?

          We are still waiting for a fix or workaround

          Stanisław added a comment - daspilker the bug is not solved at all, why you want to close it? We are still waiting for a fix or workaround

            daspilker Daniel Spilker
            yrsurya suryatej yaramada
            Votes:
            6 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: