Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39621

Pipeline job loop after polling always find changes

      Hi,

      We have a situation where my pipeline job is configured with branch "origin/develop"
      I have a master and a tests branches as well.

      Configured with polling every 5 minutes.

      There's no change in any of the branches and still polling 'detects' one on the system-tests branch...

      Polling Log example:

      Started on Nov 9, 2016 5:58:00 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/system-tests is: afac68d836bc8dcd31826fbd56cb343cf1daf3b5
      Done. Took 0.34 sec
      Changes found
      

      I tried recreating the job, test branch... no use...

      Appreciate your help here.

      Thanks

          [JENKINS-39621] Pipeline job loop after polling always find changes

          Mark Waite added a comment -

          Can you provide sufficient detail so that the problem can be duplicated in another Jenkins instance?

          For example, can you provide step by step instructions to configure a job on a newly installed Jenkins which will show the problem?

          Alternately, could you provide a job in a Docker instance which shows how you can duplicate the problem?

          Alternately, could you provide a job definition (with a Jenkinsfile) which shows the problem using a public git repository?

          Mark Waite added a comment - Can you provide sufficient detail so that the problem can be duplicated in another Jenkins instance? For example, can you provide step by step instructions to configure a job on a newly installed Jenkins which will show the problem? Alternately, could you provide a job in a Docker instance which shows how you can duplicate the problem? Alternately, could you provide a job definition (with a Jenkinsfile) which shows the problem using a public git repository?

          Leibo added a comment -

          Hi Mark,

          I configured the same pipeline job on a new server, 2.7.3 with Git plugin version 3.0.0 and the problem doesn't occur...

          The pipeline job is a basic pool SCB (every 5 minutes), pipeline script from SCM --> Git, branch origin/develop and path pipeline/pipeline for the script.

          The polling log is smaller than the one and is not repeated 3 times like the ones we see.
          Perhaps some Git cache issues on the server?

          Started on Nov 10, 2016 9:42:00 AM
          Using strategy: Default
          [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
          using GIT_SSH to set credentials 
           > git ls-remote -h ssh://my.git.server/git/tests.git # timeout=10
          Found 3 remote heads on ssh://my.git.server/git/tests.git
          [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 1
          Done. Took 0.15 sec
          No changes
          

          Any hint on how to debug this better?

          Leibo added a comment - Hi Mark, I configured the same pipeline job on a new server, 2.7.3 with Git plugin version 3.0.0 and the problem doesn't occur... The pipeline job is a basic pool SCB (every 5 minutes), pipeline script from SCM --> Git, branch origin/develop and path pipeline/pipeline for the script. The polling log is smaller than the one and is not repeated 3 times like the ones we see. Perhaps some Git cache issues on the server? Started on Nov 10, 2016 9:42:00 AM Using strategy: Default [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop) using GIT_SSH to set credentials > git ls-remote -h ssh: //my.git.server/git/tests.git # timeout=10 Found 3 remote heads on ssh: //my.git.server/git/tests.git [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 1 Done. Took 0.15 sec No changes Any hint on how to debug this better?

          Mark Waite added a comment -

          You might investigate the advanced git settings of the problem job to see if it is defined differently than the job you created to duplicate the problem. The 3 duplicated references to the same SHA1 (in the original log file) may indicate that you've inadvertently defined multiple sources or different refspecs than you intended.

          On the problem server, you could try copying the existing problem job to a temporary job to see if the new temporary job shows the same problem. If it does not, then that may indicate the problem depends on some specific operations in the history of the job. Comparing the "Git Build Data" from the problem job and the working job may provide more insights into the differences between the jobs.

          Mark Waite added a comment - You might investigate the advanced git settings of the problem job to see if it is defined differently than the job you created to duplicate the problem. The 3 duplicated references to the same SHA1 (in the original log file) may indicate that you've inadvertently defined multiple sources or different refspecs than you intended. On the problem server, you could try copying the existing problem job to a temporary job to see if the new temporary job shows the same problem. If it does not, then that may indicate the problem depends on some specific operations in the history of the job. Comparing the "Git Build Data" from the problem job and the working job may provide more insights into the differences between the jobs.

          we are having the same issue on our Multibranch-Pipeline projects.

          we do not poll our repository but have a post-receive hook that calls https://<our jenkins>/git/notifyCommit?url=<repo path>

          in the build itself creates a tag and submits that to the repository, before 3.0.1 it was not triggering a rebuild after the tag reached the server, but with 3.0.1 it starts to build even if the SHA1 are still the same.

          i reverted the plugin back to 3.0.0 and the build loop is gone

          Axel Kämpfe added a comment - we are having the same issue on our Multibranch-Pipeline projects. we do not poll our repository but have a post-receive hook that calls https://<our jenkins>/git/notifyCommit?url=<repo path> in the build itself creates a tag and submits that to the repository, before 3.0.1 it was not triggering a rebuild after the tag reached the server, but with 3.0.1 it starts to build even if the SHA1 are still the same. i reverted the plugin back to 3.0.0 and the build loop is gone

          Mark Waite added a comment -

          akaias, I don't think there were any changes between 3.0.0 and 3.0.1 which would alter handling of the post-receive hook, or which would cause jobs to run when the SHA1 is the same. I suspect that there is something else happening in your environment, and that the build loop will eventually be visible even with git plugin 3.0.0. Unfortunately, I don't know how to duplicate the problem, and other descriptions have not been sufficient to provide steps that I can use to see the problem on an independent installation.

          Mark Waite added a comment - akaias , I don't think there were any changes between 3.0.0 and 3.0.1 which would alter handling of the post-receive hook, or which would cause jobs to run when the SHA1 is the same. I suspect that there is something else happening in your environment, and that the build loop will eventually be visible even with git plugin 3.0.0. Unfortunately, I don't know how to duplicate the problem, and other descriptions have not been sufficient to provide steps that I can use to see the problem on an independent installation.

          Leibo added a comment -

          Hi,

          Not sure it is related but - Is there a way to clean Jenkins SCM cache or something alike?

          We moved from our current Git server to a new one and once creating a new job with the new SCM details it started finding changes on every poll

          Leibo added a comment - Hi, Not sure it is related but - Is there a way to clean Jenkins SCM cache or something alike? We moved from our current Git server to a new one and once creating a new job with the new SCM details it started finding changes on every poll

          Martin Sander added a comment -

          markewaite: I was able to reproduce this (or a similar issue), by using different branches on the same repository.

          Minimal example:

          node {
              stage('checkout') {
                  git 'https://github.com/jenkinsci/pipeline-examples.git'
                  git branch: 'pipeline', url: 'https://github.com/jenkinsci/pipeline-examples.git'
              }
              
              stage('check branch') {
                  sh "git branch -a"
              }
          }
          

          Git polling log:

          Using strategy: Default
          [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master)
           > git ls-remote -h https://github.com/jenkinsci/pipeline-examples.git # timeout=10
          Found 2 remote heads on https://github.com/jenkinsci/pipeline-examples.git
          [poll] Latest remote head revision on refs/heads/master is: ab36352bf2132d079ba60853a23a2aab1e7ec239 - already built by 5
          Using strategy: Default
          [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master)
           > git ls-remote -h https://github.com/jenkinsci/pipeline-examples.git # timeout=10
          Found 2 remote heads on https://github.com/jenkinsci/pipeline-examples.git
          [poll] Latest remote head revision on refs/heads/pipeline is: 909a32731f3088b463f03811a21b56ab8c88956a
          Done. Took 1.3 sec
          Changes found
          

          I.e. Jenkins only remembers one branch when checking for "Already built", not all the branches. Don't know if changing the behavior to "all the branches" would work, but at least there should be a BIG red warning when checking out the same repo twice with different branches.

          Martin Sander added a comment - markewaite : I was able to reproduce this (or a similar issue), by using different branches on the same repository. Minimal example: node { stage( 'checkout' ) { git 'https: //github.com/jenkinsci/pipeline-examples.git' git branch: 'pipeline' , url: 'https: //github.com/jenkinsci/pipeline-examples.git' } stage( 'check branch' ) { sh "git branch -a" } } Git polling log: Using strategy: Default [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master) > git ls-remote -h https: //github.com/jenkinsci/pipeline-examples.git # timeout=10 Found 2 remote heads on https: //github.com/jenkinsci/pipeline-examples.git [poll] Latest remote head revision on refs/heads/master is: ab36352bf2132d079ba60853a23a2aab1e7ec239 - already built by 5 Using strategy: Default [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master) > git ls-remote -h https: //github.com/jenkinsci/pipeline-examples.git # timeout=10 Found 2 remote heads on https: //github.com/jenkinsci/pipeline-examples.git [poll] Latest remote head revision on refs/heads/pipeline is: 909a32731f3088b463f03811a21b56ab8c88956a Done. Took 1.3 sec Changes found I.e. Jenkins only remembers one branch when checking for "Already built", not all the branches. Don't know if changing the behavior to "all the branches" would work, but at least there should be a BIG red warning when checking out the same repo twice with different branches.

          Mark Waite added a comment -

          0x89, isn't your case using a single working directory for two different branches? It seems to me that the first git command in the checkout stage will perform a checkout of the master branch, then the second command will perform a checkout of the "pipeline" branch.

          I think that \the second checkout will overwrite the first checkout (with the 'pipeline' branch as the working branch at the end of the checkout stage), but I would expect the pipeline or the git plugin may assume there are now two SCM sources, even though they are using a single directory. If they assume there are two sources, but those two sources are using the same working directory, I would think that one of the two sources would always be "out of date".

          Does the same problem occur if you use a non-default target directory in the second git checkout?

          Mark Waite added a comment - 0x89 , isn't your case using a single working directory for two different branches? It seems to me that the first git command in the checkout stage will perform a checkout of the master branch, then the second command will perform a checkout of the "pipeline" branch. I think that \the second checkout will overwrite the first checkout (with the 'pipeline' branch as the working branch at the end of the checkout stage), but I would expect the pipeline or the git plugin may assume there are now two SCM sources, even though they are using a single directory. If they assume there are two sources, but those two sources are using the same working directory, I would think that one of the two sources would always be "out of date". Does the same problem occur if you use a non-default target directory in the second git checkout?

          Martin Sander added a comment -

          Hey markewaite, I was also able to reproduce with this (should be reproducable on any Jenkins with internet access):

          node {
              stage('checkout') {
                  ws() {
                      git 'https://github.com/jenkinsci/pipeline-examples.git'
                  }
                  ws() {
                      git branch: 'pipeline', url: 'https://github.com/jenkinsci/pipeline-examples.git'
                  }
              }
              
              stage('check branch') {
                  sh "git branch -a"
              }
          }
          

          Martin Sander added a comment - Hey markewaite , I was also able to reproduce with this (should be reproducable on any Jenkins with internet access): node { stage( 'checkout' ) { ws() { git 'https: //github.com/jenkinsci/pipeline-examples.git' } ws() { git branch: 'pipeline' , url: 'https: //github.com/jenkinsci/pipeline-examples.git' } } stage( 'check branch' ) { sh "git branch -a" } }

          Mike Neary added a comment - - edited

          I have similar issues with several of my jobs. Some of them I am able to get to "reset" by unchecking polling and building manually once, before setting up polling again. I do have one particular job thought at builds every 5 minutes if I am polling no matter what I do. Attached is a polling log. Aside from time stamps, they all look identical for this job.

          gitpolling.log

          So it looks like the job isn't recognizing that revision 4a03...1f79 has already been built. For clarity, I am pulling BRANCH-49 in the pipeline section of the Jenkins UI; this is simply to retrieve the Jenkinsfile. The Jenkinsfile pulls master for myservice and builds it, then pulls master for myotherservice and builds that.

          FWIW, I have investigated moving away from polling and using a webhook, however the Bitbucket plugin for Jenkins hasn't been maintained. I was able to get a fork of the repo to build and work for a freestyle job, but the webhook will not trigger a pipeline job, so I'm back to trying to figure out what is going on here with polling.

          Mike Neary added a comment - - edited I have similar issues with several of my jobs. Some of them I am able to get to "reset" by unchecking polling and building manually once, before setting up polling again. I do have one particular job thought at builds every 5 minutes if I am polling no matter what I do. Attached is a polling log. Aside from time stamps, they all look identical for this job. gitpolling.log So it looks like the job isn't recognizing that revision 4a03...1f79 has already been built. For clarity, I am pulling BRANCH-49 in the pipeline section of the Jenkins UI; this is simply to retrieve the Jenkinsfile. The Jenkinsfile pulls master for myservice and builds it, then pulls master for myotherservice and builds that. FWIW, I have investigated moving away from polling and using a webhook, however the Bitbucket plugin for Jenkins hasn't been maintained. I was able to get a fork of the repo to build and work for a freestyle job, but the webhook will not trigger a pipeline job, so I'm back to trying to figure out what is going on here with polling.

          Martin Sander added a comment -

          racermike:

          You are checking out two different branches of the same repository, which confuses Jenkins:

          Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git
          [poll] Latest remote head revision on refs/heads/BRANCH-49 is: 2a04...53f1 - already built by 4
          ...
          Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git
          [poll] Latest remote head revision on refs/heads/master is: 4a03...1f79

          This caused the problem for us, try to organize the code in a way that you don't require this - seems that right now this is the only workaround.

          Martin Sander added a comment - racermike : You are checking out two different branches of the same repository, which confuses Jenkins: Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git [poll] Latest remote head revision on refs/heads/BRANCH-49 is: 2a04...53f1 - already built by 4 ... Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git [poll] Latest remote head revision on refs/heads/master is: 4a03...1f79 This caused the problem for us, try to organize the code in a way that you don't require this - seems that right now this is the only workaround.

          I have the same case as described:

          markewaite: I was able to reproduce this (or a similar issue), by using different branches on the same repository.

           

          In my case the tests master branch contains the code needed not just to install a given version of the product, but also upgrade from any other version (i.e. fresh install 1.0.0, then upgrade to 1.1.1, then apply patch 1.1.1-p1, etc). However every released version has a tests branch that contains the tests that apply to it.

           

          So my job needs to run the tests applicable for 1.1.1-p10  upgrading from 1.0.0 first:

          • First i need to checkout the product repo and build 1.1.1-p10
          • then i checkout the master branch of the tests and use it to setup the product to the desired version (1.1.1-p10) using GA versions + the patch that i just built
          • Then i checkout the maint-1.1.1 branch of the same tests repo to run the tests applicable for the 1.1.1 version

          So i have one repo and 2 branches. It is not so uncommon case and currently there is no workaround as far as i know....

          Now, i can use a webhook but i already use it to trigger builds in the repo that contains the actual product code (I use Multibranch pipeline). The Multibranch pipeline is setup to monitor the product repo (where the Jenkinsfile is) so the web hook will trigger multibranch scanning of the product repo only. If i add the tests repo to the git sources of the multibranch pipeline nothing happens as the acceptance projects do not have a Jenkinsfile (i really do not know what will happen if i add one but i assume nothing good).

          Thanks

          Pavel Georgiev added a comment - I have the same case as described: markewaite : I was able to reproduce this (or a similar issue), by using different branches on the same repository.   In my case the tests master branch contains the code needed not just to install a given version of the product, but also upgrade from any other version (i.e. fresh install 1.0.0, then upgrade to 1.1.1, then apply patch 1.1.1-p1, etc). However every released version has a tests branch that contains the tests that apply to it.   So my job needs to run the tests applicable for 1.1.1-p10  upgrading from 1.0.0 first: First i need to checkout the product repo and build 1.1.1-p10 then i checkout the master branch of the tests and use it to setup the product to the desired version (1.1.1-p10) using GA versions + the patch that i just built Then i checkout the maint-1.1.1 branch of the same tests repo to run the tests applicable for the 1.1.1 version So i have one repo and 2 branches. It is not so uncommon case and currently there is no workaround as far as i know.... Now, i can use a webhook but i already use it to trigger builds in the repo that contains the actual product code (I use Multibranch pipeline). The Multibranch pipeline is setup to monitor the product repo (where the Jenkinsfile is) so the web hook will trigger multibranch scanning of the product repo only. If i add the tests repo to the git sources of the multibranch pipeline nothing happens as the acceptance projects do not have a Jenkinsfile (i really do not know what will happen if i add one but i assume nothing good). Thanks

          Mark Waite added a comment -

          It sounds like pgeorgiev, racermike, and 0x89 have all encountered a case where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches), causes the continuous detection of changes in the polling loop.

          That seems (to me) like quite a different description than was submitted with this bug report. assafl can you confirm that you are using a Jenkinsfile to checkout two different branches of the same repository?

          If assafl does not confirm that, then I recommend pgeorgiev or 0x89 or racermike submit a new bug report with that information. I think that is a bug, but I am not confident it is this bug.

          Mark Waite added a comment - It sounds like pgeorgiev , racermike , and 0x89 have all encountered a case where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches), causes the continuous detection of changes in the polling loop. That seems (to me) like quite a different description than was submitted with this bug report. assafl can you confirm that you are using a Jenkinsfile to checkout two different branches of the same repository? If assafl does not confirm that, then I recommend pgeorgiev or 0x89 or racermike submit a new bug report with that information. I think that is a bug, but I am not confident it is this bug.

          Leibo added a comment -

          Hi, not on my case, we don't checkout more than one branch.

          Leibo added a comment - Hi, not on my case, we don't checkout more than one branch.

          Pavel Georgiev added a comment - https://issues.jenkins-ci.org/browse/JENKINS-44762

          Martin Sander added a comment -

          markewaite:

          where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches)

          "using a separate directory for each of the branches" is not relevant - I was able to reproduce both using different directories and using the same directory, see https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285715&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285715 and https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285698&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285698.

          Martin Sander added a comment - markewaite : where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches) "using a separate directory for each of the branches" is not relevant - I was able to reproduce both using different directories and using the same directory, see https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285715&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285715 and https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285698&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285698 .

          Martin Sander added a comment -

          assafl could you try providing a minimal example so that we can differentiate from JENKINS-44762?

          Martin Sander added a comment - assafl could you try providing a minimal example so that we can differentiate from JENKINS-44762 ?

          Mark Waite added a comment -

          0x89 I don't think you've understood why I specifically said "using a separate directory".  

          If you don't use a separate directory for each checkout, then you mingle two different git repositories into a single directory structure.  In the general case, that will cause many different surprises (like "last checkout wins" when both repos have one or more of the same file, or "damaged branch structure information when branch names are the same").  I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it.

          Mark Waite added a comment - 0x89 I don't think you've understood why I specifically said "using a separate directory".   If you don't use a separate directory for each checkout, then you mingle two different git repositories into a single directory structure.  In the general case, that will cause many different surprises (like "last checkout wins" when both repos have one or more of the same file, or "damaged branch structure information when branch names are the same").  I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it.

          Martin Sander added a comment -

          markewaite:

          Yes, I think I did - kind of - understand why you said that. I just wanted to point out that it does not make a difference if you check out into a separate directory or not - both trigger this behavior.

          I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it.

          Absolutely agree. Although if this will continue triggering JENKINS-44762, then Jenkins should at least issue a warning when you are trying to do that - it is sometimes hard for a user to recognize that they are making that mistake..

          Martin Sander added a comment - markewaite : Yes, I think I did - kind of - understand why you said that. I just wanted to point out that it does not make a difference if you check out into a separate directory or not - both trigger this behavior. I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it. Absolutely agree. Although if this will continue triggering JENKINS-44762 , then Jenkins should at least issue a warning when you are trying to do that - it is sometimes hard for a user to recognize that they are making that mistake..

            Unassigned Unassigned
            assafl Leibo
            Votes:
            8 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated: