Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39621

Pipeline job loop after polling always find changes

      Hi,

      We have a situation where my pipeline job is configured with branch "origin/develop"
      I have a master and a tests branches as well.

      Configured with polling every 5 minutes.

      There's no change in any of the branches and still polling 'detects' one on the system-tests branch...

      Polling Log example:

      Started on Nov 9, 2016 5:58:00 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/system-tests is: afac68d836bc8dcd31826fbd56cb343cf1daf3b5
      Done. Took 0.34 sec
      Changes found
      

      I tried recreating the job, test branch... no use...

      Appreciate your help here.

      Thanks

          [JENKINS-39621] Pipeline job loop after polling always find changes

          Leibo created issue -
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]

          Mark Waite added a comment -

          Can you provide sufficient detail so that the problem can be duplicated in another Jenkins instance?

          For example, can you provide step by step instructions to configure a job on a newly installed Jenkins which will show the problem?

          Alternately, could you provide a job in a Docker instance which shows how you can duplicate the problem?

          Alternately, could you provide a job definition (with a Jenkinsfile) which shows the problem using a public git repository?

          Mark Waite added a comment - Can you provide sufficient detail so that the problem can be duplicated in another Jenkins instance? For example, can you provide step by step instructions to configure a job on a newly installed Jenkins which will show the problem? Alternately, could you provide a job in a Docker instance which shows how you can duplicate the problem? Alternately, could you provide a job definition (with a Jenkinsfile) which shows the problem using a public git repository?

          Leibo added a comment -

          Hi Mark,

          I configured the same pipeline job on a new server, 2.7.3 with Git plugin version 3.0.0 and the problem doesn't occur...

          The pipeline job is a basic pool SCB (every 5 minutes), pipeline script from SCM --> Git, branch origin/develop and path pipeline/pipeline for the script.

          The polling log is smaller than the one and is not repeated 3 times like the ones we see.
          Perhaps some Git cache issues on the server?

          Started on Nov 10, 2016 9:42:00 AM
          Using strategy: Default
          [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
          using GIT_SSH to set credentials 
           > git ls-remote -h ssh://my.git.server/git/tests.git # timeout=10
          Found 3 remote heads on ssh://my.git.server/git/tests.git
          [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 1
          Done. Took 0.15 sec
          No changes
          

          Any hint on how to debug this better?

          Leibo added a comment - Hi Mark, I configured the same pipeline job on a new server, 2.7.3 with Git plugin version 3.0.0 and the problem doesn't occur... The pipeline job is a basic pool SCB (every 5 minutes), pipeline script from SCM --> Git, branch origin/develop and path pipeline/pipeline for the script. The polling log is smaller than the one and is not repeated 3 times like the ones we see. Perhaps some Git cache issues on the server? Started on Nov 10, 2016 9:42:00 AM Using strategy: Default [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop) using GIT_SSH to set credentials > git ls-remote -h ssh: //my.git.server/git/tests.git # timeout=10 Found 3 remote heads on ssh: //my.git.server/git/tests.git [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 1 Done. Took 0.15 sec No changes Any hint on how to debug this better?

          Mark Waite added a comment -

          You might investigate the advanced git settings of the problem job to see if it is defined differently than the job you created to duplicate the problem. The 3 duplicated references to the same SHA1 (in the original log file) may indicate that you've inadvertently defined multiple sources or different refspecs than you intended.

          On the problem server, you could try copying the existing problem job to a temporary job to see if the new temporary job shows the same problem. If it does not, then that may indicate the problem depends on some specific operations in the history of the job. Comparing the "Git Build Data" from the problem job and the working job may provide more insights into the differences between the jobs.

          Mark Waite added a comment - You might investigate the advanced git settings of the problem job to see if it is defined differently than the job you created to duplicate the problem. The 3 duplicated references to the same SHA1 (in the original log file) may indicate that you've inadvertently defined multiple sources or different refspecs than you intended. On the problem server, you could try copying the existing problem job to a temporary job to see if the new temporary job shows the same problem. If it does not, then that may indicate the problem depends on some specific operations in the history of the job. Comparing the "Git Build Data" from the problem job and the working job may provide more insights into the differences between the jobs.

          Axel Kämpfe added a comment -

          we are having the same issue on our Multibranch-Pipeline projects.

          we do not poll our repository but have a post-receive hook that calls https://<our jenkins>/git/notifyCommit?url=<repo path>

          in the build itself creates a tag and submits that to the repository, before 3.0.1 it was not triggering a rebuild after the tag reached the server, but with 3.0.1 it starts to build even if the SHA1 are still the same.

          i reverted the plugin back to 3.0.0 and the build loop is gone

          Axel Kämpfe added a comment - we are having the same issue on our Multibranch-Pipeline projects. we do not poll our repository but have a post-receive hook that calls https://<our jenkins>/git/notifyCommit?url=<repo path> in the build itself creates a tag and submits that to the repository, before 3.0.1 it was not triggering a rebuild after the tag reached the server, but with 3.0.1 it starts to build even if the SHA1 are still the same. i reverted the plugin back to 3.0.0 and the build loop is gone

          Mark Waite added a comment -

          akaias, I don't think there were any changes between 3.0.0 and 3.0.1 which would alter handling of the post-receive hook, or which would cause jobs to run when the SHA1 is the same. I suspect that there is something else happening in your environment, and that the build loop will eventually be visible even with git plugin 3.0.0. Unfortunately, I don't know how to duplicate the problem, and other descriptions have not been sufficient to provide steps that I can use to see the problem on an independent installation.

          Mark Waite added a comment - akaias , I don't think there were any changes between 3.0.0 and 3.0.1 which would alter handling of the post-receive hook, or which would cause jobs to run when the SHA1 is the same. I suspect that there is something else happening in your environment, and that the build loop will eventually be visible even with git plugin 3.0.0. Unfortunately, I don't know how to duplicate the problem, and other descriptions have not been sufficient to provide steps that I can use to see the problem on an independent installation.

          Leibo added a comment -

          Hi,

          Not sure it is related but - Is there a way to clean Jenkins SCM cache or something alike?

          We moved from our current Git server to a new one and once creating a new job with the new SCM details it started finding changes on every poll

          Leibo added a comment - Hi, Not sure it is related but - Is there a way to clean Jenkins SCM cache or something alike? We moved from our current Git server to a new one and once creating a new job with the new SCM details it started finding changes on every poll

          Martin Sander added a comment -

          markewaite: I was able to reproduce this (or a similar issue), by using different branches on the same repository.

          Minimal example:

          node {
              stage('checkout') {
                  git 'https://github.com/jenkinsci/pipeline-examples.git'
                  git branch: 'pipeline', url: 'https://github.com/jenkinsci/pipeline-examples.git'
              }
              
              stage('check branch') {
                  sh "git branch -a"
              }
          }
          

          Git polling log:

          Using strategy: Default
          [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master)
           > git ls-remote -h https://github.com/jenkinsci/pipeline-examples.git # timeout=10
          Found 2 remote heads on https://github.com/jenkinsci/pipeline-examples.git
          [poll] Latest remote head revision on refs/heads/master is: ab36352bf2132d079ba60853a23a2aab1e7ec239 - already built by 5
          Using strategy: Default
          [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master)
           > git ls-remote -h https://github.com/jenkinsci/pipeline-examples.git # timeout=10
          Found 2 remote heads on https://github.com/jenkinsci/pipeline-examples.git
          [poll] Latest remote head revision on refs/heads/pipeline is: 909a32731f3088b463f03811a21b56ab8c88956a
          Done. Took 1.3 sec
          Changes found
          

          I.e. Jenkins only remembers one branch when checking for "Already built", not all the branches. Don't know if changing the behavior to "all the branches" would work, but at least there should be a BIG red warning when checking out the same repo twice with different branches.

          Martin Sander added a comment - markewaite : I was able to reproduce this (or a similar issue), by using different branches on the same repository. Minimal example: node { stage( 'checkout' ) { git 'https: //github.com/jenkinsci/pipeline-examples.git' git branch: 'pipeline' , url: 'https: //github.com/jenkinsci/pipeline-examples.git' } stage( 'check branch' ) { sh "git branch -a" } } Git polling log: Using strategy: Default [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master) > git ls-remote -h https: //github.com/jenkinsci/pipeline-examples.git # timeout=10 Found 2 remote heads on https: //github.com/jenkinsci/pipeline-examples.git [poll] Latest remote head revision on refs/heads/master is: ab36352bf2132d079ba60853a23a2aab1e7ec239 - already built by 5 Using strategy: Default [poll] Last Built Revision: Revision ab36352bf2132d079ba60853a23a2aab1e7ec239 (refs/remotes/origin/master) > git ls-remote -h https: //github.com/jenkinsci/pipeline-examples.git # timeout=10 Found 2 remote heads on https: //github.com/jenkinsci/pipeline-examples.git [poll] Latest remote head revision on refs/heads/pipeline is: 909a32731f3088b463f03811a21b56ab8c88956a Done. Took 1.3 sec Changes found I.e. Jenkins only remembers one branch when checking for "Already built", not all the branches. Don't know if changing the behavior to "all the branches" would work, but at least there should be a BIG red warning when checking out the same repo twice with different branches.

          Mark Waite added a comment -

          0x89, isn't your case using a single working directory for two different branches? It seems to me that the first git command in the checkout stage will perform a checkout of the master branch, then the second command will perform a checkout of the "pipeline" branch.

          I think that \the second checkout will overwrite the first checkout (with the 'pipeline' branch as the working branch at the end of the checkout stage), but I would expect the pipeline or the git plugin may assume there are now two SCM sources, even though they are using a single directory. If they assume there are two sources, but those two sources are using the same working directory, I would think that one of the two sources would always be "out of date".

          Does the same problem occur if you use a non-default target directory in the second git checkout?

          Mark Waite added a comment - 0x89 , isn't your case using a single working directory for two different branches? It seems to me that the first git command in the checkout stage will perform a checkout of the master branch, then the second command will perform a checkout of the "pipeline" branch. I think that \the second checkout will overwrite the first checkout (with the 'pipeline' branch as the working branch at the end of the checkout stage), but I would expect the pipeline or the git plugin may assume there are now two SCM sources, even though they are using a single directory. If they assume there are two sources, but those two sources are using the same working directory, I would think that one of the two sources would always be "out of date". Does the same problem occur if you use a non-default target directory in the second git checkout?

            Unassigned Unassigned
            assafl Leibo
            Votes:
            8 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated: