Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39621

Pipeline job loop after polling always find changes

      Hi,

      We have a situation where my pipeline job is configured with branch "origin/develop"
      I have a master and a tests branches as well.

      Configured with polling every 5 minutes.

      There's no change in any of the branches and still polling 'detects' one on the system-tests branch...

      Polling Log example:

      Started on Nov 9, 2016 5:58:00 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/develop is: 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 - already built by 4
      Using strategy: Default
      [poll] Last Built Revision: Revision 9b3f801b5ce8d6649c25ddd4a573801c9a6b5a23 (refs/remotes/origin/develop)
       > /usr/bin/git ls-remote -h ssh://my.git.server/git/test.git # timeout=10
      Found 3 remote heads on ssh://my.git.server/git/test.git
      [poll] Latest remote head revision on refs/heads/system-tests is: afac68d836bc8dcd31826fbd56cb343cf1daf3b5
      Done. Took 0.34 sec
      Changes found
      

      I tried recreating the job, test branch... no use...

      Appreciate your help here.

      Thanks

          [JENKINS-39621] Pipeline job loop after polling always find changes

          Mike Neary added a comment - - edited

          I have similar issues with several of my jobs. Some of them I am able to get to "reset" by unchecking polling and building manually once, before setting up polling again. I do have one particular job thought at builds every 5 minutes if I am polling no matter what I do. Attached is a polling log. Aside from time stamps, they all look identical for this job.

          gitpolling.log

          So it looks like the job isn't recognizing that revision 4a03...1f79 has already been built. For clarity, I am pulling BRANCH-49 in the pipeline section of the Jenkins UI; this is simply to retrieve the Jenkinsfile. The Jenkinsfile pulls master for myservice and builds it, then pulls master for myotherservice and builds that.

          FWIW, I have investigated moving away from polling and using a webhook, however the Bitbucket plugin for Jenkins hasn't been maintained. I was able to get a fork of the repo to build and work for a freestyle job, but the webhook will not trigger a pipeline job, so I'm back to trying to figure out what is going on here with polling.

          Mike Neary added a comment - - edited I have similar issues with several of my jobs. Some of them I am able to get to "reset" by unchecking polling and building manually once, before setting up polling again. I do have one particular job thought at builds every 5 minutes if I am polling no matter what I do. Attached is a polling log. Aside from time stamps, they all look identical for this job. gitpolling.log So it looks like the job isn't recognizing that revision 4a03...1f79 has already been built. For clarity, I am pulling BRANCH-49 in the pipeline section of the Jenkins UI; this is simply to retrieve the Jenkinsfile. The Jenkinsfile pulls master for myservice and builds it, then pulls master for myotherservice and builds that. FWIW, I have investigated moving away from polling and using a webhook, however the Bitbucket plugin for Jenkins hasn't been maintained. I was able to get a fork of the repo to build and work for a freestyle job, but the webhook will not trigger a pipeline job, so I'm back to trying to figure out what is going on here with polling.

          Martin Sander added a comment -

          racermike:

          You are checking out two different branches of the same repository, which confuses Jenkins:

          Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git
          [poll] Latest remote head revision on refs/heads/BRANCH-49 is: 2a04...53f1 - already built by 4
          ...
          Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git
          [poll] Latest remote head revision on refs/heads/master is: 4a03...1f79

          This caused the problem for us, try to organize the code in a way that you don't require this - seems that right now this is the only workaround.

          Martin Sander added a comment - racermike : You are checking out two different branches of the same repository, which confuses Jenkins: Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git [poll] Latest remote head revision on refs/heads/BRANCH-49 is: 2a04...53f1 - already built by 4 ... Found 6 remote heads on ssh://git@bitbucket.foo:7999/ba/myservice.git [poll] Latest remote head revision on refs/heads/master is: 4a03...1f79 This caused the problem for us, try to organize the code in a way that you don't require this - seems that right now this is the only workaround.

          I have the same case as described:

          markewaite: I was able to reproduce this (or a similar issue), by using different branches on the same repository.

           

          In my case the tests master branch contains the code needed not just to install a given version of the product, but also upgrade from any other version (i.e. fresh install 1.0.0, then upgrade to 1.1.1, then apply patch 1.1.1-p1, etc). However every released version has a tests branch that contains the tests that apply to it.

           

          So my job needs to run the tests applicable for 1.1.1-p10  upgrading from 1.0.0 first:

          • First i need to checkout the product repo and build 1.1.1-p10
          • then i checkout the master branch of the tests and use it to setup the product to the desired version (1.1.1-p10) using GA versions + the patch that i just built
          • Then i checkout the maint-1.1.1 branch of the same tests repo to run the tests applicable for the 1.1.1 version

          So i have one repo and 2 branches. It is not so uncommon case and currently there is no workaround as far as i know....

          Now, i can use a webhook but i already use it to trigger builds in the repo that contains the actual product code (I use Multibranch pipeline). The Multibranch pipeline is setup to monitor the product repo (where the Jenkinsfile is) so the web hook will trigger multibranch scanning of the product repo only. If i add the tests repo to the git sources of the multibranch pipeline nothing happens as the acceptance projects do not have a Jenkinsfile (i really do not know what will happen if i add one but i assume nothing good).

          Thanks

          Pavel Georgiev added a comment - I have the same case as described: markewaite : I was able to reproduce this (or a similar issue), by using different branches on the same repository.   In my case the tests master branch contains the code needed not just to install a given version of the product, but also upgrade from any other version (i.e. fresh install 1.0.0, then upgrade to 1.1.1, then apply patch 1.1.1-p1, etc). However every released version has a tests branch that contains the tests that apply to it.   So my job needs to run the tests applicable for 1.1.1-p10  upgrading from 1.0.0 first: First i need to checkout the product repo and build 1.1.1-p10 then i checkout the master branch of the tests and use it to setup the product to the desired version (1.1.1-p10) using GA versions + the patch that i just built Then i checkout the maint-1.1.1 branch of the same tests repo to run the tests applicable for the 1.1.1 version So i have one repo and 2 branches. It is not so uncommon case and currently there is no workaround as far as i know.... Now, i can use a webhook but i already use it to trigger builds in the repo that contains the actual product code (I use Multibranch pipeline). The Multibranch pipeline is setup to monitor the product repo (where the Jenkinsfile is) so the web hook will trigger multibranch scanning of the product repo only. If i add the tests repo to the git sources of the multibranch pipeline nothing happens as the acceptance projects do not have a Jenkinsfile (i really do not know what will happen if i add one but i assume nothing good). Thanks

          Mark Waite added a comment -

          It sounds like pgeorgiev, racermike, and 0x89 have all encountered a case where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches), causes the continuous detection of changes in the polling loop.

          That seems (to me) like quite a different description than was submitted with this bug report. assafl can you confirm that you are using a Jenkinsfile to checkout two different branches of the same repository?

          If assafl does not confirm that, then I recommend pgeorgiev or 0x89 or racermike submit a new bug report with that information. I think that is a bug, but I am not confident it is this bug.

          Mark Waite added a comment - It sounds like pgeorgiev , racermike , and 0x89 have all encountered a case where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches), causes the continuous detection of changes in the polling loop. That seems (to me) like quite a different description than was submitted with this bug report. assafl can you confirm that you are using a Jenkinsfile to checkout two different branches of the same repository? If assafl does not confirm that, then I recommend pgeorgiev or 0x89 or racermike submit a new bug report with that information. I think that is a bug, but I am not confident it is this bug.

          Leibo added a comment -

          Hi, not on my case, we don't checkout more than one branch.

          Leibo added a comment - Hi, not on my case, we don't checkout more than one branch.

          Pavel Georgiev added a comment - https://issues.jenkins-ci.org/browse/JENKINS-44762

          Martin Sander added a comment -

          markewaite:

          where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches)

          "using a separate directory for each of the branches" is not relevant - I was able to reproduce both using different directories and using the same directory, see https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285715&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285715 and https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285698&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285698.

          Martin Sander added a comment - markewaite : where using a Jenkinsfile to checkout two different branches of the same repository into a workspace (using a separate directory for each of the branches) "using a separate directory for each of the branches" is not relevant - I was able to reproduce both using different directories and using the same directory, see https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285715&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285715 and https://issues.jenkins-ci.org/browse/JENKINS-39621?focusedCommentId=285698&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-285698 .

          Martin Sander added a comment -

          assafl could you try providing a minimal example so that we can differentiate from JENKINS-44762?

          Martin Sander added a comment - assafl could you try providing a minimal example so that we can differentiate from JENKINS-44762 ?

          Mark Waite added a comment -

          0x89 I don't think you've understood why I specifically said "using a separate directory".  

          If you don't use a separate directory for each checkout, then you mingle two different git repositories into a single directory structure.  In the general case, that will cause many different surprises (like "last checkout wins" when both repos have one or more of the same file, or "damaged branch structure information when branch names are the same").  I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it.

          Mark Waite added a comment - 0x89 I don't think you've understood why I specifically said "using a separate directory".   If you don't use a separate directory for each checkout, then you mingle two different git repositories into a single directory structure.  In the general case, that will cause many different surprises (like "last checkout wins" when both repos have one or more of the same file, or "damaged branch structure information when branch names are the same").  I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it.

          Martin Sander added a comment -

          markewaite:

          Yes, I think I did - kind of - understand why you said that. I just wanted to point out that it does not make a difference if you check out into a separate directory or not - both trigger this behavior.

          I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it.

          Absolutely agree. Although if this will continue triggering JENKINS-44762, then Jenkins should at least issue a warning when you are trying to do that - it is sometimes hard for a user to recognize that they are making that mistake..

          Martin Sander added a comment - markewaite : Yes, I think I did - kind of - understand why you said that. I just wanted to point out that it does not make a difference if you check out into a separate directory or not - both trigger this behavior. I consider checkout of two different repositories into the same directory unsupported and do not ever plan to support it. Absolutely agree. Although if this will continue triggering JENKINS-44762 , then Jenkins should at least issue a warning when you are trying to do that - it is sometimes hard for a user to recognize that they are making that mistake..

            Unassigned Unassigned
            assafl Leibo
            Votes:
            8 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated: