-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
we are using Jenkins LTS 2.176.3 on hosted on amazonlinux , job-dsl - 1.74 , git plugin - 4.0.0-rc
-
Powered by SuggestiMate
step 1: create a seed job
```
String basepath = 'Common/Java/testapp'
String giturl = 'git@bitbucket.org:abc/testapp'
pipelineJob("$basepath/1.Build-Dev-Int") {
logRotator {
numToKeep(10)
artifactNumToKeep(10)
}
parameters {
stringParam('USE_GIT_BRANCH', 'master')
}
throttleConcurrentBuilds {
maxTotal(1)
}
triggers {
scm('H/15 * * * *')
}
definition {
cpsScm {
scm {
git
branches('master')
extensions { }
}
scriptPath('pipeline/pipeline_dev_int.groovy')
}
}
}
}```
step 2 : create pipeline_dev_int.groovy
```@Library('sharedlibrary@master')
node()
remaining groovy script```
step 3: configured shared library in manage jenkins
even though there is no change in repo it is triggering jobs
whenever a branch/ commit made to the shared library even though there is no change in testapp repo it triggering the job for every 15 min and found this in polling history
Using strategy: Default
[poll] Last Built Revision: Revision 6717866352d0d5a5e9b41da88a799993ab1d4191 (origin/master)
No credentials specified
> git --version # timeout=10
> git ls-remote -h git@bitbucket.org:abc/testapp # timeout=10
Found 3 remote heads on git@bitbucket.org:abc/testapp
[poll] Latest remote head revision on refs/heads/master is: 6717866352d0d5a5e9b41da88a799993ab1d4191 - already built by 97
Using strategy: Specific revision
[poll] Last Built Revision: Revision a7cf4fd0c7484765c4a5166a36613c387aee8501 (surya/MCDCA-1412-ECR-NEW-ACCOUNT)
No credentials specified
> git --version # timeout=10
> git ls-remote -h git@bitbucket.org:abc/sharedlibrary # timeout=10
Found 3 remote heads on git@bitbucket.org:abc/sharedlibrary
[poll] Latest remote head revision on refs/heads/surya/MCDCA-1412-ECR-NEW-ACCOUNT is: ab8eb071afb815bca04fd4e50a991ae54ea37e8c
Using strategy: Default
[poll] Last Built Revision: Revision 6717866352d0d5a5e9b41da88a799993ab1d4191 (origin/master)
using credential 6e83116e-4fb6-491a-a2c9-9ac9d51d4765
> git --version # timeout=10
using GIT_SSH to set credentials
> git ls-remote -h git@bitbucket.org:abc/testapp # timeout=10
Found 3 remote heads on git@bitbucket.org:abc/testapp
[poll] Latest remote head revision on refs/heads/master is: 6717866352d0d5a5e9b41da88a799993ab1d4191 - already built by 97
Done. Took 8.6 sec
Changes found
Polling Log
This page captures the polling log that triggered this build.
Started on Jul 30, 2018 7:15:00 PM Using strategy: Default [poll] Last Built Revision: Revision 80ae1a7997f138af5faa0e594eca7e88eb174026 (origin/master) > git --version # timeout=10 > git ls-remote -h git@bitbucket.org:xxx/xxxxxxx # timeout=10 Found 1 remote heads on git@bitbucket.org:xx/xxxxxx[poll] Latest remote head revision on refs/heads/master is: 80ae1a7997f138af5faa0e594eca7e88eb174026 - already built by 185 no polling baseline in /var/lib/jenkins/workspace/Sandbox/xxxx/xxxx/1.Build-Dev-Int@libs/xxx on Using strategy: Default > git --version # timeout=10 using GIT_SSH to set credentials Bitbucket SSH Key > git ls-remote -h git@bitbucket.org:xxxxx.git # timeout=10 Found 1 remote heads on git@bitbucket.org:xx/xxxx.git [poll] Latest remote head revision on refs/heads/master is: 80ae1a7997f138af5faa0e594eca7e88eb174026 Done. Took 2.7 sec Changes found
[JENKINS-52816] scm poll triggers even there is no change in repository(bitbucket)
For example, what is the final definition of the job as created by Job DSL?
- declerative pipeline jobs using one checkout in job configuration for the jenkinsfile with branch */master and also multiple other git repos in the jenkinsfile with */master
Is there a webhook configured from Bitbucket to Jenkins? If not, why not (since polling is much less efficient than web hooks)?
- Yes, with http://Jenkinsurl/git/notifyCommit?url=repourl - jobs are receiving the notification from bitbucket
If you use a web hook instead of polling, does that resolve the problem?
- here it's the other way around, but nevertheless both end up after certain build with no baseline for *** and starting multiple jobs
Is the problem job a Pipeline (not multibranch) or a multibranch Pipeline?
- yes, as already mentioned before
What is the branch definition in the jobs?
- */branchname, some other jobs contain slashes based on the gitflow pattern
I hope that is enough.
i have also identical log outputs (even identical sha1) for the builds which are triggered
Started on Nov 22, 2018 8:37:06 AM
no polling baseline in <path>@libs\shared-library-repo on
Using strategy: Default
[poll] Last Built Revision: Revision <sha1> (refs/remotes/origin/master)
Using strategy: Default
[poll] Last Built Revision: Revision <sha1> (refs/remotes/origin/master)
Using strategy: Default
[poll] Last Built Revision: Revision <sha1> (refs/remotes/origin/master)
Done. Took 5 ms
Changes found
I think you're assuming that I'll do a lot of work to guess the conditions for this problem. When I make guesses, the guesses may be wrong. If I guess incorrectly, I spend time investigating something that is not the problem you are seeing. I'm a volunteer that works on the git plugin during my nights and weekends.
For example, I would not have guessed that you were using '*/master' to refer to a branch when 'master' is the shorter form and 'origin/master' is the more precise form. Using '*/master' with some configurations will increase the risk that the plugin will incorrectly match a branch name from the wrong remote repository.
As another example, I would not have guessed that you were using multiple other git repos. I assume those multiple other git repos are being cloned into separate subdirectories, but since you don't say that and don't provide steps to duplicate the problem, I cannot be sure.
As another example, I asked if the job is a Pipeline or a multibranch Pipeline. You answered "yes, as already mentioned before". That didn't answer my question. A multibranch Pipeline has some important differences compared to a Pipeline. I need to know which of those two types it is.
When you provide a series of numbered steps that will allow another person to duplicate the problem you're seeing, I'll consider further investigation.
so it's not enough...
I'll try to go into more detail:
we are using pipeline jobs and in the job configuration we checkout the jenksfile
Branch: */master or */releases/
Additial Behaviours: Don't trigger a build on commit notifications
In the jenkinsfile we checkout several git repos
Branch: */master or */releases/
Additial Behaviours:
- useSubfolder
- localBrach
- includePath
- excludePath
- UserExclusion
checkout([ $class: 'GitSCM', branches: [[name: 'origin/master']], credentialsId: <credtials>, doGenerateSubmoduleConfigurations: false, extensions: [$class: 'LocalBranch', localBranch: **], [$class: 'ScmName', name: repoName], [$class: 'CloneOption', depth: cloneDepth, honorRefspec: true, noTags: false, reference: '', shallow: false], [$class: 'UserExclusion', excludedUsers: <excluded User>] [$class: 'PathRestriction', excludedRegions: <excludePath>, includedRegions: <includePath>] [$class: 'RelativeTargetDirectory', relativeTargetDir: item] only for one repo: [$class: 'IgnoreNotifyCommit'] [$class: 'WipeWorkspace'] submoduleCfg: [], userRemoteConfigs: [[url: <repoUrl>]] ])
- we are not using the multibranch pipeline plugin
- most jobs run on slave farm using one common label
i saw also your answer in https://issues.jenkins-ci.org/browse/JENKINS-50683 and the entry in the docu
"In the event that Fast Remote Polling is detected as being not possible (branches to build contains wildcards, etc), polling will fallback to requiring a workspace."
it looks like my problem is exactly related to this behavior by using:
- wildcard branch names (changing to origin/master and origin/release/branchname is no problem)
- an exclude message or path definition
- an include region definition
- an excluded user definition
and since we delete the workspace every night, we get this behavior.
so due this facts it seems that atm frp is not possible for me. any idea how to handle this?
markewaite - I guess workspace polling always needs the same workspace and causes problems when the next build runs on a different slave, correct?
swtch3k workspace polling for Freestyle jobs does not require the same workspace for each build. If a workspace does not exist on the agent performing the poll, then a complete workspace is created for it. I have not performed deep checks of the behavior of Pipelines (that are not multibranch) with workspace based polling.
update:
in the meantime i found the problem. the problem was that changes from the repo, from which we get the jenkinsfile, were not ignored even though the additional behavior "dont trigger a build on commit notification" was selected.
it's fatal to assume that everything that is available also have to work correctly. the polling log was not conclusive at this point, it differs for repos which are checkout out outside the pipeline definition.
--> i think the issue can be closed, no activity from the reporter
I have the same problem. swtch3k It sounds exactly as you describe. Do you have a solution or at least a workaround?
In fact, I believe the causing SCM is not the one providing the Jenkins file but the one that provides the pipeline library ("@Library") that the Jenkinsfile uses (in thise case both branches are on the same repository). But I may be wrong.
But the polling protocol does say "already built by XX" for scm polls that are explicitely mentioned in the groovy code so I don't think these are causing the rebuilds.
Hey kugel, my workaround was to add an additional behavior at the pipeline definition for the jenkinsfile, where i use ignore certain path with an inclusion of "ignoreRepo". If you believe the causing repo is the shared library, you have to look at the configuration in the jenkins system configuration. Good luck!
Do you also use pipeline libraries? I tried your workaround on the job for the repo that pulls the Jenkinsfile, but the parent project (and the Jenkinsfile) specifies a pipeline library. For that library I cannot specifiy ignore patterns in the same way as for the Jenkinsfile itself.
Hi
We have the same problem and did not find any solution so far.
We set polling in job settings while job itself is pipeline style.
Shared libraries defined with polling disabled.
And jobs randomly running even if there are no pushes to git.
Based on your experience, did you resolve problems you had?
i have the same problem.Please let me know when are you planning to release this fix.
jmishra no fix has been implemented. swtch3k reported a workaround in an earlier comment. Are you absolutely certain after reading all the descriptions and comments that your issue is the same problem as described here? If so, does the workaround from swtch3k also fix it for you?
Hi markewaite yes,only diff. is we are not triggering a pipeline but a simple job.
We have a scheduled job which runs every hr to check if there is a new commit if its there,it will trigger the build process.
This was working all fine with SVN, we recently moved to Bitbucket and we see this problem, even if there is no new commit ,build would trigger.
Hey, I have managed it in the meantime that the "fast remote polling" works again. Therefore my tip, eliminate following extensions:
- wildcard branch names (changing to origin/master and origin/release/branchname is no problem)
- an exclude message or path definition
- an include region definition
- an excluded user definition
That makes everything easier. No workaround needed.
we have configured this in pipeline scm and we have SCM polling for every 15 minutes but still we seeing jobs are continuously rebuilding for every 15 min without change
what we found is we facing this situation when we create a branch in our shared pipeline repo.
this issue is not yet resolve , I can walk you through steps if need to be reproduce , we trying to explore option for reverse proxy using smee but even it had issue with Bitbucket - https://github.com/probot/smee-client/issues/116 , we need to disable many jobs to not triggered. It will be really helpful if any work around or implementation to resolve this bug.
Using strategy: Default
[poll] Last Built Revision: Revision 6717866352d0d5a5e9b41da88a799993ab1d4191 (origin/master)
No credentials specified
> git --version # timeout=10
> git ls-remote -h git@bitbucket.org:xx/test # timeout=10
Found 3 remote heads on git@bitbucket.org:xxx/test
[poll] Latest remote head revision on refs/heads/master is: 6717866352d0d5a5e9b41da88a799993ab1d4191 - already built by 97
Using strategy: Specific revision
[poll] Last Built Revision: Revision a7cf4fd0c7484765c4a5166a36613c387aee8501 (surya/MCDCA-1412-ECR-NEW-ACCOUNT)
No credentials specified
> git --version # timeout=10
> git ls-remote -h git@bitbucket.org:xxx/shared # timeout=10
Found 3 remote heads on git@bitbucket.org:xxx/shared
[poll] Latest remote head revision on refs/heads/surya/MCDCA-1412-ECR-NEW-ACCOUNT is: ab8eb071afb815bca04fd4e50a991ae54ea37e8c
Using strategy: Default
[poll] Last Built Revision: Revision 6717866352d0d5a5e9b41da88a799993ab1d4191 (origin/master)
using credential 6e83116e-4fb6-491a-a2c9-9ac9d51d4765
> git --version # timeout=10
using GIT_SSH to set credentials
> git ls-remote -h git@bitbucket.org:xxx/test # timeout=10
Found 3 remote heads on git@bitbucket.org:xxx/test
[poll] Latest remote head revision on refs/heads/master is: 6717866352d0d5a5e9b41da88a799993ab1d4191 - already built by 97
Done. Took 8.6 sec
Changes found
This is what we seeing in polling log , if you observer I created a branch in shared repo , then test repo which builts in 97 again triggered as it found changes and rebuilding many times so we need to disable and enable job whenever needed.
yrsurya if you can walk through the steps to reproduce it, then it seems like you can also provide those steps as a numbered sequence of detailed instructions written into this issue report so that someone else can use those detailed instructions to duplicate the problem. Please provide that numbered set of steps so that someone else could work on the problem without needing to make guesses about the steps.
Many times when I've tried to make guesses about the steps to duplicate the problem, I've wasted many hours investigating things that the original author of the issue report could have clarified initially.
The current descriptions in this issue report seem to be from several different people, with at least one finding a workaround and another saying "I have the same problem" but providing no additional detail. Your description does not have enough detail to duplicate the problem. swtch3k describes in much greater detail but then states that a workaround has been found. You then say that the issue is not resolved, which I assume means that your specific details are different than those provided by swtch3k .
https://issues.jenkins-ci.org/browse/JENKINS-41497 here is similar issue
markewaite We found that most proably shared pipeline libraries cause the rebuilds. Can you rule this out?
markewaite updated description with the steps to reproduce , please let me know if need more details.
Here is latest Polling change that we seeing rebuilds for every 10 min
Git Polling Log
Started on Oct 17, 2019 9:22:07 PM Using strategy: Default [poll] Last Built Revision: Revision 9af93c1a4cf6e2e8a625ee7898dcfdb2339acbc6 (origin/master) No credentials specified > git --version # timeout=10 > git ls-remote -h git@bitbucket.org:abc/testapp # timeout=10 Found 3 remote heads on git@bitbucket.org:abc/testapp [poll] Latest remote head revision on refs/heads/master is: 9af93c1a4cf6e2e8a625ee7898dcfdb2339acbc6 - already built by 161 Using strategy: Specific revision [poll] Last Built Revision: Revision a7cf4fd0c7484765c4a5166a36613c387aee8501 (master) No credentials specified > git --version # timeout=10 > git ls-remote -h git@bitbucket.org:abc/shared # timeout=10 Found 3 remote heads on git@bitbucket.org:abc/shared [poll] Latest remote head revision on refs/heads/master is: 019f70307c622c80ac33b168e7c5dc134aa5d02a Using strategy: Default [poll] Last Built Revision: Revision 9af93c1a4cf6e2e8a625ee7898dcfdb2339acbc6 (origin/master) using credential 6e83116e-4fb6-491a-a2c9-9ac9d51d4765 > git --version # timeout=10 using GIT_SSH to set credentials > git ls-remote -h git@bitbucket.org:abc/testapp # timeout=10 Found 3 remote heads on git@bitbucket.org:abc/testapp [poll] Latest remote head revision on refs/heads/master is: 9af93c1a4cf6e2e8a625ee7898dcfdb2339acbc6 - already built by 161 Done. Took 3.1 sec Changes found
we didn't have any commits to shared library just we had few open branches
Hi yrsurya you could exclude the shared library and check if this is really the reason for the rebuilds. I would also advise you to add the remote name to the branch specifier instead of using only the branch name.
If the cause is really the shared library, you could test the following:
adjust the Global Pipeline Library definition and choose legancy scm instead of mordern scm. There you have the option where you can add the additional behavior, 'Polling ignore commits of certain path'. When selected enter "ignoreRepo" under Included Region and save.
swtch3k your workaround (changing to legacy scm and effectively ignoring all commits there) seems effective, thanks! It means that we cannot trigger on pipeline library changes but that's good enough for now.
Now, does that point to a defect in the modern scm method? That ought to be fixed, right? I got the feeling that "legacy scm" is somewhat half-deprecated and might be removed in the future.
Is there any update for it? when there should be a permanent fix for it?
Hi markewaite / All,
Hope everyone is keeping safe.
We are also facing similar issue where git just keeps polling even if no new commits in the repo, just checking whether to create a new bug for the same, or it is the similar scenario as this issue?
- Git Plugin: v4.3.0
- Bitbucket Server: v6.10.4
- Webhook from Bitbucket to jenkins is not enabled due to org-wise admin policy
- Git polling enabled to poll every 10 mins
- Pipeline (declarative) script is uploaded in SCM [note that pipeline script is also in the same scm repo where the repo code lies which is being polled]
- Lightweight checkpout option ticked for pipeline script
- GitSCM plugin used to define git checkout and identify repository to be polled
- We have 2 scenarios:
- First one is to trigger build only if new tag with a matching pattern is added to a commit, using below snippet for this:
checkout([$class: 'GitSCM', branches: [[name: '**/tags/release-*']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[refspec: 'refs/tags/*:refs/remotes/origin/tags/*', url: 'ssh://git@<git-url>.git']]])
- Second scenario is to poll a branch directly for any commits, using below for this:
checkout([$class: 'GitSCM', branches: [[name: '*/develop']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[url: 'ssh://git@<git-url>.git']]])
- First one is to trigger build only if new tag with a matching pattern is added to a commit, using below snippet for this:
- In both of the scenarios, once first successful trigger is done, polling keep finding false positives and triggering the build even though no new commit/tag is pushed
- Sharing git polling log sample for the second scenario (first scenario logs are too long due to multiple tags in repo, can share if required), which should have been "no changes" but results in "changes found". Note that even on subsequent runs, it keeps producing same output, with exact same values of <prev_commit_id> and <new_commit_id>. and hence keeps finding changes.
Git Polling Log
Started on Jul 6, 2020 11:54:00 AM
Using strategy: Default
[poll] Last Built Revision: Revision <prev_commit_id>(refs/remotes/origin/<branch_name>)
No credentials specified
> git --version # timeout=10
> git ls-remote -h – ssh://git@<git_url>.git # timeout=10
Found 59 remote heads on ssh://git@<git_url>.git
[poll] Latest remote head revision on refs/heads/jenkins_polling_check is: <prev_commit_id> - already built by 58
Using strategy: Default [poll] Last Built Revision: Revision <prev_commit_id> (refs/remotes/origin/<branch_name>)
No credentials specified
> git --version # timeout=10
> git ls-remote -h – ssh://git@<git_url>.git # timeout=10
Found 59 remote heads on ssh://git@<git_url>.git
[poll] Latest remote head revision on refs/heads/develop is: <new_commit_id> Done. Took 0.86 sec
Changes found - Build time for the job is less than pipeline polling interval. Build time <1 min, polling interval 10 mins.
- Also note that if same pipeline script is used as "inline scipt" in pipeline console instead of using "pipeline script from scm", this works fine 9/10 times (no false polling), still get same issue but frequency is very less in case of inline script.
- However while using pipeline script in scm, this fails almost everytime. Our projects requires pipeline code to be in scm.
Kindly suggest? Is this known issue or new issue? Is this related to pipeline script and the actual code repo (to be polled) staying in the same git repository? Please let me know if something is not clear.
Thanks.
ashisharma888 I don't know if that is the same issue or a different issue.
I assume there is something in the workspace that is causing the git plugin to believe that the new_commit_id has not been built, even though you indicate that it has been built. You might try evaluating https://github.com/jenkinsci/git-plugin/pull/579 in your environment to see if the changes proposed there are a help for the case that you're seeing.
Thanks markewaite, so this means it is alright to keep the pipeline script in the same repo/branch as the build code? Issue might be something else.
Also, can you please help to guide how to use the pull request and update my current plugin from the same? Sorry this would be the first time. Thanks.
Yes, it is good to keep the pipeline script in the same repo and branch as the build code. I do that with many repositories on many different source control providers, including Bitbucket.
You'll need to build the git plugin locally by following the instructions in the CONTRIBUTING file:
$ git clone https://github.com/jenkinsci/git-plugin $ cd git-plugin $ mvn clean install
After you've compiled it, you'll need to include the proposed changes from the pull request:
$ git remote add jacob-keller https://github.com/jacob-keller/git-plugin $ git fetch jacob-keller $ git checkout -b testing-jk-fix-polling-local-branches $ git merge jacob-keller/jk-fix-polling-local-branches $ # Resolve the merge conflicts $ mvn clean install
I did those those steps up through the install.
You can download the file I generated as git.hpi
Thank you very much markewaite. After trying out the new hpi, and a various scenarios for a while here are my additional observations:
- All of the behaviors mentioned below are same for latest official release of plugin and the version shared using the pull request mentioned above.
- It looks like that the plugin is marking branch for polling which is specified in branch specifier field to read pipeline script from scm path
- If I need to actually poll for changes in other branch or a tag pattern, it keeps failing
- Our intention is to use branch / tag provided in GitSCM class of pipeline script to be marked for polling.
- Also again sharing the same Git polling log I shared above, I think I also misread it last time:
Started on Jul 6, 2020 11:54:00 AM
Using strategy: Default
[poll] Last Built Revision: Revision <prev_commit_id>(refs/remotes/origin/<branch_name>)
No credentials specified
> git --version # timeout=10
> git ls-remote -h – ssh://git@<git_url>.git # timeout=10
Found 59 remote heads on ssh://git@<git_url>.git
[poll] Latest remote head revision on refs/heads/jenkins_polling_check is: <prev_commit_id> - already built by 58
Using strategy: Default [poll] Last Built Revision: Revision <prev_commit_id> (refs/remotes/origin/<branch_name>)
No credentials specified
> git --version # timeout=10
> git ls-remote -h – ssh://git@<git_url>.git # timeout=10
Found 59 remote heads on ssh://git@<git_url>.git
[poll] Latest remote head revision on refs/heads/develop is: <new_commit_id> Done. Took 0.86 sec
Changes found - In this example, polling is checking against last built commit of branch refs/remotes/origin/<branch_name> where pipeline script lies, however it is comparing with the latest commit of branch refs/heads/develop which is present in the branch field of GitCSM class of the pipeline script. It keeps finding differences and hence keep marking polling as changes found.**
So I think the only question is how can I remove my pipeline script branch from git polling, so that it only polls for changes based on branch specifier provided in GitSCM of the pipeline script?
Apologies if I was not able to explain this clearly last time, it only became clear to me post trying and testing the scenarios, and trying to read polling log and build console out in more detail.
ashisharma888 there is not a way to do what you're proposing with a pipeline job that reads the Pipeline from SCM (in other words, an SCM pipeline that is not a multibranch pipeline). It might be possible from a multibranch pipeline job that is defined to select only a subset of branches.
Ohh really, I thought it might be a common use case. Let me do similar POC around multi branch plugin.
Thank you very much markewaite. You are awesome. Stay safe. Cheers.
markewaite Just to close the loop here, the use case we have is not resolved using multi branch pipeline as well as it can help to select which branch to run pipeline in but again can't solve the polling challenge I have. So to be able to get positive polling results for our case (Poll for a specific tag on new commit, and trigger build from the tag once the new commit comes), I have just pasted pipeline script directly in the jenkins pipeline script console.
This issue does not seem to be related to Job DSL. Can we close the issue?
Please provide a series of numbered steps that will allow another person to duplicate the problem you're seeing. There are too many details missing from the description to help with the issue you're seeing.
For example, what is the final definition of the job as created by Job DSL?
Is there a webhook configured from Bitbucket to Jenkins? If not, why not (since polling is much less efficient than web hooks)?
If you use a web hook instead of polling, does that resolve the problem?
Is the problem job a Pipeline (not multibranch) or a multibranch Pipeline?
What is the branch definition in the jobs?
What are the branches defined in the repository?