-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Powered by SuggestiMate
Hi,
Recently I've upgraded Jenkins to ver 1.508 and many plugins (including Jenkins GIT plugin from 1.1.25 to 1.3.0) and since that time I got a lot of my jobs using Git SCM running very frequently without any reason.
None of my jobs using "Poll SCM" in "Build Triggers". But I see them running with build cause "Started by an SCM change" though very often there are no changes at all. And Polling Log is always empty.
- View2.JPG
- 33 kB
- view1.JPG
- 20 kB
- IntegritySCM.zip
- 156 kB
- Image_1.png
- 32 kB
- is duplicated by
-
JENKINS-32422 Job without poller triggering on SCM
-
- Closed
-
- is related to
-
JENKINS-29714 infinite polling with variable in branchspec
-
- Open
-
[JENKINS-17614] Jenkins triggers builds on git SCM changes, but nothing changed
I did't mean to bold the text. I meant to put the star sign (asteriks)
Steps I took to duplicate the problem:
- Create a bare repository /var/lib/git/mwaite/bugs/
JENKINS-17614.git - Clone the bare repository, add a README, and push the README to the bare repository
- Define a Jenkins job which polls that bare repository
- Commit another change to the bare repository, confirm the polling detects the repository and builds the job
- Create a branch named "origin/master" and push it to the bare repository, confirm the job builds never stop
I'm hopeful that is not the case also seen by the original submitter of the bug report, but at least I know one way to show the problem.
markewaite, why are you hopeful it's not it? And is this working the way it should, or it is actually a bug?
I think it is a bug, but also an uncommon case. I don't plan to spend any further time investigating the bug since I believe that single case is low probability and there are other bugs which are higher risk of being seen by many users.
I think there are more scenario's then this one.
I have a job that depends on a bunch of repositories, all to branch "*/fc_1.8-bugfix" and scheduled it to check for changes on the schedule "H 18-23,0-6 * * *" and automatically delete workspace before and after.
If I then look at the detected changes the last one are listed for build 55 but I also have this list of builds:
#58 17-aug-2015 9:57
#57 16-aug-2015 23:05
#56 16-aug-2015 7:46
#55 15-aug-2015 11:04
So it started 3 more times while nothing changed.
So nothing involving "origin/master" . If I list all the branches of all the repositories involved here then I only see these 2 lines containing "master":
remotes/origin/HEAD -> origin/master
remotes/origin/master
wilcobt I also think there are more scenarios than the case of the badly named branch. However, yours is the first attempt to describe another case with enough detail that others can duplicate the problem. Have you checked that your case is repeatable on a fresh Jenkins installation?
If you'd like the convenience of a docker instance to run that test, you could use the master-with-plugins docker instance plugin that I use for my quick setup and teardown of test Jenkins instances.
Sorry, but that is not exactly trivial in the environment here and I don't see when we will have a sprint that there is time available for this, the coming weeks look overfull already.
I just encountered this problem (on Cloudbees, so I don't know the versin #) and fixed it by updating branch from the default ** to a specific branch name, "master" in my case.
I've managed to trigger this problem again on 1.598 of Jenkins with the git client plugin v1.6.4 and git plugin v2.0.4. What seems to be the trigger is doing a git cherry-pick -x from HEAD to a branch and the building of the branch via an SCM update resulting in multiple builds.
I don't know if it's the extra git commit at the bottom of the message that causes the issue but I haven't seen this issue for a while which corresponds to a lack of backports from HEAD to a version branch.
Method to re-create
1. Push a fix to HEAD
2. Cherry pick the change to a branch using
git cherry-pick -x 2bedf18808f066a29d84bcb3301308cd6b81d0fe
3. This produces a commit similar to
Commit header
Commit message
...
(cherry picked from commit 2bedf18808f066a29d84bcb3301308cd6b81d0fe)
4. Wait for the SCM update to trigger on the branch
5. As the poll is set to every 5 minutes I then get a new build started every 5 minutes as it seems that Jenkins seems to think the last build was the one done before the cherry-pick and keeps spinning more builds.
I'm wondering if the extra commit reference at the end is confusing the parser.
This bug is about Jenkins triggering builds of jobs which do not use SCM polling, yet indicating that an SCM change was the cause. Why is everyone discussing unrelated problems? Should I create a new bug to get a clean look at this?
Yes, ampleyfly, please create a separate bug report with specific steps which show the problem on a fresh installation of Jenkins and the git plugin and which specifically states that SCM polling must not be configured seems very reasonable to me.
I've made at least one mistaken comment on this bug report assuming that polling was enabled, even though the original submitter clarified that polling was not enabled on the job which originally prompted this bug report.
I'm unlikely to spend more time on this bug report without those steps, since I've not yet seen a numbered series of steps which clearly duplicate the problem.
Unfortunately, I no longer encounter this bug, and I have no idea what caused it, so can't reproduce.
I think it's a good tip for the next person to come across this to follow Mark Waite's advice and create a new, detailed bug, with steps to reproduce the problem.
I have encountered the problem I had again, and this time figured out why it happened. Of course, I cannot be sure this is what happened to the original bug reporter.
We use a parameter for specifying which branch to build. If this parameter is empty for any reason, the SCM build step starts new builds for all branches it can find in that repository. These new builds start without any parameter values, and so cause the same problem with any other repositories used in the same way. All builds started this way are marked as having been started by an SCM change.
This is not a bug, since the plugin clearly states this is what is supposed to happen if you don't supply a branch.
In my case, the cause was slightly masked by the way our jobs are set up. There's some jobs A, B and C. Job A does some work, then runs job B with some parameters, which are then forwarded to job C. If B fails for any reason, the idea is that it can be rebuilt, without having to run the steps in A again. The problem occurs when C needs some parameter which B does not list as a parameter of its own. If B is rebuilt, this parameter will not get a value when C is called, and so the whole build avalanche starts.
We have an auto merge job which gets triggered by a Build-A and Build-Test-B jobs not by scm changes, and due to this bug, it was randomly picking up commits from 6 months ago, with random git hashes. so as per THIS use case, you DO need to fix this. You should have the option of having allowing no/null branch or hash. like a checkbox. It should appropriately exit accordingly.
kamal2222ahmed you'll need to provide much more detail before anyone will be persuaded to take their time to investigate and work on the problem you're reporting. I'd advise you to create a numbered series of steps which start with a fresh Jenkins installation, the latest version of the git plugin, and show the problem. Use those steps to create a new bug report, since it is likely a very different scenario than the one described originally in this bug report.
I suggested a few months ago that anyone who encounters what they believe to be an instance of this bug should provide detailed steps that illustrate the problem and submit a new bug report. This particular bug report requires that SCM polling is not enabled, and does not seem to report anywhere that it was selecting random commits from 6 months ago. Your comments says that the job was triggered by another job and was retrieving random commits.
You might also consider evaluating the "Ancestry" build chooser which is available in the git plugin under "Strategy for choosing what to build". That alternate build chooser allows you to limit the age of commits which will be considered.
I'm seeing this with the MultiSCM plugin: My project pulls from half a dozen Git repos on two different servers. One of the servers provides post-commit notifications to Jenkins for 3 of the repos.
If commits occur to two of the notifying repos in a short timeframe (which is often the case, as one repo contains shared code and the other contains specific code+data), Jenkins picks up all changes in one build, then immediately does a second build with "no SCM changes".
Seems like there should be a way to ignore notifications iff there are no changes detected?
markewaite just wanted to say that I ran into this bug today, and your comment where you figured out that having a branch named 'origin/master' causes endless builds saved me We did indeed have such a rogue branch, and deleting it fixed the problem.
I also want to add that the "rogue" builds all said they were "Started by an SCM change," even though we do not have SCM polling enabled. And the polling log in the Jenkins UI (/pollingLog) was empty. But the polling.log files on the master, for all builds, said e.g., "This build was triggered by build 284 because more than one build candidate was found." In other words they were endlessly triggered by the previous build. Maybe that's also a bug, the fact that this was written to the polling log? And maybe it does not display in the UI because Jenkins sees that the job is not configured for polling? I'm not sure if that information is of any use to you, but I didn't see it anywhere else in this bug report. The initial run of this job was triggered by the Jenkins API.
Thanks as always for your continued work on the Git plugin!
owenmehegan thanks for the further details on the failure mode that you saw.
Reading all this because we had the same issue: a job without SCM polling configured continuously being triggered for an SCM change. And the reason was the same as for [Owen Mehegan] above:
* [new branch] origin/master -> origin/origin/master
A dev committed and pushed a new branch with name `origin/master`
Note: we have another job with SCM polling from the same repository and that was also triggered continuously for no reason. I can understand why this one was mistaken, not why the one with no SCM polling could be
I had the same issue. My branch was specified as lspa/${APP_NAME}. Polling was disabled, and my polling log didn't display any changes.
It fixed when I changed it to refs/heads/lspa/${APP_NAME}.
My jobs are also triggering on SCM change without having any triggers configured. It shows the icon with the red "–" and green "++", even though Bitbucket is configured to send commit IDs, which means another job that does trigger for the same repo shows the black diamond commit notification icon (and there's one job that only shows merge icons because it builds pull requests). Strange.
It only started happening around Mar 15. Most of the affected builds have a git branch parameter, read from git ssh://, mask passwords, execute system groovy script (to update build description), then process Job DSL, and finally Slack notification. I think I added a branch filter "origin/(.*)" to the git branch parameter around the time it started happening. I don't remember any other upgrades of Jenkins (currently 1.651.1) or plugins.
One other job that triggered once incorrectly does not have any parameters. It also does polling, but should show "commit notification" instead of "Started by an SCM change". It's a Maven job with Slack notifications and Notify Stash Instance.
Of possible interest... the extra "trigger" for the JobDSL jobs always happens immediately following a manual build. But it doesn't happen after every manual build. Hope this helps.
A few more clues...
I added a shell script to my job that would detect when the build was not triggered manually, and then cause the build to error out. For some reason, this failed build would trigger another build to run right after... I would have dozens of failed builds right in a row until I was able to disable the job.
For jobs where it wasn't happening previously, once it started it would keep happening right after all of my manual builds. Also, git seemed to be stuck on an old commit... so no matter how many commits were added to a branch, building on that branch would only use an old commit. The only way to fix was to delete the entire job and regenerate from JobDSL.
I started suspecting that the ".git" repository in the workspace wasn't being updated for some reason, so then I figured out I only had to manually delete the workspace, then the job would run normally again. I set "Delete workspace before build starts" to the Build Environment, and everything has been run smooth as a peach since then.
Running into the same issue,
• Job without SCM polling enabled
• Manually launch a build on branch release/something, Jenkins build from refs/remotes/origin/release/something
• Jenkins triggers a build on origin/develop for no reason, states it was triggered from an SCM change, polling log is empty
Facing the same issue. Think it may have to do with us having a global script library for pipeline. The polling isn't fully figuring out the build version of it, and just re-triggering the job over and over, because it doesn't know what version of the global script library was included in the last build.
Made a standard job which triggers the pipeline job and it all works well, but it would be great if we didn't have to do this workaround.
Same issue as ----:JENKINS-32422
- Jenkins 2.60.3, GIT 3.6.3
- Job with two GIT SCM's, no build triggers, but manually started
- Uses build-parameters, also to configure which GIT branch-spec and local branch name for both repo's to checkout
Yesterday problems started, it was probably the first time a BranchSpec containing '/' was set for the build at this system/job.
- Immediately after manual build, an SCM triggered build is started.
- The SCM triggered build has an empty polling log linked
- The resulting build was using older commits, instead of having updated to latest commit from remote
Before and after reading this quite old #17614 ticket, accepted as dupe of #32422, I tried the following without success:
- (system config) Cleared unused Global pipeline library (although nowhere either the library nor the job in question was used/called from a pipeline)
- (system config) Cleared fields from unused GITLab plugin
- Added script as first builder, as fall-back to prevent SCM-triggered builds to messing with first configured env, ie. default parameters.
- test "$BUILD_CAUSE" = "MANUALTRIGGER" || exit 1
I assumed given the ticket the problem would be sporadic and hard to reproduce. But it wasn't. Today the pattern continued, except it did not overwrite the default env. with old version anymore but it simply failed itself after an SCM trigger.
What I would point at the root of the issue now is the GIT BranchSpec. The job had the normal symbolic name, ie. master, feature/XYZ, release/v123. I think Jenkins failed to update this local reference from the remote reference, I would normally have included the <remote> prefix, but someone else configured this job.
After adding the remote prefix, for example 'origin/release/v123', the issue immediately disappeared. Just fyi.
chr0n1x any hints on what you're hoping to accomplish with "bump"?
Have you added the remote prefix as berendvb mentioned?
Are you using a multi-branch pipeline as described in the most recent comments on this bug report, or are you using a freestyle job as described in the bug report and the first set of comments?
Are you interested in test driving the notifyCommit with slashes pull request (PR461) and its test git plugin build?
markewaite the remote fix is something that I cannot use because I'm managing an instance for people and do not have rights to their repos. I'm using freestyle jobs. Yes, Im interested in testing but not sure if I can commit any time.
What would be nice is if the plugin itself actually just had a switch or something to turn automatic SCM change detection on (default to off). It's an SCM, not a build-trigger (as opposed to ghprbuilder).
chr0n1x since you're using freestyle jobs, and you're the administrator of the Jenkins server, you can apply the technique described by berendvb in the freestyle job definitions on the server, without access to the source code repositories. In the "Branches to build" definition of each job, add the remote name prior to the branch name. The default remote name is origin, so a typical branch to build might be feature/somebranch and you would change it to origin/feature/somebranch.
The administrators and users in the 120000+ installations of the git plugin would howl in protest if automatic SCM change detection were disabled. It is a critical part of the workflow provided by the git plugin. I won't spend time adding an option to the plugin to disable SCM change detection because it is too important for my workflow, and for the workflow of many others.
markewaite - first off - thanks for taking the time to respond. Second - to just give some background:
So on the git SCM front:
- I actually use origin/* - all branches are things that I potentially want to run a build against
- by just doing that, SCM "detects changes" (i.e.: detects branches that fulfill that glob) and runs a build - that's my main issue
So yes, technically I could just use "origin/master" or "origin/branches/" or something of the like. However, for most of my jobs, I use the ghprbuilder plugin to trigger. I need the ghprbuilder plugin to also trigger builds for pull requests that are not targeting a specific branch (i.e.: any branch from a fork may have a pull-request raised into any branch in "origin/branches/*" for the parent repo). A few other issues come up as a result:
- when I use "origin/master" for the SCM branch identifier, ghprbuilder does not trigger any builds for pull-requests into other branches in the repo, say "origin/branches/foobar-feature"
- only when I use "origin/*" does it trigger builds properly
- this kind of leaves me in a catch-22 situation, where I need ghprbuilder to be able to run builds against all PRs targeting anything in "origin/branches/" ...but then the SCM config triggers a build for everything in "origin/branches/" at least once because the SCM, "detected a change"
To reiterate (for clarity) - my main concern & reason as to why I came here first was due to the perception that an SCM config just manages pulling of code and build-triggerers handle the triggering of builds. From here, it just looks like the git SCM is seemingly taking responsibilities of both which is confusing. Originally, I wrote "bump" because I thought that this was a legitimate bug and was wondering if anyone came up with an actual fix, assuming that my thought that this is a bug was correct.
I'm not saying that we completely change the functionality of the plugin. Merely that somehow this functionality were exposed in the UI explicitly. This means having a config to the effect of "trigger when SCM changes detected" which can default to true. For me, I could then update my job templates to just leave that switch off.
We have a similar issue where we need to start pull request builds via Bitbucket. When you start the job, every branch with a unique commit will get build resulting in thousands of unnecessary builds.
I submitted the PR https://github.com/jenkinsci/git-plugin/pull/550 a while ago.
It allows the user to turn off the "Started by an SCM change" behavior via an additional behavior on a per job basis and leaves everything else untouched.
Maybe this is what you want?
I'm seeing this same behavior and we're using Subversion. Latest LTS and latest plugins. Almost every one of our trunk builds is triggered nightly and fails because the readme file wasn't update per our requirements. When I recently went to the latest Jenkins LTS and Subversion plugin, it now seems that it is happening every single night (we trigger a nightly build for the product only if there were changes). I've got some irritated managers here and I have no explanation. The changes list is empty, but I do see "Changes found" at the end of the polling log. No errors in the system log. Should I raise a separate ticket for this? I think this post is on to something (for me at least) as it does appear that the polling log is looking at everything since the last successful pipeline build: https://issues.jenkins-ci.org/browse/JENKINS-17614?focusedCommentId=190266&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-190266
kerrhome submit a separate ticket.
This ticket is assigned to the git plugin. If a change were made to address it, it would be in the git plugin. That wouldn't help your case, since you're not using the git plugin.
I did just submit a ticket (JENKINS-53839), markewaite. Thanks. I wasn't sure if there was some common SCM API that could be the culprit here.
I am seeing a very odd behavior that seems to have just started out of the blue. I have a multijob job that does NO polling at all. It is meant to be kicked off manually and depends on many parameters to be set by the user. It is being kicked off randomly saying it was started by an SCM change. since the parameters are not set properly, it fails. Why is this build getting kicked off by scm changes when I don't using polling?
when I click on the Started by an SCM change link, it brings me to a polling log that is empty.
jenkins version: 2.121.3
git plugin version: 3.9.1
mcote55 a question asked on a bug report has much less chance of a response than a question asked in one of the chat systems or on the mailing list. Fewer people read the bug reports than read the chat system and the mailing list.
I don't know why it reports "started by an SCM change". That is usually an indicator that a notifyCommit was received or that a webhook was called.
Mark,
thanks for the advice. your comment actually led me right to the culprit. a hook on one of my Bitbucket repos. thank you!
Marlene
Team,
We are also facing the same issue Job started triggering due to SCM continuously, but no actual change in code. Please can we know if any workaround or any fix is available.
Jenkins version --->2.89.4
Git Plugin version --> 3.9.1
Thanks
Senthil
spalani you can attempt the following workarounds as possible alternatives:
- Copy the existing job to a new name, check if the continuous builds happen in the new job or not
- Wipe the workspace of the existing job
- Modify the existing job to poll only a single branch rather than a wildcard branch name
- Modify the existing job to require a local workspace
Those are all only temporary workarounds, but worth trying to see if they will let you avoid the bug you're seeing.
Facing the same issue. Using Jenkins 2.167 & GitPlugin, a pipeline job without any webhook/notifiyCommit and Poll SCM (enabled to check for changes only on master every hour). However, a build is triggered nightly logging "Started by an SCM change" even though there are no changes.
markewaite - tried out the 4 suggested workarounds, but no change in behavior.
Is it possible to inspect whatever the git plugin uses to track which git hashes it hass built so far? I can't seem to find this anywhere on my jenkins master. Presumably it has to be persisted to disk on the master.. somewhere.
We continue to randomly hit this issue. When I created a new release branch, one that isn't being built by the build in question, the problem building started triggering on every poll. If I delete the release branch, triggering stops. No traces of the release branch exist within the problem build configuration OR source.
markewaite I have also tried all of the work arounds you supplied (thank you btw), but none have worked. Is any data stored on the file system that I can delete or edit which will flush this out?
dmaxfield I don't know of any other work arounds beyond those that I listed earlier.
Recently I got into this issue and fix it by identify the right 'Branch Specifier '.
Set the 'Branch Specifier ' as '*/dev' will cause the problem because I have a local branch called 'local/dev' and the auto-pull won't update the local branch.
Set the 'Branch Specifier ' as 'origin/dev', the poll SCM get back to normal.
I had the same problem with some feature branches, but after setting 'Branch Specifier' with 'refs/heads/feature/...' and not only 'feature/...' everything works fine now.
I am also facing this same issue now. In my branch specifier it is " */develop ". Any ideas on how to resolve this?
We had the same issue, removing one of the recent branches added and removing the workspace fixed it.
I am also facing this issue in my branch one branch i.e develop using freestyle job poll scm automatically trigger with in a second without any new commit i resolve this issue by below steps:
1 . Clean workspace of particular pipeline from server where Jenkins is deploy location will be (/var/lib/jenkins/workspace/) under workspace folder you will get all pipeline so remove particular job which you having a issue.
2 Perform above step if issue is resolve by performing above step thats great if not perform one more step after doing this first step in you pipeline configuration
Set the 'Branch Specifier ' as 'refs/heads/develop'
I am seeing this issue in our builds. Our specifier is `refs/remotes/origin/dev`. I created a copy of the job that was "corrupted" and it worked for 1-2 weeks and then starting building on every commit in git (not just commits to our dev branch).
The git polling log shows an incorrect git hash after several correct git hashes.
In the below log, hash 1db609585eedb7d551abe735b69b8aa08bf57b2e is a commit on May 24 and hash 13b0645d2d0eef5a33f2cca02e26ff8161fe13fd is a commit from May 26. The polling log correctly shows it finding the May 26 commit several times and then, seemingly randomly, it finds the May 24 hash.
Example output from the git polling log in Jenkins:
> git ls-remote -h – git@github.com:my-company-tools/our-code.git # timeout=10
Found 438 remote heads on git@github.com:my-company-tools/our-code.git
[poll] Latest remote head revision on refs/heads/dev is: 13b0645d2d0eef5a33f2cca02e26ff8161fe13fd
Using strategy: Default
[poll] Last Built Revision: Revision 13b0645d2d0eef5a33f2cca02e26ff8161fe13fd (refs/remotes/origin/dev)
The recommended git tool is: git
using credential theDeployKey
> git --version # timeout=10
> git --version # 'git version 2.17.1'
using GIT_SSH to set credentials Deploy key for our-code mono-repository in GitHub
Verifying host key using manually-configured host key entries
> git ls-remote -h – git@github.com:my-company-tools/our-code.git/ # timeout=10
Found 438 remote heads on git@github.com:my-company-tools/our-code.git/
Using strategy: Default
[poll] Last Built Revision: Revision 1db609585eedb7d551abe735b69b8aa08bf57b2e (refs/remotes/origin/dev)
The recommended git tool is: git
using credential theDeployKey
> git --version # timeout=10
> git --version # 'git version 2.17.1'
using GIT_SSH to set credentials Deploy key for our-code mono-repository in GitHub
Verifying host key using manually-configured host key entries
> git ls-remote -h – git@github.com:my-company-tools/our-code.git # timeout=10
Found 438 remote heads on git@github.com:my-company-tools/our-code.git
[poll] Latest remote head revision on refs/heads/dev is: 13b0645d2d0eef5a33f2cca02e26ff8161fe13fd
Done. Took 13 sec
Changes found
I found this in our groovy:
script.checkout([
$class: 'GitSCM',
branches: [[name: "*/${gitBranch}"]],
doGenerateSubmoduleConfigurations: false,
I changed it to try:
script.checkout([
$class: 'GitSCM',
branches: [[name: "refs/remotes/origin/${gitBranch}"]],
doGenerateSubmoduleConfigurations: false,
We are still seeing the issue. I'm going to have someone in IT try deleting the workspace folder to see if that helps. Any other suggestions?
Actually I do. I think I found the problem. In git plugin by default /master branch is set to build from. I've noticed that someone commited and pushed a branch named origin/master to gitlab. I think that Jenkins somehow gets confused when it finds two branches matching the pattern (/master), so it triggers the job constantly. I don't know if it should work that way, but I guess you can now try to reproduce the problem. I had the newest version of Jenkins back then, which was 1.621. Hope it will help you!