-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Powered by SuggestiMate
Hi,
Recently I've upgraded Jenkins to ver 1.508 and many plugins (including Jenkins GIT plugin from 1.1.25 to 1.3.0) and since that time I got a lot of my jobs using Git SCM running very frequently without any reason.
None of my jobs using "Poll SCM" in "Build Triggers". But I see them running with build cause "Started by an SCM change" though very often there are no changes at all. And Polling Log is always empty.
- Image_1.png
- 32 kB
- IntegritySCM.zip
- 156 kB
- view1.JPG
- 20 kB
- View2.JPG
- 33 kB
- is duplicated by
-
JENKINS-32422 Job without poller triggering on SCM
-
- Closed
-
- is related to
-
JENKINS-29714 infinite polling with variable in branchspec
-
- Open
-
[JENKINS-17614] Jenkins triggers builds on git SCM changes, but nothing changed
I find that my build gets triggered over and over by an SCM Change even when the repository has not been updated. I updated Jenkins to 1.518 today and still see this behavior. Git plugin is 1.4.0.
Same here (Jenkins ver. 1.525 with Git Plugin and several other plugins, but noticed this behaviour also in older versions), seeing "Started by an SCM change" and an empty pollingLog in the job that was triggered even though it shouldn't have been triggered at all as "Poll SCM" is disabled.
I'm also experiencing this after a recent upgrade from Jenkins 1.480 to 1.532, using v1.5.0 of the Jenkins GIT Plugin.
I'm going to try Frank's suggested workaround of cloning the job, but this is causing serious issues with our build failure notifications system.
warandpeace, just to make sure: I meant to clone the project manually, i.e. we have a shell script which does the git clone/checkout/pull command(s). We also create and update tags, branches and perform some other git commands through bash scripts, since our requirements for those are beyond what plugins currently can provide. Fortunately, we don't need to notify stash or JIRA for those parametrized builds...
frankmeissner: Ahh, understood. That doesn't really help me then, I'm afraid. Thank you, though!
I'll just have to wait for someone to sort this.
This is still causing serious issues with our downstream builds & deploy metrics.
Does anyone know if this is still occurring in 1.536 with Git 2.0? I'm on 1.532 with Git version 1.5.0
Can those who are encountering the problem identify anything specific about their configuration which exposes the problem? I've found that I see the problem with Git plugin 2.0 and Git client plugin 1.4.6 if I use JGit as the implementation, but I don't see the problem if I use Git command line as the implementation. I've seen the problem on all the platforms I tested, including Windows 8 x64, Windows 7 x86, Windows Home Server 2011, Debian Wheezy x64, and Debian Testing x64.
I can't duplicate the problem when using the git command line implementation, yet the JGit implementation did not become widely used until after the release of Git plugin 2.0.
If you can't deduce what is distinctive about your job that makes it build too frequently, would you be willing to upload sample job definitions which show the problem? The job definitions can then be compared with job definitions which do not poll too frequently in hopes there are clues in the differences between the jobs.
Some questions:
- Does the too frequent building happen even after the workspace has been wiped?
- Does the too frequent building happen even on a copy of the job?
- Does the too frequent building happen on jobs using the git command line implementation?
My situation may not precisely match the parameters of this ticket. However, FWIW, here's my observation. I have 22 jobs that are nearly identical. One was created, the others were cloned from it. (New job, copy existing job...) The only differences between these jobs are their names and the names of the GIT repositories they access. The build steps are identical and include wiping the workspace. All were set to trigger a build on a GIT change and all do that. 21 of them perform the build and then go back to an idle state. One performs the build, then about five minutes later does it again. And again. And again. (The trigger timing was set to "H/5 * * * *".) The GIT change trigger has been removed from that errant job and now I have to spawn it manually.
Solved!
Encountered the problem on Jenkins ver. 1.540 with Git plugin 2.0. In my case, the scm-polling.log in /var/lib/jenkins/jobs/jobname/ showed that Jenkins couldn't connect to the Git server (while we were having a Git server outage). Since it failed to contact Git, the workspace was the same from the previous job run, so it thought [again] that it had new code in the repo and kept running the job.
Solution:
- /sbin/service jenkins stop
- rm -rf /var/lib/jenkins/jobs/*/workspace
- /sbin/service jenkins start
(Make sure you didn't need anything that was in ...../*/workspace!!!)
You can also setup the job to clean out the workspace/ after the job runs.
To verify that you have connectivity to your Git server from the Jenkins server:
jenkins$ telnet git.yourcompany.com 9418
Eric, that technique does not solve it for me when using the JGit implementation. Were you using the command line implementation, or the JGit implementation with the plugin?
In my case, I wiped the workspace, and confirmed that the multiple executions were all performing a checkout of the same git version, even though they were reporting that changes were detected.
We have the same issue with Jenkins ver. 1.547. We updated from 1.535 to the 1.546 and suddenly this issue appeared.
We are using PTC plugin, and for all build jobs each time when a polling happened the new build is triggered, polling log tells that there is so many changes found, but SCM tells that there is no changes! See attached pictures - noticed that here "No changes"
and SCM change log tells that
Here is log files of one of the typical build started without any changes!
Grigoriy, could you submit a separate bug report for your Integrity SCM bug? This bug is specific to Git and the Git plugin and your Integrity information will be lost within this bug, since I'm sure the maintainer of the Integrity SCM plugin does not read Git plugni bug reports.
I'm having the same problem for a job where it's only build trigger is "Build after other projects are built". I'm seeing builds that are triggered on SCM changes which is problematic since this particular job is creating AWS resources for deployment.
Has anyone came up with a workaround?
amatheny I don't understand how a job which does not poll could trigger on SCM changes. Does that mean you have the job configured to both "Build after other projects are built" and to "Poll SCM"? If you don't intend to poll SCM for changes, couldn't you delete the polling from that job?
I don't understand how a job which does not poll could trigger on SCM changes.
Mark, that was a reason why I created this issue.
However it doesn't occur in my Jenkins anymore. I believe it has disappeared with other strange things when I turned Maven Integration Plugin on (I've disabled it shortly after I installed Jenkins).
[markewaite] As Alexander said, there is no other trigger on the job. Which is why it is alarming to see builds being triggered on SCM changes.
[uvizhe] Unfortunately, I already have the Maven Integration plugin enabled. I'll try recreating this job to see if that helps.
A Possible workaround to the solution is to check the ${GIT_COMMIT} and ${GIT_PREVIOUS_COMMIT} variables in a job and then based on the outcome of this job, trigger the actual job as a downstream. This would require "Run Condition Plugin" (https://wiki.jenkins-ci.org/display/JENKINS/Run+Condition+Plugin)
Steps:
1. Install the Run Condition Plugin
2. Create a Test Job, which would Poll SCM in the required interval. As a condition in the test job, you can configure to test for a "Strings Match" condition and as a result, in the "Steps to run if condition is met" option, you can execute a windows batch command "exit 1", which will simply fail this test job if the Git current commit and Previous commit matches (a possible duplicate build triggered in Jenkins).
3. If these commits do not match, then it is a valid candidate for execution and the actual job that you want to execute can be called in the "Post Build Action". So essentially, you are checking the current and previous commit and aborting the test job, if they are same, and proceed to build the actual job, if the commits are different.
Let me also try to attach a screen shot for help. Please try out and see if this going to help.
Circumventing the ghost build that Jenkins triggers, using a Run condition to check the GIT_COMMIT and GIT_PREVIOUS_COMMIT
This issue occurs when a branch with the same name exists in two different subbranches, e.g. 'origin/features/new_feature' and 'origin/new_feature'
Hope that helps!
I am seeing a similiar issue: particular builds always building twice. Still trying to narrow down a repro.
> This issue occurs when a branch with the same name exists in two different subbranches, e.g. 'origin/features/new_feature' and 'origin/new_feature'
This is not the case for my setup.
> This issue occurs when a branch with the same name exists in two different subbranches, e.g. 'origin/features/new_feature' and 'origin/new_feature'
Not the case for me as well, though I do notice that sub-branches, anything with a prefix followed by a / (feature/BRANCH_NAME) does trigger this mystery double builds, while top-level branch such as master or develop do not.
xaviershay and whoaa512 are you using Atlassian Stash and its web hook to perform a notify of the commit?
The Atlassian Stash web hook now passes two new parameters (branch and sha1) which have generated other bug reports like JENKINS-24133 (and this pull request and this pull request) which might be related.
We use Github Enterprise with a webhook.py script configured in the settings.
I'm getting this issue and i've only got master, develop and demo branches on my repo.
My polling log says that are "No Changes" but i get a build "Started by an SCM Change"
What other information do you need to resolve this?
lordmortis we need enough information to be able to duplicate your problem. In your case, if you're able to isolate a series of steps or conditions which cause the problem, then the problem also should have a new bug report.
This bug report has wandered far from its original problem description with several statements of "I'm seeing the same bug" which are then followed by statements which seem to me to contradict the original bug report.
For example, the original bug report says:
None of my jobs using "Poll SCM" in "Build Triggers". But I see them running with build cause "Started by an SCM change" though very often there are no changes at all. And Polling Log is always empty.
Later comments then include copies of a non-empty polling log even though the original report says that the polling log is always empty. Later comments refer to use of a web hook, but web hooks don't have any affect on jobs which are not configured to "Poll SCM".
Mark Waite - the issue i'm actually seeing is that Poll SCM (or Feature aware poll SCM) is enabled at an interval of 5 minutes but even though no changes are detected it continues to build. Should I create a new issue for this?
lordmortis you should create a new issue for it. Refer to how to report an issue for key items that will give your issue report a better chance of success.
A bug report to an open source project (like Jenkins) is an attempt to persuade someone to use their personal time to work on the bug you found. Every barrier you inadvertently insert between effective use of their personal time and your bug report is a risk that your bug report won't receive the attention you want.
I see similar behaviour but it seems for me to be triggered by deleting workspaces. I now in the past I saw it happen when I did it manually but now we have a bunch of variants that need to be build sporadic but have huge workspaces so I added the automatic delete final step and I have the feeling this is triggering the change detection.
Im facing similar issue. I have jenkins - 1.584 and Git plugin - 2.33.
Following are the behaviours
– No polling is configured. But build is triggered automatically as 'SCM change'. Nothing is displayed in the polling logs.
– When i trigger a build, another SCM trigger build follows up with that.
– SCM triggered build deletes the workspace.
jeevarathinamd I believe you are describing a different condition than this one. All the other descriptions of this bug require polling to be enabled. You say that polling is disabled. If polling is disabled, then builds should not be triggered as "SCM change" as far as I know.
I don't understand why an SCM triggered build would delete the workspace, unless you've configured something to delete the workspace. The git plugin does not have any ability to delete the workspace after a build. It can delete a workspace immediately before a build, but not after a build.
markewaite- My job wipes the workspace and could agree that its not from the git plugin. But definitely job doesn't have configuration for Polling.
I see someone posted similar observation before in the same thread.
"mika Michael Prokop added a comment - 08/Aug/13 11:11 AM
Same here (Jenkins ver. 1.525 with Git Plugin and several other plugins, but noticed this behaviour also in older versions), seeing "Started by an SCM change" and an empty pollingLog in the job that was triggered even though it shouldn't have been triggered at all as "Poll SCM" is disabled."
jeevarathinamd since this bug report includes a screen shot with a polling log, and the only way that I know to have a polling log is to enable polling, I assume that the comment you quoted is about a different bug. It is a bug, but I've not seen any way to duplicate the bug and I generally can't safely fix a bug which I can't duplicate.
Hey, I'm facing the same issue.
I've created new job with multiple git scm. In all repositories I checked the option "don't trigger a build on commit notifications", but job is still triggering itself, one ofter the other, with "Started by an SCM change". There is no hooks set on our gitlab. The option "Ignore post-commit hooks" in Poll SCM doesn't work too. It seems that job is triggering without any reason.
Hi,
I am also having the same issue where a build is started even though there has been no commit between the two times and the git tag is exactly the same.
So for build#263 from the "Started by SCM change" link I have
Started on 13-Aug-2015 15:39:59
Using strategy: Default
[poll] Last Built Revision: Revision 9cca12842f5b191d37e48a0ac2a1b7a2a87368e1 (origin/master)
Done. Took 29 ms
Changes found
And for build#264 I get
Started on 13-Aug-2015 16:09:59
Using strategy: Default
[poll] Last Built Revision: Revision 9cca12842f5b191d37e48a0ac2a1b7a2a87368e1 (origin/master)
Done. Took 44 ms
Changes found
Note that the GIT tag is identical in both cases and it is using the same branch. So how is this deciding that there is a change?
Actually I do. I think I found the problem. In git plugin by default /master branch is set to build from. I've noticed that someone commited and pushed a branch named origin/master to gitlab. I think that Jenkins somehow gets confused when it finds two branches matching the pattern (/master), so it triggers the job constantly. I don't know if it should work that way, but I guess you can now try to reproduce the problem. I had the newest version of Jenkins back then, which was 1.621. Hope it will help you!
I did't mean to bold the text. I meant to put the star sign (asteriks)
Steps I took to duplicate the problem:
- Create a bare repository /var/lib/git/mwaite/bugs/
JENKINS-17614.git - Clone the bare repository, add a README, and push the README to the bare repository
- Define a Jenkins job which polls that bare repository
- Commit another change to the bare repository, confirm the polling detects the repository and builds the job
- Create a branch named "origin/master" and push it to the bare repository, confirm the job builds never stop
I'm hopeful that is not the case also seen by the original submitter of the bug report, but at least I know one way to show the problem.
markewaite, why are you hopeful it's not it? And is this working the way it should, or it is actually a bug?
I think it is a bug, but also an uncommon case. I don't plan to spend any further time investigating the bug since I believe that single case is low probability and there are other bugs which are higher risk of being seen by many users.
I think there are more scenario's then this one.
I have a job that depends on a bunch of repositories, all to branch "*/fc_1.8-bugfix" and scheduled it to check for changes on the schedule "H 18-23,0-6 * * *" and automatically delete workspace before and after.
If I then look at the detected changes the last one are listed for build 55 but I also have this list of builds:
#58 17-aug-2015 9:57
#57 16-aug-2015 23:05
#56 16-aug-2015 7:46
#55 15-aug-2015 11:04
So it started 3 more times while nothing changed.
So nothing involving "origin/master" . If I list all the branches of all the repositories involved here then I only see these 2 lines containing "master":
remotes/origin/HEAD -> origin/master
remotes/origin/master
wilcobt I also think there are more scenarios than the case of the badly named branch. However, yours is the first attempt to describe another case with enough detail that others can duplicate the problem. Have you checked that your case is repeatable on a fresh Jenkins installation?
If you'd like the convenience of a docker instance to run that test, you could use the master-with-plugins docker instance plugin that I use for my quick setup and teardown of test Jenkins instances.
Sorry, but that is not exactly trivial in the environment here and I don't see when we will have a sprint that there is time available for this, the coming weeks look overfull already.
I just encountered this problem (on Cloudbees, so I don't know the versin #) and fixed it by updating branch from the default ** to a specific branch name, "master" in my case.
I've managed to trigger this problem again on 1.598 of Jenkins with the git client plugin v1.6.4 and git plugin v2.0.4. What seems to be the trigger is doing a git cherry-pick -x from HEAD to a branch and the building of the branch via an SCM update resulting in multiple builds.
I don't know if it's the extra git commit at the bottom of the message that causes the issue but I haven't seen this issue for a while which corresponds to a lack of backports from HEAD to a version branch.
Method to re-create
1. Push a fix to HEAD
2. Cherry pick the change to a branch using
git cherry-pick -x 2bedf18808f066a29d84bcb3301308cd6b81d0fe
3. This produces a commit similar to
Commit header
Commit message
...
(cherry picked from commit 2bedf18808f066a29d84bcb3301308cd6b81d0fe)
4. Wait for the SCM update to trigger on the branch
5. As the poll is set to every 5 minutes I then get a new build started every 5 minutes as it seems that Jenkins seems to think the last build was the one done before the cherry-pick and keeps spinning more builds.
I'm wondering if the extra commit reference at the end is confusing the parser.
This bug is about Jenkins triggering builds of jobs which do not use SCM polling, yet indicating that an SCM change was the cause. Why is everyone discussing unrelated problems? Should I create a new bug to get a clean look at this?
Yes, ampleyfly, please create a separate bug report with specific steps which show the problem on a fresh installation of Jenkins and the git plugin and which specifically states that SCM polling must not be configured seems very reasonable to me.
I've made at least one mistaken comment on this bug report assuming that polling was enabled, even though the original submitter clarified that polling was not enabled on the job which originally prompted this bug report.
I'm unlikely to spend more time on this bug report without those steps, since I've not yet seen a numbered series of steps which clearly duplicate the problem.
Unfortunately, I no longer encounter this bug, and I have no idea what caused it, so can't reproduce.
I think it's a good tip for the next person to come across this to follow Mark Waite's advice and create a new, detailed bug, with steps to reproduce the problem.
I have encountered the problem I had again, and this time figured out why it happened. Of course, I cannot be sure this is what happened to the original bug reporter.
We use a parameter for specifying which branch to build. If this parameter is empty for any reason, the SCM build step starts new builds for all branches it can find in that repository. These new builds start without any parameter values, and so cause the same problem with any other repositories used in the same way. All builds started this way are marked as having been started by an SCM change.
This is not a bug, since the plugin clearly states this is what is supposed to happen if you don't supply a branch.
In my case, the cause was slightly masked by the way our jobs are set up. There's some jobs A, B and C. Job A does some work, then runs job B with some parameters, which are then forwarded to job C. If B fails for any reason, the idea is that it can be rebuilt, without having to run the steps in A again. The problem occurs when C needs some parameter which B does not list as a parameter of its own. If B is rebuilt, this parameter will not get a value when C is called, and so the whole build avalanche starts.
We have an auto merge job which gets triggered by a Build-A and Build-Test-B jobs not by scm changes, and due to this bug, it was randomly picking up commits from 6 months ago, with random git hashes. so as per THIS use case, you DO need to fix this. You should have the option of having allowing no/null branch or hash. like a checkbox. It should appropriately exit accordingly.
I also have no trigger whatsoever configured in one job, since this job is parametrised. I ended up cloning the project through a shell script build step.
My jenkins version is 1.512.