Okay! I just figured out this bug is what causes massive amounts of builds to kick off whenever a slave goes down. It is also why all builds kick off when I restart Jenkins!

Lets assume we have a set of jobs, J, that require slaves of label L. If all slaves of label L go down, then there are no workspaces available for J, therefore the git plugin kicks off builds for all jobs J to get a workspace.

In addition, it seems that sometimes on restart the git plugin kicks off before the slaves come online causing all jobs on the system to kick off!

Finally, this also breaks the "only bring up slave when needed" functionality. I imagine it breaks the "provision a slave automatically" feature as well.

This really needs to be fixed. It means any maintenance task renders the build system busy for hours (we have 100s of builds, some of which are very long). It also needlessly bumps version numbers for no obvious reason which causes confusion.

I've been experimenting with a master/slave Jenkins setup to combine our 4 existing Hudson servers into one, but this issue is going to make it untenable. I don't have an easy way to count but there'd easily be more than 50 builds.

In my case, the Jenkins master has 0 executors. Any git project, no matter what the slave label target is, hits this problem on my server.

Interestingly the projects still in SVN don't have this issue; they appear on the build executor list to poll, but when there are no changes the build doesn't trigger. Looks like the git plugin doesn't check for changes until after the build has begun, meaning it would have to cancel if there are no changes.

Partially fixed by https://github.com/jenkinsci/git-plugin/commit/c750b63ecd527a5721d9d8ee426bff97dd0e7a75
using git ls-remote thanks to onemanbucket contribution
(only supports simple cases : single branch, single repo)

Cool!

I thought the git plugin only supported single branch single repo cases in general though

No, it lets you specify a wildcard to pick branches to build.

In a few releases, we should get rid of the checkbox, too. Excerpt from the IRC log:

(02:31:03 PM) kohsuke: Any reason this needs to be a checkbox?
(02:31:16 PM) One-Man-Bucket: i was thinking it could be the default behaviour in the simple use case
(02:31:18 PM) abayer: kohsuke: I'd like it to not be the default behavior quite yet.
(02:31:30 PM) kohsuke: I mean, shouldn't it just work automatically?
(02:31:57 PM) abayer: Actually, I'd be good with having the readResolve make it off and the default for new projects to be on.
(02:32:05 PM) One-Man-Bucket: as i wrote int he pull request, jgit has ls-remote since a couple of months
(02:32:22 PM) kohsuke: I think we already have enough switches and knobs in Git configuration
(02:32:28 PM) kohsuke: IMHO, that is.
(02:32:45 PM) abayer: One-Man-Bucket: I'm actually meeting with ksawicki of github tomorrow to start talking about gutting/rewriting the git plugin from the ground up, to mainly use jgit among other things. =)
(02:32:59 PM) One-Man-Bucket: abayer: oh, cool
(02:33:41 PM) abayer: I'm just wary of potential regressions in upgrades due to having introduced gobs of them myself. =)
(02:34:13 PM) One-Man-Bucket: abayer: :)
(02:34:42 PM) kohsuke: OK, I suppose you can take away the checkbox later.
(02:35:04 PM) One-Man-Bucket: one perk with the checkbox is that you get feedback if the configuration won't support remote poll
(02:35:14 PM) One-Man-Bucket: which is important for our usecase
(02:37:06 PM) kohsuke: One-Man-Bucket: by "it just works" I mean switching to ls-remote when and only when you can.
(02:37:39 PM) kohsuke: IIUC, provided that it works, there's no downside in using ls-remote.
(02:37:57 PM) One-Man-Bucket: kohsuke: fair enough
(02:38:38 PM) kohsuke: And also, IIUC, there's no inherent limitation in git ls-remote that you can't support multiple branches, right?
(02:38:49 PM) kohsuke: It's just that we haven't implemented that yet.
(02:39:13 PM) One-Man-Bucket: right, it should work for multiple branches as well
(02:39:37 PM) kohsuke: Anyway, I guess we aren't eliminating options --- as long as folks are willing to drop this checkbox at a later point, I'm happy.
(02:39:39 PM) abayer: Would it work with branch wildcards?
(02:39:58 PM) kohsuke: Yeah, you just list all the remote branches and check against the wildcard you got
(02:40:12 PM) abayer: kohsuke: That's fine by me. One release to make sure it doesn't barf, and in the meantime have an escape hatch.
(02:40:26 PM) kohsuke: Yes.
(02:40:57 PM) kohsuke: ("eliminating options" is kind of funny here because we are trying not to eliminate the option of eliminating the option (= the checkbox)

I've just updated to the new version of the git plugin, and I can confirm the fix works. Shut down all of my slaves and there are no jobs queued

I really would've preferred it to be on by default, but we have our job configs in SCM so it was a simple search/replace to turn it on for every build.

I'm seeing some strange errors in my SCM polling logs now... These errors don't show in the UI, and don't seem to impact builds, but they're still concerning.

I have 3 long builds showing an error while they are building - this doesn't happen on other active builds though:

(server name removed)

[poll] Last Built Revision: Revision a03190f9886de492a997a72c6525d65424618641 (origin/master)
ERROR: Failed to record SCM polling
hudson.plugins.git.GitException: Error performing command: git ls-remote -h git@<server> master
Command "git ls-remote -h git@<server> master" returned status code 128: error: cannot run ssh: No such file or directory
fatal: unable to fork

	at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:764)
	at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:729)
	at hudson.plugins.git.GitAPI.getHeadRev(GitAPI.java:1043)
	at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:624)
	at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:355)
	at hudson.scm.SCM.poll(SCM.java:372)
	at hudson.model.AbstractProject.poll(AbstractProject.java:1324)
	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)
Caused by: hudson.plugins.git.GitException: Command "git ls-remote -h git@<server> master" returned status code 128: error: cannot run ssh: No such file or directory
fatal: unable to fork

	at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:759)
	... 15 more

Builds that used the ** branch specifier (we want our builds to all be master-only, this has exposed a few that aren't) have this error in their polling log:

Started on 09/09/2011 5:10:18 PM
Using strategy: Default
[poll] Last Build : #38
[poll] Last Built Revision: Revision b9af602f4250b74fbf026a92716083d9b294121e (origin/master)
ERROR: Failed to record SCM polling
java.lang.NullPointerException
	at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:657)
	at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:355)
	at hudson.scm.SCM.poll(SCM.java:372)
	at hudson.model.AbstractProject.poll(AbstractProject.java:1324)
	at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
	at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
	at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:636)

Happy to help track these down if you need me to.

ok, "ssh: no such file or directory" isn't related to build length... still happening after the builds finished.

Looks like it's related to the slave node OS; it's happening on all builds that are set to only use our windows slaves. Turning off the fast remote polling for those builds stops the error.

I think this issue is related to JENKINS-9596, at least on a high level.

I just updated to Jenkins 1.447 with git plugin 1.1.15 and decided to try fast polling on the windows builds again. It works now

The second exception in Andrews comment from 09/Sep/11 is caused by JENKINS-10880

ndeloof: What is the reason behind restricting the option to the simple case of single repo, single branch? git ls-remote can retrieve the full set of branches, and revisions for each, and then compare those branch revisions against the "last built revision" for each branch it is configured for. If any of them are different, then a build would be scheduled – this is no different than what already happens with workspace polling, right?

Another thing I don't understand, is how does it determine that it's not a single repo/single branch? That is, I'm using the multiple-scms plugin, and apparently that is causing the ls-remote option to be disabled automatically. But each instance of the GitSCM in my project contains only a single repo and a single branch, so why would that be failing? Multiple-SCMs should be able to isolate each of those separately, without considering it any more complex than a single repo, single branch.

This behavior is a major problem for us, as we have the same problem that Christian reports in the first comment. We try to keep our master server free of builds, so that it can focus on its main task of "being the master." That means on restart, all of our jobs (well over 100) get scheduled immediately on startup, even though the slaves are actually coming online only seconds later. That is a huge waste of resources, and at times it actually causes the entire cluster to become unstable – simply as a result of starting up!

I'll look at modifying the ls-remote patch to see if I can get it to work with multiple branches, and also to find out why it behaves like this in the first place (since at least in my case, I actually don't have multiple branches). I just wanted to see if anyone else knows of some reason why what I said above could not work?

EDIT: bah, nevermind :/ I just realized what was causing it now... I was so focused on the "single branch" criteria, that I didn't think to look at excluded regions and users, which many of my projects use to prevent re-triggering a build when a Jenkins job is responsible for auto publishing commits (publishing things like minified JavaScript and CSS, and newly compiled Flash (swf) files). Removed those exclusions, and it works. I agree with the other feature request though, that this should be caught via the frontend before the user even clicks save, so that at least he knows why it won't work.

We're seeing this message in (git) polling log, and we're using git plugin 1.1.23.

Polling Log
View as plain text

This page captures the polling log that triggered this build.

Started on Oct 15, 2012 11:24:38 PM
Workspace is offline.
Scheduling a new build to get a workspace.
Done. Took 4 ms
Changes found

Seeing this with 1.1.25, with a single repo & single branch (so it should work according to the above).

Started on 24-Jan-2013 00:24:57
No workspace is available, so can't check for updates.
Scheduling a new build to get a workspace.
Done. Took 1 ms
Changes found

Any news on this? I can turn on fast remote polling to get around this. But that is not really a solution because then I can only use one branch (without wildcards).

Code changed in jenkins
User: Nicolas De Loof
Path:
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
http://jenkins-ci.org/commit/git-client-plugin/649c08617ff01191028e2e744013ed64e8a14431
Log:
JENKINS-10131 ls-remote for general usage to lookup all branches

I don't agree that this is an improvement. It's a genuine bug, and at the moment there is not even a real workaround for it.

For the multiple branch case, we ended up using a conditional buildstep, which compares GIT_COMMIT and GIT_PREVIOUS_COMMIT. But then you can't do manual builds anymore, because there is no way to find out the build cause inside of a build....

It's Worse Than You Think.

The ls-remote polling, as written, makes a hard build/no-build decision based on the git ls-remote output only. This interacts ... poorly with the GitSCM plugin features that filter commits (UserExclusion and so on), since they need to look at the individual commits to make their own go/no-go decisions - but that can't be done remotely.

Forcing workspace polling fixes that (and I've got a related PR open for JENKINS-20607 anyways) but reintroduces the problem Christian et al are complaining about. I suspect the real solution is to always do the polling on the master, possibly outside the workspace, and have the build runs re-fetch and check out the correct commit.

From working on JENKINS-20607 I'm well aware of how much work that is.

The 1.6.2 release of git-client-plugin fixed a bug in ls-remote polling for the command line git implementation. The bug would have manifest itself as having ls-remote based polling show that there were changes when there were no changes.

Refer to https://github.com/jenkinsci/git-client-plugin/commit/e5e75d5a100905c0d5b332bbde3642e96ffe10a9 for more details on the bug.

That fix does not change the behavior described by Owen Jacobson. Remote polling does not apply commit filtering, so if you're using inclusion or exclusion filters in your job definition, you can't use remote polling.

Now that the default behavior of the plug-in is to use remote polling, does this mean we need to not use exclusion filters?

drdt you may continue to use exclusion filters and inclusion filters, but jobs which use them must be configured to "Force polling using workspace".

Effectively, you're providing the compatibility with past behavior by forcing polling to use the workspace. Had you enabled fast remote polling with inclusion or exclusion filtering with git plugin releases prior to 2.0, you would have seen the same types of problems as exist today. The remote polling technique does not receive enough information to evaluate inclusion or exclusion filters.

Thanks, Mark. I actually never got exclusions to work in prior versions, so I wanted to try them out now I upgraded.

I am having trouble getting forced workspace polling to work at any rate; see my comments on https://issues.jenkins-ci.org/browse/JENKINS-20427.

As described in the bug report, this is resolved. The git plugin does not require a workspace for polling, and the default (as of git plugin 2.0) is to not use a workspace for polling.

The problems described by Owen Jacobson (and others) still exist for the somewhat more advanced use cases like excluding users, excluding files, and including files, but the basic issue reported by the bug is fixed.