-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins: 1.644
git: 2.3.5
git-client: 1.16.1
-
Powered by SuggestiMate
Our git plugin is set up to poll for commits whenever a commit is pushed, and when we left on Friday evening it seems that Jenkins build the same commit ~1300 times. It seemed to start at 18:34 on Friday and then finish at 05:00 this morning (Monday).
Our builds take ~1 minute to run at the moment, and a quick look through the console log seems to suggest that the pre-build made Jenkins decide it needed to do another build
[JENKINS-32707] Git plugin built the same commit 1300 times
Can you duplicate the problem on a freshly installed system, then share the duplication steps?
I have a feeling it's similar to JENKINS-31731. We've never had this happen before and since it was late on a Friday I don't know what was fully going on. I might be able to piece together what was happening in git if I have some spare time.
Is there a reason you're running the very latest weekly build of Jenkins but running git plugin and git client plugin versions from a year ago?
I'll have to ask the sysadmin I'm afraid! It's probably because we're doing significant updates to the project that we use Jenkins for and we began updating our Jenkins server to accommodate (but dropped it when we realised it'd make running CI for the old version of our project in parallel difficult). I'll update the git plugin tonight though.
You suggest that the "pre-build made Jenkins decide it needed to do another build". Can you explain further what you mean by that, and what you observed?
I had a quick scan through the console logs visible on the web page. The git plugin has output Scheduling another build to catch up with ProjectName, and seems to have done that for each of the builds. At 5:01 this morning it built a different commit which is why I'm wondering if it's a dupe of JENKINS-31731.
Is your job that ran many times configured to allow concurrent builds? Jenkins jobs by default are not configured to allow concurrent builds. That was a critical part of the race condition described in JENKINS-31731.
Yes. Our setup has two executors and both were firing at the same time. I've not been able to look into it much further since we're so busy at the moment, but I've got a note to try find out what happened on the git server at the time the builds began (though that might not happen for a few months).
This happened to us on a few jobs after upgrading our jenkins from 1.574 to 1.650 and the git plugin from 1.4.0 to 2.4.2. After a lot of digging I managed to reproduce it on a fresh setup. The trick is to have the git polling being performed on a slave with an older version of git, which in our case was 1.7.1, and have the master branch in the repository being completely detached from origin/master (the branches do not share any common history). I am not sure exactly how we have ended up in this state but the affected jobs have been around for a long time so someone may have done nasty things to the repositories and/or workspaces.
I have managed to reproduce it on a fresh setup by creating a buildslave running on Centos 6.7 which comes with git 1.7.1. I then tied a job to to that build slave. I had to edit the job's config.xml to get it into the same state as our production job and I'm not sure exactly what setting causes it to start running the git polling on the build slave but here is the config.xml which reproduces this:
<?xml version='1.0' encoding='UTF-8'?> <project> <actions/> <description></description> <keepDependencies>false</keepDependencies> <properties> <hudson.model.ParametersDefinitionProperty> <parameterDefinitions> <hudson.model.TextParameterDefinition> <name>TAG</name> <description></description> <defaultValue>master</defaultValue> </hudson.model.TextParameterDefinition> </parameterDefinitions> </hudson.model.ParametersDefinitionProperty> <hudson.plugins.throttleconcurrents.ThrottleJobProperty plugin="throttle-concurrents@1.8.4"> <maxConcurrentPerNode>0</maxConcurrentPerNode> <maxConcurrentTotal>0</maxConcurrentTotal> <throttleEnabled>false</throttleEnabled> <throttleOption>project</throttleOption> </hudson.plugins.throttleconcurrents.ThrottleJobProperty> </properties> <scm class="hudson.plugins.git.GitSCM" plugin="git@2.4.2"> <configVersion>2</configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <name>origin</name> <refspec>+refs/heads/*:refs/remotes/origin/*</refspec> <url>https://github.com/Cantemo/RSSFeedWidget.git</url> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches> <hudson.plugins.git.BranchSpec> <name>${TAG}</name> </hudson.plugins.git.BranchSpec> </branches> <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations> <submoduleCfg class="list"/> <extensions> <hudson.plugins.git.extensions.impl.PerBuildTag/> <hudson.plugins.git.extensions.impl.AuthorInChangelog/> </extensions> </scm> <assignedNode>buildslave-co67</assignedNode> <canRoam>false</canRoam> <disabled>false</disabled> <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding> <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding> <triggers> <hudson.triggers.SCMTrigger> <spec>* * * * *</spec> <ignorePostCommitHooks>false</ignorePostCommitHooks> </hudson.triggers.SCMTrigger> </triggers> <concurrentBuild>false</concurrentBuild> <builders/> <publishers/> <buildWrappers/> </project>
I then let the job check out a git repository and start polling on the slave. Once that had completed I logged into the build slave and ran
git checkout master
in the workspace. After this I changed the repository url to point to another repository and after that, I started to get new build jobs spawned every 60 seconds.
The first repository I used was https://github.com/Cantemo/PortalPluginTemplate.git and the second one was https://github.com/Cantemo/RSSFeedWidget.git
This is the git polling log:
Started on Feb 27, 2016 12:49:00 PM Polling SCM changes on buildslave-co67 Using strategy: Default [poll] Last Built Revision: Revision e8adac91f3872ad7fe6fdf4ec84951af86e03e6a (origin/master) > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > git config remote.origin.url https://github.com/Cantemo/RSSFeedWidget.git # timeout=10 Fetching upstream changes from https://github.com/Cantemo/RSSFeedWidget.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/Cantemo/RSSFeedWidget.git +refs/heads/*:refs/remotes/origin/* Polling for changes in > git rev-parse origin/master^{commit} # timeout=10 > git rev-parse master^{commit} # timeout=10 > git log --full-history --no-abbrev --format=raw -M -m --raw e8adac91f3872ad7fe6fdf4ec84951af86e03e6a..5d99e8ae6e989be1c3cd568c25701b9c15abf2e6 # timeout=10 Done. Took 0.7 sec Changes found
The output from the commands above is:
[root@buildslave-co67 testjob]# git rev-parse origin/master^{commit} e8adac91f3872ad7fe6fdf4ec84951af86e03e6a [root@buildslave-co67 testjob]# git rev-parse master^{commit} 5d99e8ae6e989be1c3cd568c25701b9c15abf2e6
commit e8adac91f3872ad7fe6fdf4ec84951af86e03e6a is master in repo https://github.com/Cantemo/RSSFeedWidget.git
commit 5d99e8ae6e989be1c3cd568c25701b9c15abf2e6 is master in repo https://github.com/Cantemo/PortalPluginTemplate.git
In the end, wiping the workspaces on the affected jobs stops the looping builds.
Sorry that it went into a loop building the same commit repeatedly.
Unfortunately, I don't know how to duplicate the problem you're describing. Can you duplicate the problem on a freshly installed system, then share the duplication steps? If you want an easy starting point, you might consider the official Jenkins docker image, or if you'd prefer an image configured with the plugins that I often use, you could start with my master with plugins docker image.
Is there a reason you're running the very latest weekly build of Jenkins but running git plugin and git client plugin versions from a year ago?
You suggest that the "pre-build made Jenkins decide it needed to do another build". Can you explain further what you mean by that, and what you observed?