-
Bug
-
Resolution: Not A Defect
-
Blocker
-
None
-
Master: Hudson ver. 1.363, RHEL Linux 2.6.18-164.el5
Slaves: RHEL Linux 2.6.9-55.0.2.ELhugemem & Windows XP SP3
After Hudson update from version 1.361 to 1.363 we occasionally see jobs being triggered twice. This seems to happen in both cases, when upstream job triggers downstream job, and remotely triggering a job via curl in a script.
All of our jobs are free-style jobs.
[JENKINS-6819] Jobs are triggered twice occasionally
For what it's worth, I'm having trouble reproducing it in a fresh install of 1.363 - I've only seen it in upgrades. I think it's not the dependency graph changes - it seems to be a race condition, possibly related to Kohsuke's refactoring of pgweiss's logic for preventing more or less this exact thing from happening.
After downgrading Hudson to 1.362, I can see the two successive builds of same job were run from the same trigger (the builds nuxeo-features-5.3-notests/973/ and 972/ said they were both run from the build nuxeo-jsf-5.3-notests/504).
These all are Maven jobs, configured to trigger a build when a snapshot dependency is built.
On the upstream project (nuxeo-jsf-5.3-notests/504) does it report that it's triggering nuxeo-features-5.3-notests twice?
abayer: I doubt if the concurrent fix change that you refer to can cause this problem. The problem there is that if you actually have two builds of the same job queued (which requires a parameterization), then it's possible that those two builds end up executed concurrently, even if it's supposed to be prohibited.
In case of this problem, there should be just one build in the queue.
Still digging deeper...
I withdraw my comment above. I just came up with a hypothesis that causes this problem.
@kohsuke: nuxeo-jsf-notests/504 says "downstream builds: nuxeo-features-5.3-notests #974-#976", nothing about builds 972 and 973; whereas nuxeo-features-5.3-notests/974 was started by an URL trigger (we've set a hook on Mercurial to trigger a build from URL when there's a commit on a given branch); nuxeo-jsf-notests/503 says "downstream builds: nuxeo-features-5.3-notests #969-#971".
Changes included in 1.364:
r32256 | kohsuke | 2010-06-23 17:17:13 -0700 (Wed, 23 Jun 2010)
Changed paths:
M /trunk/hudson/main/core/src/main/java/hudson/model/DependencyGraph.java
[JENKINS-6819] tweaking the implementation a bit so that the code never ever have multiple edges between the same src and dst.
Not that I was able to spot where the previous code did that, but this implementation is bit more straightforward.
------------------------------------------------------------------------
r32258 | kohsuke | 2010-06-23 18:40:12 -0700 (Wed, 23 Jun 2010)
Changed paths:
M /trunk/hudson/main/core/src/main/java/hudson/model/Queue.java
M /trunk/www/changelog.html
[FIXED JENKINS-6819] Here is the race condition that induces this problem.
- Task X becomes buildable
- Thread E1 enters the pop() method, assigns X to an executor E2.
Note that at this point X is not removed from the buildables list yet - Thread E1 goes to offer.event.block and blocks, losing the lock
- Thread E3 enters the pop() method, assigns X to anogher executor E4.
This is possible because X is still in the buildables list. - Thread E3 goes to offer.event.block and blocks, losing the lock.
- Thread E2 wakes up, obtains the lock and execute the rest of the
pop() method. It decides that it's assigned X, it removes X from
the buildables list. - Thread E2 leaves the pop() method, loses the lock, and starts building X
- Thread E4 wakes up, obtains the lock, and executes the rest of the
pop() method. It decides that it's assigned X. It tries to remove X
from the buildables list. This remove operation becomes no-op,
but we don't check that. - Thread E4 leaves the pop(), and start building X.
Thus we end up building X twice.
To avoid this problem, I've introduced the 4th state "pending", and moves BuildableItem from buildables to pendings as soon as we assigned it to the executor. New BuildableItem cannot be created
while its another copy is either in the buildables or pendings state.
I've upgraded to Hudson 1.364 and all of my projects (Maven) are themselves the upstream project. It leads to a infinite loop a project triggering itself as soon as it's done.
I've had to hack in a workaround in the Ivy plugin for this same issue. The fix should probably be made in core to not allow self-referencing triggers (which was the previous behaviour), but I just wanted to get my builds working again locally.
svn di -c 32210 ivy Index: ivy/src/main/java/hudson/ivy/IvyModule.java =================================================================== --- ivy/src/main/java/hudson/ivy/IvyModule.java (revision 32209) +++ ivy/src/main/java/hudson/ivy/IvyModule.java (revision 32210) @@ -418,7 +418,8 @@ upstream = src; } - if (!hasDependency(graph, upstream, downstream)) + // Create the build dependency, ignoring self-referencing or already existing deps + if (upstream != downstream && !hasDependency(graph, upstream, downstream)) graph.addDependency(new IvyThresholdDependency(upstream, downstream, Result.SUCCESS)); } }
The code here is very similar to the Hudson Maven plugin, so the same workaround could probably be applied there in MavenModule.java as a temporary measure.
I've still had to go back to Hudson version 1.362 anyway though since the locks and latches plugin has been totally broken by these queue scheduling changes and we rely on that heavily as well.
I observed the same behavior and I think I know why this is happening. Double clicking in browser can generate this, it can happen more often than you may think.
A simple solution for this would be prevent this from happening, how about http://greatwebguy.com/programming/dom/prevent-double-submit-with-jquery/ ?
This happened once to me while presenting hudson to a first time user. When I told her to start a new job she asked me why there are two running jobs. Please do not count this as an usage issue
I'm quite sure that double clicking is not the issue for us, because we are not clicking anything manually.
We are starting the chain of jobs periodically from one job, and from the first job onwards everything is done either using the "build other jobs" option of Hudson or remotely triggering the jobs using wget / curl.
With modern versions of Jenkins, the build console output reports the list of causes that triggered a build. For those who are experiencing double invocations, please report what you see there in two (or more) builds that are fired.
We just recently updated our Hudson from version 1.362 to Jenkins 1.400 through the manage section, and we haven't seen this issue anymore for some reason.
We are just glad we were finally able to update!
Issue seems to be fixed with newer versions of Jenkins. Please re-open if there are still problems.
This is also hapenning in 2.15 version. Jobs builds twice using poll scm cron or manually clicking the build. The job executes twice
Happening on 2.7.4 . I'm trying to update git plugins to see if fix this . We aren't using SCM polling. It's a task scheduled to be executed only one time per day, that was working fine before we change the git branch that is checked. It's very annoying as it's a heavy task that enters on a infinite loop. The duplicated jobs shows that Jenkins detected changes on the repository, but not show any change.
UPDATE : Updating the git plug-in don't fix this.
Looks that gets fixed if I clean the workspace. I would say tomorrow.
I have seen this happen in Jenkins 1.651.3, however, only when the Jenkins master is under heavy CPU load.
Normally, everything works as it should, but on certain times when the Jenkins master CPU is running on max, we see two downstream
builds of one project started, both builds pointing to the same parent build.
This is done using the conditional build step + parameterized trigger and a schedule for the parent project.
This is still happening with most recent versions of Jenkins, like Jenkins ver. 2.46.1
I can easily double tick the build button, especial on a MBP with touchpad, doing a double click is so easy.
I am also facing this issue with Jenkins ver. 2.46.1. I have a upstream job which is triggering a downstream job.
post build actions>Build other projects>The downstream job is getting triggered twice.
Also post build actions>Trigger parameterized build on other projects>The downstream job is getting triggered twice.
This issue is fixed for me by doing below:
Define the downstream job name in upstream job using either Build other projects OR Trigger parameterized build on other projects.
Removed the upstream job name in downstream job defined by Build Triggers>Build after other projects are built
I still see this issue. Do we have any workaround for this? @Sangeeta - I don't understand your workaround. can you please explain in detail
I'm also seeing the same issue.
updating issue description for others:
Downstream job is getting invoked twice when 'Trigger parameterized build on other projects' option is selected in upstream project.
If anybody found a solution or workaround to mitigate this, please post.
I have tried steps mentioned by sangeetawakhale , this got resolved.
Thanks !
Thanks sangeetawakhale for the tip. Your workaround fixed the issue.
My "upstream job" was triggering a "downstream job" twice all the time using "Trigger parametrized build on other projects" plugin. My Jenkins version is 2.73.2
I had to go to the "downstream job" -> Build Trigger -> remove "Build after other projects are built"
It looks like this was mainly improper upstream/downstream job relationships that were defined.
I was searching for another issue but this masked it. Adding a link if that helps someone : https://issues.jenkins-ci.org/browse/JENKINS-58296 Projects are triggered twice when using File Parameter and Firefox 67
This is still happening.
Perhaps unrelated, but this is an exception that was coincidental with the duplicate trigger:
Sep 21, 2021 5:23:44 PM SEVERE io.jenkins.blueocean.autofavorite.FavoritingScmListener onCheckoutUnexpected error when retrieving changeset hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M "--format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %ai%ncommitter %cN <%cE> %ci%n%n%w(0,4,4)%B" -n 1 1bad1afcb14a41a82963e987511d6536874709c4 in /Users/software/var/jenkins/workspace/owlogix-org-repo_flowlogix_PR-85 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$6.execute(CliGitAPIImpl.java:1272) at io.jenkins.blueocean.autofavorite.FavoritingScmListener.getChangeSet(FavoritingScmListener.java:159) at io.jenkins.blueocean.autofavorite.FavoritingScmListener.onCheckout(FavoritingScmListener.java:84) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:155) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
We encounter same issue. And when it happens the duplicated jobs' views became unreachable (neither the job view, neither any of both build views). Other views on other jobs work fine.