I have the same problem.
I am using parameterized trigger where the parameters is a SVN branch.
I start :

• job build 1 from branch1
• job build 2 from branch2
• job build 3 from branch3
• job build 4 from branch4

They are all running fine, but if job 2 or 3 or 4 finish before 1. They are really finish only when job 1 is finish.
Then problem is the node resource not available then for starting a new job, and I am waiting the result of the build.

Jenkins ver. 1.458
Still have problem: concurrent build not finished while last build ended.

Any progress in resolve issue?

We had a very similar issue, and found the problem to be in the email part of core somewhere.

When we disabled all plugins that extended email, and turned off build notification emails, our builds no longer block other concurrent builds that share the same name.

The the work around isn't ideal, as we then had to go add email code to several build scripts (that we are trying to simplify and remove maintenance from).

We have "Editable Email Notification".
It is difficult to us to use shell script for notifications.
Will be nice to fix this bug.

Any progress? Latest Jenkins has this bug.

Do anybody read this ticket?
This bug make our work slow.
Help!

Created: 08/Jun/11
Today: 08/Jun/12
Still no progress...

This is a very unfortunate bug. I observed today that Jenkins finishes builds in the exact same order as they were started. So if you have builds #1, #2, #3 running, Jenkins will always let #1 finish first, then #2, then #3. It doesn't matter which of the builds finishes its actions first.

I wonder if there's an inherent limitation in Jenkins that a build #n cannot be running while #(n+x) has already completed.

A short test confirms what Logan Mattox observed: it's due to the e-mail notifications (both with the built-in "E-Mail Notification" post-build action, and the email-ext plugin).

Some other post-build actions don't seem to trigger the problem: I tried launching a downstream job, and this allows build #2 to complete before #1.

However, the "Jenkins Text Finder" plugin ALSO triggers the problem. Maybe it's related to plugins that read the console output?

Any idea when this will be fixed? this one's a blocker for me.

I there any chance to have this fixed soon ?

I attached to screenshot to explain again what's happen.
Jobs 537, 538, 539 have finished as you can see on second screenshot.
But executors is still busy as you can see on first one.

Created: 08/Jun/11 9:43 PM

date
Mon Oct 22 16:15:51 MSK 2012

Resolution: Unresolved

A very unfortunate bug indeed, especially if one of the batch of jobs getting launched gets stuck for whatever reasons (remote execution over ssh, etc.). The stuck job will cause the whole batch to get stuck, and the possibility of this happening is exactly why we're using concurrent executions in first place...

Bitten by this issue also

Do we have a public target date for this bug to be fixed?
It's getting kinda blocker for me.

I am afraid that no one is working on a the investigation of this bug.

It would be nice to know if someone will investigate this problem.

same problem here. total blocker. task A starts, then task B starts. task B reaches the 'Recording test results' stage and hangs until task A finishes. after testing with simple timed builds with many plugins/options enabled/disabled concluded that junit is the problem. - found another issue like this here: https://issues.jenkins-ci.org/browse/JENKINS-10234

Similar to JENKINS-10234 - for some causes even the same.

There are several cause described here. Some are JUnit archiving, which is specifically handled in JENKINS-10234. Some seem to be related to email notification.
I dare to say that in almost all cases this is a feature and not a bug as the logic for email notification and JUnit result archiving needs to wait until the previous builds are finished.

It is not a feature. All concurent builds execute in separated workspace and must archive its artifacts independently. No need to wait others builds if current have finished.

Sergey, do you mean 'artifact archiving' or 'JUnit result archiving'. In the former case, you would be probably right, but I've seen no comment here that artifact archiving is blocking, too.
In the case of JUnit: yes it MUST block to calculate the diff (regressions et.al.) to the previous test results. So it is a feature.

I mean all result of concurent job.
For example. I have Jenkins job that execute E2E tests on different QA servers. I started 10 jobs. One job running 5 hours. But others takes 1 hour. Then I can't see results of finished job and have to wait 4 hours to see it. We must have instrument to prevent this.

For that you can e.g. use the xunit plugin as mentioned in JENKINS-10234.

Generally, don't use any build steps which require blocking behaviour. Yes, very unfortunately it's not visible for end users which build steps do and which don't.

How is this a feature?

I do not use JUnit. I have a gerrit plugin that triggers a (parameterized) build+test job (a build script, and another script that runs some tests, all in a small shell snippet) whenever code is pushed to the repository. When the job is completed (successfully or not), an email notification needs to go out to the authors. It doesn't get much simpler than this.

Commits are independent from each other, and as a consequence, so are the build jobs. Why would you want the email plugin to sit in some checkpoint if the job is done?

Is there a workaround for this? This causes horrendous problems in our setup: aside from the resource waste (many nodes spend long periods of time waiting for slower builds), bugs in the test code will cause the whole cluster to deadlock.

Do anybody know how to solve this problem? Any patch, workaround...?
I can't remove post build steps for the job

Happening in 1.515

Workaround is to remove all post-Build-Actions and have it trigger a project that will do the dirty work.
To ensure it runs on the same machine, use the "NodeLabel Parameter Plugin", and Add a "NodeLabel Parameter" with the textbox "name" = NODE_NAME and "node" = ${ENV,var="NODE_NAME"}

>Workaround is to remove all post-Build-Actions
It is not workaround.

Sergey, I may not have made myself clear.

For example:
Project 'A' is the project you wish to run in parallel.

Project 'A' has a bunch of really long shell/bat commands and requires some sort of post-build action that is causing this defect.
Which is: if Project 'A' is run in parallel and contains post-build actions, it will not release until it completes.

The solution: Create a second project 'B' that contains all your post-build actions from 'A'.
Finally, remove all the post-build actions in 'A' (as they are all duplicated in project 'B') and have 'A' trigger 'B' as the final step.

Additional notes:
You will need to use the "NodeLabel Parameter Plugin" to ensure that this project is run on the same machine.
You may also need to give it the ${WORKSPACE} parameter (and any others) if your post-build actions need to manipulate the artefacts generated from Project 'A'.

Thank you, 4 4. Now it's clear.
This will cause new problems, because my tests execute notification job in post-build step. This job analize what was upstream and send mail
I have to chage logic in notify job.

Not a bug per se, but Jenkins needs to make sure that post-build actions which require the previous build to be complete (a) are documented to do so, e.g. in inline help; (b) print a helpful message to the build log when waiting for a previous build. And of course wherever feasible, offer an option to not block in this way, or to perform the processing dependent on the previous build asynchronously, e.g. in a RunListener.

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/CheckPoint.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/tasks/BuildStepMonitor.java
core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
core/src/main/resources/hudson/model/Messages.properties
http://jenkins-ci.org/commit/jenkins/eec307511c80112274d27f2a840d9f96cda784d3
Log:
[FIXED JENKINS-9913] At least print a diagnostic to the build log if we are waiting on a checkpoint.

Compare: https://github.com/jenkinsci/jenkins/compare/e5f5402cc2fd...eec307511c80

Integrated in jenkins_main_trunk #2789
[FIXED JENKINS-9913] At least print a diagnostic to the build log if we are waiting on a checkpoint. (Revision eec307511c80112274d27f2a840d9f96cda784d3)

Result = SUCCESS
Jesse Glick : eec307511c80112274d27f2a840d9f96cda784d3
Files :

• core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
• core/src/main/java/hudson/tasks/BuildStepMonitor.java
• core/src/main/resources/hudson/model/Messages.properties
• core/src/main/java/hudson/model/CheckPoint.java
• core/src/main/java/hudson/model/Run.java

I am not sure how printing to logs resolves this. Can we reopen this.
Also can we make Run.waitForCheckpoint an instance method instead of a static method.

@surya548:

I am not sure how printing to logs resolves this.

As noted above, it just makes it clear why a given plugin is blocking the way it is. If there is a particular plugin which is blocking which you believe should not block, that should be filed in a separate issue.

can we make Run.waitForCheckpoint an instance method instead of a static method

I see no reason why that would be necessary, but this is probably not the place to discuss it anyway.

> If there is a particular plugin which is blocking which you believe should not block

I dont want any plugin to block on checkpoint because it doesn't make sense to compare two arbitrary branches in the repo.

> that should be filed in a separate issue.

I have so many plugins that do this here are just few example example : junitarchiver, tap, checkstyle, findbugs , corbertura

I would like a global disable checkpointing option.

it doesn't make sense to compare two arbitrary branches in the repo

True, but this needs to be solved at a higher level in Jenkins, by creating a separate AbstractProject for each branch, so that each has a linear build history. There is some work TBA that accomplishes this.

I would like a global disable checkpointing option.

Sorry, this needs to be implemented on a per-plugin basis, since plugins are not necessarily written to behave gracefully when their checkpoint expectations are unmet.

> so that each has a linear build history.
We dont care about none of this linear history, checkpointing stuff if builds are hung for days deadlocked on checkpoints.
There should be a way to turn this feature off. I think most people use git with lots of branches anyways these days and none of this is relevant/useful.

>plugins are not necessarily written to behave gracefully when their checkpoint expectations are unmet

if we just noop Run.waitForCheckpoint based on some global setting wouldn't plugins just assume there is nothing to compare against and behave gracefully?

I think most people use git with lots of branches

Exactly why separate branch projects are needed: so that each branch does not need to be marked concurrent-capable, avoiding wasted build and also avoiding waits on checkpoints as a corollary.

if we just noop Run.waitForCheckpoint based on some global setting wouldn't plugins just assume there is nothing to compare against and behave gracefully?

Depends on the plugin. Some may behave fine, but others may behave erratically because they are still looking up the “previous build” (which they assume to be complete based on their stated checkpoint semantics). Disabling the checkpoint is incompatible.

>Exactly why separate branch projects are needed.

I dont understand what you suggest that I do now. Our Jenkins Instance is basically unusable because of these useless checkpoints.
I am not sure if it is practical for me to open bugs against 9 plugins( that i know of, after looking at source code for all the plugins we use) and ask them to remove checkpoints.

Why can't we get rid of (or atleast make it optional) a obsolete feature designed for SVN/CVS .
As others have said in the comments above we would rather have stable Jenkins than some optional feature than no-one cares about.

I am not sure if it is practical for me to open bugs against 9 plugins

Why not?

feature designed for SVN/CVS

Applies equally to Git as to SVN; both support branches, and for any such SCM it is preferable to have one AbstractProject per branch so that each has a sensible linear history (as previously mentioned there is ongoing work in this regard). If you do that, and suppress parallel builds within a branch (i.e. may have concurrent builds only of distinct branches), then there is no further issue.

Now even within a branch it is sometimes desirable to permit parallel builds, when there is a premium on quick feedback over build-to-build comparisons; in that case plugins should be configurable to do no comparison to the previous build (i.e. you are intentionally waiving your right to information such as whether a given test case failure was a regression), and should not use checkpoints either.

some optional feature

The original behavior of build steps was to be unconditionally serialized. For compatibility, that default must remain. New plugins should consider their actual needs and override the relevant method to specify it (optionally introducing finer-grained checkpoints).

>Applies equally to Git as to SVN; both support branches.

There is a big big difference, svn branches are long lived and most git branches typically are short lived( pull requests for example) . And having linear history doesn't make any sense for a branch with only a few commits.
The same goes for calculating changesets, culprits etc. None of those features are relevant for short lived branches.

Changelogs and the like are relevant if there is more than one commit in the branch, which is common in Git pull requests. And SVN branches need not be long lived since it is just as cheap to create and dispose of them as it is in Git. (CVS is a different matter of course.) Whether you care about linear build history depends on your workflow, not the SCM per se.

> Whether you care about linear build history depends on your workflow, not the SCM per se.

As I've been stating over and over, if it depends on our workflow there should be an option to configure if we need linear history or not based on our workflow. Right now jenkins is assuming that everyone needs this linear history thing.

Your solution is to go though source code of each plugin and open bugs against them which is extremely impractical and it feels extremely silly to expect users to go though plugins source code before installing it.

> more than one commit in the branch, which is common in Git pull requests

I am not sure how you came up with that (misguided) conclusion.
Here are the number of commits on last few open pull requests on jenkinci/jenkins . As you can see most of them are short lived with 1 or 2 commits.

917 - 1
916 - 2
914 - 1
913 - 1
912 - 3
911 - 1
910 - 1
907 - 1
904 - 2
902 - 1
894 - 1
891 - 1
889 - 2
884 - 1
882 - 1
877 - 2
858 - 3
855 - 2
847 - 2
840 - 4
839 - 1
823 - 4
816 - 7
808 - 1
806 - 1
799 - 1
782 - 1
776 - 1

Around half of those have more than one commit, as I said.

I give up, you win man. thanks for your help.

I have to wholeheartedly agree with surya et al: There needs to be a (non-plugin) switch for this. Perhaps not a global switch (although, why not?), but at an absolute minimum a per-job switch. This is a Jenkins-wide issue and giving plugins enough rope to hang the users does not strike me as the best process.

The only "workaround" really is to throw out the entire Post-Build functionality and recreated it from whole cloth as Build Steps. That is really what this bug means: The entire Post-Build feature must be considered suspect and unreliable. At the very minimum there should be a huge user warning around the Concurrent Build check box telling users that Post-Build actions may likely serialize their jobs anyway and should be avoided.

It should also be noted that Jenkins jobs are leveraged for MUCH more than simply "pulling code from an SCM and compiling it" type jobs. There are a huge number of utility tasks around a typical development lifecycle (deployments, environment setups and refreshes, data jobs, etc) that fit very well into Jenkins. In the real world I see such utility jobs far, far outnumber "real" build jobs. I mention this because of all the debate above about Git vs Subversion workflows and the like: It doesn't matter...in real world Jenkins usage it's common to find that MOST jobs aren't calling SCM at all because they aren't "buliding" anything, rendering all such workflow debates mute.

Build step monitors are an aspect of any build step, including both “builders” and “post-build steps” (recorders and notifiers), so that is an artificial distinction. Prior to the introduction of concurrent builds, plugins were free to assume that a build with a lower number than the current one was in fact completed—and some did in fact make that assumption. Monitors were introduced to ensure full binary compatibility. All plugins with build steps need to be updated to specify the monitor style they actually require, which is usually none. This has to be done on a case-by-case basis, while reviewing the plugin code for calls such as getPreviousBuild and ensuring that the logic continues to make sense when this build might be finishing well before earlier builds.

Got bitten by the same thing. Now planning to move the post build stuff into an additional last step of the build job.
Only problem is that I need to get data from the slave to master before that. And that is also a Post build action (I use Copy-To-Slave plugin). Any way I can do it as a part of build job only?

To resolve this problem use xplugin as an intermediary to nunit/junit.

Code changed in jenkins
User: Tim-Clifford
Path:
src/main/java/jenkins/plugins/slack/ActiveNotifier.java
http://jenkins-ci.org/commit/slack-plugin/832e7b50028ca086dd49311ce454e7a19c9a3af9
Log:
Change .getPreviousBuild() to allow for higher concurrency

Jenkins changed getPreviousBuild() to halt jobs if a previous
build hasn't finished, meaning jobs of variable runtime end up
being serialized:

https://issues.jenkins-ci.org/browse/JENKINS-9913?focusedCommentId=184188&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-184188

This change switches to the latest completed build, allowing
this plugin to work with jobs of variable length.

My apologies if this is the wrong please to ask this but I really did look for the right place and couldn't find it - is there a schedule for when this fix will be released? I am running into this issue and would benefit from the fix.

ericc there are distinct issues for various plugins.