-
New Feature
-
Resolution: Fixed
-
Major
-
None
-
Dependant matrix builds across Linux and Solaris
-
Powered by SuggestiMate
When a downstream build is in progress, Hudson should prevent the upstream builds from occurring until the downstream build is complete.
For example:
CompA->CompB->CompC->CompD
If component C is building Hudson should block builds on A and B, until C has finished.
It should then rebuild A, B and C before allowing D to proceed.
As the current system stands, if a user presses "build now" on component A when component C is building, A builds immediately. This can be worked around in part by using the locks & latches plugin, but this has the side affect of disabling parallel Matrix builds which massively increases the build time (see JENKINS-2725). Further, it should NOT be necessary to use ANY plugins to provide this fundamental behaviour.
[JENKINS-7046] Block Build when downstream project is building
More specifically, that option has been available for all jobs since Hudson 1.323
I have already set "Block build when upstream project is building" and it doesn't provide the behaviour I described in my first comment. Either it isn't intended to or it doesn't work!
After re-reading you original description, I can also confirm the bug : the 'Block build when upstream project is building' does not work transitively.
Ie. in the example of the reporter, A is not blocked when C is building, because A does not directly depend on C, only transitively.
We also observe this in our setup. Our hudson build is 1.376
Seems like 2 different things are discussed here.. willemv's last comment refers to "block build when upstream project is building" not working transitively.. I have also observed this behavior. willemv, maybe file a new issue for this? JENKINS-5125 and JENKINS-5150 don't seem to specifically mention this..
However, it appears to me like the original request describes blocking in the opposite direction.. "block build when downstream project is building". Hudson currently has no such feature. Can you describe why you want this? It doesn't make sense to me.. If we start the upstream job A, we know that will eventually trigger the downstream job C.. but since C has already started, there's nothing we can do at this point to avoid another build. So why block A at this point? Might as well run it in parallel with C to get all the builds completed more quickly.
Hmm, I'm confused.
The summary says : "Hudson not correctly blocking downstream builds when upstream builds in progress"
The description says : "Hudson should prevent the upstream builds from occurring until the downstream build is complete"
Aggamemnon, which of the two do you actually need ?
"If we start the upstream job A, we know that will eventually trigger the downstream job C.. but since C has already started, there's nothing we can do at this point to avoid another build. So why block A at this point? Might as well run it in parallel with C to get all the builds completed more quickly."
Comp C may depend on objects contained within A, hence if A rebuilds C will fail, or be inconsistent.
"Aggamemnon, which of the two do you actually need ?"
Both
mindless: imagine the case where you have a job that manipulates (e.g. destroys + rebuilds) a database schema. And then you have jobs that interact with this schema (unit,integration tests etc). Logically, the latter jobs have a dependency on the database job. We have expressed this dependency in the maven POMs, i.e. all application modules have a snapshot dependency on the database module.
ofcourse we don't want the database job to kick in during the building of any of its downstream jobs because it would just rip out the database schema while integration tests are in progress. That's why the database job would need a "block build when downstream project is building".
We can express this with locks and latches (urgh) or port allocator plugin but it would be a half baked job only and taking advantage of Hudson's snapshot dependency management (which is already GREAT by the way).
I hope this clarifies the need a bit for this feature.
Thanks
Jorg
Here is a simple use case:
A: compilation of a jar
B: test the jar
B is running and locks the jar file
A is triggered: it fails to write the jar file after compilation.
For above use case I recommend using copyartifact plugin to copy the jar file to the workspace of job B, and use that when running tests. Job A can now run while B is already running for a previous build of the jar, and A/B are also more flexible in that they can run on different slaves.
That only moves the problem.
The jar copy is either done by A or B.
If done by A while B is running: the copy fails as B locks the file
If done by B while A is running: the copy fails as A locks the file. There is a shorter interval during which the file is locked, but it can still happen.
Besides that, it's an intrusive approach on tests that are not designed to run on something else that the compilation output directory.
If done by A while B is running: the copy fails as B locks the file
I don't see how a running B can lock an artifact from a previous build. Hudson makes a copy of the file when it archives the artifact.. it wouldn't be copying a file that is in-use/locked by a newly running build of B, right?
If done by B while A is running: the copy fails as A locks the file. There is a shorter interval during which the file is locked, but it can still happen.
B would never invoke a copy when A is configured with a copyartifact build step.
Here is more detail of what I was describing above:
- A runs; Hudson archives an artifact
- B runs; it has a copyartifact build step to copy the artifact from the latest completed build of A.. if A is currently running that is ok; it is copying from the archived artifacts, not the workspace that is in use by that running build where files may be locked.
- A runs; if B is currently running that is ok since it is using a copy of a previous artifact.. this build can run to completion and archive a new artifact, then trigger a new build of B.
If you have a "compilation output directory" instead of a simple artifact, I can see how this is less than ideal.
"If you have a "compilation output directory" instead of a simple artifact, I can see how this is less than ideal."
We have a compilation output directory. The issue is becoming more pressing.
Maybe it's worth prototyping a setup where you zip up the output directory and archive that zip as an artifact, in case this issue is not resolved for some time.. though maybe it's very large and that's why you haven't tried that already?
"Maybe it's worth prototyping a setup where you zip up the output directory and archive that zip as an artifact, in case this issue is not resolved for some time.. though maybe it's very large and that's why you haven't tried that already?"
Not feasible due to the large size of the build directories, the large number of them and the extra unnecessary complexity this would add to the build environment.
Have you read the description?
"This can be worked around in part by using the locks & latches plugin, but this has the side affect of disabling parallel Matrix builds which massively increases the build time (see JENKINS-2725). Further, it should NOT be necessary to use ANY plugins to provide this fundamental behaviour."
On top the "lock" is released before the artifacts are copied back and the next job in line is kicked off. That means this approach does not work.
Here's another use case I have:
Large build with a large suite of tests (unit + 1.5hrs worth of selenium browser tests).
Job A:
Builds the app, starts it up
Job B:
Matrix job using a custom workspace that points to Job A's workspace
4 elements in 1 Axis: 4 test suites that are run simultaneously
We cant have Job A run while Job B is running because it'll wipe the workspace out. Job A has an SCM poll trigger which on a busy day will already have another run queued while B is still running. Rather than space SCM polls several hours apart, we'd rather the build for Job A be blocked by its downstream job.
Locks and latches doesn't work because each instance of Job B (it's a matrix job, remember) tries to grab a lock, preventing the instances from executing concurrently.
(Also note that we've found custom workspace to be the best option because the Copy Workspace has some crappy limitations (doesnt copy permissions, doesnt copy empty directories) and tarring/untarring a 1gb build is a waste of time and space)
Here's a working implementation:
https://github.com/mleinart/hudson/tree/JENKINS-7046
It's basically copy/paste/substitute from the upstream option. A refactoring, especially of the display of the option (pulldown of Upstream, Downstream, Both?) may be desired.
Integrated in hudson_main_trunk #434
[FIXED JENKINS-7046] - Add "Block build when downstream project is building" option
recording the JENKINS-7046 RFE
mleinart :
Files :
- core/src/main/resources/hudson/model/AbstractItem/configure-common.jelly
- core/src/main/resources/hudson/model/Messages.properties
- war/src/main/webapp/help/project-config/block-downstream-building.html
- core/src/main/resources/lib/hudson/project/config-blockWhenDownstreamBuilding.jelly
- core/src/main/java/hudson/model/AbstractProject.java
Kohsuke Kawaguchi :
Files :
- changelog.html
Have you tried the 'Block build when upstream project is building' option in the 'Advanced project options' section of the configuration page of your job ? It does exactly what you appear to request, and we've been using it in my company successfully for quite a while now.