Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7046

Block Build when downstream project is building

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Dependant matrix builds across Linux and Solaris
    • Similar Issues:

      Description

      When a downstream build is in progress, Hudson should prevent the upstream builds from occurring until the downstream build is complete.

      For example:

      CompA->CompB->CompC->CompD

      If component C is building Hudson should block builds on A and B, until C has finished.

      It should then rebuild A, B and C before allowing D to proceed.

      As the current system stands, if a user presses "build now" on component A when component C is building, A builds immediately. This can be worked around in part by using the locks & latches plugin, but this has the side affect of disabling parallel Matrix builds which massively increases the build time (see JENKINS-2725). Further, it should NOT be necessary to use ANY plugins to provide this fundamental behaviour.

        Attachments

          Activity

          aggamemnon aggamemnon created issue -
          Hide
          willemv Willem Verstraeten added a comment -

          Have you tried the 'Block build when upstream project is building' option in the 'Advanced project options' section of the configuration page of your job ? It does exactly what you appear to request, and we've been using it in my company successfully for quite a while now.

          Show
          willemv Willem Verstraeten added a comment - Have you tried the 'Block build when upstream project is building' option in the 'Advanced project options' section of the configuration page of your job ? It does exactly what you appear to request, and we've been using it in my company successfully for quite a while now.
          Hide
          willemv Willem Verstraeten added a comment -

          More specifically, that option has been available for all jobs since Hudson 1.323

          Show
          willemv Willem Verstraeten added a comment - More specifically, that option has been available for all jobs since Hudson 1.323
          Hide
          aggamemnon aggamemnon added a comment -

          I have already set "Block build when upstream project is building" and it doesn't provide the behaviour I described in my first comment. Either it isn't intended to or it doesn't work!

          Show
          aggamemnon aggamemnon added a comment - I have already set "Block build when upstream project is building" and it doesn't provide the behaviour I described in my first comment. Either it isn't intended to or it doesn't work!
          Hide
          willemv Willem Verstraeten added a comment -

          After re-reading you original description, I can also confirm the bug : the 'Block build when upstream project is building' does not work transitively.

          Ie. in the example of the reporter, A is not blocked when C is building, because A does not directly depend on C, only transitively.

          We also observe this in our setup. Our hudson build is 1.376

          Show
          willemv Willem Verstraeten added a comment - After re-reading you original description, I can also confirm the bug : the 'Block build when upstream project is building' does not work transitively. Ie. in the example of the reporter, A is not blocked when C is building, because A does not directly depend on C, only transitively. We also observe this in our setup. Our hudson build is 1.376
          Hide
          mindless Alan Harder added a comment -

          Seems like 2 different things are discussed here.. willemv's last comment refers to "block build when upstream project is building" not working transitively.. I have also observed this behavior. willemv, maybe file a new issue for this? JENKINS-5125 and JENKINS-5150 don't seem to specifically mention this..

          However, it appears to me like the original request describes blocking in the opposite direction.. "block build when downstream project is building". Hudson currently has no such feature. Can you describe why you want this? It doesn't make sense to me.. If we start the upstream job A, we know that will eventually trigger the downstream job C.. but since C has already started, there's nothing we can do at this point to avoid another build. So why block A at this point? Might as well run it in parallel with C to get all the builds completed more quickly.

          Show
          mindless Alan Harder added a comment - Seems like 2 different things are discussed here.. willemv's last comment refers to "block build when upstream project is building" not working transitively.. I have also observed this behavior. willemv, maybe file a new issue for this? JENKINS-5125 and JENKINS-5150 don't seem to specifically mention this.. However, it appears to me like the original request describes blocking in the opposite direction.. "block build when downstream project is building". Hudson currently has no such feature. Can you describe why you want this? It doesn't make sense to me.. If we start the upstream job A, we know that will eventually trigger the downstream job C.. but since C has already started, there's nothing we can do at this point to avoid another build. So why block A at this point? Might as well run it in parallel with C to get all the builds completed more quickly.
          Hide
          willemv Willem Verstraeten added a comment -

          Hmm, I'm confused.

          The summary says : "Hudson not correctly blocking downstream builds when upstream builds in progress"
          The description says : "Hudson should prevent the upstream builds from occurring until the downstream build is complete"

          Aggamemnon, which of the two do you actually need ?

          Show
          willemv Willem Verstraeten added a comment - Hmm, I'm confused. The summary says : "Hudson not correctly blocking downstream builds when upstream builds in progress" The description says : "Hudson should prevent the upstream builds from occurring until the downstream build is complete" Aggamemnon, which of the two do you actually need ?
          Hide
          aggamemnon aggamemnon added a comment -

          "If we start the upstream job A, we know that will eventually trigger the downstream job C.. but since C has already started, there's nothing we can do at this point to avoid another build. So why block A at this point? Might as well run it in parallel with C to get all the builds completed more quickly."

          Comp C may depend on objects contained within A, hence if A rebuilds C will fail, or be inconsistent.

          "Aggamemnon, which of the two do you actually need ?"

          Both

          Show
          aggamemnon aggamemnon added a comment - "If we start the upstream job A, we know that will eventually trigger the downstream job C.. but since C has already started, there's nothing we can do at this point to avoid another build. So why block A at this point? Might as well run it in parallel with C to get all the builds completed more quickly." Comp C may depend on objects contained within A, hence if A rebuilds C will fail, or be inconsistent. "Aggamemnon, which of the two do you actually need ?" Both
          Hide
          heymjo Jorg Heymans added a comment -

          mindless: imagine the case where you have a job that manipulates (e.g. destroys + rebuilds) a database schema. And then you have jobs that interact with this schema (unit,integration tests etc). Logically, the latter jobs have a dependency on the database job. We have expressed this dependency in the maven POMs, i.e. all application modules have a snapshot dependency on the database module.

          ofcourse we don't want the database job to kick in during the building of any of its downstream jobs because it would just rip out the database schema while integration tests are in progress. That's why the database job would need a "block build when downstream project is building".

          We can express this with locks and latches (urgh) or port allocator plugin but it would be a half baked job only and taking advantage of Hudson's snapshot dependency management (which is already GREAT by the way).

          I hope this clarifies the need a bit for this feature.

          Thanks
          Jorg

          Show
          heymjo Jorg Heymans added a comment - mindless: imagine the case where you have a job that manipulates (e.g. destroys + rebuilds) a database schema. And then you have jobs that interact with this schema (unit,integration tests etc). Logically, the latter jobs have a dependency on the database job. We have expressed this dependency in the maven POMs, i.e. all application modules have a snapshot dependency on the database module. ofcourse we don't want the database job to kick in during the building of any of its downstream jobs because it would just rip out the database schema while integration tests are in progress. That's why the database job would need a "block build when downstream project is building". We can express this with locks and latches (urgh) or port allocator plugin but it would be a half baked job only and taking advantage of Hudson's snapshot dependency management (which is already GREAT by the way). I hope this clarifies the need a bit for this feature. Thanks Jorg
          Hide
          heymjo Jorg Heymans added a comment -

          (i took the liberty to update the subject to make it less confusing)

          Show
          heymjo Jorg Heymans added a comment - (i took the liberty to update the subject to make it less confusing)
          heymjo Jorg Heymans made changes -
          Field Original Value New Value
          Issue Type Bug [ 1 ] New Feature [ 2 ]
          Summary Hudson not correctly blocking downstream builds when upstream builds in progress Block Build when downstream project is building
          Hide
          dmichonneau dmichonneau added a comment -

          Here is a simple use case:

          A: compilation of a jar
          B: test the jar

          B is running and locks the jar file
          A is triggered: it fails to write the jar file after compilation.

          Show
          dmichonneau dmichonneau added a comment - Here is a simple use case: A: compilation of a jar B: test the jar B is running and locks the jar file A is triggered: it fails to write the jar file after compilation.
          Hide
          mindless Alan Harder added a comment -

          For above use case I recommend using copyartifact plugin to copy the jar file to the workspace of job B, and use that when running tests. Job A can now run while B is already running for a previous build of the jar, and A/B are also more flexible in that they can run on different slaves.

          Show
          mindless Alan Harder added a comment - For above use case I recommend using copyartifact plugin to copy the jar file to the workspace of job B, and use that when running tests. Job A can now run while B is already running for a previous build of the jar, and A/B are also more flexible in that they can run on different slaves.
          Hide
          dmichonneau dmichonneau added a comment -

          That only moves the problem.

          The jar copy is either done by A or B.

          If done by A while B is running: the copy fails as B locks the file
          If done by B while A is running: the copy fails as A locks the file. There is a shorter interval during which the file is locked, but it can still happen.

          Besides that, it's an intrusive approach on tests that are not designed to run on something else that the compilation output directory.

          Show
          dmichonneau dmichonneau added a comment - That only moves the problem. The jar copy is either done by A or B. If done by A while B is running: the copy fails as B locks the file If done by B while A is running: the copy fails as A locks the file. There is a shorter interval during which the file is locked, but it can still happen. Besides that, it's an intrusive approach on tests that are not designed to run on something else that the compilation output directory.
          Hide
          mindless Alan Harder added a comment -

          If done by A while B is running: the copy fails as B locks the file

          I don't see how a running B can lock an artifact from a previous build. Hudson makes a copy of the file when it archives the artifact.. it wouldn't be copying a file that is in-use/locked by a newly running build of B, right?

          If done by B while A is running: the copy fails as A locks the file. There is a shorter interval during which the file is locked, but it can still happen.

          B would never invoke a copy when A is configured with a copyartifact build step.

          Here is more detail of what I was describing above:

          1. A runs; Hudson archives an artifact
          2. B runs; it has a copyartifact build step to copy the artifact from the latest completed build of A.. if A is currently running that is ok; it is copying from the archived artifacts, not the workspace that is in use by that running build where files may be locked.
          3. A runs; if B is currently running that is ok since it is using a copy of a previous artifact.. this build can run to completion and archive a new artifact, then trigger a new build of B.

          If you have a "compilation output directory" instead of a simple artifact, I can see how this is less than ideal.

          Show
          mindless Alan Harder added a comment - If done by A while B is running: the copy fails as B locks the file I don't see how a running B can lock an artifact from a previous build. Hudson makes a copy of the file when it archives the artifact.. it wouldn't be copying a file that is in-use/locked by a newly running build of B, right? If done by B while A is running: the copy fails as A locks the file. There is a shorter interval during which the file is locked, but it can still happen. B would never invoke a copy when A is configured with a copyartifact build step. Here is more detail of what I was describing above: A runs; Hudson archives an artifact B runs; it has a copyartifact build step to copy the artifact from the latest completed build of A.. if A is currently running that is ok; it is copying from the archived artifacts, not the workspace that is in use by that running build where files may be locked. A runs; if B is currently running that is ok since it is using a copy of a previous artifact.. this build can run to completion and archive a new artifact, then trigger a new build of B. If you have a "compilation output directory" instead of a simple artifact, I can see how this is less than ideal.
          Hide
          aggamemnon aggamemnon added a comment -

          "If you have a "compilation output directory" instead of a simple artifact, I can see how this is less than ideal."

          We have a compilation output directory. The issue is becoming more pressing.

          Show
          aggamemnon aggamemnon added a comment - "If you have a "compilation output directory" instead of a simple artifact, I can see how this is less than ideal." We have a compilation output directory. The issue is becoming more pressing.
          Hide
          mindless Alan Harder added a comment -

          Maybe it's worth prototyping a setup where you zip up the output directory and archive that zip as an artifact, in case this issue is not resolved for some time.. though maybe it's very large and that's why you haven't tried that already?

          Show
          mindless Alan Harder added a comment - Maybe it's worth prototyping a setup where you zip up the output directory and archive that zip as an artifact, in case this issue is not resolved for some time.. though maybe it's very large and that's why you haven't tried that already?
          Hide
          aggamemnon aggamemnon added a comment -

          "Maybe it's worth prototyping a setup where you zip up the output directory and archive that zip as an artifact, in case this issue is not resolved for some time.. though maybe it's very large and that's why you haven't tried that already?"

          Not feasible due to the large size of the build directories, the large number of them and the extra unnecessary complexity this would add to the build environment.

          Show
          aggamemnon aggamemnon added a comment - "Maybe it's worth prototyping a setup where you zip up the output directory and archive that zip as an artifact, in case this issue is not resolved for some time.. though maybe it's very large and that's why you haven't tried that already?" Not feasible due to the large size of the build directories, the large number of them and the extra unnecessary complexity this would add to the build environment.
          Hide
          mindless Alan Harder added a comment -

          Have you tried a setup with locks&latches plugin?

          Show
          mindless Alan Harder added a comment - Have you tried a setup with locks&latches plugin?
          Hide
          aggamemnon aggamemnon added a comment -

          Have you read the description?

          "This can be worked around in part by using the locks & latches plugin, but this has the side affect of disabling parallel Matrix builds which massively increases the build time (see JENKINS-2725). Further, it should NOT be necessary to use ANY plugins to provide this fundamental behaviour."

          Show
          aggamemnon aggamemnon added a comment - Have you read the description? "This can be worked around in part by using the locks & latches plugin, but this has the side affect of disabling parallel Matrix builds which massively increases the build time (see JENKINS-2725 ). Further, it should NOT be necessary to use ANY plugins to provide this fundamental behaviour."
          Hide
          robsimon robsimon added a comment -

          On top the "lock" is released before the artifacts are copied back and the next job in line is kicked off. That means this approach does not work.

          Show
          robsimon robsimon added a comment - On top the "lock" is released before the artifacts are copied back and the next job in line is kicked off. That means this approach does not work.
          Hide
          mindless Alan Harder added a comment -

          yes, read it long ago..

          Show
          mindless Alan Harder added a comment - yes, read it long ago..
          Hide
          mleinart mleinart added a comment -

          Here's another use case I have:

          Large build with a large suite of tests (unit + 1.5hrs worth of selenium browser tests).

          Job A:
          Builds the app, starts it up
          Job B:
          Matrix job using a custom workspace that points to Job A's workspace
          4 elements in 1 Axis: 4 test suites that are run simultaneously

          We cant have Job A run while Job B is running because it'll wipe the workspace out. Job A has an SCM poll trigger which on a busy day will already have another run queued while B is still running. Rather than space SCM polls several hours apart, we'd rather the build for Job A be blocked by its downstream job.

          Locks and latches doesn't work because each instance of Job B (it's a matrix job, remember) tries to grab a lock, preventing the instances from executing concurrently.

          (Also note that we've found custom workspace to be the best option because the Copy Workspace has some crappy limitations (doesnt copy permissions, doesnt copy empty directories) and tarring/untarring a 1gb build is a waste of time and space)

          Show
          mleinart mleinart added a comment - Here's another use case I have: Large build with a large suite of tests (unit + 1.5hrs worth of selenium browser tests). Job A: Builds the app, starts it up Job B: Matrix job using a custom workspace that points to Job A's workspace 4 elements in 1 Axis: 4 test suites that are run simultaneously We cant have Job A run while Job B is running because it'll wipe the workspace out. Job A has an SCM poll trigger which on a busy day will already have another run queued while B is still running. Rather than space SCM polls several hours apart, we'd rather the build for Job A be blocked by its downstream job. Locks and latches doesn't work because each instance of Job B (it's a matrix job, remember) tries to grab a lock, preventing the instances from executing concurrently. (Also note that we've found custom workspace to be the best option because the Copy Workspace has some crappy limitations (doesnt copy permissions, doesnt copy empty directories) and tarring/untarring a 1gb build is a waste of time and space)
          Hide
          mleinart mleinart added a comment -

          Here's a working implementation:
          https://github.com/mleinart/hudson/tree/JENKINS-7046

          It's basically copy/paste/substitute from the upstream option. A refactoring, especially of the display of the option (pulldown of Upstream, Downstream, Both?) may be desired.

          Show
          mleinart mleinart added a comment - Here's a working implementation: https://github.com/mleinart/hudson/tree/JENKINS-7046 It's basically copy/paste/substitute from the upstream option. A refactoring, especially of the display of the option (pulldown of Upstream, Downstream, Both?) may be desired.
          Hide
          aggamemnon aggamemnon added a comment -

          Fantastic, any idea on when this will make it into a release?

          Show
          aggamemnon aggamemnon added a comment - Fantastic, any idea on when this will make it into a release?
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          The change merged in 1.394. Sorry for the delay!

          Show
          kohsuke Kohsuke Kawaguchi added a comment - The change merged in 1.394. Sorry for the delay!
          kohsuke Kohsuke Kawaguchi made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          dogfood dogfood added a comment -

          Integrated in hudson_main_trunk #434
          [FIXED JENKINS-7046] - Add "Block build when downstream project is building" option
          recording the JENKINS-7046 RFE

          mleinart :
          Files :

          • core/src/main/resources/hudson/model/AbstractItem/configure-common.jelly
          • core/src/main/resources/hudson/model/Messages.properties
          • war/src/main/webapp/help/project-config/block-downstream-building.html
          • core/src/main/resources/lib/hudson/project/config-blockWhenDownstreamBuilding.jelly
          • core/src/main/java/hudson/model/AbstractProject.java

          Kohsuke Kawaguchi :
          Files :

          • changelog.html
          Show
          dogfood dogfood added a comment - Integrated in hudson_main_trunk #434 [FIXED JENKINS-7046] - Add "Block build when downstream project is building" option recording the JENKINS-7046 RFE mleinart : Files : core/src/main/resources/hudson/model/AbstractItem/configure-common.jelly core/src/main/resources/hudson/model/Messages.properties war/src/main/webapp/help/project-config/block-downstream-building.html core/src/main/resources/lib/hudson/project/config-blockWhenDownstreamBuilding.jelly core/src/main/java/hudson/model/AbstractProject.java Kohsuke Kawaguchi : Files : changelog.html
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 137143 ] JNJira + In-Review [ 187381 ]

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            aggamemnon aggamemnon
            Votes:
            21 Vote for this issue
            Watchers:
            20 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: