Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8929

Deadlock when upstream and downstream jobs are blocked on each other

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      Job A (deploy to functional test) is upstream of job B (run functional tests).

      We need job A to block while its downstream job builds - otherwise a deployment while the functional tests are running will fail job B.

      We need job B to block while its upstream job builds - otherwise trying to run the tests while the deployment is running will fail.

      Unfortunately, if you configure A to block on downstream builds and B to block on upstream builds and A and B are scheduled at the same time they both block each other and are queued indefinitely, requiring manual intervention to clear the deadlock.

      In this situation job A should build and B should remain blocked until A completes.

        Attachments

          Activity

          mahoney Robert Elliot created issue -
          Hide
          mahoney Robert Elliot added a comment -

          (In the example above A and B need to be separate jobs - if a change is made to a test only, only B should run, there is no need to redeploy the application. However, if the application changes it needs to be redeployed and the tests need to be run after that deploy.)

          Show
          mahoney Robert Elliot added a comment - (In the example above A and B need to be separate jobs - if a change is made to a test only, only B should run, there is no need to redeploy the application. However, if the application changes it needs to be redeployed and the tests need to be run after that deploy.)
          Hide
          mahoney Robert Elliot added a comment -

          For reference, I found this thread describing the same problem with a possible solution:
          http://jenkins.361315.n4.nabble.com/Upstream-Downstream-Deadlock-td3307350.html

          Show
          mahoney Robert Elliot added a comment - For reference, I found this thread describing the same problem with a possible solution: http://jenkins.361315.n4.nabble.com/Upstream-Downstream-Deadlock-td3307350.html
          Hide
          mahoney Robert Elliot added a comment -

          Fix made on my clone of jenkins and pull request made here:
          https://github.com/jenkinsci/jenkins/pull/69

          Show
          mahoney Robert Elliot added a comment - Fix made on my clone of jenkins and pull request made here: https://github.com/jenkinsci/jenkins/pull/69
          mahoney Robert Elliot made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          mahoney Robert Elliot made changes -
          Assignee Robert Elliot [ mahoney ]
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #565
          [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable
          JENKINS-8929 follow up changes
          recording JENKINS-8929 fix

          Kohsuke Kawaguchi : b40e7b4228a6a5979fa484b2d2b15259bfab0ff5
          Files :

          • core/src/main/java/hudson/model/AbstractProject.java
          • core/src/main/java/hudson/model/Queue.java

          Kohsuke Kawaguchi : 86d88e5e7b341c884b156185051945678ce81ae8
          Files :

          • core/src/main/java/hudson/model/AbstractProject.java
          • core/src/main/java/hudson/model/Queue.java

          Kohsuke Kawaguchi : 05b9727edf1089b1c5e6e9461b4f62c71790065e
          Files :

          • changelog.html
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #565 [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable JENKINS-8929 follow up changes recording JENKINS-8929 fix Kohsuke Kawaguchi : b40e7b4228a6a5979fa484b2d2b15259bfab0ff5 Files : core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java Kohsuke Kawaguchi : 86d88e5e7b341c884b156185051945678ce81ae8 Files : core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java Kohsuke Kawaguchi : 05b9727edf1089b1c5e6e9461b4f62c71790065e Files : changelog.html
          Hide
          mahoney Robert Elliot added a comment -

          Thanks - though I'm not totally convinced the change should be symmetric; I think there may be a case for blocking downstream builds when upstream builds are blocked, so that it is certain that if both are scheduled at the same time then it is definitely the upstream one that goes first.

          Show
          mahoney Robert Elliot added a comment - Thanks - though I'm not totally convinced the change should be symmetric; I think there may be a case for blocking downstream builds when upstream builds are blocked, so that it is certain that if both are scheduled at the same time then it is definitely the upstream one that goes first.
          Hide
          cbos Cees Bos added a comment -

          We upgrade to version 1.403, since we also faced this issue.
          But now the block does not work anymore in all situations.

          We see situations were this fails.
          If both upstream and downstream job are queue and the downstream project finishes, both project get started.

          Do I need to log a new issue for that?

          As soon as a upstream project is running the downstream gets waiting in the queue.

          Show
          cbos Cees Bos added a comment - We upgrade to version 1.403, since we also faced this issue. But now the block does not work anymore in all situations. We see situations were this fails. If both upstream and downstream job are queue and the downstream project finishes, both project get started. Do I need to log a new issue for that? As soon as a upstream project is running the downstream gets waiting in the queue.
          Hide
          mahoney Robert Elliot added a comment -

          My fault for not writing the test cases that would have shown Kohsuke why I did not make the change symmetrical. I wanted to but I haven't managed to get my head round the testing strategy for Jenkins yet.

          Show
          mahoney Robert Elliot added a comment - My fault for not writing the test cases that would have shown Kohsuke why I did not make the change symmetrical. I wanted to but I haven't managed to get my head round the testing strategy for Jenkins yet.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Robert Elliot
          Path:
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/bddc4f2c1ae569dbf7af4bc73f8dcf667bf407bb
          Log:
          [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable
          but NOT blocked. This prevents the deadlock condition where upstream project A is blocked on downstream project B, which is
          blocked on upstream project A - now in this scenario B will remain blocked but A will build, eventually freeing up B.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Robert Elliot Path: core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/bddc4f2c1ae569dbf7af4bc73f8dcf667bf407bb Log: [FIXED JENKINS-8929] Changed the definition of a building downstream project to one that is Building, Waiting, Pending or Buildable but NOT blocked. This prevents the deadlock condition where upstream project A is blocked on downstream project B, which is blocked on upstream project A - now in this scenario B will remain blocked but A will build, eventually freeing up B.
          scm_issue_link SCM/JIRA link daemon made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/1eff6776e81756b5f15c0b599210e433d366f11d
          Log:
          JENKINS-8929 follow up changes

          • AbstractProject.getBuildingUpstream/getBuildingDownstream()
            should be symmetric.
          • It's unlikely for us to add another category of items,
            so reverted getUnblockedItems() implementation to the older one
            that was more efficient.
          • ItemList was meant to be internal, so keep it as such,
            at least for now.
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/1eff6776e81756b5f15c0b599210e433d366f11d Log: JENKINS-8929 follow up changes AbstractProject.getBuildingUpstream/getBuildingDownstream() should be symmetric. It's unlikely for us to add another category of items, so reverted getUnblockedItems() implementation to the older one that was more efficient. ItemList was meant to be internal, so keep it as such, at least for now.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/37f9f722fd15096066062931bfa264d8d8c31952
          Log:
          recording JENKINS-8929 fix

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html http://jenkins-ci.org/commit/jenkins/37f9f722fd15096066062931bfa264d8d8c31952 Log: recording JENKINS-8929 fix
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 139073 ] JNJira + In-Review [ 188322 ]

            People

            Assignee:
            mahoney Robert Elliot
            Reporter:
            mahoney Robert Elliot
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: