Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27708

Concurrent build limits not honored on Jenkins 1.607

      After upgrading to 1.607 we noticed the throttle plugin doesn't always prevent jobs from running in parallel as expected.

      Steps to reproduce:
      1. Create a throttle category with global limit 1, per-node limit 1.
      2. Create 3 jobs using the category, restricted to a node with two executors, sleep 60 as a build step.
      3. Request builds of all 3 jobs.

      What should happen:
      4. Jobs run in sequence; 1 then 2 then 3.

      What actually happens:
      5. Job 1 starts building, jobs 2 and 3 wait in queue (OK).
      6. After job 1 finishes, both job 2 and 3 start running (not OK).

      Plugin version is 1.8.4. Issue does not appear in Jenkins 1.606 with the same version of the plugin. Issue is reproducible on a fresh install.

      We were forced to downgrade back to 1.606 as a workaround (which is unfortunately not trivial due to JENKINS-27700).

          [JENKINS-27708] Concurrent build limits not honored on Jenkins 1.607

          It seems to be fixed for me as well (I reported duplicate JENKINS-28084).

          But now the same (as in above comment) happened to me multiple times: jobs get stuck from time to time in queue. Nothing else running, but they just won't start. If I cancel ('x' button) them from queue and start them again manually they do start.

          I don't have a Build Blocker Plugin installed.

          Can I check something more when I see this happening again to pinpoint the cause?
          Should we reopen or create separate bug report?

          Andrei Costescu added a comment - It seems to be fixed for me as well (I reported duplicate JENKINS-28084 ). But now the same (as in above comment) happened to me multiple times: jobs get stuck from time to time in queue. Nothing else running, but they just won't start. If I cancel ('x' button) them from queue and start them again manually they do start. I don't have a Build Blocker Plugin installed. Can I check something more when I see this happening again to pinpoint the cause? Should we reopen or create separate bug report?

          Seems like we have similar issues with only BuildBlocker (not the throttle concurrent builds plugin) installed. But we´re still running on 1610. Just looks like queued blocked builds are not triggert (or the regex check) again after one blocking build finished.

          Dirk von Husen added a comment - Seems like we have similar issues with only BuildBlocker (not the throttle concurrent builds plugin) installed. But we´re still running on 1610. Just looks like queued blocked builds are not triggert (or the regex check) again after one blocking build finished.

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/11b89d9c0b46f0afad0ca86a11264b503b799c2e
          Log:
          [FIXED JENKINS-27708][FIXED JENKINS-27871] Ensure that identification of blocked tasks is using the live state.

          • The creation of a snapshot itself should be relatively cheap given the expected rate of
            job execution. You probably would need 100's of jobs starting execution every iteration
            of maintain() before this could even start to become an issue and likely the calculation
            of isBuildBlocked(p) will become a bottleneck before updateSnapshot() will. Additionally
            since the snapshot itself only ever has at most one reference originating outside of the stack
            it should remain in the eden space and thus be cheap to GC.
          • JENKINS-27708 comments 225819 and 225906 provide more complex but logically equivalent fixes of
            this issue. I am favouring this approach as it is simpler and provides less scope for error as any
            new helper methods can just rely on the snapshot being up to date whereas with the other
            two candidates if a new helper method is introduced there is the potential to miss adding support
            for the live view. The comment 225819 has the risk of introducing extra lock contention while
            the comment 225906 version forces every access to the helper methods to pass a second memory
            barrier

          (cherry picked from commit 5880ed830201f9349ae9def6653c19a186e1eb18)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/11b89d9c0b46f0afad0ca86a11264b503b799c2e Log: [FIXED JENKINS-27708] [FIXED JENKINS-27871] Ensure that identification of blocked tasks is using the live state. The creation of a snapshot itself should be relatively cheap given the expected rate of job execution. You probably would need 100's of jobs starting execution every iteration of maintain() before this could even start to become an issue and likely the calculation of isBuildBlocked(p) will become a bottleneck before updateSnapshot() will. Additionally since the snapshot itself only ever has at most one reference originating outside of the stack it should remain in the eden space and thus be cheap to GC. JENKINS-27708 comments 225819 and 225906 provide more complex but logically equivalent fixes of this issue. I am favouring this approach as it is simpler and provides less scope for error as any new helper methods can just rely on the snapshot being up to date whereas with the other two candidates if a new helper method is introduced there is the potential to miss adding support for the live view. The comment 225819 has the risk of introducing extra lock contention while the comment 225906 version forces every access to the helper methods to pass a second memory barrier (cherry picked from commit 5880ed830201f9349ae9def6653c19a186e1eb18)

          Dirk, did you find a solution to the new problem - builds not starting when they should?
          I didn't update Jenkins recently to avoid hitting new bugs.

          Andrei Costescu added a comment - Dirk, did you find a solution to the new problem - builds not starting when they should? I didn't update Jenkins recently to avoid hitting new bugs.

          Dirk von Husen added a comment - - edited

          No, did not find a solution so far. I am going to create an issue here soon. Since we were still using an older version, I wanted to update first, check if the problem is still there and then come back to this issue.
          Unfortunately the problem still exists, even after the update to version 1.613.

          -> please notice JENKINS-28376

          Dirk von Husen added a comment - - edited No, did not find a solution so far. I am going to create an issue here soon. Since we were still using an older version, I wanted to update first, check if the problem is still there and then come back to this issue. Unfortunately the problem still exists, even after the update to version 1.613. -> please notice JENKINS-28376

          I added my vote to it. Thanks.

          Andrei Costescu added a comment - I added my vote to it. Thanks.

          This is fundamentally an issue in t-c-b-p making incorrect assumptions

          Stephen Connolly added a comment - This is fundamentally an issue in t-c-b-p making incorrect assumptions

          Oleg Nenashev added a comment -

          costescuandrei , stephenconnolly, dvh_yxlon
          Is it a subject for a new issue?

          Oleg Nenashev added a comment - costescuandrei , stephenconnolly , dvh_yxlon Is it a subject for a new issue?

          Dirk already created JENKINS-28376 for the new problem.
          Don't know how related it is in code to the current one.

          Andrei Costescu added a comment - Dirk already created JENKINS-28376 for the new problem. Don't know how related it is in code to the current one.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4292
          [FIXED JENKINS-27708][FIXED JENKINS-27871] Ensure that identification of blocked tasks is using the live state. (Revision 11b89d9c0b46f0afad0ca86a11264b503b799c2e)

          Result = UNSTABLE
          ogondza : 11b89d9c0b46f0afad0ca86a11264b503b799c2e
          Files :

          • core/src/main/java/hudson/model/Queue.java

          dogfood added a comment - Integrated in jenkins_main_trunk #4292 [FIXED JENKINS-27708] [FIXED JENKINS-27871] Ensure that identification of blocked tasks is using the live state. (Revision 11b89d9c0b46f0afad0ca86a11264b503b799c2e) Result = UNSTABLE ogondza : 11b89d9c0b46f0afad0ca86a11264b503b799c2e Files : core/src/main/java/hudson/model/Queue.java

            oleg_nenashev Oleg Nenashev
            tsniatowski Tomasz Śniatowski
            Votes:
            2 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved: