• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • None
    • Platform: All, OS: All

      Starting from version 1.452, a job in queue waiting for the quiet period to pass is blocking all other jobs that have been queued after it.

      Example:

      1. Job A has quiet period set to 10 seconds
      2. Job B has quiet period set to 300 seconds
      3. Job C has quiet period set to 100 seconds
      4. Jobs get queued in A-B-C order.
      5. Job A starts executing after 10 seconds
      6. After 100 seconds, status of job C changes to "pending - Waiting for next available executor" even if free executors are available
      7. After 300 seconds both jobs B and C start executing

      This problem is hardly visible when short quiet periods are used but becomes problematic with longer quiet periods or plugins like Naginator that will increase the quiet period to hours if the builds fail enough.

      Version 1.451 is the last release where this problem isn't visible. From public releases, at least 1.452 and 1.454 are affected.

          [JENKINS-12994] Quiet period is blocking other jobs in queue

          I'm seeing this, or something like it, as well.

          I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.

          Matthew Webber added a comment - I'm seeing this, or something like it, as well. I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.

          allenservedio added a comment -

          I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).

          allenservedio added a comment - I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).

          Dirk Kuypers added a comment -

          I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.

          Dirk Kuypers added a comment - I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.

          Sami Tikka added a comment -

          I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.

          Sami Tikka added a comment - I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.

          Vincent Latombe added a comment - - edited

          I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5

          You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list.
          waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.

          Vincent Latombe added a comment - - edited I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5 You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list. waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.

          Chris Nogradi added a comment -

          I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.

          Chris Nogradi added a comment - I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.

          allenservedio added a comment -

          To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.

          allenservedio added a comment - To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.

          Daniel Tkatch added a comment -

          Daniel Tkatch added a comment - I might have the same problem I described here: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/df72335ffe030639

          wbauer added a comment -

          I am seeing the same issue with "Waiting for next available executor" when several machines are actually completely free. Issues happens since 1.452.

          wbauer added a comment - I am seeing the same issue with "Waiting for next available executor" when several machines are actually completely free. Issues happens since 1.452.

          This started for us with our upgrade to 1.455 and continues with 1.456

          DevTools Active added a comment - This started for us with our upgrade to 1.455 and continues with 1.456

          Shawn Garner added a comment -

          I've noted this issue on 1.452.
          We have job a with a quiet period and three idle executors.
          Then job b comes with no wait period.
          Job b waits until job a is done.
          I've also noticed that job b even when it is on the front of the queue that it seems to wait even though all three executors are free.
          It's almost as if it thinks job b has a quiet period too.
          They eventually all go through but just not quite the throughput you'd expect with three idle executors.
          I've noticed as many as 4 or 5 jobs backed up with all three executors idle.

          Shawn Garner added a comment - I've noted this issue on 1.452. We have job a with a quiet period and three idle executors. Then job b comes with no wait period. Job b waits until job a is done. I've also noticed that job b even when it is on the front of the queue that it seems to wait even though all three executors are free. It's almost as if it thinks job b has a quiet period too. They eventually all go through but just not quite the throughput you'd expect with three idle executors. I've noticed as many as 4 or 5 jobs backed up with all three executors idle.

          Same issue here. All builds are blocked until all quiet periods end. This is a serious disadvantage and I would like to vote for fixing this in the first place.

          Krzysztof Malinowski added a comment - Same issue here. All builds are blocked until all quiet periods end. This is a serious disadvantage and I would like to vote for fixing this in the first place.

          Waldek M added a comment -

          Same here. In case of many scheduled builds, this may cause them never to happen.

          Waldek M added a comment - Same here. In case of many scheduled builds, this may cause them never to happen.

          Bumping priority from Major to Critical, as this problem is adversely affecting a lot of people.

          Matthew Webber added a comment - Bumping priority from Major to Critical, as this problem is adversely affecting a lot of people.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          changelog.html
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/e4abef20fbdb71971085e7507d7fff4b92676368
          Log:
          Merge pull request #419 from Vlatombe/jenkins-12994

          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: changelog.html core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/e4abef20fbdb71971085e7507d7fff4b92676368 Log: Merge pull request #419 from Vlatombe/jenkins-12994 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue

          dogfood added a comment -

          Integrated in jenkins_main_trunk #1623
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179)
          Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b)

          Result = SUCCESS
          Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179
          Files :

          • core/src/main/java/hudson/model/Queue.java

          Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #1623 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179) Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b) Result = SUCCESS Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179 Files : core/src/main/java/hudson/model/Queue.java Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b Files : changelog.html

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/394e9d6c0488fae6834d97a158a018abb31f3179
          Log:
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
          Queue#maintain() was returning too soon

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/394e9d6c0488fae6834d97a158a018abb31f3179 Log: [FIX JENKINS-12994] Quiet period is blocking other jobs in queue Queue#maintain() was returning too soon

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/07b3f2cccb077df85617f2748f9b329528bc263b
          Log:
          Update changelog for JENKINS-12994

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html http://jenkins-ci.org/commit/jenkins/07b3f2cccb077df85617f2748f9b329528bc263b Log: Update changelog for JENKINS-12994

          dogfood added a comment -

          Integrated in jenkins_ui-changes_branch #21
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179)
          Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b)

          Result = SUCCESS
          Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179
          Files :

          • core/src/main/java/hudson/model/Queue.java

          Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_ui-changes_branch #21 [FIX JENKINS-12994] Quiet period is blocking other jobs in queue (Revision 394e9d6c0488fae6834d97a158a018abb31f3179) Update changelog for JENKINS-12994 (Revision 07b3f2cccb077df85617f2748f9b329528bc263b) Result = SUCCESS Vincent Latombe : 394e9d6c0488fae6834d97a158a018abb31f3179 Files : core/src/main/java/hudson/model/Queue.java Vincent Latombe : 07b3f2cccb077df85617f2748f9b329528bc263b Files : changelog.html

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          core/src/main/java/hudson/model/Queue.java
          http://jenkins-ci.org/commit/jenkins/210e50fc8547c99d66b033787371dae115f03b7e
          Log:
          [FIX JENKINS-12994] Quiet period is blocking other jobs in queue
          Queue#maintain() was returning too soon(cherry picked from commit 394e9d6c0488fae6834d97a158a018abb31f3179)

          Conflicts:

          core/src/main/java/hudson/model/Queue.java

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: core/src/main/java/hudson/model/Queue.java http://jenkins-ci.org/commit/jenkins/210e50fc8547c99d66b033787371dae115f03b7e Log: [FIX JENKINS-12994] Quiet period is blocking other jobs in queue Queue#maintain() was returning too soon(cherry picked from commit 394e9d6c0488fae6834d97a158a018abb31f3179) Conflicts: core/src/main/java/hudson/model/Queue.java

          Code changed in jenkins
          User: Vincent Latombe
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/a86887043605c6070c81894bf38e4c6084e1f229
          Log:
          Update changelog for JENKINS-12994(cherry picked from commit 07b3f2cccb077df85617f2748f9b329528bc263b)

          Conflicts:

          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vincent Latombe Path: changelog.html http://jenkins-ci.org/commit/jenkins/a86887043605c6070c81894bf38e4c6084e1f229 Log: Update changelog for JENKINS-12994 (cherry picked from commit 07b3f2cccb077df85617f2748f9b329528bc263b) Conflicts: changelog.html

          Antonio Rojas added a comment -

          Was it fixed and how to apply the fix if that is the case ? thanks for your help

          Antonio Rojas added a comment - Was it fixed and how to apply the fix if that is the case ? thanks for your help

          Daniel Beck added a comment -

          bqe_antonio_rojas: This was fixed years ago. It's really time to update your Jenkins if you're still affected.

          However if you think this happens in a recent Jenkins version, it's almost guaranteed an entirely different issue (e.g. JENKINS-27871) and has nothing to do with this one.

          Daniel Beck added a comment - bqe_antonio_rojas : This was fixed years ago. It's really time to update your Jenkins if you're still affected. However if you think this happens in a recent Jenkins version, it's almost guaranteed an entirely different issue (e.g. JENKINS-27871 ) and has nothing to do with this one.

            Unassigned Unassigned
            teetoivo Teemu Toivola
            Votes:
            32 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: