• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • None
    • Platform: All, OS: All

      Starting from version 1.452, a job in queue waiting for the quiet period to pass is blocking all other jobs that have been queued after it.

      Example:

      1. Job A has quiet period set to 10 seconds
      2. Job B has quiet period set to 300 seconds
      3. Job C has quiet period set to 100 seconds
      4. Jobs get queued in A-B-C order.
      5. Job A starts executing after 10 seconds
      6. After 100 seconds, status of job C changes to "pending - Waiting for next available executor" even if free executors are available
      7. After 300 seconds both jobs B and C start executing

      This problem is hardly visible when short quiet periods are used but becomes problematic with longer quiet periods or plugins like Naginator that will increase the quiet period to hours if the builds fail enough.

      Version 1.451 is the last release where this problem isn't visible. From public releases, at least 1.452 and 1.454 are affected.

          [JENKINS-12994] Quiet period is blocking other jobs in queue

          Teemu Toivola created issue -

          I'm seeing this, or something like it, as well.

          I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.

          Matthew Webber added a comment - I'm seeing this, or something like it, as well. I have also recently installed the "Throttle Concurrent Builds Plugin" and the "Exclusion-Plugin", so initially I though it might be related to that, but this ticket suggests it's a core problem.

          allenservedio added a comment -

          I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).

          allenservedio added a comment - I am seeing the same thing in 1.454. Even worse, I am starting to see deadlocks around this. I had to go in and kill a three jobs where two of them were stuck in an infinite loop of quiet periods and the third (a scheduled job) was stuck waiting for an executor. As I was not watching the builds, they were stuck in that state for over an hour before I caught them (the quiet period for the other two jobs was 190 seconds, it just kept cycling through quiet periods).

          Dirk Kuypers added a comment -

          I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.

          Dirk Kuypers added a comment - I am seeing this, too, since 1.452 I think. Maybe this is a hint (or just coincidence?): I never had to kill the pending jobs, I went into "Configure Jenkins->Configure System" and just pressed back, without changing/saving anything. After that the queues got filled again. Maybe others could try this, too, to confirm if this helps.

          Sami Tikka added a comment -

          I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.

          Sami Tikka added a comment - I upgraded from 1.451 to 1.453 and all my builds started staying in the queue forever. Exactly same symptom: Build says it is waiting for available executor even though all slaves were online with free executors. Restarting Jenkins did not help. Downgraded back to 1.451 and the problem went away.

          Vincent Latombe added a comment - - edited

          I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5

          You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list.
          waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.

          Vincent Latombe added a comment - - edited I've dug in the history between 1.451 and 1.452, and it seems the issue is coming from the following commit : https://github.com/jenkinsci/jenkins/commit/1ac4872ecdc500c63c0dbfce5e4beb5ce78369e5 You can see at line 920-921 that the method returns if the first element of the waiting list is not ready to build, so it doesn't consider other elements in the waiting list. waitingList is a TreeSet<WaitingItem> and it looks like it is not properly sorted by timestamp, meaning that the compareTo method is not properly implemented.

          Chris Nogradi added a comment -

          I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.

          Chris Nogradi added a comment - I tried upgrading to 1.452 and 1.455 and both behave as described above with all jobs remaining in 'Waiting for next executor' state with all nodes free. Downgrading to 1.451 restores functionality.

          allenservedio added a comment -

          To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.

          allenservedio added a comment - To work around this, I have removed the quiet period from all of my jobs (running on 1.455 now). This will mean we will get out of order builds sometimes as it will not necessarily see changes from dependent repos - but we can live with that for a bit until this is fixed.
          sogabe made changes -
          Link New: This issue is duplicated by JENKINS-13095 [ JENKINS-13095 ]

          Daniel Tkatch added a comment -

          Daniel Tkatch added a comment - I might have the same problem I described here: http://groups.google.com/group/jenkinsci-users/browse_thread/thread/df72335ffe030639

            Unassigned Unassigned
            teetoivo Teemu Toivola
            Votes:
            32 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: