Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44756

throttle(...) step can be ineffective at restart time

    XMLWordPrintable

Details

    Description

      See discussion at JENKINS-44747 and https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/50 - I'm legitimately not sure whether anyone will ever actually encounter it, but it's a theoretical possibility that throttled Pipeline threads could fail to actually throttle upon master restart due to the fix for JENKINS-44747.

      Attachments

        Issue Links

          Activity

            basil Basil Crow added a comment -

            I observed this issue recently after restarting one of my Jenkins masters that had Throttle Concurrents in use. Throttle Concurrents was supposed to keep the # of builds that use a specific external resource down to 30. After the restart, I verified that at least 40 builds were using that external resource, indicating that the throttle was not taking effect. I only observed this after the restart; before the restart, things were fine. The end result for me as a user was that the external resource was overwhelmed and all my jobs failed. It's disappointing to me that bugs like this have existed for over a year without resolution. It effectively means the Throttle Concurrents plugin is unusable, at least in my use case.

            basil Basil Crow added a comment - I observed this issue recently after restarting one of my Jenkins masters that had Throttle Concurrents in use. Throttle Concurrents was supposed to keep the # of builds that use a specific external resource down to 30. After the restart, I verified that at least 40 builds were using that external resource, indicating that the throttle was not taking effect. I only observed this after the restart; before the restart, things were fine. The end result for me as a user was that the external resource was overwhelmed and all my jobs failed. It's disappointing to me that bugs like this have existed for over a year without resolution. It effectively means the Throttle Concurrents plugin is unusable, at least in my use case.
            oleg_nenashev Oleg Nenashev added a comment -

            > It's disappointing to me that bugs like this have existed for over a year without resolution

            Agreed, it is a disappointment. The plugin is open for adoption, and anybody is more than welcome to take an ownership or to propose a fix. Contributors move on sometimes, so plugins may stay in stale until somebody has interest to maintain them

             

            oleg_nenashev Oleg Nenashev added a comment - > It's disappointing to me that bugs like this have existed for over a year without resolution Agreed, it is a disappointment. The plugin is open for adoption, and anybody is more than welcome to take an ownership or to propose a fix. Contributors move on sometimes, so plugins may stay in stale until somebody has interest to maintain them  
            vizzard Aron Smyk added a comment -

            It's not only during the master restart. It affects every build started before restart.

            Due to some other issues I have to restart Jenkins every week (number of open files is rising to the point where Jenkins is not usable). As a result I'm observing this issue every week, and have to cancel builds started before reboot.

            My observations are a little different than the issue description. For me, throttle for the running pipelines is forgotten during the reboot. All new Jobs after the reboot are throttling correctly but they are not respecting throttling that happened before reboot. This leads to builds deadlocks and is very frustrating because builds takes a really long time in my case.

            I did not observed throttling problem during or after the restart but I haven't done much testing either. For me, it affects every single build started before reboot, making the Pipeline durability totally useless (meaning it can resume after master restart). I have to abort such build anyway because new builds will not see throttling of that build.

            vizzard Aron Smyk added a comment - It's not only during the master restart. It affects every build started before restart. Due to some other issues I have to restart Jenkins every week (number of open files is rising to the point where Jenkins is not usable). As a result I'm observing this issue every week, and have to cancel builds started before reboot. My observations are a little different than the issue description. For me, throttle for the running pipelines is forgotten during the reboot. All new Jobs after the reboot are throttling correctly but they are not respecting throttling that happened before reboot. This leads to builds deadlocks and is very frustrating because builds takes a really long time in my case. I did not observed throttling problem during or after the restart but I haven't done much testing either. For me, it affects every single build started before reboot, making the Pipeline durability totally useless (meaning it can resume after master restart). I have to abort such build anyway because new builds will not see throttling of that build.
            cng Wilson Ng added a comment -

            Do we have anyone fixing this bug ? This is really annoying if the Jenkins got restart, suddenly another set of builds got throttle, and max out our pool resource. 

            cng Wilson Ng added a comment - Do we have anyone fixing this bug ? This is really annoying if the Jenkins got restart, suddenly another set of builds got throttle, and max out our pool resource. 
            tkrah Torsten Krah added a comment -

            Found that one and wanted to add that we are facing this bug too, everytime master is restarting (reason for restart does not matter) - all builds scheduled before that restart fails to honor the throttle settings - which is a death sentence to all the slaves because they get easily overloaded because of that because of resource exhaustion.

            tkrah Torsten Krah added a comment - Found that one and wanted to add that we are facing this bug too, everytime master is restarting (reason for restart does not matter) - all builds scheduled before that restart fails to honor the throttle settings - which is a death sentence to all the slaves because they get easily overloaded because of that because of resource exhaustion.

            People

              Unassigned Unassigned
              abayer Andrew Bayer
              Votes:
              7 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated: