Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44756

throttle(...) step can be ineffective at restart time

      See discussion at JENKINS-44747 and https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/50 - I'm legitimately not sure whether anyone will ever actually encounter it, but it's a theoretical possibility that throttled Pipeline threads could fail to actually throttle upon master restart due to the fix for JENKINS-44747.

          [JENKINS-44756] throttle(...) step can be ineffective at restart time

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/080f4768bfd31d55c717f904c223bfb6119a89fe
          Log:
          [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/080f4768bfd31d55c717f904c223bfb6119a89fe Log: [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          CHANGELOG.md
          http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/d58cb2873a58ed59323b033aecd3c2536c569ad3
          Log:
          Merge pull request #52 from jenkinsci/changelog/2.0.1

          [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice

          Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/cfc775b523fe...d58cb2873a58

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: CHANGELOG.md http://jenkins-ci.org/commit/throttle-concurrent-builds-plugin/d58cb2873a58ed59323b033aecd3c2536c569ad3 Log: Merge pull request #52 from jenkinsci/changelog/2.0.1 [JENKINS-44756, JENKINS-44747] - 2.0.1 changelog + Errata notice Compare: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/compare/cfc775b523fe...d58cb2873a58

          Jesse Glick added a comment -

          casz I think are talking about JENKINS-44747, not this issue which would be tricky to even observe, and remains hypothetical.

          Jesse Glick added a comment - casz I think are talking about  JENKINS-44747 , not this issue which would be tricky to even observe, and remains hypothetical.

          Your correct Jesse, my bad

          Joseph Petersen (old) added a comment - Your correct Jesse, my bad

          Oleg Nenashev added a comment -

          No plan to work on it anytime soon. Pull requests will be appreciated, reviewed and integrated if possible.

          Oleg Nenashev added a comment - No plan to work on it anytime soon. Pull requests will be appreciated, reviewed and integrated if possible.

          Basil Crow added a comment -

          I observed this issue recently after restarting one of my Jenkins masters that had Throttle Concurrents in use. Throttle Concurrents was supposed to keep the # of builds that use a specific external resource down to 30. After the restart, I verified that at least 40 builds were using that external resource, indicating that the throttle was not taking effect. I only observed this after the restart; before the restart, things were fine. The end result for me as a user was that the external resource was overwhelmed and all my jobs failed. It's disappointing to me that bugs like this have existed for over a year without resolution. It effectively means the Throttle Concurrents plugin is unusable, at least in my use case.

          Basil Crow added a comment - I observed this issue recently after restarting one of my Jenkins masters that had Throttle Concurrents in use. Throttle Concurrents was supposed to keep the # of builds that use a specific external resource down to 30. After the restart, I verified that at least 40 builds were using that external resource, indicating that the throttle was not taking effect. I only observed this after the restart; before the restart, things were fine. The end result for me as a user was that the external resource was overwhelmed and all my jobs failed. It's disappointing to me that bugs like this have existed for over a year without resolution. It effectively means the Throttle Concurrents plugin is unusable, at least in my use case.

          Oleg Nenashev added a comment -

          > It's disappointing to me that bugs like this have existed for over a year without resolution

          Agreed, it is a disappointment. The plugin is open for adoption, and anybody is more than welcome to take an ownership or to propose a fix. Contributors move on sometimes, so plugins may stay in stale until somebody has interest to maintain them

           

          Oleg Nenashev added a comment - > It's disappointing to me that bugs like this have existed for over a year without resolution Agreed, it is a disappointment. The plugin is open for adoption, and anybody is more than welcome to take an ownership or to propose a fix. Contributors move on sometimes, so plugins may stay in stale until somebody has interest to maintain them  

          Aron Smyk added a comment -

          It's not only during the master restart. It affects every build started before restart.

          Due to some other issues I have to restart Jenkins every week (number of open files is rising to the point where Jenkins is not usable). As a result I'm observing this issue every week, and have to cancel builds started before reboot.

          My observations are a little different than the issue description. For me, throttle for the running pipelines is forgotten during the reboot. All new Jobs after the reboot are throttling correctly but they are not respecting throttling that happened before reboot. This leads to builds deadlocks and is very frustrating because builds takes a really long time in my case.

          I did not observed throttling problem during or after the restart but I haven't done much testing either. For me, it affects every single build started before reboot, making the Pipeline durability totally useless (meaning it can resume after master restart). I have to abort such build anyway because new builds will not see throttling of that build.

          Aron Smyk added a comment - It's not only during the master restart. It affects every build started before restart. Due to some other issues I have to restart Jenkins every week (number of open files is rising to the point where Jenkins is not usable). As a result I'm observing this issue every week, and have to cancel builds started before reboot. My observations are a little different than the issue description. For me, throttle for the running pipelines is forgotten during the reboot. All new Jobs after the reboot are throttling correctly but they are not respecting throttling that happened before reboot. This leads to builds deadlocks and is very frustrating because builds takes a really long time in my case. I did not observed throttling problem during or after the restart but I haven't done much testing either. For me, it affects every single build started before reboot, making the Pipeline durability totally useless (meaning it can resume after master restart). I have to abort such build anyway because new builds will not see throttling of that build.

          Wilson Ng added a comment -

          Do we have anyone fixing this bug ? This is really annoying if the Jenkins got restart, suddenly another set of builds got throttle, and max out our pool resource. 

          Wilson Ng added a comment - Do we have anyone fixing this bug ? This is really annoying if the Jenkins got restart, suddenly another set of builds got throttle, and max out our pool resource. 

          Torsten Krah added a comment -

          Found that one and wanted to add that we are facing this bug too, everytime master is restarting (reason for restart does not matter) - all builds scheduled before that restart fails to honor the throttle settings - which is a death sentence to all the slaves because they get easily overloaded because of that because of resource exhaustion.

          Torsten Krah added a comment - Found that one and wanted to add that we are facing this bug too, everytime master is restarting (reason for restart does not matter) - all builds scheduled before that restart fails to honor the throttle settings - which is a death sentence to all the slaves because they get easily overloaded because of that because of resource exhaustion.

            Unassigned Unassigned
            abayer Andrew Bayer
            Votes:
            7 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: