Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66534

throttle current build not working when 2 or more build starts at same time with same parameter

      Dear Team,

      We have implemented a throttle concurrent build plugin in our prod Jenkins environment and are facing issues due to that. We are using the project category with prevent parameter option.

      In this, when 3build are coming at the same time with the same parameter it's not able to throttle the build. all builds are started same time and creating issues.

       

      Observation is when the first job is yet to clone on master other jobs are started running. If the first job is a clone on master then other jobs are get queued.

      For e.g: I have 3 jobs that started on the same time 12:00 with parameter name defaulttenantname and the value is dep011test-one-dev-ee. then while 1st job is getting clone the rest of the other jobs are started working.

      Here, If 1st job is cloned then the rest of the other jobs are getting queued.

       

      One more bug is if 1 job is in the queue and we trigger another job with the same parameter then that job does not get queued and even it does not show in Jenkins and never gets triggered.

      Please help me to solve this issue. As we are blocked due to this.

       

      Thanks,

      Jayesh Mahajan

       

          [JENKINS-66534] throttle current build not working when 2 or more build starts at same time with same parameter

          Team

          Please update.

          Jayesh Mahajan added a comment - Team Please update.

          Marco Jacob added a comment -

          We encountered a similar behaviour only with pipeline jobs.

          Until yesterday we had 10 maven build jobs with throttle of 0 for the same category. Only one job should run because they use the same resource.

          Now we switched from maven jobs to pipeline jobs with inline script and use the same configuration at the job configuration and in the pipeline script itself.

          The result is, that 2 jobs starting at the same time not blocked.
          In this case the server started should only be used by one of the two jobs.

          One of the jobs will now fail.

          The same example with maven jobs worked and there was no concurrent run.

          Marco Jacob added a comment - We encountered a similar behaviour only with pipeline jobs. Until yesterday we had 10 maven build jobs with throttle of 0 for the same category. Only one job should run because they use the same resource. Now we switched from maven jobs to pipeline jobs with inline script and use the same configuration at the job configuration and in the pipeline script itself. The result is, that 2 jobs starting at the same time not blocked. In this case the server started should only be used by one of the two jobs. One of the jobs will now fail. The same example with maven jobs worked and there was no concurrent run.

          Marco Jacob added a comment -

          This seems only to happen when the jobs are scheduled by cron at the same time, not when I start them manually after each other.

          Marco Jacob added a comment - This seems only to happen when the jobs are scheduled by cron at the same time, not when I start them manually after each other.

          Xiaoyi added a comment - - edited

          wow, 2 years later. same problem here.

          And I found where the bug is from.

          When jenkins maintains the build queue, it will ask the throttle plugin for if the job could be dispatched (canTake). This is done with a lock, so no race.

          However, it doesn't immediately dispatch the allowed job to executor, but batch them up until all the allowed items are dequeued first.

          When making the canTake decision, our lovely throttle plugin queries the executor for the number of jobs running there. But that number doesn't include the changes introduced from the previous decision.

          Which means, when multiple jobs are dispatched in the same maintain() cycle (5 sec window), the throttle plugin will be happily allowing all jobs to run at the same node regardless its per node limit, as it won't be able to tell if there are other jobs being dispatched there already.

          The apparent fix would be doing all the dispatch work sequentially, one job at a time, but I guess the current code is the result of past queue performance optimization efforts. So don't expect a fix anytime soon.

          Xiaoyi added a comment - - edited wow, 2 years later. same problem here. And I found where the bug is from . When jenkins maintains the build queue, it will ask the throttle plugin for if the job could be dispatched (canTake). This is done with a lock, so no race. However, it doesn't immediately dispatch the allowed job to executor, but batch them up until all the allowed items are dequeued first . When making the canTake decision, our lovely throttle plugin queries the executor for the number of jobs running there . But that number doesn't include the changes introduced from the previous decision. Which means, when multiple jobs are dispatched in the same maintain() cycle ( 5 sec window ), the throttle plugin will be happily allowing all jobs to run at the same node regardless its per node limit, as it won't be able to tell if there are other jobs being dispatched there already. The apparent fix would be doing all the dispatch work sequentially, one job at a time, but I guess the current code is the result of past queue performance optimization efforts. So don't expect a fix anytime soon.

            Unassigned Unassigned
            jayesh007 Jayesh Mahajan
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: