Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44085

have a simple way to limit the number of parallel branches that run concurrently

      If you have a large number of branches for a parallel step you may want to be nice and just run a maximum of 10 branches at any one time.

      For a single pipeline project you could use the lockable-resource-plugin to achieve this but for a multi-branch you need to create a lock for each PR in advance. (You would want all PRs / SCM branch jobs) to be able to run 10 of the branches in parallel not just 10 across all of the the jobs.

      You could also use the throttle-concurrent-builds but that only works with nodes and has the same limitations - that the categories need to be set up ahead of time.

      Even if the categories/resources would not need to be setup ahead of time (e.g. enhance the creation of the resource to let it have a capacity of N) you would end up polluting the global configuration for something that is inherently Run based.

      Therefore it would be desirable to implement an extra option to the parallel step to limit that maximum number of branches that would be run at one time.

      e.g. given the following code only 10 branches would run at once.

      def branches [:]
      for (int i=0;i<1000;i++) {
        def thing = "$i"
        map[thing] = {
          echo "$thing"
        }
      }
      branches[maxConcurrent] = 10
      parallel branches
      

      the current workaround would be to use a BlockingDeque and manually adding in / removing values when starting and exiting the parallel block, or by using the waitUntil and something like an AtomicInteger

          [JENKINS-44085] have a simple way to limit the number of parallel branches that run concurrently

          Michael Who added a comment -

          I am wondering if withLocks is being used whether the thread is still created but stay being blocked until the lock is released. Probably the usage of a queue is still necessary.

          Michael Who added a comment - I am wondering if withLocks is being used whether the thread is still created but stay being blocked until the lock is released. Probably the usage of a queue is still necessary.

          Sam Gleske added a comment -

          From what I can tell in source code it doesn't actually create a concurrency lock.  Lockable resources plugin queues "threads" as lightweight jobs.  Eventually, the jobs get scheduled.  So it doesn't work in the traditional sense of what you think of as locks in concurrent high performance programming.

          Sam Gleske added a comment - From what I can tell in source code it doesn't actually create a concurrency lock.  Lockable resources plugin queues "threads" as lightweight jobs.  Eventually, the jobs get scheduled.  So it doesn't work in the traditional sense of what you think of as locks in concurrent high performance programming.

          This looks like what the https://github.com/jenkinsci/concurrent-step-plugin wants to achieve. locakable-resources is more for locks between different jobs/runs. cuncurrent-step exposes Java concurrent primitives inside a pipeline run.

          Tobias Gruetzmacher added a comment - This looks like what the https://github.com/jenkinsci/concurrent-step-plugin wants to achieve. locakable-resources is more for locks between different jobs/runs. cuncurrent-step exposes Java concurrent primitives inside a pipeline run.

          jerry wiltse added a comment -

          Yes, i opened issue in February asking for release here:  https://github.com/jenkinsci/concurrent-step-plugin/issues/5

          jerry wiltse added a comment - Yes, i opened issue in February asking for release here:   https://github.com/jenkinsci/concurrent-step-plugin/issues/5

          jerry wiltse added a comment -

          jerry wiltse added a comment - Officially released!   https://github.com/jenkinsci/concurrent-step-plugin/issues/5#issuecomment-627217753

          bright.ma added a comment - - edited

          here is my workaround:

          I transform the task list to  2D list .

           

          // code placeholder
                 stage('4. parallel') {
          
                      steps {
                          echo "will run for loop to run scripts on more node"
                          script {
                              def nodeListList = [
                                         ["bf-01", "bf-02", "bf-03"], 
                                         ["bf-03", "bf-02", "bf-01"],
                                       ]                    
                              
                              nodeListList.eachWithIndex { nodeList, i -> // loop the 2D list
                                  stage("pre nodeList ${i}") {
                                      echo "pre stage nodeList: ${nodeList}  ${i}"
                                  }
          
                                  stage("loop nodeList ${i}") {
                                      def jobs = [:]                            
                                      nodeList.eachWithIndex { nodeName, j ->
                                          jobs["on_${nodeName}_${i}_${j}"] = {
                                              node(nodeName) {
                                                  stage("run ${i}_${j}") {
                                                      echo "${nodeName}-${i}-${j}, ${NODE_NAME}, ${env.WORKSPACE}"
                                                      sh   "sleep 60"
                                                  }
                                              }//end node()
                                          }//end jobs[]
                                      } // end nodeList.eachWithIndex 
                                      
                                      println "this jobs is: " + jobs
                                      parallel jobs
                                  }// end stage("loop nodeList")
          
                                  stage("post nodeList  ${i}") {
                                      echo "post stage nodeList: ${nodeList}  ${i}"
                                  }
          
                              }//end nodeListList.eachWithIndex 
                          }//end script
          
                          echo "end run for loop to run scripts on more node"
                      }
          
                  }
          

           

           def nodeListList = [ 
                       ["bf-01", "bf-02", "bf-03"],
                       ["bf-03", "bf-02", "bf-01"],
           ] 
          // update the 2D list , the pipeline will like below:
          

           

           

           

          // update the 2D list then will get different parallel pipeline
          def nodeListList = [
               ["bf-01", "bf-02", "bf-03", "bf-04"],
                ["bf-11", "bf-12", "bf-13", "bf-14"],
                ["bf-05", "bf-06"],
                ["bf-01", "bf-02", "bf-03", "bf-04", "bf-07", "bf-08"],
                 ["bf-gb-01", "bf-gb-02", "bf-gb-03", "bf-gb-04", "bf-gb-04","bf-gb-02","bf-gb-01"],
           ]
          

           

           

          bright.ma added a comment - - edited here is my workaround: I transform the task list to  2D list .   // code placeholder stage( '4. parallel' ) { steps { echo "will run for loop to run scripts on more node" script { def nodeListList = [ [ "bf-01" , "bf-02" , "bf-03" ], [ "bf-03" , "bf-02" , "bf-01" ], ] nodeListList.eachWithIndex { nodeList, i -> // loop the 2D list stage( "pre nodeList ${i}" ) { echo "pre stage nodeList: ${nodeList} ${i}" } stage( "loop nodeList ${i}" ) { def jobs = [:] nodeList.eachWithIndex { nodeName, j -> jobs[ "on_${nodeName}_${i}_${j}" ] = { node(nodeName) { stage( "run ${i}_${j}" ) { echo "${nodeName}-${i}-${j}, ${NODE_NAME}, ${env.WORKSPACE}" sh "sleep 60" } } //end node() } //end jobs[] } // end nodeList.eachWithIndex println " this jobs is: " + jobs parallel jobs } // end stage( "loop nodeList" ) stage( "post nodeList ${i}" ) { echo "post stage nodeList: ${nodeList} ${i}" } } //end nodeListList.eachWithIndex } //end script echo "end run for loop to run scripts on more node" } }   def nodeListList = [ [ "bf-01" , "bf-02" , "bf-03" ], [ "bf-03" , "bf-02" , "bf-01" ], ] // update the 2D list , the pipeline will like below:       // update the 2D list then will get different parallel pipeline def nodeListList = [ [ "bf-01" , "bf-02" , "bf-03" , "bf-04" ], [ "bf-11" , "bf-12" , "bf-13" , "bf-14" ], [ "bf-05" , "bf-06" ], [ "bf-01" , "bf-02" , "bf-03" , "bf-04" , "bf-07" , "bf-08" ], [ "bf-gb-01" , "bf-gb-02" , "bf-gb-03" , "bf-gb-04" , "bf-gb-04" , "bf-gb-02" , "bf-gb-01" ], ]    

          Jesse Glick added a comment -

          Do not use java.util.concurrent from (CPS-transformed) Pipeline script. It is not going to do what you think it does.

          One valid solution requiring no special plugins: https://github.com/jenkinsci/bom/blob/542369a68d4c8b604626b4dbc00a109cc8833836/Jenkinsfile#L47-L71

          Jesse Glick added a comment - Do not use java.util.concurrent from (CPS-transformed) Pipeline script. It is not going to do what you think it does. One valid solution requiring no special plugins: https://github.com/jenkinsci/bom/blob/542369a68d4c8b604626b4dbc00a109cc8833836/Jenkinsfile#L47-L71

          jerry wiltse added a comment -

          jglick sorry to bother but I'd like to clarify:

          Is it unsafe/misleading to use this plugin: https://github.com/jenkinsci/concurrent-step-plugin ?

          I know you didn't say exactly that, but I don't fully understand what you were saying, so I wanted to ask for clarification about this plugin directly.

          jerry wiltse added a comment - jglick sorry to bother but I'd like to clarify: Is it unsafe/misleading to use this plugin: https://github.com/jenkinsci/concurrent-step-plugin ? I know you didn't say exactly that, but I don't fully understand what you were saying, so I wanted to ask for clarification about this plugin directly.

          Jesse Glick added a comment -

          From a brief glance at https://github.com/jenkinsci/concurrent-step-plugin I would say that it is designed incorrectly (confuses “native” Java threads with “virtual” CPS VM threads) and should not be used. Most or all of its steps probably could be reimplemented correctly while using the same Pipeline script interface.

          Jesse Glick added a comment - From a brief glance at https://github.com/jenkinsci/concurrent-step-plugin I would say that it is designed incorrectly (confuses “native” Java threads with “virtual” CPS VM threads) and should not be used. Most or all of its steps probably could be reimplemented correctly while using the same Pipeline script interface.

          We're running into this again and would love to see it implemented.

          Jared Kauppila added a comment - We're running into this again and would love to see it implemented.

            Unassigned Unassigned
            teilo James Nord
            Votes:
            63 Vote for this issue
            Watchers:
            65 Start watching this issue

              Created:
              Updated: