Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41334

Support parallel execution of stages in Declarative

    • Declarative - 1.2

      Improvement on roadmap

      This improvement is on the Blue Ocean project roadmap. Check the roadmap page for updates.

      Proposal
      Note - this is a post-1.0 feature.

      Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

      I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:

      pipeline {
        agent any
        
        stages {
          stage('first') {
            steps {
              echo 'first, non-parallel stage'
            }
          }
      
          stage('top-parallel') {
            stages {
              stage('first-parallel') {
                steps {
                  echo 'First of the parallel stages without further nesting'
                  sleep 60
                }
              }
              stage('second-parallel') {
                stages {
                  stage('first-nested-parallel') {
                    steps {
                       echo 'the first of the nested parallel stages'
                       sleep 30
                    }
                 }
                 stage('second-nested-parallel') {
                    steps {
                       echo 'the second of the nested parallel stages'
                       sleep 30
                    }
                 }
              }
           }
        }
      }
      

      So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.

      I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.

      Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.

          [JENKINS-41334] Support parallel execution of stages in Declarative

          Andrew Bayer created issue -
          Andrew Bayer made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          Andrew Bayer added a comment -

          Very preliminary initial work up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/98 - right now it's just the infrastructure/parsing/schema/etc for making this work without actually testing the new parallelism directly, 'cos I wanted to see what else blew up as a result. =)

          Andrew Bayer added a comment - Very preliminary initial work up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/98 - right now it's just the infrastructure/parsing/schema/etc for making this work without actually testing the new parallelism directly, 'cos I wanted to see what else blew up as a result. =)

          Andrew Bayer added a comment -

          cc @vivek jamesdumay kzantow hrmpw michaelneale for your thoughts. Again, remember this is not targeted for 1.0 so don't freak out. =)

          Andrew Bayer added a comment - cc @vivek jamesdumay kzantow hrmpw michaelneale for your thoughts. Again, remember this is not targeted for 1.0 so don't freak out. =)

          Patrick Wolf added a comment -

          Nesting and intermixing of stages and stage seems overly complicated but I'm not sure I have something better right now. At least with parallel the fact that is something different visually alerts you that something different is going on. With this, you have to track indentation and number of } to know what to expect.

          Patrick Wolf added a comment - Nesting and intermixing of stages and stage seems overly complicated but I'm not sure I have something better right now. At least with parallel the fact that is something different visually alerts you that something different is going on. With this, you have to track indentation and number of } to know what to expect.

          Andrew Bayer added a comment -

          Yeah, I get what you mean. I'm open to calling it something other than stages but not parallel since that's already a meaningful term in Scripted Pipeline.

          Andrew Bayer added a comment - Yeah, I get what you mean. I'm open to calling it something other than stages but not parallel since that's already a meaningful term in Scripted Pipeline.

          Keith Zantow added a comment -

          Well, I understand the possible issue if this is basically run through the existing groovy interpreter, but I'd like to see something like:

          stage('Build') {
            parallel { // this indicates running stages in parallel
              stage('Windows') {
                 ...
              }
              stage('Linux') {
                ...
              }
            }
          }
          

          Or more simply, just have a rule that all stages within a stage must finish before it completes, and add a parallel flag for nested stages?

          stage('Build', parallel) {
            stage('Windows') {
               ...
            }
            stage('Linux') {
              ...
            }
          }
          

          ... just some thoughts on this. I'm not a big fan of huge nesting, I already feel like it's getting a bit too nested for my tastes in places. Might as well just use LISP if we wanted that!

          Keith Zantow added a comment - Well, I understand the possible issue if this is basically run through the existing groovy interpreter, but I'd like to see something like: stage( 'Build' ) { parallel { // this indicates running stages in parallel stage( 'Windows' ) { ... } stage( 'Linux' ) { ... } } } Or more simply, just have a rule that all stages within a stage must finish before it completes, and add a parallel flag for nested stages? stage( 'Build' , parallel) { stage( 'Windows' ) { ... } stage( 'Linux' ) { ... } } ... just some thoughts on this. I'm not a big fan of huge nesting, I already feel like it's getting a bit too nested for my tastes in places. Might as well just use LISP if we wanted that!

          Andrew Bayer added a comment -

          So I am vehemently opposed to adding more arguments to stage because I'm still not happy about there being any arguments to stage. =) Not to mention that would be a pain to translate between Groovy syntax and JSON syntax. kzantow's first example is basically the one I've proposed but with a different name for the block (parallel vs stages) and, well, I kinda have that working, so my bias is definitely in that direction, modulo naming. We can technically handle parallel as the name, just as we handle stage when it's not really StageStep on the backend, I just prefer to avoid duplication of names if possible.

          Andrew Bayer added a comment - So I am vehemently opposed to adding more arguments to stage because I'm still not happy about there being any arguments to stage . =) Not to mention that would be a pain to translate between Groovy syntax and JSON syntax. kzantow 's first example is basically the one I've proposed but with a different name for the block ( parallel vs stages ) and, well, I kinda have that working, so my bias is definitely in that direction, modulo naming. We can technically handle parallel as the name, just as we handle stage when it's not really StageStep on the backend, I just prefer to avoid duplication of names if possible.

          Patrick Wolf added a comment -

          I think the first example from kzantow is pretty easy to grok but it does reuse parallel in a different way than in steps . If we take the original example from abayer in description we get:

          pipeline {
            agent any
            
            stages {
              stage('first') {
                steps {
                  echo 'first, non-parallel stage'
                }
              }
          
              stage('top-parallel') {
                parallel {
                  stage('first-parallel') {
                    steps {
                      echo 'First of the parallel stages without further nesting'
                      sleep 60
                    }
                  }
                  stage('second-parallel') {
                    parallel {
                      stage('first-nested-parallel') {
                        steps {
                           echo 'the first of the nested parallel stages'
                           sleep 30
                        }
                     }
                     stage('second-nested-parallel') {
                        steps {
                           echo 'the second of the nested parallel stages'
                           sleep 30
                        }
                     }
                  }
               }
            }
          }
          

          Patrick Wolf added a comment - I think the first example from kzantow is pretty easy to grok but it does reuse parallel in a different way than in steps . If we take the original example from abayer in description we get: pipeline { agent any stages { stage( 'first' ) { steps { echo 'first, non-parallel stage' } } stage( 'top-parallel' ) { parallel { stage( 'first-parallel' ) { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage( 'second-parallel' ) { parallel { stage( 'first-nested-parallel' ) { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage( 'second-nested-parallel' ) { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } }

          Andrew Bayer added a comment -

          Worth mentioning that my indentation sucked. =)

          Andrew Bayer added a comment - Worth mentioning that my indentation sucked. =)

            abayer Andrew Bayer
            abayer Andrew Bayer
            Votes:
            42 Vote for this issue
            Watchers:
            86 Start watching this issue

              Created:
              Updated:
              Resolved: