Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41334

Support parallel execution of stages in Declarative

    • Declarative - 1.2

      Improvement on roadmap

      This improvement is on the Blue Ocean project roadmap. Check the roadmap page for updates.

      Proposal
      Note - this is a post-1.0 feature.

      Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

      I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:

      pipeline {
        agent any
        
        stages {
          stage('first') {
            steps {
              echo 'first, non-parallel stage'
            }
          }
      
          stage('top-parallel') {
            stages {
              stage('first-parallel') {
                steps {
                  echo 'First of the parallel stages without further nesting'
                  sleep 60
                }
              }
              stage('second-parallel') {
                stages {
                  stage('first-nested-parallel') {
                    steps {
                       echo 'the first of the nested parallel stages'
                       sleep 30
                    }
                 }
                 stage('second-nested-parallel') {
                    steps {
                       echo 'the second of the nested parallel stages'
                       sleep 30
                    }
                 }
              }
           }
        }
      }
      

      So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.

      I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.

      Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.

          [JENKINS-41334] Support parallel execution of stages in Declarative

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          content/doc/book/pipeline/syntax.adoc
          http://jenkins-ci.org/commit/jenkins.io/35bd40690e95e96239bd9e07922d3f47a0ab407e
          Log:
          JENKINS-41334 Add parallel stages documentation

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/35bd40690e95e96239bd9e07922d3f47a0ab407e Log: JENKINS-41334 Add parallel stages documentation

          Code changed in jenkins
          User: R. Tyler Croy
          Path:
          content/doc/book/pipeline/shared-libraries.adoc
          content/doc/book/pipeline/syntax.adoc
          http://jenkins-ci.org/commit/jenkins.io/4c98e01767006f7c86310037cb0e2121c52192cd
          Log:
          Merge pull request #1046 from abayer/jenkins-41334

          JENKINS-41334 Add parallel stages documentation

          Compare: https://github.com/jenkins-infra/jenkins.io/compare/ee9271b021c8...4c98e0176700

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/doc/book/pipeline/shared-libraries.adoc content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/4c98e01767006f7c86310037cb0e2121c52192cd Log: Merge pull request #1046 from abayer/jenkins-41334 JENKINS-41334 Add parallel stages documentation Compare: https://github.com/jenkins-infra/jenkins.io/compare/ee9271b021c8...4c98e0176700

          Tang Ming added a comment - - edited

          kshultz Thanks. I have a main pipeline like below, It can be achieved by using build flow type project now. But build flow project can't trigger Pipeline type project.

           

          p1.1
          p1.2  p1.2.1
                p1.2.2
                p1.2.3
          p1.3  p1.3.1
                p1.3.2  p1.3.1.1
                        p1.3.1.2
                        p1.3.1.3
                p1.3.3

          p1.1, p1.2, p1.3 are in parallel. p1.2.1, p1.2.2, p1.2.3 are in parallel. and so on.

           

          Is there any way to achieve this ues Declarative Pipeline? I can achieve it  use scripted Pipeline, but it is hard to learn for other people.

          Tang Ming added a comment - - edited kshultz  Thanks. I have a main pipeline like below, It can be achieved by using build flow type project now. But build flow project can't trigger Pipeline type project.   p1.1 p1.2 p1.2.1 p1.2.2 p1.2.3 p1.3 p1.3.1 p1.3.2 p1.3.1.1 p1.3.1.2 p1.3.1.3 p1.3.3 p1.1, p1.2, p1.3 are in parallel. p1.2.1, p1.2.2, p1.2.3 are in parallel. and so on.   Is there any way to achieve this ues Declarative Pipeline? I can achieve it  use scripted Pipeline, but it is hard to learn for other people.

          Jakub Pawlinski added a comment - - edited

          Hi, thanks for this fix, but trying to use it and I cannot actually understand the difference. Sorry, maybe its just me, but it seems like I was hoping for much more than we actually got. Don't get me wrong, I appreciate the work and assume that a lot was changed under the hood to actually accommodate parallel in a similar manner to stage/step processes. Maybe I'm getting this wrong, take a look on OldVsNew diff, maybe I could update it more, but there are no examples of it.

           

          Additionally I tried  to use second level of parallel processing but ended up with 
          Parallel stages or branches can only be included in a top-level stage.
          I can attach example, but I guess this is not implemented, is it?

           

          Other issue was addressed in https://issues.jenkins-ci.org/browse/JENKINS-47219

           

           

          Jakub Pawlinski added a comment - - edited Hi, thanks for this fix, but trying to use it and I cannot actually understand the difference. Sorry, maybe its just me, but it seems like I was hoping for much more than we actually got. Don't get me wrong, I appreciate the work and assume that a lot was changed under the hood to actually accommodate parallel in a similar manner to stage/step processes. Maybe I'm getting this wrong, take a look on OldVsNew diff , maybe I could update it more, but there are no examples of it.   Additionally I tried  to use second level of parallel processing but ended up with  Parallel stages or branches can only be included in a top-level stage. I can attach example, but I guess this is not implemented, is it?   Other issue was addressed in  https://issues.jenkins-ci.org/browse/JENKINS-47219    

          James Dumay added a comment -

          quas we introduced a cleaner syntax for what you had previously using script blocks. You can also now define environment variables, agents, etc at a parallel stage level just like regular stages. What were you expecting instead?

          James Dumay added a comment - quas we introduced a cleaner syntax for what you had previously using script blocks. You can also now define environment variables, agents, etc at a parallel stage level just like regular stages. What were you expecting instead?

          Jakub Pawlinski added a comment - - edited

          I was hoping for nested parallels and parallel pipelines, so I could not only do:

          parallel {do {a} do {b}}

          but also:

          parallel {do {a, b, c} do {d}}

          I know that above can be achieved on step level, but step seems to be something small and is not visualised in blue ocean 

          parallel {do { do {a} do {b}} do {c}}

          I'm now aware that those are separated issues and watching them, but initially is seemed like this one could solve those too.

          Jakub Pawlinski added a comment - - edited I was hoping for nested parallels and parallel pipelines, so I could not only do: parallel { do {a} do {b}} but also: parallel { do {a, b, c} do {d}} I know that above can be achieved on step level, but step seems to be something small and is not visualised in blue ocean  parallel { do { do {a} do {b}} do {c}} I'm now aware that those are separated issues and watching them, but initially is seemed like this one could solve those too.

          Andrew Bayer added a comment -

          quas Yeah, we're adding these things gradually, to make sure the execution, syntax, and visualization all are working together at each step. I expect to get

          parallel {do {a, b, c} do {d}}
          

          before the end of the year, most likely - the nested parallels may be further out, though.

          Andrew Bayer added a comment - quas Yeah, we're adding these things gradually, to make sure the execution, syntax, and visualization all are working together at each step. I expect to get parallel { do {a, b, c} do {d}} before the end of the year, most likely - the nested parallels may be further out, though.

          Matt Kunze added a comment - - edited

          Is there another another issue to track the progress for having multiple steps within a parallel stream (do {a, b, c})? 

          I think I'm looking for something very similar to others - the ability to use parallel to split to different environments (Linux, Windows, etc) and then run multiple stages in sequence on each (Build, Test, Archive)

          Matt Kunze added a comment - - edited Is there another another issue to track the progress for having multiple steps within a parallel stream ( do {a, b, c })?  I think I'm looking for something very similar to others - the ability to use parallel to split to different environments (Linux, Windows, etc) and then run multiple stages in sequence on each (Build, Test, Archive)

          Andrew Bayer added a comment -

          mattkunze, quas Yup, JENKINS-46809 is what you're looking for.

          Andrew Bayer added a comment - mattkunze , quas Yup, JENKINS-46809 is what you're looking for.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

            abayer Andrew Bayer
            abayer Andrew Bayer
            Votes:
            42 Vote for this issue
            Watchers:
            86 Start watching this issue

              Created:
              Updated:
              Resolved: