Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41334

Support parallel execution of stages in Declarative

XMLWordPrintable

    • Declarative - 1.2

      Improvement on roadmap

      This improvement is on the Blue Ocean project roadmap. Check the roadmap page for updates.

      Proposal
      Note - this is a post-1.0 feature.

      Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.

      I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:

      pipeline {
  agent any
  
  stages {
    stage('first') {
      steps {
        echo 'first, non-parallel stage'
      }
    }

    stage('top-parallel') {
      stages {
        stage('first-parallel') {
          steps {
            echo 'First of the parallel stages without further nesting'
            sleep 60
          }
        }
        stage('second-parallel') {
          stages {
            stage('first-nested-parallel') {
              steps {
                 echo 'the first of the nested parallel stages'
                 sleep 30
              }
           }
           stage('second-nested-parallel') {
              steps {
                 echo 'the second of the nested parallel stages'
                 sleep 30
              }
           }
        }
     }
  }
}

      So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.

      I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.

      Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.

        1. 2017-09-29 12_34_42-jenkins _ TestScript _ #90.png
          2017-09-29 12_34_42-jenkins _ TestScript _ #90.png
          14 kB
        2. OldVsNew.htm
          27 kB
        3. screenshot-1.png
          screenshot-1.png
          593 kB

          [JENKINS-41334] Support parallel execution of stages in Declarative

          Andrew Bayer added a comment -

          Very preliminary initial work up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/98 - right now it's just the infrastructure/parsing/schema/etc for making this work without actually testing the new parallelism directly, 'cos I wanted to see what else blew up as a result. =)

          Andrew Bayer added a comment - Very preliminary initial work up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/98 - right now it's just the infrastructure/parsing/schema/etc for making this work without actually testing the new parallelism directly, 'cos I wanted to see what else blew up as a result. =)

          Andrew Bayer added a comment -

          cc @vivek jamesdumay kzantow hrmpw michaelneale for your thoughts. Again, remember this is not targeted for 1.0 so don't freak out. =)

          Andrew Bayer added a comment - cc @vivek jamesdumay kzantow hrmpw michaelneale for your thoughts. Again, remember this is not targeted for 1.0 so don't freak out. =)

          Patrick Wolf added a comment -

          Nesting and intermixing of stages and stage seems overly complicated but I'm not sure I have something better right now. At least with parallel the fact that is something different visually alerts you that something different is going on. With this, you have to track indentation and number of } to know what to expect.

          Patrick Wolf added a comment - Nesting and intermixing of stages and stage seems overly complicated but I'm not sure I have something better right now. At least with parallel the fact that is something different visually alerts you that something different is going on. With this, you have to track indentation and number of } to know what to expect.

          Andrew Bayer added a comment -

          Yeah, I get what you mean. I'm open to calling it something other than stages but not parallel since that's already a meaningful term in Scripted Pipeline.

          Andrew Bayer added a comment - Yeah, I get what you mean. I'm open to calling it something other than stages but not parallel since that's already a meaningful term in Scripted Pipeline.

          Keith Zantow added a comment -

          Well, I understand the possible issue if this is basically run through the existing groovy interpreter, but I'd like to see something like:

          stage('Build') {
  parallel { // this indicates running stages in parallel
    stage('Windows') {
       ...
    }
    stage('Linux') {
      ...
    }
  }
}

          Or more simply, just have a rule that all stages within a stage must finish before it completes, and add a parallel flag for nested stages?

          stage('Build', parallel) {
  stage('Windows') {
     ...
  }
  stage('Linux') {
    ...
  }
}

          ... just some thoughts on this. I'm not a big fan of huge nesting, I already feel like it's getting a bit too nested for my tastes in places. Might as well just use LISP if we wanted that!

          Keith Zantow added a comment - Well, I understand the possible issue if this is basically run through the existing groovy interpreter, but I'd like to see something like: stage( 'Build' ) { parallel { // this indicates running stages in parallel stage( 'Windows' ) { ... } stage( 'Linux' ) { ... } } } Or more simply, just have a rule that all stages within a stage must finish before it completes, and add a parallel flag for nested stages? stage( 'Build' , parallel) { stage( 'Windows' ) { ... } stage( 'Linux' ) { ... } } ... just some thoughts on this. I'm not a big fan of huge nesting, I already feel like it's getting a bit too nested for my tastes in places. Might as well just use LISP if we wanted that!

          Andrew Bayer added a comment -

          So I am vehemently opposed to adding more arguments to stage because I'm still not happy about there being any arguments to stage. =) Not to mention that would be a pain to translate between Groovy syntax and JSON syntax. kzantow's first example is basically the one I've proposed but with a different name for the block (parallel vs stages) and, well, I kinda have that working, so my bias is definitely in that direction, modulo naming. We can technically handle parallel as the name, just as we handle stage when it's not really StageStep on the backend, I just prefer to avoid duplication of names if possible.

          Andrew Bayer added a comment - So I am vehemently opposed to adding more arguments to stage because I'm still not happy about there being any arguments to stage . =) Not to mention that would be a pain to translate between Groovy syntax and JSON syntax. kzantow 's first example is basically the one I've proposed but with a different name for the block ( parallel vs stages ) and, well, I kinda have that working, so my bias is definitely in that direction, modulo naming. We can technically handle parallel as the name, just as we handle stage when it's not really StageStep on the backend, I just prefer to avoid duplication of names if possible.

          Patrick Wolf added a comment -

          I think the first example from kzantow is pretty easy to grok but it does reuse parallel in a different way than in steps . If we take the original example from abayer in description we get:

          pipeline {
  agent any
  
  stages {
    stage('first') {
      steps {
        echo 'first, non-parallel stage'
      }
    }

    stage('top-parallel') {
      parallel {
        stage('first-parallel') {
          steps {
            echo 'First of the parallel stages without further nesting'
            sleep 60
          }
        }
        stage('second-parallel') {
          parallel {
            stage('first-nested-parallel') {
              steps {
                 echo 'the first of the nested parallel stages'
                 sleep 30
              }
           }
           stage('second-nested-parallel') {
              steps {
                 echo 'the second of the nested parallel stages'
                 sleep 30
              }
           }
        }
     }
  }
}

          Patrick Wolf added a comment - I think the first example from kzantow is pretty easy to grok but it does reuse parallel in a different way than in steps . If we take the original example from abayer in description we get: pipeline { agent any stages { stage( 'first' ) { steps { echo 'first, non-parallel stage' } } stage( 'top-parallel' ) { parallel { stage( 'first-parallel' ) { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage( 'second-parallel' ) { parallel { stage( 'first-nested-parallel' ) { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage( 'second-nested-parallel' ) { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } }

          Andrew Bayer added a comment -

          Worth mentioning that my indentation sucked. =)

          Andrew Bayer added a comment - Worth mentioning that my indentation sucked. =)

          James Dumay added a comment -

          abayer BTW I have not forgotten about this. Will come back to it when I come up for air...

          James Dumay added a comment - abayer BTW I have not forgotten about this. Will come back to it when I come up for air...

          Yannick Koehler added a comment - - edited

          I am interested in a scenario that looks like this:

            stages {
    stage('extract') {
    }
    stage('build') {
      parallel {
        stage('build product A') {
          stage('build module set 1') {
          }
          stage('build module set 2') {
          }
        }
        stage('build product B') {
        }
      }
    }
    stage('finalize') {
    }
  }

          The outcome order would be that build only occurs after extract, that finalize only occurs after all build sub-stage are completed. That build product A and build procduct B are done in parallel. That build module set 2 only occurs after build module set 1 has completed (since it has some dependencies).

          Now, I feel that using "stages" inside a "stage" or a "stages" tag make no sense. Also, indenting those "stage" looks messy.

          A suggestion would be to do this operation like Make does, by stating the stages first, then adjusting the dependencies later, i.e.:

            stages {
    stage('extract') { }
    stage('build product A') { } 
    stage('build module set 1') { } 
    stage('build module set 2') { } 
    stage('build product B') { } 
    stage('finalize') { }
  }

          This would be default create a set of dependencies such as 

             <target>: <dependencies>
   'build product A': 'extract'
   'build module set 1': 'build product A'
   'build module set 2': 'build module set 1'
   'build product B': 'build module set 2'
   'finalize': 'build product B'

          Same as today, the author can then change those dependencies from the default to what he wants:

            stages {
    stage('extract') { }
    stage('build product A') { } 
    stage('build module set 1') { } 
    stage('build module set 2') { } 
    stage('build product B') { 
      dependencies: [ 'extract' ]
    } 
    stage('finalize') { 
      dependencies: ['build product A', 'build product B']
    }
  }

          This effectively transforming the dependency table to 

             <target>: <dependencies>
   'build product A': 'extract'
   'build module set 1': 'build product A'
   'build module set 2': 'build module set 1'
   'build product B': 'extract'
   'finalize': 'build product A', 'build product B'

          Using this set of dependency, the parallelism used is imply in the dependency, as soon as a target dependency are met, the associated stage can start.

          This is backward compatible, get rid of the "parallel" declarative statement and add a minimum set of keywords "dependencies" to the current DSL.

          Anyway, just a need of mine and an opinion to share on how to do it.

          Yannick Koehler added a comment - - edited I am interested in a scenario that looks like this: stages { stage('extract') { } stage('build') { parallel { stage('build product A') { stage('build module set 1') { } stage('build module set 2') { } } stage('build product B') { } } } stage('finalize') { } } The outcome order would be that build only occurs after extract, that finalize only occurs after all build sub-stage are completed. That build product A and build procduct B are done in parallel. That build module set 2 only occurs after build module set 1 has completed (since it has some dependencies). Now, I feel that using "stages" inside a "stage" or a "stages" tag make no sense. Also, indenting those "stage" looks messy. A suggestion would be to do this operation like Make does, by stating the stages first, then adjusting the dependencies later, i.e.: stages { stage('extract') { } stage('build product A') { } stage('build module set 1') { } stage('build module set 2') { } stage('build product B') { } stage('finalize') { } } This would be default create a set of dependencies such as <target>: <dependencies> 'build product A': 'extract' 'build module set 1': 'build product A' 'build module set 2': 'build module set 1' 'build product B': 'build module set 2' 'finalize': 'build product B' Same as today, the author can then change those dependencies from the default to what he wants: stages { stage('extract') { } stage('build product A') { } stage('build module set 1') { } stage('build module set 2') { } stage('build product B') { dependencies: [ 'extract' ] } stage('finalize') { dependencies: ['build product A', 'build product B'] } } This effectively transforming the dependency table to <target>: <dependencies> 'build product A': 'extract' 'build module set 1': 'build product A' 'build module set 2': 'build module set 1' 'build product B': 'extract' 'finalize': 'build product A', 'build product B' Using this set of dependency, the parallelism used is imply in the dependency, as soon as a target dependency are met, the associated stage can start. This is backward compatible, get rid of the "parallel" declarative statement and add a minimum set of keywords "dependencies" to the current DSL. Anyway, just a need of mine and an opinion to share on how to do it.

          Liam Newman added a comment -

          abayer

          I think name-wise I'm against reusing stages to mean "parallel" without some extra marking. Children of stages are executed in serial at top level, it would be strange to make them execute in parallel at the non-root level. The behavior should not change so radically with some indicator, something like stages (parallel: true)? That would mean I could use the same marker for the top level stages.

          Any plans to have some sort of design meeting?

          Liam Newman added a comment - abayer I think name-wise I'm against reusing stages to mean "parallel" without some extra marking. Children of stages are executed in serial at top level, it would be strange to make them execute in parallel at the non-root level. The behavior should not change so radically with some indicator, something like stages (parallel: true) ? That would mean I could use the same marker for the top level stages . Any plans to have some sort of design meeting?

          Yannick Koehler added a comment -

          Could be wrong, but the concept of Jenkins stage was always about concurrency... To me, parallel and concurrent is pretty much the same thing. Before stage concurrency was limited to different build same stage name. In my proposal, concurrency is between different stage within the same build.

          Maybe this shows that we may need to augment stage to allow control of exactly how the stage's concurrency is defined, same build, between builds, same stage, different stage name...

          Note also that in my proposal, stage is not used to mean parallel, without explicit additionnal dependencies keywords, the stages are serialized. Only when a dependency indicate the 2 stage share the same dependency that they can then be run concurrently (or in parallel).

          Having said all that, I am not against adding a "parallel: true", or... restore the "concurrency" option and allowing to specify a maximum value to that same concurrency.

              stage('build product B') { 
      dependencies: [ 'extract' ]
      concurrency: infinite
    }

          Yannick Koehler added a comment - Could be wrong, but the concept of Jenkins stage was always about concurrency... To me, parallel and concurrent is pretty much the same thing. Before stage concurrency was limited to different build same stage name. In my proposal, concurrency is between different stage within the same build. Maybe this shows that we may need to augment stage to allow control of exactly how the stage's concurrency is defined, same build, between builds, same stage, different stage name... Note also that in my proposal, stage is not used to mean parallel, without explicit additionnal dependencies keywords, the stages are serialized. Only when a dependency indicate the 2 stage share the same dependency that they can then be run concurrently (or in parallel). Having said all that, I am not against adding a "parallel: true", or... restore the "concurrency" option and allowing to specify a maximum value to that same concurrency. stage('build product B') { dependencies: [ 'extract' ] concurrency: infinite }

          Andrew Bayer added a comment -

          I've posted an RFC doc for review up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/126

          Andrew Bayer added a comment - I've posted an RFC doc for review up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/126

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          rfc/JENKINS-41334-parallel-stage-execution.md
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/27e41ddb34b7a12307a8cba94e3c8f9126fe4c4e
          Log:
          RFC: JENKINS-41334: Parallel stage execution

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: rfc/ JENKINS-41334 -parallel-stage-execution.md http://jenkins-ci.org/commit/pipeline-model-definition-plugin/27e41ddb34b7a12307a8cba94e3c8f9126fe4c4e Log: RFC: JENKINS-41334 : Parallel stage execution

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          rfc/JENKINS-41334-parallel-stage-execution.md
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/c033644c282e0f1727fdb0b6beddaff6a5819723
          Log:
          Merge pull request #126 from abayer/parallel-stages-rfc

          RFC: JENKINS-41334: Parallel stage execution

          Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/00dc23399128...c033644c282e

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: rfc/ JENKINS-41334 -parallel-stage-execution.md http://jenkins-ci.org/commit/pipeline-model-definition-plugin/c033644c282e0f1727fdb0b6beddaff6a5819723 Log: Merge pull request #126 from abayer/parallel-stages-rfc RFC: JENKINS-41334 : Parallel stage execution Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/00dc23399128...c033644c282e

          Andrew Bayer added a comment -

          Actual PR up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/152!

          Andrew Bayer added a comment - Actual PR up at  https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/152 !

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidator.java
          pipeline-model-api/src/main/resources/ast-schema.json
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/f2211f5e7c487dbbe742c3e9d02e357a34aa09e8
          Log:
          [FIXED JENKINS-41334] Adding parallel stages.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidator.java pipeline-model-api/src/main/resources/ast-schema.json pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/f2211f5e7c487dbbe742c3e9d02e357a34aa09e8 Log: [FIXED JENKINS-41334] Adding parallel stages.

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidator.java
          pipeline-model-api/src/main/resources/ast-schema.json
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/f2211f5e7c487dbbe742c3e9d02e357a34aa09e8
          Log:
          [FIXED JENKINS-41334] Adding parallel stages.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidator.java pipeline-model-api/src/main/resources/ast-schema.json pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/f2211f5e7c487dbbe742c3e9d02e357a34aa09e8 Log: [FIXED JENKINS-41334] Adding parallel stages.

          Craig Rodrigues added a comment - - edited

          Is this fixed, or is there more work to be done on this?

          This comment:

          https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/152/#issuecomment-307122085

           

          seems to indicate that a new release of pipeline-stage-step-plugin is needed.  Is that right?

           

          Craig Rodrigues added a comment - - edited Is this fixed, or is there more work to be done on this? This comment: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/152/#issuecomment-307122085   seems to indicate that a new release of pipeline-stage-step-plugin is needed.  Is that right?  

          James Dumay added a comment -

          It's still in progress

          James Dumay added a comment - It's still in progress

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidator.java
          pipeline-model-api/src/main/resources/ast-schema.json
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/1755d0e8533184c10625006154bcc390360b88f0
          Log:
          [FIXED JENKINS-41334] Adding parallel stages.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidator.java pipeline-model-api/src/main/resources/ast-schema.json pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/1755d0e8533184c10625006154bcc390360b88f0 Log: [FIXED JENKINS-41334] Adding parallel stages.

          Andrew Bayer added a comment -

          This'll be in the Declarative 1.2 release.

          Andrew Bayer added a comment - This'll be in the Declarative 1.2 release.

          Frederick Vanderhaeghe added a comment -

          Is there already a target release date for the Declarative 1.2 release?

          Frederick Vanderhaeghe added a comment - Is there already a target release date for the Declarative 1.2 release?

          Andrew Bayer added a comment -

          quirexx - Nothing concrete - at least in theory, we need to wait for the Blue Ocean Editor to add support for this, but we may jump the gun on that since the editor already is missing support for a good number of syntactic features. =) My guess is early September, most likely - https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/174 for JENKINS-42753 has to wind up too.

          Andrew Bayer added a comment - quirexx - Nothing concrete - at least in theory, we need to wait for the Blue Ocean Editor to add support for this, but we may jump the gun on that since the editor already is missing support for a good number of syntactic features. =) My guess is early September, most likely - https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/174 for JENKINS-42753 has to wind up too.

          Cesar Salazar added a comment - - edited

          This is still not working in Blue Ocean 1.2, is it? I have tried with both the proposed solution here, and the example at https://github.com/jenkinsci/pipeline-examples/blob/master/declarative-examples/jenkinsfile-examples/mavenDocker.groovy#L52

          None of them work.

          When I used the syntax proposed here the error is:

          WorkflowScript: 27: Unknown stage section "stages". Starting with version 0.5, steps in a stage must be in a steps block. @ line 27, column 9.
 stage('build') {
 ^WorkflowScript: 27: Nothing to execute within stage "build" @ line 27, column 9.
 stage('build') {
 ^

          when using the example in the github repo:

          org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
 WorkflowScript: 27: Unknown stage section "parallel". Starting with version 0.5, steps in a stage must be in a steps block. @ line 27, column 9.
 stage('build') {
 ^
WorkflowScript: 27: Nothing to execute within stage "build" @ line 27, column 9.
 stage('build') {
 ^

          It doesn't work neither when I use stage inside parallel() inside steps. It fails with error:

          org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
 WorkflowScript: 40: unexpected token: stage @ line 40, column 17.
 stage ('prod') {
 ^

          Cesar Salazar added a comment - - edited This is still not working in Blue Ocean 1.2, is it? I have tried with both the proposed solution here, and the example at https://github.com/jenkinsci/pipeline-examples/blob/master/declarative-examples/jenkinsfile-examples/mavenDocker.groovy#L52 None of them work. When I used the syntax proposed here the error is: WorkflowScript: 27: Unknown stage section "stages" . Starting with version 0.5, steps in a stage must be in a steps block. @ line 27, column 9. stage( 'build' ) { ^WorkflowScript: 27: Nothing to execute within stage "build" @ line 27, column 9. stage( 'build' ) { ^ when using the example in the github repo: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 27: Unknown stage section "parallel" . Starting with version 0.5, steps in a stage must be in a steps block. @ line 27, column 9. stage( 'build' ) { ^ WorkflowScript: 27: Nothing to execute within stage "build" @ line 27, column 9. stage( 'build' ) { ^ It doesn't work neither when I use stage inside parallel() inside steps. It fails with error: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 40: unexpected token: stage @ line 40, column 17. stage ( 'prod' ) { ^

          Craig Rodrigues added a comment -

          This depends on newer versions of the Pipeline plugin that haven't been released.

          I'm looking here:

           

          https://wiki.jenkins.io/display/JENKINS/Pipeline+Plugin

          https://github.com/jenkinsci/pipeline-model-definition-plugin/releases

           

           

          Craig Rodrigues added a comment - This depends on newer versions of the Pipeline plugin that haven't been released. I'm looking here:   https://wiki.jenkins.io/display/JENKINS/Pipeline+Plugin https://github.com/jenkinsci/pipeline-model-definition-plugin/releases    

          Liam Newman added a comment -

          Declarative 1.2 has not been released yet it looks like.

          Liam Newman added a comment - Declarative 1.2 has not been released yet it looks like.

          Len Tan added a comment -

          after investing a lot to change to declarative, assuming this would come soon,
          is there an indication when 1.2 will be released?

          Len Tan added a comment - after investing a lot to change to declarative, assuming this would come soon, is there an indication when 1.2 will be released?

          Karl Shultz added a comment -

          len, 1.2 has been released and is now available.

          Karl Shultz added a comment - len , 1.2 has been released and is now available.

          Tang Ming added a comment -

          kshultz I can ues parallel stages now but I can't use parallel branch at the same. Are they conflicting?

          My Pipeline code:

          pipeline {
    agent any
    stages {
        stage('parallel') {
            parallel {
                stage('firstParallelStage') {
                    steps {
                        parallel firstBranch: {
                            node('master') {
                            build 'pipeline_test_1'
                            }
                        },
                        secondBranch: {
                            node('master') {
                            build 'pipeline_test_2'
                            }
                       },
                       failFast: false
                   }
               }
               stage('secondParallelStage') {
                   steps {
                       echo "firstParallelStage"
                   }
               }
           }
       }
    }
}

          Console Output:

          org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 4: Unknown stage section "parallel". Starting with version 0.5, steps in a stage must be in a steps block. @ line 4, column 9.
           stage('parallel') {
           ^

WorkflowScript: 4: Nothing to execute within stage "parallel" @ line 4, column 9.
           stage('parallel') {
           ^

2 errors

	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1073)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:67)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:429)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:392)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:221)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Finished: FAILURE

          Tang Ming added a comment - kshultz  I can ues parallel stages now but I can't use parallel branch at the same. Are they conflicting? My Pipeline code: pipeline { agent any stages { stage( 'parallel' ) { parallel { stage( 'firstParallelStage' ) { steps { parallel firstBranch: { node( 'master' ) { build 'pipeline_test_1' } }, secondBranch: { node( 'master' ) { build 'pipeline_test_2' } }, failFast: false } } stage( 'secondParallelStage' ) { steps { echo "firstParallelStage" } } } } } } Console Output: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 4: Unknown stage section "parallel" . Starting with version 0.5, steps in a stage must be in a steps block. @ line 4, column 9. stage( 'parallel' ) { ^ WorkflowScript: 4: Nothing to execute within stage "parallel" @ line 4, column 9. stage( 'parallel' ) { ^ 2 errors at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1073) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:67) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:429) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:392) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:221) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:404) Finished: FAILURE

          Matt F added a comment -

          Installed latest Pipeline plugins after the 1.2 release. Would have expected the following Jenkinsfile to work according to the changes in 1.2:

          pipeline {
    agent { label 'myslave' }
    stages {
        stage('Parallel Test') {
            parallelStages {
                stage('Second Parallel') {
                    steps {
                        sh "sleep 15 && echo done second parallel"
                    }
                }
                stage('Third Parallel') {
                    steps {
                        sh "sleep 20 && echo done third parallel"
                    }
                }
            }
        }
    }
}

          yields:

          org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 14: Unknown stage section "parallelStages". Starting with version 0.5, steps in a stage must be in a steps block. @ line 14, column 9.
           stage('Parallel Test') {
           ^

WorkflowScript: 14: No "steps" or "parallel" to execute within stage "Parallel Test" @ line 14, column 9.
           stage('Parallel Test') {
           ^

2 errors

	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:419)
Finished: FAILURE

          Matt F added a comment - Installed latest Pipeline plugins after the 1.2 release. Would have expected the following Jenkinsfile to work according to the changes in 1.2: pipeline { agent { label 'myslave' } stages { stage( 'Parallel Test' ) { parallelStages { stage( 'Second Parallel' ) { steps { sh "sleep 15 && echo done second parallel" } } stage( 'Third Parallel' ) { steps { sh "sleep 20 && echo done third parallel" } } } } } } yields: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 14: Unknown stage section "parallelStages" . Starting with version 0.5, steps in a stage must be in a steps block. @ line 14, column 9. stage( 'Parallel Test' ) { ^ WorkflowScript: 14: No "steps" or "parallel" to execute within stage "Parallel Test" @ line 14, column 9. stage( 'Parallel Test' ) { ^ 2 errors at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419) Finished: FAILURE

          Andrew Bayer added a comment -

          matf - try just parallel instead of parallelStages

          Andrew Bayer added a comment - matf - try just parallel instead of parallelStages

          Karl Shultz added a comment -

          matf - the parallel block should be defined with the string parallel, not parallelStages. Try it this way, substituting your preferred agent label for my own, which is linux:

          pipeline {
    agent { label 'linux' }
    stages {
        stage('Parallel Test') {
            parallel {
                stage('Second Parallel') {
                    steps {
                        sh "sleep 15 && echo done second parallel"
                    }
                }
                stage('Third Parallel') {
                    steps {
                        sh "sleep 20 && echo done third parallel"
                    }
                }
            }
        }
    }
}

          Karl Shultz added a comment - matf - the parallel block should be defined with the string parallel , not parallelStages . Try it this way, substituting your preferred agent label for my own, which is linux : pipeline { agent { label 'linux' } stages { stage('Parallel Test') { parallel { stage('Second Parallel') { steps { sh "sleep 15 && echo done second parallel" } } stage('Third Parallel') { steps { sh "sleep 20 && echo done third parallel" } } } } } }

          Matt F added a comment -

          Cool, that works, thanks!

          Matt F added a comment - Cool, that works, thanks!

          Karl Shultz added a comment - - edited

          hummerstudio,

          Are you running Declarative 1.2? That message looks, to me, like the plugins haven't been upgraded. Otherwise I don't think you'd see Unknown stage section "parallel". Do us a favor, and check http://your-jenkins:your-port/pluginManager/installed. Should look like this:

          Karl Shultz added a comment - - edited hummerstudio , Are you running Declarative 1.2? That message looks, to me, like the plugins haven't been upgraded. Otherwise I don't think you'd see Unknown stage section "parallel" . Do us a favor, and check http://your-jenkins:your-port/pluginManager/installed . Should look like this:

          Karl Shultz added a comment -

          hummerstudio,

          Also, your suspicion is correct. Running parallel branches inside a set of parallel stages isn't supported. You should get an error message back explaining it:

          WorkflowScript: 23: Parallel stages or branches can only be included in a top-level stage. @ line 23, column 29.
 node('master') \{
 ^ 

          That's not to say you can't have both styles of parallelism, though, even inside the same Jenkinsfile. Here's an example that I created, starting with your original as a basis, and then modifying it to match up to my own agent labels, branch names, and so on. It works. In fact, the branch ten-parallels has its own set of parallel stages inside of it.

          pipeline {
    agent any
    stages {
        stage('parallel') {
            parallel {
                stage('firstParallelStage') {
                    // This is the supported syntax for specifying
                    // an agent label inside of a stage:
                    agent { label ('linux') }
                    steps {
                        build 'JENKINS-41334-Triage'
                    }
                }
                stage('secondParallelStage') {
                    agent { label ('linux') } 
                    steps {
                        build 'JENKINS-41334-Triage'
                    }
                }
            } //   this ends the parallel block...
        } //       ... and this ends the wrapper stage.

        // From here you could do a second parallel block, 
        // they just can't be nested inside of each other.
        // Here's the 'old' syntax. I've changed the 
        // jobs being built to reflect the names of some 
        // branches I have in my repo.
        stage('Parallel Steps As Before') {
            steps {
                parallel firstBranch: {
                    node('master') {
                        // This is a different branch in my multibranch 
                        // project. It has its own set of ten (10) parallel 
                        // stages. 
                        build 'ten-parallels'
                    }
                },
                secondBranch: {
                    node('master') {
                        // Same as above.
                        build 'ten-parallels'
                    }
                },
                failFast: false
            }
        }
    }
}

          Karl Shultz added a comment - hummerstudio , Also, your suspicion is correct. Running parallel branches inside a set of parallel stages isn't supported. You should get an error message back explaining it: WorkflowScript: 23: Parallel stages or branches can only be included in a top-level stage. @ line 23, column 29. node('master') \{ ^  That's not to say you can't have both styles of parallelism, though, even inside the same Jenkinsfile. Here's an example that I created, starting with your original as a basis, and then modifying it to match up to my own agent labels, branch names, and so on. It works. In fact, the branch ten-parallels has its own set of parallel stages inside of it. pipeline { agent any stages { stage('parallel') { parallel { stage('firstParallelStage') { // This is the supported syntax for specifying // an agent label inside of a stage: agent { label ('linux') } steps { build 'JENKINS-41334-Triage' } } stage('secondParallelStage') { agent { label ('linux') } steps { build 'JENKINS-41334-Triage' } } } // this ends the parallel block... } // ... and this ends the wrapper stage. // From here you could do a second parallel block, // they just can't be nested inside of each other. // Here's the 'old' syntax. I've changed the // jobs being built to reflect the names of some // branches I have in my repo. stage('Parallel Steps As Before') { steps { parallel firstBranch: { node('master') { // This is a different branch in my multibranch // project. It has its own set of ten (10) parallel // stages. build 'ten-parallels' } }, secondBranch: { node('master') { // Same as above. build 'ten-parallels' } }, failFast: false } } } }

          Tang Ming added a comment -

          kshultz  Thank for your reply! Sorry, I checked and found that I copied the console output from wrong page.

          the right console output is :

          org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 9: Parallel stages or branches can only be included in a top-level stage. @ line 9, column 29.
                               node('master') {
                               ^

1 error

	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1073)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:253)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:404)
Finished: FAILURE

          Below pipeline works good:

          pipeline {
agent any
stages {
    stage('parallel') {
        parallel {
            stage('first') {
                steps {
                    echo "First branch"
                    }
                }
            stage('second') {
                steps {
                    echo "Second branch"
                }
             }
         }
     }
   }
}

          and

          pipeline 
{
agent any
stages {
    stage('parallel build') {
        steps {
            parallel firstBranch: {
                node('master') {
                    echo 'pipeline_test_1'
                    }
                },
                secondBranch: {
                    node('master') {
                        echo 'pipeline_test_2'
                    }
                },
                failFast: false
            }
        }
    }
}

          But when I use parallel stages and parallel branches in one pipeline, it can't work. Does it not support nested?

          pipeline {
    agent any
    stages {
        stage('parallel') {
            parallel {
                stage('firstParallelStage') {
                    steps {
                        parallel firstBranch: {
                            node('master') {
                            build 'pipeline_test_1'
                            }
                        },
                        secondBranch: {
                            node('master') {
                            build 'pipeline_test_2'
                            }
                       },
                       failFast: false
                   }
               }
               stage('secondParallelStage') {
                   steps {
                       echo "firstParallelStage"
                   }
               }
           }
       }
    }
}

          Tang Ming added a comment - kshultz   Thank for your reply! Sorry, I checked and found that I copied the console output from wrong page. the right console output is : org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 9: Parallel stages or branches can only be included in a top-level stage. @ line 9, column 29. node( 'master' ) { ^ 1 error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1073) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:129) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:123) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:517) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:480) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:253) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:404) Finished: FAILURE Below pipeline works good: pipeline { agent any stages { stage( 'parallel' ) { parallel { stage( 'first' ) { steps { echo "First branch" } } stage( 'second' ) { steps { echo "Second branch" } } } } } } and pipeline { agent any stages { stage( 'parallel build' ) { steps { parallel firstBranch: { node( 'master' ) { echo 'pipeline_test_1' } }, secondBranch: { node( 'master' ) { echo 'pipeline_test_2' } }, failFast: false } } } } But when I use parallel stages and parallel branches in one pipeline, it can't work.  Does it not support nested? pipeline { agent any stages { stage( 'parallel' ) { parallel { stage( 'firstParallelStage' ) { steps { parallel firstBranch: { node( 'master' ) { build 'pipeline_test_1' } }, secondBranch: { node( 'master' ) { build 'pipeline_test_2' } }, failFast: false } } stage( 'secondParallelStage' ) { steps { echo "firstParallelStage" } } } } } }

          Karl Shultz added a comment -

          hummerstudio, you've got it basically correct. Nesting of parallel steps inside a set of parallel stages is not supported.

          However, you can mix and match both "styles" of parallelism in the same Pipeline. If you take a look at my previous comment in this issue, there is a sample Jenkinsfile which does exactly this - it mixes both styles. But they are not nested one inside of the other.

          Karl Shultz added a comment - hummerstudio , you've got it basically correct. Nesting of parallel steps inside a set of parallel stages is not supported. However, you can mix and match both "styles" of parallelism in the same Pipeline. If you take a look at my previous comment in this issue, there is a sample Jenkinsfile which does exactly this - it mixes both styles. But they are not nested one inside of the other.

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          content/doc/book/pipeline/syntax.adoc
          http://jenkins-ci.org/commit/jenkins.io/35bd40690e95e96239bd9e07922d3f47a0ab407e
          Log:
          JENKINS-41334 Add parallel stages documentation

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/35bd40690e95e96239bd9e07922d3f47a0ab407e Log: JENKINS-41334 Add parallel stages documentation

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: R. Tyler Croy
          Path:
          content/doc/book/pipeline/shared-libraries.adoc
          content/doc/book/pipeline/syntax.adoc
          http://jenkins-ci.org/commit/jenkins.io/4c98e01767006f7c86310037cb0e2121c52192cd
          Log:
          Merge pull request #1046 from abayer/jenkins-41334

          JENKINS-41334 Add parallel stages documentation

          Compare: https://github.com/jenkins-infra/jenkins.io/compare/ee9271b021c8...4c98e0176700

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/doc/book/pipeline/shared-libraries.adoc content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/4c98e01767006f7c86310037cb0e2121c52192cd Log: Merge pull request #1046 from abayer/jenkins-41334 JENKINS-41334 Add parallel stages documentation Compare: https://github.com/jenkins-infra/jenkins.io/compare/ee9271b021c8...4c98e0176700

          Tang Ming added a comment - - edited

          kshultz Thanks. I have a main pipeline like below, It can be achieved by using build flow type project now. But build flow project can't trigger Pipeline type project.

           

          p1.1
p1.2  p1.2.1
      p1.2.2
      p1.2.3
p1.3  p1.3.1
      p1.3.2  p1.3.1.1
              p1.3.1.2
              p1.3.1.3
      p1.3.3

          p1.1, p1.2, p1.3 are in parallel. p1.2.1, p1.2.2, p1.2.3 are in parallel. and so on.

           

          Is there any way to achieve this ues Declarative Pipeline? I can achieve it  use scripted Pipeline, but it is hard to learn for other people.

          Tang Ming added a comment - - edited kshultz  Thanks. I have a main pipeline like below, It can be achieved by using build flow type project now. But build flow project can't trigger Pipeline type project.   p1.1 p1.2 p1.2.1 p1.2.2 p1.2.3 p1.3 p1.3.1 p1.3.2 p1.3.1.1 p1.3.1.2 p1.3.1.3 p1.3.3 p1.1, p1.2, p1.3 are in parallel. p1.2.1, p1.2.2, p1.2.3 are in parallel. and so on.   Is there any way to achieve this ues Declarative Pipeline? I can achieve it  use scripted Pipeline, but it is hard to learn for other people.

          Jakub Pawlinski added a comment - - edited

          Hi, thanks for this fix, but trying to use it and I cannot actually understand the difference. Sorry, maybe its just me, but it seems like I was hoping for much more than we actually got. Don't get me wrong, I appreciate the work and assume that a lot was changed under the hood to actually accommodate parallel in a similar manner to stage/step processes. Maybe I'm getting this wrong, take a look on OldVsNew diff, maybe I could update it more, but there are no examples of it.

           

          Additionally I tried  to use second level of parallel processing but ended up with 
          Parallel stages or branches can only be included in a top-level stage.
          I can attach example, but I guess this is not implemented, is it?

           

          Other issue was addressed in https://issues.jenkins-ci.org/browse/JENKINS-47219

           

           

          Jakub Pawlinski added a comment - - edited Hi, thanks for this fix, but trying to use it and I cannot actually understand the difference. Sorry, maybe its just me, but it seems like I was hoping for much more than we actually got. Don't get me wrong, I appreciate the work and assume that a lot was changed under the hood to actually accommodate parallel in a similar manner to stage/step processes. Maybe I'm getting this wrong, take a look on OldVsNew diff , maybe I could update it more, but there are no examples of it.   Additionally I tried  to use second level of parallel processing but ended up with  Parallel stages or branches can only be included in a top-level stage. I can attach example, but I guess this is not implemented, is it?   Other issue was addressed in  https://issues.jenkins-ci.org/browse/JENKINS-47219    

          James Dumay added a comment -

          quas we introduced a cleaner syntax for what you had previously using script blocks. You can also now define environment variables, agents, etc at a parallel stage level just like regular stages. What were you expecting instead?

          James Dumay added a comment - quas we introduced a cleaner syntax for what you had previously using script blocks. You can also now define environment variables, agents, etc at a parallel stage level just like regular stages. What were you expecting instead?

          Jakub Pawlinski added a comment - - edited

          I was hoping for nested parallels and parallel pipelines, so I could not only do:

          parallel {do {a} do {b}}

          but also:

          parallel {do {a, b, c} do {d}}

          I know that above can be achieved on step level, but step seems to be something small and is not visualised in blue ocean 

          parallel {do { do {a} do {b}} do {c}}

          I'm now aware that those are separated issues and watching them, but initially is seemed like this one could solve those too.

          Jakub Pawlinski added a comment - - edited I was hoping for nested parallels and parallel pipelines, so I could not only do: parallel { do {a} do {b}} but also: parallel { do {a, b, c} do {d}} I know that above can be achieved on step level, but step seems to be something small and is not visualised in blue ocean  parallel { do { do {a} do {b}} do {c}} I'm now aware that those are separated issues and watching them, but initially is seemed like this one could solve those too.

          Andrew Bayer added a comment -

          quas Yeah, we're adding these things gradually, to make sure the execution, syntax, and visualization all are working together at each step. I expect to get 

          parallel {do {a, b, c} do {d}}

          before the end of the year, most likely - the nested parallels may be further out, though.

          Andrew Bayer added a comment - quas Yeah, we're adding these things gradually, to make sure the execution, syntax, and visualization all are working together at each step. I expect to get parallel { do {a, b, c} do {d}} before the end of the year, most likely - the nested parallels may be further out, though.

          Matt Kunze added a comment - - edited

          Is there another another issue to track the progress for having multiple steps within a parallel stream (do {a, b, c})? 

          I think I'm looking for something very similar to others - the ability to use parallel to split to different environments (Linux, Windows, etc) and then run multiple stages in sequence on each (Build, Test, Archive)

          Matt Kunze added a comment - - edited Is there another another issue to track the progress for having multiple steps within a parallel stream ( do {a, b, c })?  I think I'm looking for something very similar to others - the ability to use parallel to split to different environments (Linux, Windows, etc) and then run multiple stages in sequence on each (Build, Test, Archive)

          Andrew Bayer added a comment -

          mattkunze, quas Yup, JENKINS-46809 is what you're looking for.

          Andrew Bayer added a comment - mattkunze , quas Yup, JENKINS-46809 is what you're looking for.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

            abayer Andrew Bayer
            abayer Andrew Bayer
            Votes:
            42 Vote for this issue
            Watchers:
            86 Start watching this issue

              Created:
              Updated:
              Resolved: