-
Story
-
Resolution: Fixed
-
Critical
-
None
-
-
Declarative - 1.2
This improvement is on the Blue Ocean project roadmap. Check the roadmap page for updates.
Proposal
Note - this is a post-1.0 feature.
Issues like JENKINS-41198 and JENKINS-40699 are among the drivers for this - in the Declarative model, parallel doesn't quite fit in smoothly. We need a better answer for this so that more complicated parallel execution of stages is possible within the model.
I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested stages sections, like this:
pipeline { agent any stages { stage('first') { steps { echo 'first, non-parallel stage' } } stage('top-parallel') { stages { stage('first-parallel') { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage('second-parallel') { stages { stage('first-nested-parallel') { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage('second-nested-parallel') { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } }
So in this scenario, stage('first') runs first. When it completes, stage('top-parallel') starts and immediately goes into its nested stages in parallel. stage('first-parallel') starts and goes for 60 seconds, while stage('second-parallel') starts at the same time and descends into its nested stages in parallel as well.
I don't know yet where I'd allow agent and friends, but within a stage, you would need to have one and only one of steps or stages - i.e., a stage either could have steps it executes or it could be a container for parallel stages.
Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down.
- is blocked by
-
JENKINS-42362 Investigate impact of parallel changes for declarative
-
- Resolved
-
-
JENKINS-42364 Editor adaptations for stage based parallel declarative
-
- Resolved
-
- is duplicated by
-
JENKINS-41198 Per parallel branch agent configuration
-
- Closed
-
-
JENKINS-40699 Post-stage/build test results etc doesn't play well with parallel on multiple nodes
-
- Closed
-
- relates to
-
JENKINS-40986 Matrix structure for Declarative Pipeline
-
- Resolved
-
- links to
[JENKINS-41334] Support parallel execution of stages in Declarative
Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Comment |
[ Would also love to be able to code those parallel: {code} pipeline { stages { def parallelstages = [:] paralellstages['foo'] = { stage { agent {docker 'hello' } } } parallel parallelstages } } {code} ] |
Link |
New:
This issue is duplicated by |
Remote Link | New: This issue links to "RFC doc for review (Web Link)" [ 15496 ] |
Link |
New:
This issue is blocked by |
Link |
New:
This issue is blocked by |
Description |
Original:
Note - this is a post-1.0 feature. Issues like I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this: {code} pipeline { agent any stages { stage('first') { steps { echo 'first, non-parallel stage' } } stage('top-parallel') { stages { stage('first-parallel') { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage('second-parallel') { stages { stage('first-nested-parallel') { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage('second-nested-parallel') { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } } {code} So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well. I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages. Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down. |
New:
*Scope* * *Proposal* Note - this is a post-1.0 feature. Issues like I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this: {code} pipeline { agent any stages { stage('first') { steps { echo 'first, non-parallel stage' } } stage('top-parallel') { stages { stage('first-parallel') { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage('second-parallel') { stages { stage('first-nested-parallel') { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage('second-nested-parallel') { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } } {code} So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well. I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages. Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down. |
Status | Original: In Progress [ 3 ] | New: In Review [ 10005 ] |
Remote Link | New: This issue links to "PR #152 (Web Link)" [ 16107 ] |
Description |
Original:
*Scope* * *Proposal* Note - this is a post-1.0 feature. Issues like I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this: {code} pipeline { agent any stages { stage('first') { steps { echo 'first, non-parallel stage' } } stage('top-parallel') { stages { stage('first-parallel') { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage('second-parallel') { stages { stage('first-nested-parallel') { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage('second-nested-parallel') { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } } {code} So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well. I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages. Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down. |
New:
{panel:title=Improvement on roadmap|titleBGColor=#3878de|titleColor=white} This improvement is on the Blue Ocean project roadmap. Check the [roadmap page|https://jenkins.io/projects/blueocean/roadmap/] for updates. {panel} *Proposal* Note - this is a post-1.0 feature. Issues like I'd previously dabbled with a stage execution dependency graph, originally in Plumber, but am easing away from that now. While I love the idea, I can't find a comfortable way to make it work without it being required for every stage, which is a non-option - i.e., I don't think it's a good idea to always require that every single stage has a marker for what stage(s) it can run before or after. So now I'm leaning in the direction of nested {{stages}} sections, like this: {code} pipeline { agent any stages { stage('first') { steps { echo 'first, non-parallel stage' } } stage('top-parallel') { stages { stage('first-parallel') { steps { echo 'First of the parallel stages without further nesting' sleep 60 } } stage('second-parallel') { stages { stage('first-nested-parallel') { steps { echo 'the first of the nested parallel stages' sleep 30 } } stage('second-nested-parallel') { steps { echo 'the second of the nested parallel stages' sleep 30 } } } } } } {code} So in this scenario, {{stage('first')}} runs first. When it completes, {{stage('top-parallel')}} starts and immediately goes into its nested {{stages}} in parallel. {{stage('first-parallel')}} starts and goes for 60 seconds, while {{stage('second-parallel')}} starts at the same time and descends into *its* nested {{stages}} in parallel as well. I don't know yet where I'd allow {{agent}} and friends, but within a {{stage}}, you would need to have one and only one of {{steps}} or {{stages}} - i.e., a {{stage}} either could have steps it executes or it could be a container for parallel stages. Actually implementing this will need to be done in tandem with Blue Ocean visualization, of course. And this is for now just raw thoughts, but I wanted to get it written down. |