-
New Feature
-
Resolution: Fixed
-
Minor
-
None
-
Powered by SuggestiMate -
Pipeline - April 2018
Currently with --, we set one or more stages to be executed in parallel but no "branches". Meaning that we cannot parallelise a sequence of stages.JENKINS-41334
Example:
pipeline { agent none stages { stage('Parallel stuff') { parallel 'branch 1' : { // Sequencial stages stage('Branch 1 stage 1'){ agent any steps { echo "In branch 1 stage 1" } } stage('Branch 1 stage 2'){ agent none // With that kind of sequencial stage, we can change the agent to run on steps { sleep 30 } } }, 'branch 2': { // Parallel execution stage('Branch 2 stage 1'){ agent any steps { echo "In branch 2 stage 1" sleep 60 } } } } } }
Blue ocean possible view:
- is duplicated by
-
JENKINS-43336 Allow locking multiple stages in declarative pipeline
-
- Closed
-
-
JENKINS-47163 Use same workspace or node in multiple stages in pipeline
-
- Closed
-
-
JENKINS-48422 Support parallel execution of sequence of stages
-
- Resolved
-
-
JENKINS-47532 Add possibility to run several stages 1-by-1 in a 'parallel' thread
-
- Resolved
-
-
JENKINS-39119 Can't lock nodes between stages
-
- Closed
-
- is related to
-
JENKINS-52084 Sequential stages post section does not execute in the expected context
-
- Closed
-
- relates to
-
JENKINS-39932 Support more arbitrary Pipeline definitions
-
- Closed
-
-
JENKINS-40986 Matrix structure for Declarative Pipeline
-
- Resolved
-
-
JENKINS-52025 I can edit my sequential declarative pipeline stages in the pipeline visualization plugin
-
- Open
-
-
JENKINS-49050 I can view my sequential declarative pipeline stages in the pipeline visualization plugin
-
- Closed
-
-
JENKINS-39119 Can't lock nodes between stages
-
- Closed
-
- links to
[JENKINS-46809] Allow sequential stages inside parallel in Declarative syntax
FYI, I'm leaning strongly towards the "Mixed list" approach right now...doing preliminary implementation work to make sure it's viable.
yeah I have trouble imagining it in mh mind - is it possible to have it as example/visual pairs? I think I understand, but my assumption may be wrong.
And if you're talking about what ends up being executed, it's the same either way - parallel branches with 1..N stages run sequentially inside them as specified.
And to make sure everyone knows the direction I'm leaning, it's this:
stage('mixed-list') { parallel { stage('a') { ... } group('b-and-c') { stage('b') { ... } stage('c') { ... } } } }
(again, naming of group can definitely be changed, it's the overall syntax that matters)
That'll run two parallel branches - one with Stage a, and the second with Stage b followed by Stage c.
abayer oh right - they all result in the same visual... nevermind..
Do the groups even need to be named? they aren't in the visual side of things right?
Yeah, we need some identifier for them, since they end up in a parallel branch that needs a name. I suppose in theory we could try to auto-generate a name randomly, but I'd prefer requiring one be specified.
Ok, preliminary work is up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227. I did go with the "mixed" parallel block contents (i.e., both stage and group can be in there), and since I've cracked that implementation, the other syntax options are kinda pointless, since the "mixed" one was superior in the first place (and backwards compatible!).
This is very much a work in progress - I've got all of one test for this currently, and wouldn't be shocked if some existing tests get broken by the change. But hey, let's get the ball rolling.
FYI, https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227 is probably the right place to discuss implementation details now - I've got a pile o' TODOs in the PR description, including a couple questions that need to be answered, along with a syntax example and a "diagram" of what this actually translates to at runtime.
My idea for the syntax. Sorry for crossposting.
parallel { stages { stage('Build on Linux') { steps { echo 'Building on Linux' } } stage('Test on Linux') { steps { echo 'Testing on Linux' } } } stages { stage('Build on macOS') { steps { echo 'Building on macOS' } } stage('Test on Linux') { steps { echo 'Testing on macOS' } } } }
I support above syntax. Groups already exist and are named "stages", no need to add extra block type.
I can see this issue has been in status Review for a month now. Could I hope for some progress anytime soon?
I am also curious if an "options { lock resource: '...' }" can be used on the new stage group. For now I have to stick with a "parallel" construct, which cannot solve all use cases.
We’re waiting on Blue Ocean to add support for the visualization and in the editor. And yes, you’ll be able to do options on the parent of the sequential group.
Cool.
Yes I think there is visualising and also editing - they are the "hard" bits (mostly fiddly I expect)
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
pipeline-model-api/src/main/resources/ast-schema.json
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/PostStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ValidatorTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/WhenStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ExecuteConvertedTest.java
pipeline-model-definition/src/test/resources/agentOnGroup.groovy
pipeline-model-definition/src/test/resources/environmentInGroup.groovy
pipeline-model-definition/src/test/resources/errors/emptyStagesInGroup.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/topLevelStageGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/json/agentOnGroup.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/topLevelStageGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/parallelStagesGroupsAndStages.json
pipeline-model-definition/src/test/resources/json/topLevelStageGroup.json
pipeline-model-definition/src/test/resources/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest/parallelAddsGroupsExecutionModelActionUpgrade.zip
pipeline-model-definition/src/test/resources/parallelStagesGroupsAndStages.groovy
pipeline-model-definition/src/test/resources/postStage/groupLocalAll.groovy
pipeline-model-definition/src/test/resources/postStage/postInParallelAndSequential.groovy
pipeline-model-definition/src/test/resources/toolsInGroup.groovy
pipeline-model-definition/src/test/resources/topLevelStageGroup.groovy
pipeline-model-definition/src/test/resources/when/simpleGroupWhen.groovy
pom.xml
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/e080af72f68c685bc32346bfd0e9fb87867eb4f6
Log:
[FIXED JENKINS-46809] Add sequential groups of parallel stages
Before anyone gets too excited, this isn't actually merged yet - I just pushed to a branch in the main repo so that JENKINS-45455's PR and someone's work-in-progress on matrix syntax can target that branch directly.
abayer Would you dare to guess for an ETA on this? I'm think I'm basically looking at the same use case as mattkunze (as he states here), where I'd split first by platform (and thereby node) and then perform sequential steps for that platform/node.
danielresolutiongames Same here. I am also looking for the same use case.
Is there any way we can pitch in to speed it up?
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
pipeline-model-api/src/main/resources/ast-schema.json
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/PostStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ValidatorTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/WhenStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ExecuteConvertedTest.java
pipeline-model-definition/src/test/resources/agentOnGroup.groovy
pipeline-model-definition/src/test/resources/environmentInGroup.groovy
pipeline-model-definition/src/test/resources/errors/emptyStagesInGroup.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/topLevelStageGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/json/agentOnGroup.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/topLevelStageGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/parallelStagesGroupsAndStages.json
pipeline-model-definition/src/test/resources/json/topLevelStageGroup.json
pipeline-model-definition/src/test/resources/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest/parallelAddsGroupsExecutionModelActionUpgrade.zip
pipeline-model-definition/src/test/resources/parallelStagesGroupsAndStages.groovy
pipeline-model-definition/src/test/resources/postStage/groupLocalAll.groovy
pipeline-model-definition/src/test/resources/postStage/postInParallelAndSequential.groovy
pipeline-model-definition/src/test/resources/toolsInGroup.groovy
pipeline-model-definition/src/test/resources/topLevelStageGroup.groovy
pipeline-model-definition/src/test/resources/when/simpleGroupWhen.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/78500b34f4c4f7c2b7089ac995c0269c32da644b
Log:
[FIXED JENKINS-46809] Add sequential groups of parallel stages
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
pipeline-model-api/src/main/resources/ast-schema.json
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/PostStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ValidatorTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/WhenStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ExecuteConvertedTest.java
pipeline-model-definition/src/test/resources/agentOnGroup.groovy
pipeline-model-definition/src/test/resources/environmentInGroup.groovy
pipeline-model-definition/src/test/resources/errors/emptyStagesInGroup.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/topLevelStageGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/json/agentOnGroup.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/topLevelStageGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/parallelStagesGroupsAndStages.json
pipeline-model-definition/src/test/resources/json/topLevelStageGroup.json
pipeline-model-definition/src/test/resources/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest/parallelAddsGroupsExecutionModelActionUpgrade.zip
pipeline-model-definition/src/test/resources/parallelStagesGroupsAndStages.groovy
pipeline-model-definition/src/test/resources/postStage/groupLocalAll.groovy
pipeline-model-definition/src/test/resources/postStage/postInParallelAndSequential.groovy
pipeline-model-definition/src/test/resources/toolsInGroup.groovy
pipeline-model-definition/src/test/resources/topLevelStageGroup.groovy
pipeline-model-definition/src/test/resources/when/simpleGroupWhen.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/c87840505e3a02e9da37416d8dd65f0240a34eb4
Log:
[FIXED JENKINS-46809] Add sequential groups of parallel stages
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
pipeline-model-api/src/main/resources/ast-schema.json
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/PostStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ValidatorTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/WhenStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ExecuteConvertedTest.java
pipeline-model-definition/src/test/resources/agentOnGroup.groovy
pipeline-model-definition/src/test/resources/environmentInGroup.groovy
pipeline-model-definition/src/test/resources/errors/emptyStagesInGroup.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/topLevelStageGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/json/agentOnGroup.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/topLevelStageGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/parallelStagesGroupsAndStages.json
pipeline-model-definition/src/test/resources/json/topLevelStageGroup.json
pipeline-model-definition/src/test/resources/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest/parallelAddsGroupsExecutionModelActionUpgrade.zip
pipeline-model-definition/src/test/resources/parallelStagesGroupsAndStages.groovy
pipeline-model-definition/src/test/resources/postStage/groupLocalAll.groovy
pipeline-model-definition/src/test/resources/postStage/postInParallelAndSequential.groovy
pipeline-model-definition/src/test/resources/toolsInGroup.groovy
pipeline-model-definition/src/test/resources/topLevelStageGroup.groovy
pipeline-model-definition/src/test/resources/when/simpleGroupWhen.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/9511756ebf7dcc2dc2fc5ed286ef1ba11525b3e8
Log:
[FIXED JENKINS-46809] Add sequential groups of parallel stages
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
pipeline-model-api/src/main/resources/ast-schema.json
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/PostStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ValidatorTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/WhenStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ExecuteConvertedTest.java
pipeline-model-definition/src/test/resources/agentOnGroup.groovy
pipeline-model-definition/src/test/resources/environmentInGroup.groovy
pipeline-model-definition/src/test/resources/errors/emptyStagesInGroup.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/topLevelStageGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/json/agentOnGroup.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/topLevelStageGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/parallelStagesGroupsAndStages.json
pipeline-model-definition/src/test/resources/json/topLevelStageGroup.json
pipeline-model-definition/src/test/resources/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest/parallelAddsGroupsExecutionModelActionUpgrade.zip
pipeline-model-definition/src/test/resources/parallelStagesGroupsAndStages.groovy
pipeline-model-definition/src/test/resources/postStage/groupLocalAll.groovy
pipeline-model-definition/src/test/resources/postStage/postInParallelAndSequential.groovy
pipeline-model-definition/src/test/resources/toolsInGroup.groovy
pipeline-model-definition/src/test/resources/topLevelStageGroup.groovy
pipeline-model-definition/src/test/resources/when/simpleGroupWhen.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/19a6837416add076568627cfc494c71efd6a1819
Log:
[FIXED JENKINS-46809] Add sequential groups of parallel stages
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStage.java
pipeline-model-api/src/main/resources/ast-schema.json
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/JSONParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/validator/ModelValidatorImpl.groovy
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/Messages.properties
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AbstractModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/EnvironmentTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/PostStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ValidatorTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/WhenStageTest.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ExecuteConvertedTest.java
pipeline-model-definition/src/test/resources/agentOnGroup.groovy
pipeline-model-definition/src/test/resources/environmentInGroup.groovy
pipeline-model-definition/src/test/resources/errors/emptyStagesInGroup.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/errors/parallelStagesStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/parallelStepsAndGroups.groovy
pipeline-model-definition/src/test/resources/errors/topLevelStageGroupsDeepNesting.groovy
pipeline-model-definition/src/test/resources/json/agentOnGroup.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/errors/parallelStagesStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/parallelStepsAndGroups.json
pipeline-model-definition/src/test/resources/json/errors/topLevelStageGroupsDeepNesting.json
pipeline-model-definition/src/test/resources/json/parallelStagesGroupsAndStages.json
pipeline-model-definition/src/test/resources/json/topLevelStageGroup.json
pipeline-model-definition/src/test/resources/org/jenkinsci/plugins/pipeline/modeldefinition/DeclarativeUpgradeTest/parallelAddsGroupsExecutionModelActionUpgrade.zip
pipeline-model-definition/src/test/resources/parallelStagesGroupsAndStages.groovy
pipeline-model-definition/src/test/resources/postStage/groupLocalAll.groovy
pipeline-model-definition/src/test/resources/postStage/postInParallelAndSequential.groovy
pipeline-model-definition/src/test/resources/toolsInGroup.groovy
pipeline-model-definition/src/test/resources/topLevelStageGroup.groovy
pipeline-model-definition/src/test/resources/when/simpleGroupWhen.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/79cad4ee2026e2123dcf13c6614c1988303fc034
Log:
[FIXED JENKINS-46809] Add sequential groups of parallel stages
*NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
I see there are multiple PRs raised. Is it available now? Anyway we can use this functionality?
I am facing the similar problem as mentioned here - https://issues.jenkins-ci.org/browse/JENKINS-39119 - when having the node block inside the stage block in a parallel tasks (why node block inside stage block, because we don't want to create duplicate stage for each task), the node get freed after the first stage and it become available for other jobs. Please find the below code, where the node get occupied by another job after the stage `first`.
def stages = ["first","second"] def tasks = [:] for (item in stages) { stage (item) { tasks["win"] = { node(mywinnode) { bat 'echo "check"' } } tasks["mac"] = { node(mymacnode) { sh 'echo "check"' } } parallel task } }
And in the above ticket its mentioned the fix will stated in this ticket. However I want to check is the fix is only for the declarative pipeline syntax?.
abayer: Would it be possible to provide a rough ETA on this issue? I need to consider writing my own multi-stage locking mechanism if this one won't arrive soon. BR.
It looks like it's getting closer - it's code review at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227 with some suggestion it might be released in the next week or two.
(finally) merged! JENKINS-45455 should be ready to merge shortly as well, and then I'd expect 1.3 to be out within a couple days (gotta finish docs polish/work for JENKINS-45455 first).
abayer will this resolve my requirement here https://issues.jenkins-ci.org/browse/JENKINS-46809?focusedCommentId=338694&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-338694
dineshk7 - yes, you will be able to have a parent stage with an agent and then a sequence of stages within that parent that automatically use the same agent/workspace/executor without freeing up the executor in between.
I testest shortly before, but I guess there's a visualization problem in here.
I use Jenkinsfile using official doc
https://jenkins.io/doc/book/pipeline/syntax/#parallel-stages-example
and it shows like below. I want to see nested stage.
horangs - the visualization is coming later on the Blue Ocean side - you can follow JENKINS-49050 for progress on that.
nimrod - you can already do nested stages in Scripted, though as with Declarative sequential stages, you won't get visualization for it in Blue Ocean until JENKINS-49050 is done.
When attempting a slightly complex flow, I get:
WorkflowScript: 954: Parallel stages or branches can only be included in a top-level stage. @ line 954, column 17.
stage('Destroy old QA1 deployment')
abayer Is there a good reason for such a restriction?
For example, when using this to resolve https://issues.jenkins-ci.org/browse/JENKINS-43336 it's desirable to have an outer stage which holds a lock, which contains two stages, one of which does a couple of things in parallel, and the other of which runs some stuff - e.g. setting up two test rigs in parallel, and then running some tests across both of them.
However, this is blocked, and it's not obvious to me why such a restriction exists.
We don't support arbitrarily deep nesting of parallel or sequential stages for a number of reasons - visualization challenges most prominently.
You make a good point and I think there's other discussion to be had around this. Could you open a JIRA for this?
richardwhiuk, can you please give me an example of why you would set up your pipeline that way? Why not create a sibling stage with nested stages? It seems unnecessarily complex. (I'm the blue ocean and pipeline product manager, which is why I need to better understand the use case.)
By creating a Pipeline job with the content of the ticket description:
pipeline { agent none stages { stage('Parallel stuff') { parallel 'branch 1' : { // Sequencial stages stage('Branch 1 stage 1'){ agent any steps { echo "In branch 1 stage 1" } } stage('Branch 1 stage 2'){ agent none // With that kind of sequencial stage, we can change the agent to run on steps { sleep 30 } } }, 'branch 2': { // Parallel execution stage('Branch 2 stage 1'){ agent any steps { echo "In branch 2 stage 1" sleep 60 } } } } } }
i get following errors:
WorkflowScript: 4: Expected a block for parallel @ line 4, column 5. stage('Parallel stuff') { ^ WorkflowScript: 4: Expected one of "steps", "stages", or "parallel" for stage "Parallel stuff" @ line 4, column 5. stage('Parallel stuff') { ^ 2 errors
i am using
Jenkins 2.121.1 + Blue Ocean 1.6.0 + Pipeline 2.5 +
Pipeline Graph Analysis Plugin Provides a REST API to access pipeline and pipeline run data. |
1.6 |
Pipeline implementation for Blue Ocean This plugin is a part of BlueOcean Plugin |
1.6.0 |
Pipeline SCM API for Blue Ocean This plugin is a part of BlueOcean Plugin |
1.6.0 |
Pipeline: API Plugin that defines Pipeline API. |
2.28 |
Pipeline: Basic Steps Commonly used steps for Pipelines. |
2.9 |
Pipeline: Build Step Adds the Pipeline step build to trigger builds of other jobs. |
2.7 |
Pipeline: Declarative An opinionated, declarative Pipeline. |
1.3 |
Pipeline: Declarative Agent API Replaced by Pipeline: Declarative Extension Points API plugin. |
1.1.1 |
Pipeline: Declarative Extension Points API APIs for extension points used in Declarative Pipelines. |
1.3 |
Pipeline: GitHub Groovy Libraries Allows Pipeline Grrovy libraries to be loaded on the fly from GitHub. |
1.0 |
Pipeline: Groovy Pipeline execution engine based on continuation passing style transformation of Groovy scripts. |
2.53 |
Pipeline: Input Step Adds the Pipeline step input to wait for human input or approval. |
2.8 |
Pipeline: Job Defines a new job type for pipelines and provides their generic user interface. |
2.21 |
Pipeline: Milestone Step Plugin that provides the milestone step |
1.3.1 |
Pipeline: Model API Model API for Declarative Pipeline. |
1.3 |
Pipeline: Multibranch Enhances Pipeline plugin to handle branches better by automatically grouping builds from different branches. |
2.19 |
Pipeline: Nodes and Processes Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection. |
2.19 |
Pipeline: REST API Plugin Provides a REST API to access pipeline and pipeline run data. |
2.10 |
Pipeline: SCM Step Adds a Pipeline step to check out or update working sources from various SCMs (version control). |
2.6 |
Pipeline: Shared Groovy Libraries Shared libraries for Pipeline scripts. |
2.9 |
Pipeline: Stage Step Adds the Pipeline step stage to delineate portions of a build. |
2.3 |
Pipeline: Stage Tags Metadata Library plugin for Pipeline stage tag metadata. |
1.3 |
Pipeline: Stage View Plugin Pipeline Stage View Plugin. |
2.10 |
Pipeline: Step API API for asynchronous build step primitive. |
2.15 |
Pipeline: Supporting APIs Common utility implementations to build Pipeline Plugin |
2.18 |
can you please give me an example of why you would set up your pipeline that way? Why not create a sibling stage with nested stages? It seems unnecessarily complex.
The question you should be asking is "why are we creating arbitrary restrictions ?".
Restricting actual functionality for the sake of easier visualization in Blue Ocean is not an answer you should find acceptable.
The fact that you see a pipeline as unnecessarily complex doesn't mean that it is, or that it's wrong, or that it should be prohibited.
If you're having trouble with presentation, look how TeamCity does it - it can present any pipeline you can imagine, with no issues.
jbriden One use reason is to better control the executing node.
In my example I have a sequential stage where each stage in the sequence may execute on a different node. One of those stages that I would like to split into two stages, but can only do so if they execute in the same workspace because the second stage will depend on the local output of the first. If I could have a nested sequential stage, I could define the agent on the nested sequential stage each of its stages would then share that workspace.
My exiting alternative would be to stash the outputs form the first stage and unstash them on the 2nd, which (untested) seems like a lot more overhead and complexity than I justify for splitting the stage (especially if I were to split it into more than 2 stages).
Does this example works for anyone?
Do I need a special version for a special plugin?
For what is is worth, I can't figure out if this was done, and if it was, when it was, and what version it is in or how to use it. And also, it seems like if we had the ability to have multiple pipelines for one push, that would help with all of this workload management. I would like to have nested parallelism as well as nested sequence.
The idea of parallelGroups is that groups would need to be in that, while parallel would have only stages. It'd simplify parsing and JSON translation significantly over the other options.