Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46809

Allow sequential stages inside parallel in Declarative syntax

    • Pipeline - April 2018

      Currently with -JENKINS-41334-, we set one or more stages to be executed in parallel but no "branches". Meaning that we cannot parallelise a sequence of stages.

       

      Example:

      pipeline {
        agent none
        stages {
          stage('Parallel stuff') {
            parallel 'branch 1' : {
              // Sequencial stages
              stage('Branch 1 stage 1'){
                  agent any
                  steps {
                      echo "In branch 1 stage 1"
                  }
              }
              stage('Branch 1 stage 2'){
                  agent none // With that kind of sequencial stage, we can change the agent to run on
                  steps {
                      sleep 30
                  }
              }
            }, 'branch 2': { // Parallel execution
              stage('Branch 2 stage 1'){
                  agent any
                  steps {
                      echo "In branch 2 stage 1"
                      sleep 60
                  }
              }
            }
          }
        }
      }
      

      Blue ocean possible view:

          [JENKINS-46809] Allow sequential stages inside parallel in Declarative syntax

          Andrew Bayer added a comment -

          We don't support arbitrarily deep nesting of parallel or sequential stages for a number of reasons - visualization challenges most prominently.

          Andrew Bayer added a comment - We don't support arbitrarily deep nesting of parallel or sequential stages for a number of reasons - visualization challenges most prominently.

          Liam Newman added a comment -

          richardwhiuk 

          You make a good point and I think there's other discussion to be had around this.  Could you open a JIRA for this?  

           

          Liam Newman added a comment - richardwhiuk   You make a good point and I think there's other discussion to be had around this.  Could you open a JIRA for this?    

          Jenn Briden added a comment -

          richardwhiuk, can you please give me an example of why you would set up your pipeline that way? Why not create a sibling stage with nested stages? It seems unnecessarily complex. (I'm the blue ocean and pipeline product manager, which is why I need to better understand the use case.)

          Jenn Briden added a comment - richardwhiuk , can you please give me an example of why you would set up your pipeline that way? Why not create a sibling stage with nested stages? It seems unnecessarily complex. (I'm the blue ocean and pipeline product manager, which is why I need to better understand the use case.)

          chanti vlad added a comment - - edited

          By creating a Pipeline job with the content of the ticket description:

          pipeline {
            agent none
            stages {
              stage('Parallel stuff') {
                parallel 'branch 1' : {
                  // Sequencial stages
                  stage('Branch 1 stage 1'){
                      agent any
                      steps {
                          echo "In branch 1 stage 1"
                      }
                  }
                  stage('Branch 1 stage 2'){
                      agent none // With that kind of sequencial stage, we can change the agent to run on
                      steps {
                          sleep 30
                      }
                  }
                }, 'branch 2': { // Parallel execution
                  stage('Branch 2 stage 1'){
                      agent any
                      steps {
                          echo "In branch 2 stage 1"
                          sleep 60
                      }
                  }
                }
              }
            }
          }
          

          i get following errors:

           

          WorkflowScript: 4: Expected a block for parallel @ line 4, column 5.
                 stage('Parallel stuff') {
                 ^
          
          WorkflowScript: 4: Expected one of "steps", "stages", or "parallel" for stage "Parallel stuff" @ line 4, column 5.
                 stage('Parallel stuff') {
                 ^
          
          2 errors
          

          i am using

          Jenkins 2.121.1 + Blue Ocean 1.6.0 +  Pipeline 2.5 +

          Pipeline Graph Analysis Plugin
          Provides a REST API to access pipeline and pipeline run data.
          1.6
          Pipeline implementation for Blue Ocean
          This plugin is a part of BlueOcean Plugin
          1.6.0
          Pipeline SCM API for Blue Ocean
          This plugin is a part of BlueOcean Plugin
          1.6.0
          Pipeline: API
          Plugin that defines Pipeline API.
          2.28
          Pipeline: Basic Steps
          Commonly used steps for Pipelines.
          2.9
          Pipeline: Build Step
          Adds the Pipeline step build to trigger builds of other jobs.
          2.7
          Pipeline: Declarative
          An opinionated, declarative Pipeline.
          1.3
          Pipeline: Declarative Agent API
          Replaced by Pipeline: Declarative Extension Points API plugin.
          1.1.1
          Pipeline: Declarative Extension Points API
          APIs for extension points used in Declarative Pipelines.
          1.3
          Pipeline: GitHub Groovy Libraries
          Allows Pipeline Grrovy libraries to be loaded on the fly from GitHub.
          1.0
          Pipeline: Groovy
          Pipeline execution engine based on continuation passing style transformation of Groovy scripts.
          2.53
          Pipeline: Input Step
          Adds the Pipeline step input to wait for human input or approval.
          2.8
          Pipeline: Job
          Defines a new job type for pipelines and provides their generic user interface.
          2.21
          Pipeline: Milestone Step
          Plugin that provides the milestone step
          1.3.1
          Pipeline: Model API
          Model API for Declarative Pipeline.
          1.3
          Pipeline: Multibranch
          Enhances Pipeline plugin to handle branches better by automatically grouping builds from different branches.
          2.19
          Pipeline: Nodes and Processes
          Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection.
          2.19
          Pipeline: REST API Plugin
          Provides a REST API to access pipeline and pipeline run data.
          2.10
          Pipeline: SCM Step
          Adds a Pipeline step to check out or update working sources from various SCMs (version control).
          2.6
          Pipeline: Shared Groovy Libraries
          Shared libraries for Pipeline scripts.
          2.9
          Pipeline: Stage Step
          Adds the Pipeline step stage to delineate portions of a build.
          2.3
          Pipeline: Stage Tags Metadata
          Library plugin for Pipeline stage tag metadata.
          1.3
          Pipeline: Stage View Plugin
          Pipeline Stage View Plugin.
          2.10
          Pipeline: Step API
          API for asynchronous build step primitive.
          2.15
          Pipeline: Supporting APIs
          Common utility implementations to build Pipeline Plugin
          2.18

          chanti vlad added a comment - - edited By creating a Pipeline job with the content of the ticket description: pipeline { agent none stages { stage( 'Parallel stuff' ) { parallel 'branch 1' : { // Sequencial stages stage( 'Branch 1 stage 1' ){ agent any steps { echo "In branch 1 stage 1" } } stage( 'Branch 1 stage 2' ){ agent none // With that kind of sequencial stage, we can change the agent to run on steps { sleep 30 } } }, 'branch 2' : { // Parallel execution stage( 'Branch 2 stage 1' ){ agent any steps { echo "In branch 2 stage 1" sleep 60 } } } } } } i get following errors:   WorkflowScript: 4: Expected a block for parallel @ line 4, column 5. stage( 'Parallel stuff' ) { ^ WorkflowScript: 4: Expected one of "steps" , "stages" , or "parallel" for stage "Parallel stuff" @ line 4, column 5. stage( 'Parallel stuff' ) { ^ 2 errors i am using Jenkins 2.121.1 + Blue Ocean 1.6.0 +  Pipeline 2.5 + Pipeline Graph Analysis Plugin Provides a REST API to access pipeline and pipeline run data. 1.6 Pipeline implementation for Blue Ocean This plugin is a part of BlueOcean Plugin 1.6.0 Pipeline SCM API for Blue Ocean This plugin is a part of BlueOcean Plugin 1.6.0 Pipeline: API Plugin that defines Pipeline API. 2.28 Pipeline: Basic Steps Commonly used steps for Pipelines. 2.9 Pipeline: Build Step Adds the Pipeline step  build  to trigger builds of other jobs. 2.7 Pipeline: Declarative An opinionated, declarative Pipeline. 1.3 Pipeline: Declarative Agent API Replaced by Pipeline: Declarative Extension Points API plugin. 1.1.1 Pipeline: Declarative Extension Points API APIs for extension points used in Declarative Pipelines. 1.3 Pipeline: GitHub Groovy Libraries Allows Pipeline Grrovy libraries to be loaded on the fly from GitHub. 1.0 Pipeline: Groovy Pipeline execution engine based on continuation passing style transformation of Groovy scripts. 2.53 Pipeline: Input Step Adds the Pipeline step  input  to wait for human input or approval. 2.8 Pipeline: Job Defines a new job type for pipelines and provides their generic user interface. 2.21 Pipeline: Milestone Step Plugin that provides the milestone step 1.3.1 Pipeline: Model API Model API for Declarative Pipeline. 1.3 Pipeline: Multibranch Enhances Pipeline plugin to handle branches better by automatically grouping builds from different branches. 2.19 Pipeline: Nodes and Processes Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection. 2.19 Pipeline: REST API Plugin Provides a REST API to access pipeline and pipeline run data. 2.10 Pipeline: SCM Step Adds a Pipeline step to check out or update working sources from various SCMs (version control). 2.6 Pipeline: Shared Groovy Libraries Shared libraries for Pipeline scripts. 2.9 Pipeline: Stage Step Adds the Pipeline step  stage  to delineate portions of a build. 2.3 Pipeline: Stage Tags Metadata Library plugin for Pipeline stage tag metadata. 1.3 Pipeline: Stage View Plugin Pipeline Stage View Plugin. 2.10 Pipeline: Step API API for asynchronous build step primitive. 2.15 Pipeline: Supporting APIs Common utility implementations to build Pipeline Plugin 2.18

          jbriden

          can you please give me an example of why you would set up your pipeline that way? Why not create a sibling stage with nested stages? It seems unnecessarily complex.

          The question you should be asking is "why are we creating arbitrary restrictions ?".

          Restricting actual functionality for the sake of easier visualization in Blue Ocean is not an answer you should find acceptable.

          The fact that you see a pipeline as unnecessarily complex doesn't mean that it is, or that it's wrong, or that it should be prohibited.

          If you're having trouble with presentation, look how TeamCity does it - it can present any pipeline you can imagine, with no issues.

          wknapik wknapik added a comment - jbriden can you please give me an example of why you would set up your pipeline that way? Why not create a sibling stage with nested stages? It seems unnecessarily complex. The question you should be asking is "why are we creating arbitrary restrictions ?". Restricting actual functionality for the sake of easier visualization in Blue Ocean is not an answer you should find acceptable. The fact that you see a pipeline as unnecessarily complex doesn't mean that it is, or that it's wrong, or that it should be prohibited. If you're having trouble with presentation, look how TeamCity does it - it can present any pipeline you can imagine, with no issues.

          Adam Voss added a comment -

          jbriden One use reason is to better control the executing node.

          In my example I have a sequential stage where each stage in the sequence may execute on a different node.  One of those stages that I would like to split into two stages, but can only do so if they execute in the same workspace because the second stage will depend on the local output of the first.  If I could have a nested sequential stage, I could define the agent on the nested sequential stage each of its stages would then share that workspace.

          My exiting alternative would be to stash the outputs form the first stage and unstash them on the 2nd, which (untested) seems like a lot more overhead and complexity than I justify for splitting the stage (especially if I were to split it into more than 2 stages).

          Adam Voss added a comment - jbriden One use reason is to better control the executing node. In my example I have a sequential stage where each stage in the sequence may execute on a different node.  One of those stages that I would like to split into two stages, but can only do so if they execute in the same workspace because the second stage will depend on the local output of the first.  If I could have a nested sequential stage, I could define the agent on the nested sequential stage each of its stages would then share that workspace. My exiting alternative would be to stash the outputs form the first stage and unstash them on the 2nd, which (untested) seems like a lot more overhead and complexity than I justify for splitting the stage (especially if I were to split it into more than 2 stages).

          Does this example works for anyone? 

          Do I need a special version for a special plugin?

          Oliver Santschi added a comment - Does this example works for anyone?  Do I need a special version for a special plugin?

          Leonard Lee added a comment -

          olle71, please see JENKINS-49050. Also, see this gist.

          Leonard Lee added a comment - olle71 , please see JENKINS-49050 . Also, see this gist .

          Hokwang Lee added a comment -

          Please be interested in this JENKINS-53127 related issue.

          Hokwang Lee added a comment - Please be interested in this JENKINS-53127 related issue.

          For what is is worth, I can't figure out if this was done, and if it was, when it was, and what version it is in or how to use it. And also, it seems like if we had the ability to have multiple pipelines for one push, that would help with all of this workload management. I would like to have nested parallelism as well as nested sequence.

           

          Nick Roosevelt added a comment - For what is is worth, I can't figure out if this was done, and if it was, when it was, and what version it is in or how to use it. And also, it seems like if we had the ability to have multiple pipelines for one push, that would help with all of this workload management. I would like to have nested parallelism as well as nested sequence.  

            abayer Andrew Bayer
            banst Bastien Arata
            Votes:
            66 Vote for this issue
            Watchers:
            103 Start watching this issue

              Created:
              Updated:
              Resolved: