Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58236

Provide mechanism to define dynamic stages in declarative pipelines

      Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

      pipeline {
        agent any
        stages {
          stage('build') {
            steps {
              runParallel items: ("a".."f").collect { "Stage ${it}" }
            }
          }
        }
      }
      
      def runParallel(args) {
        parallel args.items.collectEntries { name -> [ "${name}": {
          stage("${name}") {
            echo name
          }
        }]}
      }
      

      This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a separate function (I couldn't get it working without).
      On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

      pipeline {
        agent none
        stages {
          stage('build') {
            stages {
              ("A".."F").collect {
                stage("Stage ${it}") {
                  agent any
                  steps {
                    echo "Hello from stage ${it}!"
                  }
                }
              }
            }
          }
        }
      }
      

          [JENKINS-58236] Provide mechanism to define dynamic stages in declarative pipelines

          Hokwang Lee added a comment -

          The number of Dockerfile is increasing more and more, so this feature is really needed and important to all users.

          Hokwang Lee added a comment - The number of Dockerfile is increasing more and more, so this feature is really needed and important to all users.

          KY Lee added a comment -

          There is a way to achieve this. See this response: https://stackoverflow.com/a/51308400/795137

          The way he described is to create these dynamic stages within script closure. Wrap it around in a parallel closure and it should get what you want.

          KY Lee added a comment - There is a way to achieve this. See this response: https://stackoverflow.com/a/51308400/795137 The way he described is to create these dynamic stages within script closure. Wrap it around in a parallel closure and it should get what you want.

          This is pretty much the same as my first example that I don't want because of the explained reasons.

          Hendrik Halkow added a comment - This is pretty much the same as my first example that I don't want because of the explained reasons.

          Shantur Rathore added a comment - - edited

          hendrikhalkow :

          You can have sub-stages and agent like below.

          You need to use Scripted way of using a node with `node('MyNodeLabel')`

          pipeline {
            agent any
            stages {
                stage('BuildAll') {
                    parallel {
                      stage('build-win') {
                        steps {
                          sh "env | grep -i NODE_NAME"
                          runParallel items: ("a".."d").collect { "Win-${it}" }
                        }
                      }
                      
                      stage('build-Mac') {
                        steps {
                          sh "env | grep -i NODE_NAME"
                          runParallel items: ("a".."d").collect { "Mac-${it}" }
                        }
                      }
                  }
              }
            }
          }
          
          def runParallel(args) {
           parallel args.items.collectEntries { name -> [ "${name}": { node('nodeLabel') {
              stage("${name}") {
                  stage("${name}-a") {
                      sh "env | grep -i NODE_NAME"
                  }
                  stage("${name}-b") {
                      sh "env | grep -i NODE_NAME"
                  }
              }}
            }]}
          }

           

           

           

          Shantur Rathore added a comment - - edited hendrikhalkow : You can have sub-stages and agent like below. You need to use Scripted way of using a node with `node('MyNodeLabel')` pipeline { agent any stages { stage( 'BuildAll' ) { parallel { stage( 'build-win' ) { steps { sh "env | grep -i NODE_NAME" runParallel items: ( "a" .. "d" ).collect { "Win-${it}" } } } stage( 'build-Mac' ) { steps { sh "env | grep -i NODE_NAME" runParallel items: ( "a" .. "d" ).collect { "Mac-${it}" } } } } } } } def runParallel(args) { parallel args.items.collectEntries { name -> [ "${name}" : { node( 'nodeLabel' ) { stage( "${name}" ) { stage( "${name}-a" ) { sh "env | grep -i NODE_NAME" } stage( "${name}-b" ) { sh "env | grep -i NODE_NAME" } }} }]} }      

          Having said that, the Blue Ocean goes crazy while its being built. It shows everything correctly once its built.

          A proper support for dynamic parallel stages in declarative pipeline would be really nice.

          Shantur Rathore added a comment - Having said that, the Blue Ocean goes crazy while its being built. It shows everything correctly once its built. A proper support for dynamic parallel stages in declarative pipeline would be really nice.

          Fabio Sbano added a comment -

          I solved it as follows:

          1. Jenkinsfile

          1. Groovy

          Fabio Sbano added a comment - I solved it as follows: Jenkinsfile Groovy

          Robin added a comment -

          shantur Your solution is also pretty much what hendrikhalkow presents in his first example. I agree you can specify the agent with `node`. But it doesn't mean we have all the features Declarative Pipeline offers. For example, another feature that would be appreciated is being able of adding a `post` in the sub-stage.

          Robin added a comment - shantur Your solution is also pretty much what hendrikhalkow presents in his first example. I agree you can specify the agent with `node`. But it doesn't mean we have all the features Declarative Pipeline offers. For example, another feature that would be appreciated is being able of adding a `post` in the sub-stage.

            Unassigned Unassigned
            hendrikhalkow Hendrik Halkow
            Votes:
            13 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: