Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58236

Provide mechanism to define dynamic stages in declarative pipelines

      Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

      pipeline {
        agent any
        stages {
          stage('build') {
            steps {
              runParallel items: ("a".."f").collect { "Stage ${it}" }
            }
          }
        }
      }
      
      def runParallel(args) {
        parallel args.items.collectEntries { name -> [ "${name}": {
          stage("${name}") {
            echo name
          }
        }]}
      }
      

      This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a separate function (I couldn't get it working without).
      On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

      pipeline {
        agent none
        stages {
          stage('build') {
            stages {
              ("A".."F").collect {
                stage("Stage ${it}") {
                  agent any
                  steps {
                    echo "Hello from stage ${it}!"
                  }
                }
              }
            }
          }
        }
      }
      

          [JENKINS-58236] Provide mechanism to define dynamic stages in declarative pipelines

          Hendrik Halkow created issue -
          Hendrik Halkow made changes -
          Description Original: Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

          {code}
          pipeline {
            agent any
            stages {
              stage('build') {
                steps {
                  runParallel items: ("a".."f").collect { "Stage ${it}" }
                }
              }
            }
          }

          def runParallel(args) {
            parallel args.items.collectEntries { name -> [ "${name}": {
              stage("${name}") {
                echo name
              }
            }]}
          }
          {code}

          This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a
          On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

          {code}
          pipeline {
            agent none
            stages {
              stage('build') {
                stages {
                  ("A".."F").collect {
                    stage("Stage ${it}") {
                      agent any
                      steps {
                        echo "Hello from stage ${it}!"
                      }
                    }
                  }
                }
              }
            }
          }
          {code}
          New: Hello, it would be great if we had a mechanism to define dynamic stages within declarative pipelines. The use case is that I want to create a pipeline function for micro service repositories. Each repository contains a number of container images that needs to be built in parallel. The only way I found is using the these pseudo stages wrapped by the `parallel` step:

          {code}
          pipeline {
            agent any
            stages {
              stage('build') {
                steps {
                  runParallel items: ("a".."f").collect { "Stage ${it}" }
                }
              }
            }
          }

          def runParallel(args) {
            parallel args.items.collectEntries { name -> [ "${name}": {
              stage("${name}") {
                echo name
              }
            }]}
          }
          {code}

          This is bad because the sub-stages can neither have an agent nor contain other sub-stages. And it requires a separate function (I couldn't get it working without).
          On the other hand, this shouldn't make the declarative pipeline too scripty. A minimal scriptiness approach would look like this:

          {code}
          pipeline {
            agent none
            stages {
              stage('build') {
                stages {
                  ("A".."F").collect {
                    stage("Stage ${it}") {
                      agent any
                      steps {
                        echo "Hello from stage ${it}!"
                      }
                    }
                  }
                }
              }
            }
          }
          {code}

          Hokwang Lee added a comment -

          The number of Dockerfile is increasing more and more, so this feature is really needed and important to all users.

          Hokwang Lee added a comment - The number of Dockerfile is increasing more and more, so this feature is really needed and important to all users.

          KY Lee added a comment -

          There is a way to achieve this. See this response: https://stackoverflow.com/a/51308400/795137

          The way he described is to create these dynamic stages within script closure. Wrap it around in a parallel closure and it should get what you want.

          KY Lee added a comment - There is a way to achieve this. See this response: https://stackoverflow.com/a/51308400/795137 The way he described is to create these dynamic stages within script closure. Wrap it around in a parallel closure and it should get what you want.

          This is pretty much the same as my first example that I don't want because of the explained reasons.

          Hendrik Halkow added a comment - This is pretty much the same as my first example that I don't want because of the explained reasons.
          Andrew Bayer made changes -
          Assignee Original: Andrew Bayer [ abayer ]
          Shantur Rathore made changes -
          Attachment New: image-2020-08-21-11-00-07-514.png [ 52405 ]
          Shantur Rathore made changes -
          Attachment New: Screenshot 2020-08-21 at 11.13.42.png [ 52407 ]
          Shantur Rathore made changes -
          Attachment Original: image-2020-08-21-11-00-07-514.png [ 52405 ]
          Shantur Rathore made changes -
          Attachment New: image-2020-08-21-11-14-53-397.png [ 52408 ]

            Unassigned Unassigned
            hendrikhalkow Hendrik Halkow
            Votes:
            13 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: