Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53032

Support dynamic definition of parallel running stages

      I would like to define parallel stages{} blocks that are executed in parallel dynamically during runtime.

      The stages{} blocks should run in parallel, the stage{} definitions in the parallel stages{} blocks should run sequential.

       

      A usecase for this is that the pipeline determines which applications changed in a branch and then creates parallel stages{} block to run build, checks, test etc per application.
      Per application tests should only run if the previous build stage for the app succeeded.

      It seems to be not supported currently. I couldn't find a way to realize it. I tried different variations of the syntax but couldn't make it work.

      Example of what I would like to do:

      pipeline {
        agent none
      
        stages {
          stage('Discover Changed Applications') {
            agent any
            steps {
              script {
                apps = ["app-one", "app-two"]
                }
              }
          }
      
          stage('parallel') {
            steps {
              script {
                pstages = [:]
      
                for (int i = 0; i < apps.size(); i++) {
                  app = apps[i]
      
                  pstages[app] = {
                    stages {
                      stage('Build application ') {
                        agent { label 'build' }
      
                        steps{
                          sh 'echo build'
                        }
                      }
                      stage('Test application ') {
                        agent { label 'test' }
      
                        steps{
                          sh 'echo test'
                        }
                      }
                    }
                  }
                }
      
                parallel(pstages)
              }
            }
          }
        }
      }
      

          [JENKINS-53032] Support dynamic definition of parallel running stages

          Fabian Holler created issue -
          Fabian Holler made changes -
          Description Original: We often have the usecase that we need to define stages that we run in parallel during runtime.
          E.g.: we run a script to determine which application changed in a branch and want to run in parallel for each application stages like Build, Test, Check etc.

          It seems to be not supported currently. I couldn't find a way to realize it. I tried different variations of the syntax but couldn't make it work.


          Example of what I would like to do:


          {code:java}
          pipeline {
          agent none

          stages {
          stage('Discover Changed Applications') {
          steps {
          script {
          apps = ["app-one", "app-two"]
          }
          }
          }

          stage('parallel') {
          steps {
          script {
          pstages = [:]

          for (int i = 0; i < apps.size(); i++) {
          app = apps[i]

          pstages[app] = {
          stages {
          stage('Build application ') {
          agent { label 'build' }

          steps{
          sh 'echo build'
          }
          }
          stage('Test application ') {
          agent { label 'test' }

          steps{
          sh 'echo test'
          }
          }
          }
          }
          }

          parallel(pstages)
          }
          }
          }
          }
          }
          {code}
          New: We often have the usecase that we need to define stages that we run in parallel during runtime.
          E.g.: we run a script to determine which application changed in a branch and want to run in parallel for each application stages like Build, Test, Check etc.

          It seems to be not supported currently. I couldn't find a way to realize it. I tried different variations of the syntax but couldn't make it work.


          Example of what I would like to do:


          {code:java}
          pipeline {
          agent none

          stages {
          stage('Discover Changed Applications') {
                                  agent any
          steps {
          script {
          apps = ["app-one", "app-two"]
          }
          }
          }

          stage('parallel') {
          steps {
          script {
          pstages = [:]

          for (int i = 0; i < apps.size(); i++) {
          app = apps[i]

          pstages[app] = {
          stages {
          stage('Build application ') {
          agent { label 'build' }

          steps{
          sh 'echo build'
          }
          }
          stage('Test application ') {
          agent { label 'test' }

          steps{
          sh 'echo test'
          }
          }
          }
          }
          }

          parallel(pstages)
          }
          }
          }
          }
          }
          {code}
          Fabian Holler made changes -
          Summary Original: Support dynamic defintion of parallel running stages New: Support dynamic definition of parallel running stages
          Andrew Bayer made changes -
          Component/s New: pipeline-model-definition-plugin [ 21706 ]
          Component/s Original: pipeline [ 21692 ]

          Steve Todorov added a comment - - edited

          I've answered a similar question at devops.stackexchange.com. I believe this is a bit overcomplicated and the Jenkins team should probably think of a way to simplify how dynamic stages (be they parallel or not) can be used in a declarative pipeline.

          Steve Todorov added a comment - - edited I've answered a similar question at devops.stackexchange.com . I believe this is a bit overcomplicated and the Jenkins team should probably think of a way to simplify how dynamic stages (be they parallel or not) can be used in a declarative pipeline.
          Ugur OZCAN made changes -
          Attachment New: image-2020-07-22-16-22-03-982.png [ 51964 ]
          Ugur OZCAN made changes -
          Attachment New: image-2020-07-22-16-23-38-411.png [ 51965 ]

          Ugur OZCAN added a comment -

          Steve's solution looks like a workaround we need a similar solution but we dont want to create a pipeline with groovy script.

          Why this is not a use case reading from a Listmap and using those values in parallel stages? I bet this is a very useful feature and valid usecase which tousands can benefit from.

          For example we have a device list and we can keep it as simple  [[name: AA, id:123][name:BB , id:234], ....] So as a listmap and we want to use device list (ids) each in different parallel stages like:

           

          Ugur OZCAN added a comment - Steve's solution looks like a workaround we need a similar solution but we dont want to create a pipeline with groovy script. Why this is not a use case reading from a Listmap and using those values in parallel stages? I bet this is a very useful feature and valid usecase which tousands can benefit from. For example we have a device list and we can keep it as simple  [ [name: AA, id:123] ,  [name:BB , id:234] , ....] So as a listmap and we want to use device list (ids) each in different parallel stages like:  
          Ugur OZCAN made changes -
          Priority Original: Minor [ 4 ] New: Major [ 3 ]
          Ugur OZCAN made changes -
          Assignee New: A A [ a ]

            Unassigned Unassigned
            fho Fabian Holler
            Votes:
            8 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: