Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54010

Support the visualization of two levels of parallelity in stages

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Support scripted parallel blocks in parallel stages in a declarative pipeline.

      Example pipeline:

      pipeline {
          agent none
          stages {
              stage('Parallel Stage') {
                  parallel {
                      stage('Stage 1') {
                          steps {
                              echo "Stage 1"
                          }
                      }
                      stage('Stage 2') {
                          steps {
                              script {
                                  parallel (
                                      "Stage 2.1.": {
                                          echo "Stage 2.1."
                                      },
                                      "Stage 2.2.": {
                                          echo "Stage 2.2."
                                      }
                                  )
                              }
                          }
                      }
                  }
              }
          }
      }
      

      Currently it is not properly visualized in blue ocean as shown in parallel-stages.png

       

      The idea how it should be visualized is shown in parallel-stages-new.png

        Attachments

        1. TwoPipelinesInSerial.png
          TwoPipelinesInSerial.png
          94 kB
        2. TwoPipelinesInParallel.png
          TwoPipelinesInParallel.png
          64 kB
        3. Selection_093.png
          Selection_093.png
          23 kB
        4. Selection_090.png
          Selection_090.png
          80 kB
        5. Screen Shot 2019-05-20 at 8.55.46 PM.png
          Screen Shot 2019-05-20 at 8.55.46 PM.png
          49 kB
        6. Screen Shot 2019-05-14 at 9.49.06 AM.png
          Screen Shot 2019-05-14 at 9.49.06 AM.png
          52 kB
        7. parallel-stages-new.png
          parallel-stages-new.png
          10 kB
        8. parallel-stages.PNG
          parallel-stages.PNG
          9 kB
        9. image-2019-12-18-10-50-54-224.png
          image-2019-12-18-10-50-54-224.png
          41 kB

          Issue Links

            Activity

            Hide
            amidar Amit Dar added a comment -

            Hi,

             

            any update regarding this issue?

            Show
            amidar Amit Dar added a comment - Hi,   any update regarding this issue?
            Hide
            henryborchers Henry Borchers added a comment -

            Having multiple levels of parallel stages would optimize the use of my executors and significantly reduce my pipeline times. 

            This would be very helpful to me for stages that create multiple platform specific artifacts. When I do this now, I have to have one matrix stage that builds Windows and Linux packages (using creative use of Dockerfile agents, I'm able to get these to run in parallel), and one stage dedicated to building Mac packages. Because matrix stages seem to use up a parallel stage under the hood, I can only run these stages one after another, even though the physical servers could easily run these steps independently .  

            Show
            henryborchers Henry Borchers added a comment - Having multiple levels of parallel stages would optimize the use of my executors and significantly reduce my pipeline times.  This would be very helpful to me for stages that create multiple platform specific artifacts. When I do this now, I have to have one matrix stage that builds Windows and Linux packages (using creative use of Dockerfile agents, I'm able to get these to run in parallel), and one stage dedicated to building Mac packages. Because matrix stages seem to use up a parallel stage under the hood, I can only run these stages one after another, even though the physical servers could easily run these steps independently .  
            Hide
            alonmiz alon miz added a comment -

            that's a really important feature for our team as well. we have an optimized ci with multiple parallelized steps, which look really awkward without that capability

            Henry Borchers in your case there's no limitation in using parallelism inside another. the downside is that it's really hard to view the process in the UI and in case of a failure step, the UI just don't understand how to show it.

            Show
            alonmiz alon miz added a comment - that's a really important feature for our team as well. we have an optimized ci with multiple parallelized steps, which look really awkward without that capability Henry Borchers in your case there's no limitation in using parallelism inside another. the downside is that it's really hard to view the process in the UI and in case of a failure step, the UI just don't understand how to show it.
            Hide
            djviking Sverre Moe added a comment - - edited

            Will this fix the problem of visually showing two parallel Pipelines we have now run into?

            We have a Jenkinsfile which runs two Pipelines in serial. For efficiency these can be run in parallel, but that will not look good din Blue Ocean.
            This is how the two Pipelines are visualized in Blue Ocean (The black circle is the first pipeline, the blue is the second).

            This is how the two Pipelines are visualized when running them in parallel:

            The Scripted Pipeline in Seriel:

            def maven = new com.company.MavenBuild()
            maven.executeBuild()
            def gradle = new com.company.GradleBuild()
            gradle.executeBuild()
            

            The Scripted Pipeline in Parallel:

            parallel {
                'Maven': {
                    def maven = new com.company.MavenBuild()
                    maven.executeBuild()
                },
                'Gradle': {
                    def gradle = new com.company.GradleBuild()
                    gradle.executeBuild()
                }
            }
            
            Show
            djviking Sverre Moe added a comment - - edited Will this fix the problem of visually showing two parallel Pipelines we have now run into? We have a Jenkinsfile which runs two Pipelines in serial. For efficiency these can be run in parallel, but that will not look good din Blue Ocean. This is how the two Pipelines are visualized in Blue Ocean (The black circle is the first pipeline, the blue is the second). This is how the two Pipelines are visualized when running them in parallel: The Scripted Pipeline in Seriel: def maven = new com.company.MavenBuild() maven.executeBuild() def gradle = new com.company.GradleBuild() gradle.executeBuild() The Scripted Pipeline in Parallel: parallel { 'Maven' : { def maven = new com.company.MavenBuild() maven.executeBuild() }, 'Gradle' : { def gradle = new com.company.GradleBuild() gradle.executeBuild() } }
            Hide
            llibicpep Dmytro Kryvenko added a comment -

            Here's a use case that might help with design.

            Consider a typical pipeline that build an artifact[s] and deploys it to sequential environments (let's say Dev and QA for the sake of example).

            Consider that you might want to deploy to multiple locations, for instance - to aws, azure and gcp. These are parallel independent worlds, so environments progression would be happening in parallel there.

            Consider that within each location, you also want to deploy into multiple regions. Regions are also isolated independent worlds, so they would have to be deployed in parallel within it's location (aka aws, gcp, azure etc).

            That creates at least two levels of nesting, but generally you can extrapolate this idea, for instance to deploy to each AZ separately, or maybe - deploy to multiple groups of locations or something like that, I do not see why there should be a limit of just two.

            Anyway, here is a scripted pipeline representation of that use case (just two levels for the sake of simplicity):

             stage('Realms') {
                parallel 'aws': {
                    stage('Regions (aws)') {
                        parallel 'us-east-1 (aws)': {
                            stage('Dev (aws us-east-1)') {
                                echo 'aws us-east-1 dev'
                            }
                            stage('QA (aws us-east-1)') {
                                input message: 'Approve aws us-east-1 qa?', ok: 'Approve aws us-east-1 qa!'
                                echo 'aws us-east-1 qa'
                            }
                        }, 'us-west-2 (aws)': {
                            stage('Dev (aws us-west-2)') {
                                echo 'aws us-west-2 dev'
                            }
                            stage('QA (aws us-west-2)') {
                                input message: 'Approve aws us-west-2 qa?', ok: 'Approve aws us-west-2 qa!'
                                echo 'aws us-west-2 qa'
                            }
                        }, failFast: false
                    }
                }, 'gcp': {
                    stage('Regions (gcp)') {
                        parallel 'us-east-1 (gcp)': {
                            stage('Dev (gcp us-east4)') {
                                echo 'gcp us-east4 dev'
                            }
                            stage('QA (gcp us-east4)') {
                                input message: 'Approve gcp us-east4 qa?', ok: 'Approve gcp us-east4 qa!'
                                echo 'gcp us-east4 qa'
                            }
                        }, 'us-west-2 (gcp)': {
                            stage('Dev (gcp us-west4)') {
                                echo 'gcp us-west4 dev'
                            }
                            stage('QA (gcp us-west4)') {
                                input message: 'Approve gcp us-west4 qa?', ok: 'Approve gcp us-west4 qa!'
                                echo 'gcp us-west4 qa'
                            }
                        }, failFast: false
                    }
                }, failFast: false
            }
            

            Also consider a deployment stage itself (currently represented with just one simple echo step) - it will most likely have a lot of low level steps such as sh and so on. From the pipeline user perspective - it doesn't makes any sense to see all that implementation details. The user will want to see aggregation of steps at some level of abstractions, for instance if deployment is implemented with Terraform - the user will want to see things like plan, display diff and apply. The user might want to dig into any of these abstractions to the level of actual steps, of course, so that should be an expandable option.
            I think implementing more than two levels of nesting using a use case from above will also solve that second use case.

            Show
            llibicpep Dmytro Kryvenko added a comment - Here's a use case that might help with design. Consider a typical pipeline that build an artifact [s] and deploys it to sequential environments (let's say Dev and QA for the sake of example). Consider that you might want to deploy to multiple locations, for instance - to aws, azure and gcp. These are parallel independent worlds, so environments progression would be happening in parallel there. Consider that within each location, you also want to deploy into multiple regions. Regions are also isolated independent worlds, so they would have to be deployed in parallel within it's location (aka aws, gcp, azure etc). That creates at least two levels of nesting, but generally you can extrapolate this idea, for instance to deploy to each AZ separately, or maybe - deploy to multiple groups of locations or something like that, I do not see why there should be a limit of just two. Anyway, here is a scripted pipeline representation of that use case (just two levels for the sake of simplicity):  stage( 'Realms' ) { parallel 'aws' : { stage( 'Regions (aws)' ) { parallel 'us-east-1 (aws)' : { stage( 'Dev (aws us-east-1)' ) { echo 'aws us-east-1 dev' } stage( 'QA (aws us-east-1)' ) { input message: 'Approve aws us-east-1 qa?' , ok: 'Approve aws us-east-1 qa!' echo 'aws us-east-1 qa' } }, 'us-west-2 (aws)' : { stage( 'Dev (aws us-west-2)' ) { echo 'aws us-west-2 dev' } stage( 'QA (aws us-west-2)' ) { input message: 'Approve aws us-west-2 qa?' , ok: 'Approve aws us-west-2 qa!' echo 'aws us-west-2 qa' } }, failFast: false } }, 'gcp' : { stage( 'Regions (gcp)' ) { parallel 'us-east-1 (gcp)' : { stage( 'Dev (gcp us-east4)' ) { echo 'gcp us-east4 dev' } stage( 'QA (gcp us-east4)' ) { input message: 'Approve gcp us-east4 qa?' , ok: 'Approve gcp us-east4 qa!' echo 'gcp us-east4 qa' } }, 'us-west-2 (gcp)' : { stage( 'Dev (gcp us-west4)' ) { echo 'gcp us-west4 dev' } stage( 'QA (gcp us-west4)' ) { input message: 'Approve gcp us-west4 qa?' , ok: 'Approve gcp us-west4 qa!' echo 'gcp us-west4 qa' } }, failFast: false } }, failFast: false } Also consider a deployment stage itself (currently represented with just one simple echo step) - it will most likely have a lot of low level steps such as sh and so on. From the pipeline user perspective - it doesn't makes any sense to see all that implementation details. The user will want to see aggregation of steps at some level of abstractions, for instance if deployment is implemented with Terraform - the user will want to see things like plan, display diff and apply. The user might want to dig into any of these abstractions to the level of actual steps, of course, so that should be an expandable option. I think implementing more than two levels of nesting using a use case from above will also solve that second use case.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              kurzy Daniel Kurzynski
              Votes:
              114 Vote for this issue
              Watchers:
              119 Start watching this issue

                Dates

                Created:
                Updated: