Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54010

Support the visualization of two levels of parallelity in stages

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Support scripted parallel blocks in parallel stages in a declarative pipeline.

      Example pipeline:

      pipeline {
          agent none
          stages {
              stage('Parallel Stage') {
                  parallel {
                      stage('Stage 1') {
                          steps {
                              echo "Stage 1"
                          }
                      }
                      stage('Stage 2') {
                          steps {
                              script {
                                  parallel (
                                      "Stage 2.1.": {
                                          echo "Stage 2.1."
                                      },
                                      "Stage 2.2.": {
                                          echo "Stage 2.2."
                                      }
                                  )
                              }
                          }
                      }
                  }
              }
          }
      }
      

      Currently it is not properly visualized in blue ocean as shown in parallel-stages.png

       

      The idea how it should be visualized is shown in parallel-stages-new.png

        Attachments

        1. image-2019-12-18-10-50-54-224.png
          image-2019-12-18-10-50-54-224.png
          41 kB
        2. parallel-stages.PNG
          parallel-stages.PNG
          9 kB
        3. parallel-stages-new.png
          parallel-stages-new.png
          10 kB
        4. Screen Shot 2019-05-14 at 9.49.06 AM.png
          Screen Shot 2019-05-14 at 9.49.06 AM.png
          52 kB
        5. Screen Shot 2019-05-20 at 8.55.46 PM.png
          Screen Shot 2019-05-20 at 8.55.46 PM.png
          49 kB
        6. Selection_090.png
          Selection_090.png
          80 kB
        7. Selection_093.png
          Selection_093.png
          23 kB
        8. TwoPipelinesInParallel.png
          TwoPipelinesInParallel.png
          64 kB
        9. TwoPipelinesInSerial.png
          TwoPipelinesInSerial.png
          94 kB

          Issue Links

            Activity

            Hide
            stuartrowe Stuart Rowe added a comment -

            There hasn't been any active development on Blue Ocean for a while. See https://groups.google.com/g/jenkinsci-users/c/xngZrSsXIjc/m/-_aIrWyICgAJ.

            The pipeline-graph-view-plugin is bringing the pipeline visualization from Blue Ocean into the classic UI. It might be worth creating a new issue for that plugin?

            Show
            stuartrowe Stuart Rowe added a comment - There hasn't been any active development on Blue Ocean for a while. See https://groups.google.com/g/jenkinsci-users/c/xngZrSsXIjc/m/-_aIrWyICgAJ. The pipeline-graph-view-plugin  is bringing the pipeline visualization from Blue Ocean into the classic UI. It might be worth creating a new issue for that plugin?
            Hide
            llibicpep Dee Kryvenko added a comment - - edited

            Well said Henry Borchers. Additionally I would point out that classic UI does not have visualization support for parallel stages at all, in any shape or form, not even single leveled. It is all unreadable linear mess in the classic UI.

            Show
            llibicpep Dee Kryvenko added a comment - - edited Well said Henry Borchers . Additionally I would point out that classic UI does not have visualization support for parallel stages at all, in any shape or form, not even single leveled. It is all unreadable linear mess in the classic UI.
            Hide
            henryborchers Henry Borchers added a comment -

            Graham McGregor, I get the feeling the project does not have anybody working on Blue Ocean development anymore. My guess is that whoever was in charge of it has left the company or moved on to other projects because I haven't seen any real improvements done to it in years. It's a shame because Blue Ocean was a GAME CHANGER!!! I personally love how Blue Ocean displays my pipeline and helps me quickly figure out an issue when something goes wrong.  However, it seems to have issues that don't aren't being addressed and missing features (like this one) that could make it much. (In addition, Blue Ocean takes a very long time to load!)

            I would LOVE first-class support for multiple levels of parallelism (such as nested parallel commands or nested matrixes in the declarative pipelines). Having that would let me load-balance and greatly reduce the duration of my pipelines. However, it just doesn't seem like a priority for anybody who has the skills to do anything about it.

            I don't blame anybody working on Jenkins. In fact, I love everything they do regardless if they address stuff that make my life better or someone else's. No, I just wish my own skills included what was needed to help out in the same way the Jenkins has helped me. 

            Show
            henryborchers Henry Borchers added a comment - Graham McGregor , I get the feeling the project does not have anybody working on Blue Ocean development anymore. My guess is that whoever was in charge of it has left the company or moved on to other projects because I haven't seen any real improvements done to it in years. It's a shame because Blue Ocean was a GAME CHANGER!!! I personally love how Blue Ocean displays my pipeline and helps me quickly figure out an issue when something goes wrong.  However, it seems to have issues that don't aren't being addressed and missing features (like this one) that could make it much. (In addition, Blue Ocean takes a very long time to load!) I would LOVE first-class support for multiple levels of parallelism (such as nested parallel commands or nested matrixes in the declarative pipelines). Having that would let me load-balance and greatly reduce the duration of my pipelines. However, it just doesn't seem like a priority for anybody who has the skills to do anything about it. I don't blame anybody working on Jenkins. In fact, I love everything they do regardless if they address stuff that make my life better or someone else's. No, I just wish my own skills included what was needed to help out in the same way the Jenkins has helped me. 
            Hide
            gmtm Graham McGregor added a comment -

            Does anyone know if this is purely a visualization issue? or it is an issue with the underlying data structure? If someone was wanting to get started looking into this issue, where should they start?

            Show
            gmtm Graham McGregor added a comment - Does anyone know if this is purely a visualization issue? or it is an issue with the underlying data structure? If someone was wanting to get started looking into this issue, where should they start?
            Hide
            llibicpep Dee Kryvenko added a comment -

            Here's a use case that might help with design.

            Consider a typical pipeline that build an artifact[s] and deploys it to sequential environments (let's say Dev and QA for the sake of example).

            Consider that you might want to deploy to multiple locations, for instance - to aws, azure and gcp. These are parallel independent worlds, so environments progression would be happening in parallel there.

            Consider that within each location, you also want to deploy into multiple regions. Regions are also isolated independent worlds, so they would have to be deployed in parallel within it's location (aka aws, gcp, azure etc).

            That creates at least two levels of nesting, but generally you can extrapolate this idea, for instance to deploy to each AZ separately, or maybe - deploy to multiple groups of locations or something like that, I do not see why there should be a limit of just two.

            Anyway, here is a scripted pipeline representation of that use case (just two levels for the sake of simplicity):

             stage('Realms') {
                parallel 'aws': {
                    stage('Regions (aws)') {
                        parallel 'us-east-1 (aws)': {
                            stage('Dev (aws us-east-1)') {
                                echo 'aws us-east-1 dev'
                            }
                            stage('QA (aws us-east-1)') {
                                input message: 'Approve aws us-east-1 qa?', ok: 'Approve aws us-east-1 qa!'
                                echo 'aws us-east-1 qa'
                            }
                        }, 'us-west-2 (aws)': {
                            stage('Dev (aws us-west-2)') {
                                echo 'aws us-west-2 dev'
                            }
                            stage('QA (aws us-west-2)') {
                                input message: 'Approve aws us-west-2 qa?', ok: 'Approve aws us-west-2 qa!'
                                echo 'aws us-west-2 qa'
                            }
                        }, failFast: false
                    }
                }, 'gcp': {
                    stage('Regions (gcp)') {
                        parallel 'us-east-1 (gcp)': {
                            stage('Dev (gcp us-east4)') {
                                echo 'gcp us-east4 dev'
                            }
                            stage('QA (gcp us-east4)') {
                                input message: 'Approve gcp us-east4 qa?', ok: 'Approve gcp us-east4 qa!'
                                echo 'gcp us-east4 qa'
                            }
                        }, 'us-west-2 (gcp)': {
                            stage('Dev (gcp us-west4)') {
                                echo 'gcp us-west4 dev'
                            }
                            stage('QA (gcp us-west4)') {
                                input message: 'Approve gcp us-west4 qa?', ok: 'Approve gcp us-west4 qa!'
                                echo 'gcp us-west4 qa'
                            }
                        }, failFast: false
                    }
                }, failFast: false
            }
            

            Also consider a deployment stage itself (currently represented with just one simple echo step) - it will most likely have a lot of low level steps such as sh and so on. From the pipeline user perspective - it doesn't makes any sense to see all that implementation details. The user will want to see aggregation of steps at some level of abstractions, for instance if deployment is implemented with Terraform - the user will want to see things like plan, display diff and apply. The user might want to dig into any of these abstractions to the level of actual steps, of course, so that should be an expandable option.
            I think implementing more than two levels of nesting using a use case from above will also solve that second use case.

            Show
            llibicpep Dee Kryvenko added a comment - Here's a use case that might help with design. Consider a typical pipeline that build an artifact [s] and deploys it to sequential environments (let's say Dev and QA for the sake of example). Consider that you might want to deploy to multiple locations, for instance - to aws, azure and gcp. These are parallel independent worlds, so environments progression would be happening in parallel there. Consider that within each location, you also want to deploy into multiple regions. Regions are also isolated independent worlds, so they would have to be deployed in parallel within it's location (aka aws, gcp, azure etc). That creates at least two levels of nesting, but generally you can extrapolate this idea, for instance to deploy to each AZ separately, or maybe - deploy to multiple groups of locations or something like that, I do not see why there should be a limit of just two. Anyway, here is a scripted pipeline representation of that use case (just two levels for the sake of simplicity):  stage( 'Realms' ) { parallel 'aws' : { stage( 'Regions (aws)' ) { parallel 'us-east-1 (aws)' : { stage( 'Dev (aws us-east-1)' ) { echo 'aws us-east-1 dev' } stage( 'QA (aws us-east-1)' ) { input message: 'Approve aws us-east-1 qa?' , ok: 'Approve aws us-east-1 qa!' echo 'aws us-east-1 qa' } }, 'us-west-2 (aws)' : { stage( 'Dev (aws us-west-2)' ) { echo 'aws us-west-2 dev' } stage( 'QA (aws us-west-2)' ) { input message: 'Approve aws us-west-2 qa?' , ok: 'Approve aws us-west-2 qa!' echo 'aws us-west-2 qa' } }, failFast: false } }, 'gcp' : { stage( 'Regions (gcp)' ) { parallel 'us-east-1 (gcp)' : { stage( 'Dev (gcp us-east4)' ) { echo 'gcp us-east4 dev' } stage( 'QA (gcp us-east4)' ) { input message: 'Approve gcp us-east4 qa?' , ok: 'Approve gcp us-east4 qa!' echo 'gcp us-east4 qa' } }, 'us-west-2 (gcp)' : { stage( 'Dev (gcp us-west4)' ) { echo 'gcp us-west4 dev' } stage( 'QA (gcp us-west4)' ) { input message: 'Approve gcp us-west4 qa?' , ok: 'Approve gcp us-west4 qa!' echo 'gcp us-west4 qa' } }, failFast: false } }, failFast: false } Also consider a deployment stage itself (currently represented with just one simple echo step) - it will most likely have a lot of low level steps such as sh and so on. From the pipeline user perspective - it doesn't makes any sense to see all that implementation details. The user will want to see aggregation of steps at some level of abstractions, for instance if deployment is implemented with Terraform - the user will want to see things like plan, display diff and apply. The user might want to dig into any of these abstractions to the level of actual steps, of course, so that should be an expandable option. I think implementing more than two levels of nesting using a use case from above will also solve that second use case.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              kurzy Daniel Kurzynski
              Votes:
              125 Vote for this issue
              Watchers:
              130 Start watching this issue

                Dates

                Created:
                Updated: