Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54010

Support the visualization of two levels of parallelity in stages

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • blueocean-plugin
    • None

      Support scripted parallel blocks in parallel stages in a declarative pipeline.

      Example pipeline:

      pipeline {
          agent none
          stages {
              stage('Parallel Stage') {
                  parallel {
                      stage('Stage 1') {
                          steps {
                              echo "Stage 1"
                          }
                      }
                      stage('Stage 2') {
                          steps {
                              script {
                                  parallel (
                                      "Stage 2.1.": {
                                          echo "Stage 2.1."
                                      },
                                      "Stage 2.2.": {
                                          echo "Stage 2.2."
                                      }
                                  )
                              }
                          }
                      }
                  }
              }
          }
      }
      

      Currently it is not properly visualized in blue ocean as shown in parallel-stages.png

       

      The idea how it should be visualized is shown in parallel-stages-new.png

        1. image-2019-12-18-10-50-54-224.png
          image-2019-12-18-10-50-54-224.png
          41 kB
        2. parallel-stages.PNG
          parallel-stages.PNG
          9 kB
        3. parallel-stages-new.png
          parallel-stages-new.png
          10 kB
        4. Screen Shot 2019-05-14 at 9.49.06 AM.png
          Screen Shot 2019-05-14 at 9.49.06 AM.png
          52 kB
        5. Screen Shot 2019-05-20 at 8.55.46 PM.png
          Screen Shot 2019-05-20 at 8.55.46 PM.png
          49 kB
        6. Selection_090.png
          Selection_090.png
          80 kB
        7. Selection_093.png
          Selection_093.png
          23 kB
        8. TwoPipelinesInParallel.png
          TwoPipelinesInParallel.png
          64 kB
        9. TwoPipelinesInSerial.png
          TwoPipelinesInSerial.png
          94 kB

          [JENKINS-54010] Support the visualization of two levels of parallelity in stages

          Having multiple levels of parallel stages would optimize the use of my executors and significantly reduce my pipeline times. 

          This would be very helpful to me for stages that create multiple platform specific artifacts. When I do this now, I have to have one matrix stage that builds Windows and Linux packages (using creative use of Dockerfile agents, I'm able to get these to run in parallel), and one stage dedicated to building Mac packages. Because matrix stages seem to use up a parallel stage under the hood, I can only run these stages one after another, even though the physical servers could easily run these steps independently .  

          Henry Borchers added a comment - Having multiple levels of parallel stages would optimize the use of my executors and significantly reduce my pipeline times.  This would be very helpful to me for stages that create multiple platform specific artifacts. When I do this now, I have to have one matrix stage that builds Windows and Linux packages (using creative use of Dockerfile agents, I'm able to get these to run in parallel), and one stage dedicated to building Mac packages. Because matrix stages seem to use up a parallel stage under the hood, I can only run these stages one after another, even though the physical servers could easily run these steps independently .  

          alon miz added a comment -

          that's a really important feature for our team as well. we have an optimized ci with multiple parallelized steps, which look really awkward without that capability

          henryborchers in your case there's no limitation in using parallelism inside another. the downside is that it's really hard to view the process in the UI and in case of a failure step, the UI just don't understand how to show it.

          alon miz added a comment - that's a really important feature for our team as well. we have an optimized ci with multiple parallelized steps, which look really awkward without that capability henryborchers in your case there's no limitation in using parallelism inside another. the downside is that it's really hard to view the process in the UI and in case of a failure step, the UI just don't understand how to show it.

          Sverre Moe added a comment - - edited

          Will this fix the problem of visually showing two parallel Pipelines we have now run into?

          We have a Jenkinsfile which runs two Pipelines in serial. For efficiency these can be run in parallel, but that will not look good din Blue Ocean.
          This is how the two Pipelines are visualized in Blue Ocean (The black circle is the first pipeline, the blue is the second).

          This is how the two Pipelines are visualized when running them in parallel:

          The Scripted Pipeline in Seriel:

          def maven = new com.company.MavenBuild()
          maven.executeBuild()
          def gradle = new com.company.GradleBuild()
          gradle.executeBuild()
          

          The Scripted Pipeline in Parallel:

          parallel {
              'Maven': {
                  def maven = new com.company.MavenBuild()
                  maven.executeBuild()
              },
              'Gradle': {
                  def gradle = new com.company.GradleBuild()
                  gradle.executeBuild()
              }
          }
          

          Sverre Moe added a comment - - edited Will this fix the problem of visually showing two parallel Pipelines we have now run into? We have a Jenkinsfile which runs two Pipelines in serial. For efficiency these can be run in parallel, but that will not look good din Blue Ocean. This is how the two Pipelines are visualized in Blue Ocean (The black circle is the first pipeline, the blue is the second). This is how the two Pipelines are visualized when running them in parallel: The Scripted Pipeline in Seriel: def maven = new com.company.MavenBuild() maven.executeBuild() def gradle = new com.company.GradleBuild() gradle.executeBuild() The Scripted Pipeline in Parallel: parallel { 'Maven' : { def maven = new com.company.MavenBuild() maven.executeBuild() }, 'Gradle' : { def gradle = new com.company.GradleBuild() gradle.executeBuild() } }

          Dee Kryvenko added a comment -

          Here's a use case that might help with design.

          Consider a typical pipeline that build an artifact[s] and deploys it to sequential environments (let's say Dev and QA for the sake of example).

          Consider that you might want to deploy to multiple locations, for instance - to aws, azure and gcp. These are parallel independent worlds, so environments progression would be happening in parallel there.

          Consider that within each location, you also want to deploy into multiple regions. Regions are also isolated independent worlds, so they would have to be deployed in parallel within it's location (aka aws, gcp, azure etc).

          That creates at least two levels of nesting, but generally you can extrapolate this idea, for instance to deploy to each AZ separately, or maybe - deploy to multiple groups of locations or something like that, I do not see why there should be a limit of just two.

          Anyway, here is a scripted pipeline representation of that use case (just two levels for the sake of simplicity):

           stage('Realms') {
              parallel 'aws': {
                  stage('Regions (aws)') {
                      parallel 'us-east-1 (aws)': {
                          stage('Dev (aws us-east-1)') {
                              echo 'aws us-east-1 dev'
                          }
                          stage('QA (aws us-east-1)') {
                              input message: 'Approve aws us-east-1 qa?', ok: 'Approve aws us-east-1 qa!'
                              echo 'aws us-east-1 qa'
                          }
                      }, 'us-west-2 (aws)': {
                          stage('Dev (aws us-west-2)') {
                              echo 'aws us-west-2 dev'
                          }
                          stage('QA (aws us-west-2)') {
                              input message: 'Approve aws us-west-2 qa?', ok: 'Approve aws us-west-2 qa!'
                              echo 'aws us-west-2 qa'
                          }
                      }, failFast: false
                  }
              }, 'gcp': {
                  stage('Regions (gcp)') {
                      parallel 'us-east-1 (gcp)': {
                          stage('Dev (gcp us-east4)') {
                              echo 'gcp us-east4 dev'
                          }
                          stage('QA (gcp us-east4)') {
                              input message: 'Approve gcp us-east4 qa?', ok: 'Approve gcp us-east4 qa!'
                              echo 'gcp us-east4 qa'
                          }
                      }, 'us-west-2 (gcp)': {
                          stage('Dev (gcp us-west4)') {
                              echo 'gcp us-west4 dev'
                          }
                          stage('QA (gcp us-west4)') {
                              input message: 'Approve gcp us-west4 qa?', ok: 'Approve gcp us-west4 qa!'
                              echo 'gcp us-west4 qa'
                          }
                      }, failFast: false
                  }
              }, failFast: false
          }
          

          Also consider a deployment stage itself (currently represented with just one simple echo step) - it will most likely have a lot of low level steps such as sh and so on. From the pipeline user perspective - it doesn't makes any sense to see all that implementation details. The user will want to see aggregation of steps at some level of abstractions, for instance if deployment is implemented with Terraform - the user will want to see things like plan, display diff and apply. The user might want to dig into any of these abstractions to the level of actual steps, of course, so that should be an expandable option.
          I think implementing more than two levels of nesting using a use case from above will also solve that second use case.

          Dee Kryvenko added a comment - Here's a use case that might help with design. Consider a typical pipeline that build an artifact [s] and deploys it to sequential environments (let's say Dev and QA for the sake of example). Consider that you might want to deploy to multiple locations, for instance - to aws, azure and gcp. These are parallel independent worlds, so environments progression would be happening in parallel there. Consider that within each location, you also want to deploy into multiple regions. Regions are also isolated independent worlds, so they would have to be deployed in parallel within it's location (aka aws, gcp, azure etc). That creates at least two levels of nesting, but generally you can extrapolate this idea, for instance to deploy to each AZ separately, or maybe - deploy to multiple groups of locations or something like that, I do not see why there should be a limit of just two. Anyway, here is a scripted pipeline representation of that use case (just two levels for the sake of simplicity):  stage( 'Realms' ) { parallel 'aws' : { stage( 'Regions (aws)' ) { parallel 'us-east-1 (aws)' : { stage( 'Dev (aws us-east-1)' ) { echo 'aws us-east-1 dev' } stage( 'QA (aws us-east-1)' ) { input message: 'Approve aws us-east-1 qa?' , ok: 'Approve aws us-east-1 qa!' echo 'aws us-east-1 qa' } }, 'us-west-2 (aws)' : { stage( 'Dev (aws us-west-2)' ) { echo 'aws us-west-2 dev' } stage( 'QA (aws us-west-2)' ) { input message: 'Approve aws us-west-2 qa?' , ok: 'Approve aws us-west-2 qa!' echo 'aws us-west-2 qa' } }, failFast: false } }, 'gcp' : { stage( 'Regions (gcp)' ) { parallel 'us-east-1 (gcp)' : { stage( 'Dev (gcp us-east4)' ) { echo 'gcp us-east4 dev' } stage( 'QA (gcp us-east4)' ) { input message: 'Approve gcp us-east4 qa?' , ok: 'Approve gcp us-east4 qa!' echo 'gcp us-east4 qa' } }, 'us-west-2 (gcp)' : { stage( 'Dev (gcp us-west4)' ) { echo 'gcp us-west4 dev' } stage( 'QA (gcp us-west4)' ) { input message: 'Approve gcp us-west4 qa?' , ok: 'Approve gcp us-west4 qa!' echo 'gcp us-west4 qa' } }, failFast: false } }, failFast: false } Also consider a deployment stage itself (currently represented with just one simple echo step) - it will most likely have a lot of low level steps such as sh and so on. From the pipeline user perspective - it doesn't makes any sense to see all that implementation details. The user will want to see aggregation of steps at some level of abstractions, for instance if deployment is implemented with Terraform - the user will want to see things like plan, display diff and apply. The user might want to dig into any of these abstractions to the level of actual steps, of course, so that should be an expandable option. I think implementing more than two levels of nesting using a use case from above will also solve that second use case.

          Does anyone know if this is purely a visualization issue? or it is an issue with the underlying data structure? If someone was wanting to get started looking into this issue, where should they start?

          Graham McGregor added a comment - Does anyone know if this is purely a visualization issue? or it is an issue with the underlying data structure? If someone was wanting to get started looking into this issue, where should they start?

          gmtm, I get the feeling the project does not have anybody working on Blue Ocean development anymore. My guess is that whoever was in charge of it has left the company or moved on to other projects because I haven't seen any real improvements done to it in years. It's a shame because Blue Ocean was a GAME CHANGER!!! I personally love how Blue Ocean displays my pipeline and helps me quickly figure out an issue when something goes wrong.  However, it seems to have issues that don't aren't being addressed and missing features (like this one) that could make it much. (In addition, Blue Ocean takes a very long time to load!)

          I would LOVE first-class support for multiple levels of parallelism (such as nested parallel commands or nested matrixes in the declarative pipelines). Having that would let me load-balance and greatly reduce the duration of my pipelines. However, it just doesn't seem like a priority for anybody who has the skills to do anything about it.

          I don't blame anybody working on Jenkins. In fact, I love everything they do regardless if they address stuff that make my life better or someone else's. No, I just wish my own skills included what was needed to help out in the same way the Jenkins has helped me. 

          Henry Borchers added a comment - gmtm , I get the feeling the project does not have anybody working on Blue Ocean development anymore. My guess is that whoever was in charge of it has left the company or moved on to other projects because I haven't seen any real improvements done to it in years. It's a shame because Blue Ocean was a GAME CHANGER!!! I personally love how Blue Ocean displays my pipeline and helps me quickly figure out an issue when something goes wrong.  However, it seems to have issues that don't aren't being addressed and missing features (like this one) that could make it much. (In addition, Blue Ocean takes a very long time to load!) I would LOVE first-class support for multiple levels of parallelism (such as nested parallel commands or nested matrixes in the declarative pipelines). Having that would let me load-balance and greatly reduce the duration of my pipelines. However, it just doesn't seem like a priority for anybody who has the skills to do anything about it. I don't blame anybody working on Jenkins. In fact, I love everything they do regardless if they address stuff that make my life better or someone else's. No, I just wish my own skills included what was needed to help out in the same way the Jenkins has helped me. 

          Dee Kryvenko added a comment - - edited

          Well said henryborchers. Additionally I would point out that classic UI does not have visualization support for parallel stages at all, in any shape or form, not even single leveled. It is all unreadable linear mess in the classic UI.

          Dee Kryvenko added a comment - - edited Well said henryborchers . Additionally I would point out that classic UI does not have visualization support for parallel stages at all, in any shape or form, not even single leveled. It is all unreadable linear mess in the classic UI.

          Stuart Rowe added a comment -

          There hasn't been any active development on Blue Ocean for a while. See https://groups.google.com/g/jenkinsci-users/c/xngZrSsXIjc/m/-_aIrWyICgAJ.

          The pipeline-graph-view-plugin is bringing the pipeline visualization from Blue Ocean into the classic UI. It might be worth creating a new issue for that plugin?

          Stuart Rowe added a comment - There hasn't been any active development on Blue Ocean for a while. See https://groups.google.com/g/jenkinsci-users/c/xngZrSsXIjc/m/-_aIrWyICgAJ. The pipeline-graph-view-plugin  is bringing the pipeline visualization from Blue Ocean into the classic UI. It might be worth creating a new issue for that plugin?

          Edgars Batna added a comment - - edited

          +100500 Jenkins shared libraries and Jenkinsfiles are awesome, but the UI around them is a terrible mess, especially when it comes to stage display and generally trying to understand what logs are where etc. I think it's easier to rewrite the UI for pipeline builds from scratch to somehow allow browsing through everything in a visually pleasing execution step graph, not a flat representation like a list or a table.

          Edgars Batna added a comment - - edited +100500 Jenkins shared libraries and Jenkinsfiles are awesome, but the UI around them is a terrible mess, especially when it comes to stage display and generally trying to understand what logs are where etc. I think it's easier to rewrite the UI for pipeline builds from scratch to somehow allow browsing through everything in a visually pleasing execution step graph, not a flat representation like a list or a table.

          Dante Kiaunis added a comment -

          Is this still being worked on?

          Dante Kiaunis added a comment - Is this still being worked on?

            Unassigned Unassigned
            kurzy Daniel Kurzynski
            Votes:
            132 Vote for this issue
            Watchers:
            141 Start watching this issue

              Created:
              Updated: