Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33185

Visualize parallel steps within a Pipeline Stage

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Support for visualizing parallel steps is unplanned

      A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

      The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

      There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.

      From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

      For the time being solving this problem has been delayed by CloudBees. However, we encourage any community members who are interested in solving this problem to pick it up.

      Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called Blue Ocean.

      Blue Ocean offers an entirely new way of visualizing Pipelines including parallel steps , and makes it much easier to navigate the Pipelines log by Stage and Step.

      We aim that over time that many of the use cases you have come to enjoy in Stage View will be available in Blue Ocean. We would like to encourage people interested in this feature to try out Blue Ocean and provide us feedback.

      You can find out more about Blue Ocean at https://jenkins.io/projects/blueocean/

      Thanks,
      James Dumay
      jdumay@cloudbees.com

      Original Request
      1) Parallel steps are the common workflow case
      2) Build logs for parallel steps are not informative
      3) A user would expect Stage View to somehow address the issue

      Current state:

      • stage declarations within parallel sections are not being considered as parallel
      • steps within one parallelized step are being mixed => no info in the view (see the screenshot)

      Test script (move stage declarations where you want):

      for (int i=0; i< 2; ++i) {
        stage "Stage #"+i
        print 'Hello, world $i!'
      }
      
      stage "Stage Parallel"
      def branches = [:]
      for (int i = 0; i < numHelloMessages.toInteger(); i++) {
        branches["split${i}"] = {
          stage "Stage parallel- #"+i
          node('remote') {
           echo  'Starting sleep'
           sleep 10
           echo  'Finished sleep'
          }
        }
      }
      parallel branches
      

      comment from JGlick:
      So concretely: the stage view would, for a given build (row), continue to display a linear sequence of stages (columns) with their associated times and so on; but each cell (stage of a build) would be some rendering of an st-planar graph, with structure determined according to the presence of steps of interest like parallel or a proposed label. Sort of like the existing Workflow Steps, but rendered as a graph rather than a tree table, restricted to displaying certain steps, and divided by stage.
      A simpler request would be to only display branches of one top-level parallel step, if any. (I.e., an st-planar graph which is either trivial—one vertex—or where all vertices other than the source and sink are directly connected to both the source and the sink.) This more limited UI would match that of the Build Pipeline plugin.
      Note that there is also a request JENKINS-29892 which would allow several stages to share concurrency semantics, but I think that need not have any effect on visualization; it would just be an internal change in the stage step.

        Attachments

        1. blue-ocean-parallel-generated-view.png
          blue-ocean-parallel-generated-view.png
          76 kB
        2. blue-ocean-parallel-WORKS.png
          blue-ocean-parallel-WORKS.png
          8 kB
        3. parallel.png
          parallel.png
          58 kB
        4. pipeline.png
          pipeline.png
          15 kB
        5. Screen Shot 2016-11-21 at 15.41.33.png
          Screen Shot 2016-11-21 at 15.41.33.png
          14 kB
        6. stage-duration.png
          stage-duration.png
          52 kB
        7. stages.png
          stages.png
          359 kB
        8. successful-pipeline.png
          successful-pipeline.png
          140 kB

          Issue Links

            Activity

            Hide
            jeinstei Joshua Einstein-Curtis added a comment -

            I was wondering whatever happened to this? We have an architecture like the pseudo-code below, where the parallel is a stage, but the stages in the parallel aren't broken out. We need to do this to get different nodes within a stage. If there is a better way to do this so the visualization works, we'd love to know, but it is currently an ugly visual representation with only the first stage and the parallel stage, basically.

             

             

            def jobMap (
                  some['thing'] = {node (label) {
                     stage1
                     stage2
                  } }
            )
             
            node {
               def jobs = jobMap()
               stage1
               stage2 {
                  parallel jobs
               }
            }
            

             

            Show
            jeinstei Joshua Einstein-Curtis added a comment - I was wondering whatever happened to this? We have an architecture like the pseudo-code below, where the parallel is a stage, but the stages in the parallel aren't broken out. We need to do this to get different nodes within a stage. If there is a better way to do this so the visualization works, we'd love to know, but it is currently an ugly visual representation with only the first stage and the parallel stage, basically.     def jobMap (       some[ 'thing' ] = {node (label) { stage1 stage2       } } )   node { def jobs = jobMap()    stage1    stage2 {       parallel jobs    } }  
            Hide
            jk Jan Klass added a comment -

            Blue Ocean is unusable to me for its page load time alone.

            I can see how the tabular view can be useful to people.

            I have on many occasions seen the table data disappear completely, because one stage was renamed or moved. If the tabular overview is regarded this highly I would expect that to be stable and merging as well; fill previously or afterwards non-existing columns with empty cells. But it does not. Instead it throws away all of it except the last build information.

            As it is not stable in this case, the table stability is regarded too highly and inconsistently in the reasoning in this ticket (ticket conclusion). At least for me definitely so.

            Especially so as I can still see steps with multiple substeps in the tabular format be comparable and parseable very well. Not having the parallel information is a much bigger hit on the overview.

            If a cell contains multiple sub-steps I don't see how it is much worse for tabular overview? It will still be tabular, in columns. And it can still prepare and fill cells as if it were just one level.

            Show
            jk Jan Klass added a comment - Blue Ocean is unusable to me for its page load time alone. I can see how the tabular view can be useful to people. I have on many occasions seen the table data disappear completely, because one stage was renamed or moved. If the tabular overview is regarded this highly I would expect that to be stable and merging as well; fill previously or afterwards non-existing columns with empty cells. But it does not. Instead it throws away all of it except the last build information. As it is not stable in this case, the table stability is regarded too highly and inconsistently in the reasoning in this ticket (ticket conclusion). At least for me definitely so. Especially so as I can still see steps with multiple substeps in the tabular format be comparable and parseable very well. Not having the parallel information is a much bigger hit on the overview. If a cell contains multiple sub-steps I don't see how it is much worse for tabular overview? It will still be tabular, in columns. And it can still prepare and fill cells as if it were just one level.
            Hide
            jk Jan Klass added a comment - - edited

            With parallel the order in which the steps report their existence is not deterministic. Hence, the linear steps order is not stable. This results in the entire table history being dropped. So even by the argument of keeping the overview in a tabular format, this is not consistently possible, but only randomly!? Seems like a very weak argument.

            I just saw the table disappear on me (presumably) because of this.

            /edit: Okay, it looks like it appeared again? Very confusing. Is there a reordering-according-to-config-declaration at the end??

            Show
            jk Jan Klass added a comment - - edited With parallel the order in which the steps report their existence is not deterministic. Hence, the linear steps order is not stable. This results in the entire table history being dropped. So even by the argument of keeping the overview in a tabular format, this is not consistently possible, but only randomly!? Seems like a very weak argument. I just saw the table disappear on me (presumably) because of this. /edit: Okay, it looks like it appeared again? Very confusing. Is there a reordering-according-to-config-declaration at the end??
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I would like to add this item into the Jenkins roadmap: https://jenkins.io/project/roadmap/

            Would be everyone fine if I reword it to "Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI"? I would also remove the component, because it is likely to become a separate plugin if it gets implemented.

             

             

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - I would like to add this item into the Jenkins roadmap:  https://jenkins.io/project/roadmap/ Would be everyone fine if I reword it to "Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI"? I would also remove the component, because it is likely to become a separate plugin if it gets implemented.      
            Hide
            jglick Jesse Glick added a comment -

            Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI

            Makes no sense. B.O. already supports this.

            I would also remove the component, because it is likely to become a separate plugin if it gets implemented.

            Also makes no sense. The limitation is specifically in the stage view plugin.

            Show
            jglick Jesse Glick added a comment - Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI Makes no sense. B.O. already supports this. I would also remove the component, because it is likely to become a separate plugin if it gets implemented. Also makes no sense. The limitation is specifically in the stage view plugin.

              People

              Assignee:
              cloudbees CloudBees Inc.
              Reporter:
              hrmpw Patrick Wolf
              Votes:
              207 Vote for this issue
              Watchers:
              214 Start watching this issue

                Dates

                Created:
                Updated: