Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33185

Visualize parallel steps within a Pipeline Stage

      Support for visualizing parallel steps is unplanned

      A common thread in the feedback we have received is to represent parallel steps in the Stage View as cells within the Stage column.

      The use cases supported by Pipeline Stage View, such as visualizing stages across multiple runs and detecting changes in execution time or stages that fail regularly, work well due to the use of a table to visualize this data. Tables are excellent at allowing the eye to scan over a lot of related data.

      There has some debate within CloudBees on how parallel could be added to Pipeline Stage View without breaking the advantages that a table brings to the visualization. We believe that adding additional cells within the table column would reduce the usability of the overall experience, as scanning many rows becomes much more difficult.

      From a technical point of view it would be non-trivial to retrofit the Stage View with the parallel information.

      For the time being solving this problem has been delayed by CloudBees. However, we encourage any community members who are interested in solving this problem to pick it up.

      Over the last year CloudBees and the Jenkins community have been heavily invested in bringing a new user interface to Jenkins that is designed for Pipeline called Blue Ocean.

      Blue Ocean offers an entirely new way of visualizing Pipelines including parallel steps , and makes it much easier to navigate the Pipelines log by Stage and Step.

      We aim that over time that many of the use cases you have come to enjoy in Stage View will be available in Blue Ocean. We would like to encourage people interested in this feature to try out Blue Ocean and provide us feedback.

      You can find out more about Blue Ocean at https://jenkins.io/projects/blueocean/

      Thanks,
      James Dumay
      jdumay@cloudbees.com

      Original Request
      1) Parallel steps are the common workflow case
      2) Build logs for parallel steps are not informative
      3) A user would expect Stage View to somehow address the issue

      Current state:

      • stage declarations within parallel sections are not being considered as parallel
      • steps within one parallelized step are being mixed => no info in the view (see the screenshot)

      Test script (move stage declarations where you want):

      for (int i=0; i< 2; ++i) {
        stage "Stage #"+i
        print 'Hello, world $i!'
      }
      
      stage "Stage Parallel"
      def branches = [:]
      for (int i = 0; i < numHelloMessages.toInteger(); i++) {
        branches["split${i}"] = {
          stage "Stage parallel- #"+i
          node('remote') {
           echo  'Starting sleep'
           sleep 10
           echo  'Finished sleep'
          }
        }
      }
      parallel branches
      

      comment from JGlick:
      So concretely: the stage view would, for a given build (row), continue to display a linear sequence of stages (columns) with their associated times and so on; but each cell (stage of a build) would be some rendering of an st-planar graph, with structure determined according to the presence of steps of interest like parallel or a proposed label. Sort of like the existing Workflow Steps, but rendered as a graph rather than a tree table, restricted to displaying certain steps, and divided by stage.
      A simpler request would be to only display branches of one top-level parallel step, if any. (I.e., an st-planar graph which is either trivial—one vertex—or where all vertices other than the source and sink are directly connected to both the source and the sink.) This more limited UI would match that of the Build Pipeline plugin.
      Note that there is also a request JENKINS-29892 which would allow several stages to share concurrency semantics, but I think that need not have any effect on visualization; it would just be an internal change in the stage step.

        1. blue-ocean-parallel-generated-view.png
          blue-ocean-parallel-generated-view.png
          76 kB
        2. blue-ocean-parallel-WORKS.png
          blue-ocean-parallel-WORKS.png
          8 kB
        3. parallel.png
          parallel.png
          58 kB
        4. pipeline.png
          pipeline.png
          15 kB
        5. Screen Shot 2016-11-21 at 15.41.33.png
          Screen Shot 2016-11-21 at 15.41.33.png
          14 kB
        6. stage-duration.png
          stage-duration.png
          52 kB
        7. stages.png
          stages.png
          359 kB
        8. successful-pipeline.png
          successful-pipeline.png
          140 kB

          [JENKINS-33185] Visualize parallel steps within a Pipeline Stage

          trejkaz added a comment -

          It turned out that "Blue Ocean" plugin itself was never installed, but all the other parts somehow ended up being on the system even though nobody had manually installed them. I'm assuming Jenkins itself pulled in the individual parts without pulling in the parent. After manually adding it, the page finally loads, after never working the entire time since the link appeared on the site.

           

          trejkaz added a comment - It turned out that "Blue Ocean" plugin itself was never installed, but all the other parts somehow ended up being on the system even though nobody had manually installed them. I'm assuming Jenkins itself pulled in the individual parts without pulling in the parent. After manually adding it, the page finally loads, after never working the entire time since the link appeared on the site.  

          Michael Neale added a comment -

          trejkaz ok that is good to hear - although I don't think other plugins yet depend on blue ocean parts, so not quite sure how it ended up doing that. There are other tickets on better generic error reporting that may make this error more obvious (although like with all error cases, edge cases mean it sometimes just wont work,but hopefully will at least give a hint).

          Michael Neale added a comment - trejkaz ok that is good to hear - although I don't think other plugins yet depend on blue ocean parts, so not quite sure how it ended up doing that. There are other tickets on better generic error reporting that may make this error more obvious (although like with all error cases, edge cases mean it sometimes just wont work,but hopefully will at least give a hint).

          trejkaz added a comment -

          I'm not sure this view is really much better. I have to move down multiple pages of "pipelines" which aren't even pipelines to get to the only actual pipeline we have configured so far ... then when we do get to the one we want and look at some of the builds, the visualisation on each page doesn't even seem to show all the stages, just the one it's currently working on and then "End".

          Oh well.

          trejkaz added a comment - I'm not sure this view is really much better. I have to move down multiple pages of "pipelines" which aren't even pipelines to get to the only actual pipeline we have configured so far ... then when we do get to the one we want and look at some of the builds, the visualisation on each page doesn't even seem to show all the stages, just the one it's currently working on and then "End". Oh well.

          I was wondering whatever happened to this? We have an architecture like the pseudo-code below, where the parallel is a stage, but the stages in the parallel aren't broken out. We need to do this to get different nodes within a stage. If there is a better way to do this so the visualization works, we'd love to know, but it is currently an ugly visual representation with only the first stage and the parallel stage, basically.

           

           

          def jobMap (
                some['thing'] = {node (label) {
                   stage1
                   stage2
                } }
          )
           
          node {
             def jobs = jobMap()
             stage1
             stage2 {
                parallel jobs
             }
          }
          

           

          Joshua Einstein-Curtis added a comment - I was wondering whatever happened to this? We have an architecture like the pseudo-code below, where the parallel is a stage, but the stages in the parallel aren't broken out. We need to do this to get different nodes within a stage. If there is a better way to do this so the visualization works, we'd love to know, but it is currently an ugly visual representation with only the first stage and the parallel stage, basically.     def jobMap (       some[ 'thing' ] = {node (label) { stage1 stage2       } } )   node { def jobs = jobMap()    stage1    stage2 {       parallel jobs    } }  

          Jan Klass added a comment -

          Blue Ocean is unusable to me for its page load time alone.

          I can see how the tabular view can be useful to people.

          I have on many occasions seen the table data disappear completely, because one stage was renamed or moved. If the tabular overview is regarded this highly I would expect that to be stable and merging as well; fill previously or afterwards non-existing columns with empty cells. But it does not. Instead it throws away all of it except the last build information.

          As it is not stable in this case, the table stability is regarded too highly and inconsistently in the reasoning in this ticket (ticket conclusion). At least for me definitely so.

          Especially so as I can still see steps with multiple substeps in the tabular format be comparable and parseable very well. Not having the parallel information is a much bigger hit on the overview.

          If a cell contains multiple sub-steps I don't see how it is much worse for tabular overview? It will still be tabular, in columns. And it can still prepare and fill cells as if it were just one level.

          Jan Klass added a comment - Blue Ocean is unusable to me for its page load time alone. I can see how the tabular view can be useful to people. I have on many occasions seen the table data disappear completely, because one stage was renamed or moved. If the tabular overview is regarded this highly I would expect that to be stable and merging as well; fill previously or afterwards non-existing columns with empty cells. But it does not. Instead it throws away all of it except the last build information. As it is not stable in this case, the table stability is regarded too highly and inconsistently in the reasoning in this ticket (ticket conclusion). At least for me definitely so. Especially so as I can still see steps with multiple substeps in the tabular format be comparable and parseable very well. Not having the parallel information is a much bigger hit on the overview. If a cell contains multiple sub-steps I don't see how it is much worse for tabular overview? It will still be tabular, in columns. And it can still prepare and fill cells as if it were just one level.

          Jan Klass added a comment - - edited

          With parallel the order in which the steps report their existence is not deterministic. Hence, the linear steps order is not stable. This results in the entire table history being dropped. So even by the argument of keeping the overview in a tabular format, this is not consistently possible, but only randomly!? Seems like a very weak argument.

          I just saw the table disappear on me (presumably) because of this.

          /edit: Okay, it looks like it appeared again? Very confusing. Is there a reordering-according-to-config-declaration at the end??

          Jan Klass added a comment - - edited With parallel the order in which the steps report their existence is not deterministic. Hence, the linear steps order is not stable. This results in the entire table history being dropped. So even by the argument of keeping the overview in a tabular format, this is not consistently possible, but only randomly!? Seems like a very weak argument. I just saw the table disappear on me (presumably) because of this. /edit: Okay, it looks like it appeared again? Very confusing. Is there a reordering-according-to-config-declaration at the end??

          Oleg Nenashev added a comment -

          I would like to add this item into the Jenkins roadmap: https://jenkins.io/project/roadmap/

          Would be everyone fine if I reword it to "Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI"? I would also remove the component, because it is likely to become a separate plugin if it gets implemented.

           

           

           

          Oleg Nenashev added a comment - I would like to add this item into the Jenkins roadmap:  https://jenkins.io/project/roadmap/ Would be everyone fine if I reword it to "Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI"? I would also remove the component, because it is likely to become a separate plugin if it gets implemented.      

          Jesse Glick added a comment -

          Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI

          Makes no sense. B.O. already supports this.

          I would also remove the component, because it is likely to become a separate plugin if it gets implemented.

          Also makes no sense. The limitation is specifically in the stage view plugin.

          Jesse Glick added a comment - Support Blue Ocean - alike browsing of Pipelines inside Jenkins web UI Makes no sense. B.O. already supports this. I would also remove the component, because it is likely to become a separate plugin if it gets implemented. Also makes no sense. The limitation is specifically in the stage view plugin.

          Dee Kryvenko added a comment -

          I am kind of lost here. I just learned about this https://groups.google.com/g/jenkinsci-users/c/xngZrSsXIjc/m/btasuPpYCgAJ and it was discussed a bit in the https://issues.jenkins.io/browse/JENKINS-54010. That leaves us in a weird state where B.O. is half baked and still missing pieces of functionality (like aforementioned ticket) and get very little support, but the classic UI has no support for parallel stages and steps whatsoever. Running stuff in parallel is kind of basic fundamental necessity that any other CI system has support for. What is the current direction with this ticket? Obviously current description encouraging to use B.O. is outdated.

          Dee Kryvenko added a comment - I am kind of lost here. I just learned about this https://groups.google.com/g/jenkinsci-users/c/xngZrSsXIjc/m/btasuPpYCgAJ  and it was discussed a bit in the https://issues.jenkins.io/browse/JENKINS-54010 . That leaves us in a weird state where B.O. is half baked and still missing pieces of functionality (like aforementioned ticket) and get very little support, but the classic UI has no support for parallel stages and steps whatsoever. Running stuff in parallel is kind of basic fundamental necessity that any other CI system has support for. What is the current direction with this ticket? Obviously current description encouraging to use B.O. is outdated.

          Stuart Rowe added a comment -

          I mentioned pipeline-graph-view-plugin in my comment on https://issues.jenkins.io/browse/JENKINS-54010. That plugin brings visualization of parallel steps into the classic UI.

          Stuart Rowe added a comment - I mentioned  pipeline-graph-view-plugin in my comment on  https://issues.jenkins.io/browse/JENKINS-54010 . That plugin brings visualization of parallel steps into the classic UI.

            cloudbees CloudBees Inc.
            hrmpw Patrick Wolf
            Votes:
            217 Vote for this issue
            Watchers:
            221 Start watching this issue

              Created:
              Updated: