Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35836

Steps outside a stage or within a stage but outside a parallel do not get visualized

    • 1.0-japan-m9, 1.0-m7

      This feature request is unplanned and is not actively being worked on

      We are unable to merge orphaned steps (that is, steps that exist outside of a stage or outside of a parallel within a stage) into the closest stage or a synthetic without considerable engineering effort to ensure that all steps are available in the visualization.

      As a workaround we recommend that you wrap any steps within stage and do not include steps sitting outside a parallel in the same stage or try Declarative Pipeline which was designed not to have this problem.

      Scope
      For visualisation purposes:

      • Step outside stage is relocated to the previous stage or next stage (depending if at start, middle or end of the pipeline)
      • Step before a parallel block is relocated to the first parallel branch
      • Step after a parallel block is relocated to the last parallel branch

      Problem
      It is possible that a stage, that has parallel branches under it, also has steps before or after a parallel section.

      Example 1 - step outside stage block
      The output of the step that echos never going to let you down will never be visible from the UI.

      node {
        stage "Stage 1" {
          sh "echo never going to give you up"
        }
        sh "echo never going to let you down"
        stage "Stage 2" {
          parallel (
            "Firefox" : { echo "Never run around" },
            "Edge" : { echo "Never desert you" }
          ) 
        }
        
        stage "Stage 3" {
          echo "Make you cry (never)"
          echo "Tell a lie and hurt you (never)"
        }
      }
      
      

      Example 2 - step inside stage but not within parallel
      The output of the step that echos never going to let you down will never be visible from the UI.

      node {
        stage "Stage 1" {
          sh "echo never going to give you up"
        }
        
        stage "Stage 2" {
          sh "echo never going to let you down"
          parallel (
            "Firefox" : { echo "Never run around" },
            "Edge" : { echo "Never desert you" }
          ) 
        }
        
        stage "Stage 3" {
          echo "Make you cry (never)"
          echo "Tell a lie and hurt you (never)"
        }
      }
      

          [JENKINS-35836] Steps outside a stage or within a stage but outside a parallel do not get visualized

          James Dumay added a comment -

          vivek in this scenario could we fold steps outside the parallel or stage into the nearest stage or parallel ?

          James Dumay added a comment - vivek in this scenario could we fold steps outside the parallel or stage into the nearest stage or parallel ?

          Vivek Pandey added a comment -

          Stages are guaranteed to be ordered so folding steps outside stages in to nearest one is doable.

          However steps outside parallel I am not sure. In DAG returned via API, we only capture parallel branches. To fold such steps in to nearest parallel branch, we need parallelBranch event from graph-analysis API to be in order of declaration which is not the case as order of parallel branch execution are not guaranteed to be in order of declaration. svanoort can confirm or tell us if there is some way we can capture it.

          Vivek Pandey added a comment - Stages are guaranteed to be ordered so folding steps outside stages in to nearest one is doable. However steps outside parallel I am not sure. In DAG returned via API, we only capture parallel branches. To fold such steps in to nearest parallel branch, we need parallelBranch event from graph-analysis API to be in order of declaration which is not the case as order of parallel branch execution are not guaranteed to be in order of declaration. svanoort can confirm or tell us if there is some way we can capture it.

          Julien Pivotto added a comment - - edited

          If you look at my proposal closer:

          the horizontal axis means "time", not "stage". It means that groups of steps outside of parallel (or stage) are represented by nodes.

          If there are 2 x 2 parallel, they are displayed

          1 3
          2 4

          and not

          1
          2
          3
          4

          Again, it is about time and no longer about stage.

          You can identify the stage by the line on top of the nodes, if the stage covers multiple columns.

          Julien Pivotto added a comment - - edited If you look at my proposal closer: the horizontal axis means "time", not "stage". It means that groups of steps outside of parallel (or stage) are represented by nodes. If there are 2 x 2 parallel, they are displayed 1 3 2 4 and not 1 2 3 4 Again, it is about time and no longer about stage. You can identify the stage by the line on top of the nodes, if the stage covers multiple columns.

          James Dumay added a comment - - edited

          roidelapluie I have updated the ticket with a bit of a description of how we would like to handle this use case.

          EDIT: also for your last comment that problem is covered in depth in JENKINS-38442.

          James Dumay added a comment - - edited roidelapluie I have updated the ticket with a bit of a description of how we would like to handle this use case. EDIT: also for your last comment that problem is covered in depth in JENKINS-38442 .

          Michael Neale added a comment -

          I have made this one dependent on nested stages, as it seems after a closer look to be very very similar (at least the solutions will likely be similar). Ie if we can support nested stages, we can support nested synthetic stages for these post/pre parallel steps

          Michael Neale added a comment - I have made this one dependent on nested stages, as it seems after a closer look to be very very similar (at least the solutions will likely be similar). Ie if we can support nested stages, we can support nested synthetic stages for these post/pre parallel steps

          Julien Pivotto added a comment - - edited

          jamesdumay This is pretty bad. I think as an immediate action we could at least print a warning when some steps are not displayed – because that will confuse lots of users:

          «Warning! This pipeline contains steps that can not be displayed in Blue Ocean. Please go to [classic UI] to see all the steps. Read more in JENKINS-35836

          JENKINS-35836 is a link to this,
          [classic UI] is a link to the build "pipeline steps" view.

          Julien Pivotto added a comment - - edited jamesdumay This is pretty bad. I think as an immediate action we could at least print a warning when some steps are not displayed – because that will confuse lots of users: «Warning! This pipeline contains steps that can not be displayed in Blue Ocean. Please go to [classic UI] to see all the steps. Read more in JENKINS-35836 .» JENKINS-35836 is a link to this, [classic UI] is a link to the build "pipeline steps" view.

          James Dumay added a comment -

          roidelapluie we were discussing warnings today. Stay tuned!

          James Dumay added a comment - roidelapluie we were discussing warnings today. Stay tuned!

          James Dumay added a comment -

          Hi all,

          I want to be honest that it is unlikely that we will be able to satisfy everyone with any of the suggested resolutions on this ticket and that I will be closing this out.

          However, if any community member has a bright idea on how to solve it and wants to try to make it work we'd be more than happy to guide and assist.

          As per the description, there are two workarounds given that help solve this problem - and one is the brand new Declarative Pipeline syntax which was designed to solve this exact problem.

          Thanks,
          James

          James Dumay added a comment - Hi all, I want to be honest that it is unlikely that we will be able to satisfy everyone with any of the suggested resolutions on this ticket and that I will be closing this out. However, if any community member has a bright idea on how to solve it and wants to try to make it work we'd be more than happy to guide and assist. As per the description, there are two workarounds given that help solve this problem - and one is the brand new Declarative Pipeline syntax which was designed to solve this exact problem. Thanks, James

          Giorgio Sironi added a comment - - edited

          I'm not sure how the Declarative Pipeline Syntax solves this problem - when I try to define:

          pipeline {
            agent none
            stages {
              stage('My stage') {
                steps {
                  echo 'outside'
                  parallel {
                      stage('A') {
                          echo 'a'
                      }
                      stage('B') {
                          echo 'a'
                      }
                  }
                }
              }
            }
          }
          

          I get a syntax error:

          Invalid step "parallel" used - not allowed in this context - The parallel step can only be used as the only top-level step in a stages step block

          My workaround is possibly to use xargs -P or GNU parallel, but it would be a pain if the branches are Groovy code.

          Giorgio Sironi added a comment - - edited I'm not sure how the Declarative Pipeline Syntax solves this problem - when I try to define: pipeline { agent none stages { stage( 'My stage' ) { steps { echo 'outside' parallel { stage( 'A' ) { echo 'a' } stage( 'B' ) { echo 'a' } } } } } } I get a syntax error: Invalid step "parallel" used - not allowed in this context - The parallel step can only be used as the only top-level step in a stages step block My workaround is possibly to use xargs -P or GNU parallel, but it would be a pain if the branches are Groovy code.

          Also, if you have an error like a test failure in the outside are that is not inside parallel, opening the build actually sends you to /blue/organizations/jenkins/:pipeline/detail/:pipeline/:build_number/pipeline which shows everything. So it doesn't seem true that the output is not visualized, just that it is not reachable through navigation. Try this to reproduce:

          node {
              stage 'My stage', {
                  def actions = [:]
                  actions['a'] = {
                      echo 'a'
                  }
                  actions['b'] = {
                      echo 'b'
                  }
                  parallel actions
                  sh 'echo feels good to be outside parallel'
                  throw new RuntimeException("outside, later error")
              }
          }
          

          It gives this perfectly good output:

          However, it only renders this if there is a failure outside parallel.

          As for ideas:

          • make this output available even if there is no failure in that area rather than selecting the last parallel stage by default.
          • make the stage name (on the Start...End line) clickable, to open this full output.

          Giorgio Sironi added a comment - Also, if you have an error like a test failure in the outside are that is not inside parallel , opening the build actually sends you to /blue/organizations/jenkins/:pipeline/detail/:pipeline/:build_number/pipeline which shows everything. So it doesn't seem true that the output is not visualized, just that it is not reachable through navigation. Try this to reproduce: node { stage 'My stage' , { def actions = [:] actions[ 'a' ] = { echo 'a' } actions[ 'b' ] = { echo 'b' } parallel actions sh 'echo feels good to be outside parallel' throw new RuntimeException( "outside, later error" ) } } It gives this perfectly good output: However, it only renders this if there is a failure outside parallel . As for ideas: make this output available even if there is no failure in that area rather than selecting the last parallel stage by default. make the stage name (on the Start ... End line) clickable, to open this full output.

            Unassigned Unassigned
            jamesdumay James Dumay
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: