Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39464

Pipeline with parallel and no stages does not get visualised correctly

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • blueocean-plugin
    • None
    • tethys

      Problem
      If you have a parallel declaration, but no stages, in classic stage view nothing shows up (you just have to look at log). In blue ocean, you get a single node (stage) which shows only the first branch.
      Expected behavior: should show 3 parallel branches (ideally)

      Example

      parallel 'branch1': {
              node('node1') {
                  stage('Setup') {
                      checkout([details removed])
                  }
                  stage('Unit and Integration Tests') {
                      bat '"my command to execute tests"'
                  }
              }
      }, 'branch2': {
              node('node2') {
                  stage('Setup') {
                      checkout([details removed])
                  }
                  stage('Unit and Integration Tests') {
                      bat '"my command to execute tests"'
                  }
              }
      }
      

      Workaround
      Simply wrap the parallels in a stage and it will work correctly.

      stage('Build and Test') {
        parallel 'branch1': {
                node('node1') {
                    stage('Setup') {
                        checkout([details removed])
                    }
                    stage('Unit and Integration Tests') {
                        bat '"my command to execute tests"'
                    }
                }
        }, 'branch2': {
                node('node2') {
                    stage('Setup') {
                        checkout([details removed])
                    }
                    stage('Unit and Integration Tests') {
                        bat '"my command to execute tests"'
                    }
                }
        }
      }
      

      It will then be visualised like:

      Other examples

      Parallel defined between stages but not nested in its own stage

      node {
      	stage('Checkout') {
      		checkout scm
      	}
      }
      
      parallel linux: {
      	node('Linux') {
      		stage('Build') {
      			echo 'Build linux'
      		}
      		stage('Tests') {
      			echo 'Tests linux'
      		}
      		stage('Static analysis') {
      			echo 'Static analysis linux'
      		}
      	}
      },
      windows: {
      	node('Windows') {
      		stage('Build') {
      			echo 'Build windows'
      		}
      		stage('Tests') {
      			echo 'Tests windows'
      		}
      		stage('Static analysis') {
      			echo 'Static analysis windows'
      		}
      	}
      }
      

      Result looks like this:

          [JENKINS-39464] Pipeline with parallel and no stages does not get visualised correctly

          Michael Neale added a comment -

          Not sure if this should be fixed in API or front end. Leaning to former.

          Michael Neale added a comment - Not sure if this should be fixed in API or front end. Leaning to former.

          James Dumay added a comment -

          rtyler thanks for reporting this one. We are likely to take this on in the coming weeks. As a work around you can wrap your parallel in a stage.

          James Dumay added a comment - rtyler thanks for reporting this one. We are likely to take this on in the coming weeks. As a work around you can wrap your parallel in a stage.

          James Dumay added a comment -

          tscherler / vivek is this likely to be a frontend or backend issue?

          James Dumay added a comment - tscherler / vivek is this likely to be a frontend or backend issue?

          James Dumay added a comment -

          Now we have a start dot we can fix this problem as part of JENKINS-39628

          James Dumay added a comment - Now we have a start dot we can fix this problem as part of JENKINS-39628

          Michael Neale added a comment -

          yeah this comes up again and again... so worth catering to IMO

          Michael Neale added a comment - yeah this comes up again and again... so worth catering to IMO

          James Dumay added a comment - - edited

          svanoort vivek is there some way we could have this show up as if it was wrapped in a stage? not sure if we should do this in the frontend code, Blue Ocean or bismuth level.

          James Dumay added a comment - - edited svanoort vivek is there some way we could have this show up as if it was wrapped in a stage? not sure if we should do this in the frontend code, Blue Ocean or bismuth level.

          jlpinardon added a comment -

          Dear all,

          I made a trial
          stagesNestedinParallel.xml
          that is equivalent to the example given here. And, for sure, I felt into the same bug.
          Here is what it is displayed :

          I suggest that such a parallel pipeline (configure, make and test sequential stages run within several parallel "branches") could be displayed as :

          Note that the "legacy" visualization is even unable to correctly display the duration of the stages :

          jlpinardon added a comment - Dear all, I made a trial stagesNestedinParallel.xml that is equivalent to the example given here. And, for sure, I felt into the same bug. Here is what it is displayed : I suggest that such a parallel pipeline (configure, make and test sequential stages run within several parallel "branches") could be displayed as : Note that the "legacy" visualization is even unable to correctly display the duration of the stages :

          James Dumay added a comment - - edited

          jlpinardon could you share the Jenkinsfile/Script please? You might also want try the workaround provided in the ticket description.

          James Dumay added a comment - - edited jlpinardon could you share the Jenkinsfile/Script please? You might also want try the workaround provided in the ticket description.

          jlpinardon added a comment -

          Here it is :

          env.sleepTime=10
          node('MTS_RH6_X64') {
              env.initialName='Level1'
              // Initializing level 1
              stage('Start') {
                  env.stageName=env.initialName
                  sh('''
                          #!/bn/bash
                          echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                          sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                  ''')
              }
          
              parallel 'MRS_LDB':{
          
                  node('ENODEB_MRS_LDB') {
                      env.stageName='configure'
                      stage ('configure') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')
                      }
                      env.stageName='make'
                      stage ('make') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')        }
                      env.stageName='test'
                      stage ('test') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')
                      }
                  }
                 
              }, 
              'MTS_LRH5':{
                  node('MTS_LRH5') {
                      env.stageName='configure'
                      stage ('configure') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')
                      }
                      env.stageName='make'
                      stage ('make') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')        }
                      env.stageName='test'
                      stage ('test') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')
                      }
                  }
              },
              'LB': {
                  node('MTS_RH6_X64') {
                      env.stageName='configure'
                      stage ('configure') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')
                      }
                      env.stageName='make'
                      stage ('make') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')        }
                      env.stageName='test'
                      stage ('test') {
                      sh('''
                              #!/bn/bash
                              echo "Starting ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
                              sleep $(( 1 + $RANDOM % ${sleepTime} )) 
                      ''')
                      }
                  }
              }
              env.stageName=env.initialName+'_Deploy'
              stage('Deploy') {
              sh('''
                  #!/bin/bash
                  sleep $(( 1 + $RANDOM % ${sleepTime} ))
                  echo "This is the end for ${JOB_NAME}.${stageName} at $(date +'%s') on ${NODE_NAME}"
              ''')          
              }       
          }
          

          jlpinardon added a comment - Here it is : env.sleepTime=10 node( 'MTS_RH6_X64' ) { env.initialName= 'Level1' // Initializing level 1 stage( 'Start' ) { env.stageName=env.initialName sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } parallel 'MRS_LDB' :{ node( 'ENODEB_MRS_LDB' ) { env.stageName= 'configure' stage ( 'configure' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'make' stage ( 'make' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'test' stage ( 'test' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } } }, 'MTS_LRH5' :{ node( 'MTS_LRH5' ) { env.stageName= 'configure' stage ( 'configure' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'make' stage ( 'make' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'test' stage ( 'test' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } } }, 'LB' : { node( 'MTS_RH6_X64' ) { env.stageName= 'configure' stage ( 'configure' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'make' stage ( 'make' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } env.stageName= 'test' stage ( 'test' ) { sh(''' #!/bn/bash echo "Starting ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" sleep $(( 1 + $RANDOM % ${sleepTime} )) ''') } } } env.stageName=env.initialName+ '_Deploy' stage( 'Deploy' ) { sh(''' #!/bin/bash sleep $(( 1 + $RANDOM % ${sleepTime} )) echo "This is the end for ${JOB_NAME}.${stageName} at $(date + '%s' ) on ${NODE_NAME}" ''') } }

          jlpinardon added a comment - - edited

          With the workaround, the display is indeed "correct". I mean it is exact. But I think (as far as it is feasible) that the sequential stuff within could be displayed as in the my proposed schema. At least for the first level.

          Nevertheless, the display during the build run is not correct as shown below :

          Parallel blocks are shown as running (which is correct) but stages within are all displayed sequentially after, and lokke like achieved (which is for sure totally wrong).

          jlpinardon added a comment - - edited With the workaround, the display is indeed "correct". I mean it is exact. But I think (as far as it is feasible) that the sequential stuff within could be displayed as in the my proposed schema. At least for the first level. Nevertheless, the display during the build run is not correct as shown below : Parallel blocks are shown as running (which is correct) but stages within are all displayed sequentially after, and lokke like achieved (which is for sure totally wrong).

          James Dumay added a comment - - edited

          jlpinardon absolutely - we can't render those stage's within the parallels (see JENKINS-38442).

          The reason why it doesn't show correctly after you change the structure of the Pipeline while executing is that we rely on the previous run to "preview" what the flow of the Pipeline will be. Unfortunately as Pipeline is a scripting language we can't read ahead the structure without executing it. The good news is that the new declarative syntax for Pipeline doesn't have this problem - we can read the Jenkinsfile like a config file

          If you run it a second time you shouldn't see the problem.

          James Dumay added a comment - - edited jlpinardon absolutely - we can't render those stage 's within the parallels (see JENKINS-38442 ). The reason why it doesn't show correctly after you change the structure of the Pipeline while executing is that we rely on the previous run to "preview" what the flow of the Pipeline will be. Unfortunately as Pipeline is a scripting language we can't read ahead the structure without executing it. The good news is that the new declarative syntax for Pipeline doesn't have this problem - we can read the Jenkinsfile like a config file If you run it a second time you shouldn't see the problem.

          jlpinardon added a comment -

          Thanks for the good news I will try this as soon as possible.
          But concerning the display while running,the bad news is that it is still broken after several run.

          jlpinardon added a comment - Thanks for the good news I will try this as soon as possible. But concerning the display while running,the bad news is that it is still broken after several run.

          James Dumay added a comment - - edited

          svanoort vivek would it be feasible to synthesise a stage around a parallel when it is not within a stage itself? We're starting to see more and more questions about why the visualization doesn't work (obviously not an issue with Declarative). I would like to get an idea of how feasible this is and a rough estimate.

          James Dumay added a comment - - edited svanoort vivek would it be feasible to synthesise a stage around a parallel when it is not within a stage itself? We're starting to see more and more questions about why the visualization doesn't work (obviously not an issue with Declarative). I would like to get an idea of how feasible this is and a rough estimate.

          Vivek Pandey added a comment -

          jamesdumay I will start with what I think is work involved or issues we need to solve and then make determination:

          • This virtual stage would collate all orphan (with no parent stage) consecutive top level parallels (with no stages in between)
          • What would be the display name or label of each virtual stage?
          • Current implementation is event based parsing API (bismuth), it sends events for all nodes. For stage it sends events that help compute stages, status, timing info etc.
          • * Computation of stage would require copying/collating status from underlying parallel
          • * Computation of timing will involve capture timing from underlying parallel blocks. This might be incorrect unless we capture all out-of-stage nodes (steps as well) and then compute timing info (duration, start and end)
          • Sythentic stage id generation. node id is generated during execution, its simple integer. I am hoping CpsFlowExecution.iota() might just do it but maybe there are edge cases where there are chances of id conflict - like pipeline is midway execution so id of synthetic stage could be reused later? Or we generate UUID for these nodes ourselves?

          svanoort are there other complications it might cause or better approach to handle it?

          Vivek Pandey added a comment - jamesdumay I will start with what I think is work involved or issues we need to solve and then make determination: This virtual stage would collate all orphan (with no parent stage) consecutive top level parallels (with no stages in between) What would be the display name or label of each virtual stage? Current implementation is event based parsing API (bismuth), it sends events for all nodes. For stage it sends events that help compute stages, status, timing info etc. * Computation of stage would require copying/collating status from underlying parallel * Computation of timing will involve capture timing from underlying parallel blocks. This might be incorrect unless we capture all out-of-stage nodes (steps as well) and then compute timing info (duration, start and end) Sythentic stage id generation. node id is generated during execution, its simple integer. I am hoping CpsFlowExecution.iota() might just do it but maybe there are edge cases where there are chances of id conflict - like pipeline is midway execution so id of synthetic stage could be reused later? Or we generate UUID for these nodes ourselves? svanoort are there other complications it might cause or better approach to handle it?

          James Dumay added a comment -

          vivek if we have to synthesise a stage I am happy with it being called "Parallel". Would it be possible to attach some data to the synthesised node so that we can show a warning in the UI about how it has been transformed? I'd like to educate users more about the usage of Stage.

          James Dumay added a comment - vivek if we have to synthesise a stage I am happy with it being called "Parallel". Would it be possible to attach some data to the synthesised node so that we can show a warning in the UI about how it has been transformed? I'd like to educate users more about the usage of Stage.

          James Dumay added a comment -

          This fix will be released in 1.0.0-b19 next week.

          James Dumay added a comment - This fix will be released in 1.0.0-b19 next week.

          Jakub Pawlinski added a comment - - edited

          Hi, running Blue Ocean 1.1.4 in declarative pipeline I can't make it work:

           
          stages {  stage('Parallel1') { 
              parallel 'branch1': {
                node('n1') { stage('Unit 1') { echo "1" }}{{} }}{
              }, 'branch2': {
                node('n2') { stage('Unit 2') { echo "2" }
          }}

          • Result: Starting with version 0.5, steps in a stage must be in a steps block
             
            stages {
              steps {
                parallel 'branch1': { 
                  node('n1') { stage('Unit 1') { echo "1" } } 
                }, 'branch2': {
                  node('n2') { stage('Unit 2') { echo "2" } }
            }}
          • Result: Invalid step "stage" used - not allowed in this context - The stage step cannot be used in Declarative Pipelines
            Running any of above outside of stages{} results with – Expected a stage exception
             
            I have created https://groups.google.com/forum/#!topic/jenkinsci-users/ObAZcvsiZA8 so hopefully will find workaround there
             

          Jakub Pawlinski added a comment - - edited Hi, running Blue Ocean 1.1.4 in declarative pipeline I can't make it work:   stages {   stage('Parallel1') {       parallel 'branch1': {       node('n1') {  stage('Unit 1') {  echo "1" }}{{} }}{     }, 'branch2': {       node('n2') {  stage('Unit 2') {  echo "2"  }  } }} Result: Starting with version 0.5, steps in a stage must be in a steps block   stages {   steps {     parallel 'branch1': {        node('n1') {  stage('Unit 1') {  echo "1"  }  }      }, 'branch2': {       node('n2') {  stage('Unit 2') {  echo "2"  }   } }} Result: Invalid step "stage" used - not allowed in this context - The stage step cannot be used in Declarative Pipelines Running any of above outside of stages{} results with – Expected a stage exception   I have created https://groups.google.com/forum/#!topic/jenkinsci-users/ObAZcvsiZA8  so hopefully will find workaround there  

          Hi, I'm running Blue Ocean 1.3.5 and am building the following:

          stages {
            stage('Build') {
              parallel {
                stage('Synthesis') {
                  ...
                }
                stage('Compilation') {
                  // Generate compilation matrix
                  ...
                  catchError {
                    parallel compilations
                  }
                }
              }
            }
          }
          

          I have 3 issues:

          1. All parallel stages are flattened into a long list. Preferably Synthesis and Compilation should split from the main pipeline, and then each of them should split into their own matrices.
          2. The stage that is failing is not the dot that is red. When clicking on the dot in the graph, I get the correct stage, but the color does not reflect the result. See the following screenshots.
          3. A minor issue: the stage name doesn't fit in the column. (but that's unrelated to here).

          Selected the red dot but the stage passed:

          Selected the green dot underneath it, which is the actual stage that failed.

          Tsvi Mostovicz added a comment - Hi, I'm running Blue Ocean 1.3.5 and am building the following: stages { stage( 'Build' ) { parallel { stage( 'Synthesis' ) { ... } stage( 'Compilation' ) { // Generate compilation matrix ... catchError { parallel compilations } } } } } I have 3 issues: All parallel stages are flattened into a long list. Preferably Synthesis and Compilation should split from the main pipeline, and then each of them should split into their own matrices. The stage that is failing is not the dot that is red. When clicking on the dot in the graph, I get the correct stage, but the color does not reflect the result. See the following screenshots. A minor issue: the stage name doesn't fit in the column. (but that's unrelated to here). Selected the red dot but the stage passed: Selected the green dot underneath it, which is the actual stage that failed.

          vivek
          I'm facing the same problem. Any progress here?

          See my complete issue described here: JENKINS-39464

          Taras Postument added a comment - vivek I'm facing the same problem. Any progress here? See my complete issue described here:  JENKINS-39464

          Rishi Thakkar added a comment -

          Why does the resolution to this issue say "Fixed" but the issue itself is still "In Porgress"?

          Rishi Thakkar added a comment - Why does the resolution to this issue say "Fixed" but the issue itself is still "In Porgress"?

            vivek Vivek Pandey
            michaelneale Michael Neale
            Votes:
            8 Vote for this issue
            Watchers:
            23 Start watching this issue

              Created:
              Updated:
              Resolved: