When running pipeline job with parallel stages, the order of stages returned by the endpoint '/job/<jobname>/wfapi/runs' is inconsistent with the way the stages are created and displayed when completed. The ui is designed to display  runs from the newest to the oldest with stages provided in the same order. Therefore when parallel stages are used, the ui will almost always only display the last run.

       

      Attached screenshots will show :

      • Before running the job (Consistent Before Run.png), the order of stages of the completed runs allow to display all previous runs.
      • When running the job (Inconsistent During Run.png), the different order of stages of the running job discontinue the display of the previous runs.
      • After running the job (Consistent After Run.png), the order of stages of the newly completed run follows its natural creation order and allow the ui to display the previous runs.

      The previous exemple was made with the script (parallelPipelineScriptTest.txt) attached and was tested on :

      • A Jenkins server version : 2.235.1 with the Pipeline plugin version 2.6 (containing plugin Pipeline: Stage View Plugin version 2.13)
      • A local development post with pipeline-stage-view-plugin, branch : master, current version : 2.14-SNAPSHOT. Some changes in pom.xml files had to be maid to create the pipeline, 'node' being not recognized in the goovy script with the vanilla dependencies provided by the poms, see changes in patchPoms.txt.

        1. Consistent After Run.png
          Consistent After Run.png
          330 kB
        2. Consistent Before Run.png
          Consistent Before Run.png
          306 kB
        3. Inconsistent During Run.png
          Inconsistent During Run.png
          220 kB
        4. parallelPipelineScriptTest.txt
          0.4 kB
        5. patchPoms.txt
          5 kB
        6. stages-order-on-blue-ocean.png
          stages-order-on-blue-ocean.png
          23 kB
        7. stages-order-on-console-log.png
          stages-order-on-console-log.png
          27 kB
        8. stages-order-on-stage-view.png
          stages-order-on-stage-view.png
          29 kB

          [JENKINS-63115] Inconsistent Parallel Stage Order

          Ludovic Souchard added a comment - - edited

          I have created the PR 88 to resolve the issue.

          Ludovic Souchard added a comment - - edited I have created the PR  88  to resolve the issue.

          Adam Gabryś added a comment -

          The merged PR doesn't solve the issue. If I understand the code correctly, stages are sorted depending on the nodes execution start up. When parallel stages require heavy nodes, the order is not always the same as the nodes requesting order. Some pods are created on slower machines and start up later. It causes that the order is invalid. I would suggest use the same approach which is used by BlueOcean - sort stages alphabetically. It guarantees that the order is always the same, no matter when the stages are started (nodes are allocated).

          Correct order:

          • BlueOcean:
          • Console Log:

          Incorrect order:

          • Stage View:

          Adam Gabryś added a comment - The merged PR doesn't solve the issue. If I understand the code correctly, stages are sorted depending on the nodes execution start up. When parallel stages require heavy nodes, the order is not always the same as the nodes requesting order. Some pods are created on slower machines and start up later. It causes that the order is invalid. I would suggest use the same approach which is used by BlueOcean - sort stages alphabetically. It guarantees that the order is always the same, no matter when the stages are started (nodes are allocated). Correct order: BlueOcean: Console Log: Incorrect order: Stage View:

          Well, it fixes it as long as Jenkins is not waiting to start the job... No, of course, I agree, the solution is too limited. I overlooked what I took from Nuh and I didn't have a proper environment to test when Jenkins or its nodes are busy.

           

          However, I disagree with the solution you suggested. Applying directly a lexicographical sorting would mess up the chronological order of serial stages. If we had the relations between the stages, we could apply that rule locally (event tho I still don't like it). Right now I don't see how to gather that data with the ChunkVisitor.

          Ludovic Souchard added a comment - Well, it fixes it as long as Jenkins is not waiting to start the job... No, of course, I agree, the solution is too limited. I overlooked what I took from Nuh and I didn't have a proper environment to test when Jenkins or its nodes are busy.   However, I disagree with the solution you suggested. Applying directly a lexicographical sorting would mess up the chronological order of serial stages. If we had the relations between the stages, we could apply that rule locally (event tho I still don't like it). Right now I don't see how to gather that data with the ChunkVisitor.

          Ludovic Souchard added a comment - - edited

          Sorry for the long hiatus. It took me some time before actually using what I contributed. However, I do not have the same issue you are describing. I am using a script equivalent to the one you'll find attached to this issue. The main thing to recall is that what must be run in parallel must be "stages". Furthermore, when running those stages from Jenkins' master node (I've not tested on other nodes), no executor is used by the stages, hence the stages are always started immediately and are not affected by any waiting for a job to start within any stage.

          Ludovic Souchard added a comment - - edited Sorry for the long hiatus. It took me some time before actually using what I contributed. However, I do not have the same issue you are describing. I am using a script equivalent to the one you'll find attached to this issue. The main thing to recall is that what must be run in parallel must be "stages". Furthermore, when running those stages from Jenkins' master node (I've not tested on other nodes), no executor is used by the stages, hence the stages are always started immediately and are not affected by any waiting for a job to start within any stage.

          Allan BURDAJEWICZ added a comment - - edited

          This is still inconsistent. If using parallel with sequential stages, the ordering is unlikely to be preserved if the stage start times are not always consistent. Here is a simple reproducer:

          def branches = [
              'branch1': branch(1),
              'branch2': branch(2)
          ]
          
          def branch(int index) {
              return {
                  node('default-java') {
          
                      // if build number is even, the stages of one branch sleep less 5s less
                      // if build number is odd, the stages of the other branch sleep 5s less
                      def delta = (1-((env.BUILD_NUMBER.toInteger()+index)%2))*5
          
                      stage('branch' + index + '-stage1') {
                          sh "sleep ${10+delta}"
                      }
          
                      stage('branch' + index + '-stage2') {
                          sh "sleep ${10+delta}"
                      }
                  }
              }
          }
          
          parallel branches
          

          You will never see the same sequence twice in a row:

          • build #1: branch-2-stage-1,branch-1-stage-1,branch-2-stage-2,branch-1-stage-2
          • build #2: branch-1-stage-1,branch-2-stage-1,branch-1-stage-2,branch-2-stage-2
          • build #3: branch-2-stage-1,branch-1-stage-1,branch-2-stage-2,branch-1-stage-2

          And stage view will therefore only display the latest build.

          For stage view, maybe it makes more sense to follow the stage definition order (if that makes any sense in the context of stage view and the workflow api) rather than the chronological order. But maybe that's the whole issue here, that when traversing the graph we don't always get the same order..

          Allan BURDAJEWICZ added a comment - - edited This is still inconsistent. If using parallel with sequential stages, the ordering is unlikely to be preserved if the stage start times are not always consistent. Here is a simple reproducer: def branches = [ 'branch1' : branch(1), 'branch2' : branch(2) ] def branch( int index) { return { node( ' default -java' ) { // if build number is even, the stages of one branch sleep less 5s less // if build number is odd, the stages of the other branch sleep 5s less def delta = (1-((env.BUILD_NUMBER.toInteger()+index)%2))*5 stage( 'branch' + index + '-stage1' ) { sh "sleep ${10+delta}" } stage( 'branch' + index + '-stage2' ) { sh "sleep ${10+delta}" } } } } parallel branches You will never see the same sequence twice in a row: build #1: branch-2-stage-1,branch-1-stage-1,branch-2-stage-2,branch-1-stage-2 build #2: branch-1-stage-1,branch-2-stage-1,branch-1-stage-2,branch-2-stage-2 build #3: branch-2-stage-1,branch-1-stage-1,branch-2-stage-2,branch-1-stage-2 … And stage view will therefore only display the latest build. For stage view, maybe it makes more sense to follow the stage definition order (if that makes any sense in the context of stage view and the workflow api) rather than the chronological order. But maybe that's the whole issue here, that when traversing the graph we don't always get the same order..

          Jim D added a comment -

          allan_burdajewicz  - Thanks, following the stage definition order makes sense to me.  I think that prior to the above PR 88, Jenkins more or less did that, except there was some detail where the stages (or workflow nodes) for a job still running don't look the same as the stages (or nodes) for a completed job, causing the original issue here - the stage view used to reset while a pipeline is running and only go back to normal after completion.  So I think the solution might have to find a way to use stage definition order while still working for the running pipeline.

           

          Jim D added a comment - allan_burdajewicz   - Thanks, following the stage definition order makes sense to me.  I think that prior to the above PR 88, Jenkins more or less did that, except there was some detail where the stages (or workflow nodes) for a job still running don't look the same as the stages (or nodes) for a completed job, causing the original issue here - the stage view used to reset while a pipeline is running and only go back to normal after completion.  So I think the solution might have to find a way to use stage definition order while still working for the running pipeline.  

          Allan BURDAJEWICZ added a comment - I have WIP https://github.com/jenkinsci/pipeline-stage-view-plugin/pull/286

            allan_burdajewicz Allan BURDAJEWICZ
            lsouchard Ludovic Souchard
            Votes:
            6 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: