Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61280

Allow dynamic stage names

    XMLWordPrintable

Details

    Description

      I would like the ability to have stage names be dynamic strings. This is useful, for instance, when defining stages within a matrix, so that the stage names can contain the values of the axes. (Part of the problem here is that Blue Ocean doesn't render matrixes well at all.) Right now attempting to do this yields an error because stage names can only be string literals, not Groovy strings (interpolation).

      Note that I am not looking for the steps within the stage to be dynamic.

      Example:

      matrix {
          axes {
              axis {
                  name 'PLATFORM'
                  values 'linux', 'mac', 'windows'
              }
          }
          stages {
              stage("Build ${PLATFORM}") {
                  // ...
              }
              stage("Test ${PLATFORM}") {
                  // ...
              }
              stage("Deploy ${PLATFORM}") {
                  // ...
              }
          }
      }
      

      Attachments

        Activity

          rittneje Jesse Rittner created issue -
          bitwiseman Liam Newman made changes -
          Field Original Value New Value
          Assignee Andrew Bayer [ abayer ] Liam Newman [ bitwiseman ]
          bitwiseman Liam Newman added a comment -

          Stage names must be resolved before the pipeline starts.

          Since matrix construction is also done before pipeline start, the exact scenario above could be supported.
          However, users would then quite reasonably assume that they could put other variables into those interpolations and would get problems.

          I do not know the reason stage names must be resolved before the pipeline starts, but figuring out how to change that more generally would require some digging.

          bitwiseman Liam Newman added a comment - Stage names must be resolved before the pipeline starts. Since matrix construction is also done before pipeline start, the exact scenario above could be supported. However, users would then quite reasonably assume that they could put other variables into those interpolations and would get problems. I do not know the reason stage names must be resolved before the pipeline starts, but figuring out how to change that more generally would require some digging.
          smd Stefan Drissen made changes -
          Attachment image-2020-03-19-17-20-23-015.png [ 50709 ]
          smd Stefan Drissen made changes -
          Attachment image-2020-03-19-17-20-50-002.png [ 50710 ]
          smd Stefan Drissen made changes -
          Attachment image-2020-03-19-17-22-05-861.png [ 50711 ]

          I'm revisiting replacing a big xml template based multi-job project with a declarative matrix pipeline. While the matrix implementation was really nice and quick to setup, the visualization was a bit of let down. Dynamic stage names would really really really help.

          Current state:

          With stage names based on matrix values:

           

          smd Stefan Drissen added a comment - I'm revisiting replacing a big xml template based multi-job project with a declarative matrix pipeline. While the matrix implementation was really nice and quick to setup, the visualization was a bit of let down. Dynamic stage names would really really really help. Current state: With stage names based on matrix values:  
          smd Stefan Drissen made changes -
          Attachment image-2020-03-19-17-20-50-002.png [ 50710 ]

          Big +1 on this. I was able to get a massive job chopped way down thanks to matrix magic, but now the display on Blue Ocean leaves a lot to be desired. Our pipeline view looks very similar to Stefan's ^^ - there's not a clear indicator when one test fails of which one it was.

          I think either:

          a) Display the axes value(s) next to the stage name

          b) Allow for the option to add a variable to a stage name - only invalid inside a matrix directive

          c) Add an option to rename a stage - only valid inside a matrix directive

          tophergopher Topher Sterling added a comment - Big +1 on this. I was able to get a massive job chopped way down thanks to matrix magic, but now the display on Blue Ocean leaves a lot to be desired. Our pipeline view looks very similar to Stefan's ^^ - there's not a clear indicator when one test fails of which one it was. I think either: a) Display the axes value(s) next to the stage name b) Allow for the option to add a variable to a stage name - only invalid inside a matrix directive c) Add an option to rename a stage - only valid inside a matrix directive

          I think option (a) above would cover enough use cases without making it confusing why dynamic stage names only work inside a matrix.

          allout58 James Hollowell added a comment - I think option (a) above would cover enough use cases without making it confusing why dynamic stage names only work inside a matrix.
          krobarr9 James added a comment -

          Another big +1 to this. Implementing one of my regression runs as a matrix just shaved almost 10 hours off of a test regression run, but the results in Blue Ocean are a pain to navigate.

           

          I agree with tophergopher and allout58 above that option a) would solve all my problems.

          krobarr9 James added a comment - Another big +1 to this. Implementing one of my regression runs as a matrix just shaved almost 10 hours off of a test regression run, but the results in Blue Ocean are a pain to navigate.   I agree with tophergopher  and allout58  above that option a) would solve all my problems.
          bitwiseman Liam Newman added a comment -

          krobarr9allout58tophergopher
          What you are mostly describing is a Blue Ocean display issue. If you hover over the labels with ellipses, you'll see the text is there. If you go to classic view you'll be able to see it.

          I would love to see this fixed as well, but Blue Ocean is in maintenance mode, so it is unlikely that the display issue you describe this will get fixed. Due to the Blue Ocean design it isn't as simple as it might seem. I tried manually turning off the "whitespace: nowrap" that those label have on them and it doesn't improve matters. If any of you want to devote time to this, I'd be happy to point you in the right direction.

          bitwiseman Liam Newman added a comment - krobarr9 allout58 tophergopher What you are mostly describing is a Blue Ocean display issue. If you hover over the labels with ellipses, you'll see the text is there. If you go to classic view you'll be able to see it. I would love to see this fixed as well, but Blue Ocean is in maintenance mode, so it is unlikely that the display issue you describe this will get fixed. Due to the Blue Ocean design it isn't as simple as it might seem. I tried manually turning off the "whitespace: nowrap" that those label have on them and it doesn't improve matters. If any of you want to devote time to this, I'd be happy to point you in the right direction.
          smd Stefan Drissen made changes -
          Attachment image-2021-01-27-08-24-59-432.png [ 53889 ]

          bitwiseman

          Are you sure that is just a Blue Ocean display issue and not some underlying matrix issue?

          If I look at the classic display of a matrix pipeline it is rather unusable as well , each matrix combination is started (in 1s) with a matrix description, followed by the actual work in a block with the stage name:

          There is no easy way to see what these steps were. So while the Blue Ocean view may be limited, it beats the classic view hands down.

           

          smd Stefan Drissen added a comment - bitwiseman Are you sure that is just a Blue Ocean display issue and not some underlying matrix issue? If I look at the classic display of a matrix pipeline it is rather unusable as well , each matrix combination is started (in 1s) with a matrix description, followed by the actual work in a block with the stage name: There is no easy way to see what these steps were. So while the Blue Ocean view may be limited, it beats the classic view hands down.  
          jncarlic John added a comment -

          Adding my vote for this feature.  It makes both the stage view and blue ocean views ugly and overly complicated for no reason.

          jncarlic John added a comment - Adding my vote for this feature.  It makes both the stage view and blue ocean views ugly and overly complicated for no reason.
          faroguy0 Ian Panzer added a comment -

          I would love for this to happen. Thanks!

          faroguy0 Ian Panzer added a comment - I would love for this to happen. Thanks!

          bitwiseman, just the axis values would be already helpful. as currently. 

           

          rthurner rupert je thurner added a comment - bitwiseman , just the axis values would be already helpful. as currently.   
          merlachandra Chandra Babu added a comment -

          smd Could you please explain how you achieved "With stage names based on matrix values:" as in your second screenshot?

          merlachandra Chandra Babu added a comment - smd  Could you please explain how you achieved "With stage names based on matrix values:" as in your second screenshot?

          merlachandra It's just a mockup with labels modified using developer console to show what it could look like.

          smd Stefan Drissen added a comment - merlachandra  It's just a mockup with labels modified using developer console to show what it could look like.
          rajivece Rajiv Rajaian added a comment -

          +1 Vote

          rajivece Rajiv Rajaian added a comment - +1 Vote

          bitwiseman

          I'm interesting in poking around the UI code - would you still be able to point me in the right direction in the codebase?

          tophergopher Topher Sterling added a comment - bitwiseman I'm interesting in poking around the UI code - would you still be able to point me in the right direction in the codebase?
          jielpe_fr38 Jean-Luc Pé added a comment -

          Some sort of dynamic stage name would really be appreciate. At least, either build parameter or even environment variable declared in environment{} block should be avaible in all pipeline stage names (not only matrix).
          So ... +1

          jielpe_fr38 Jean-Luc Pé added a comment - Some sort of dynamic stage name would really be appreciate. At least, either build parameter or even environment variable declared in environment{} block should be avaible in all pipeline stage names (not only matrix). So ... +1
          gdemengin Guillaume DeMengin made changes -
          Attachment workaround1.png [ 56745 ]
          gdemengin Guillaume DeMengin added a comment - - edited

          just found this workaround : one can add a dummy stage inside "script" step and use a dynamic name
          ... not great but it does allow to show some dynamic name in the stage list (I use it to announce what's coming up in the stage after)

          pipeline {
              agent none 
              stages {
                  stage('build') { 
                     matrix {
                          axes {
                              axis {
                                  name 'OS'
                                  values 'windows', 'linux', 'macos'
                              }
                              axis {
                                  name 'ARCH'
                                  values 'x64', 'arm64'
                              }
                          }
                          stages {
                              stage('axis') {
                                  agent none
                                  steps {
                                      script {
                                          stage("${OS}-${ARCH}") {
                                              print "${OS}-${ARCH}"
                                          }
                                      }
                                  }
                              }
                              stage('build') {
                                  agent { label OS }
                                  steps {
                                      // build stuff
                                  }
                              }
                          }
                      }
                  }
              }
          } 

          gdemengin Guillaume DeMengin added a comment - - edited just found this workaround : one can add a dummy stage inside "script" step and use a dynamic name ... not great but it does allow to show some dynamic name in the stage list (I use it to announce what's coming up in the stage after) pipeline { agent none stages { stage( 'build' ) { matrix { axes { axis { name 'OS' values 'windows' , 'linux' , 'macos' } axis { name 'ARCH' values 'x64' , 'arm64' } } stages { stage( 'axis' ) { agent none steps { script { stage( "${OS}-${ARCH}" ) { print "${OS}-${ARCH}" } } } } stage( 'build' ) { agent { label OS } steps { // build stuff } } } } } } }

          This issue making some problems in many pipelines I manage ... and till it's fixed in the upstream I've implemented a custom step dynamicMatrix.

          That custom step mimics Jenkins declarative matrix but with better visualization and extra customization options. It makes it easier to run a parallel matrix with different combinations and groups.

          Here is an example:

          stage('single_matrix') {
              steps {
                  dynamicMatrix([
                      failFast: false,
                      axes: [
                          PLATFORM: ['linux', 'mac', 'windows'],
                          BROWSER: ['chrome', 'firefox', 'safari']
                      ],
                      actions: {
                          stage("${PLATFORM}_${BROWSER}") {
                              sh 'echo ${PLATFORM} - ${BROWSER}'
                              sh 'echo ${MATRIX_STAGE_NAME}'
                          }
                      }
                  ])
              }
          }
          

          It also has 2 other varieties to work with different combinations and groups.

          If you have different axes combinations but the same stages then dynamicMatrixMultiCombinations could be used for that case.

          stage('multi_combinations_matrix') {
              steps {
                  dynamicMatrixMultiCombinations([
                      failFast: false,
                      axes: [
                          [
                              PLATFORM: ['linux', 'mac', 'windows'],
                              BROWSER: ['chrome', 'firefox']
                          ],
                          [
                              PLATFORM: ['mac', 'windows']
                              BROWSER: ['safari'],
                          ],
                      ],
                      actions: {
                          stage("${PLATFORM}_${BROWSER}") {
                              sh 'echo ${PLATFORM}, ${BROWSER}'
                              sh 'echo ${MATRIX_STAGE_NAME}'
                          }
                      }
                  ])
              }
          }
          

          If you have different groups or different stages that should run in parallel at the same time, then dynamicMatrixMultiGroups could be used for that case.

          stage('multi_groups_matrix') {
              steps {
                  dynamicMatrixMultiGroups([
                      [
                          failFast: false,
                          axes: [
                              PLATFORM: ['windows']
                              BROWSER: ['edge', 'safari'],
                          ],
                          actions: {
                              stage("${PLATFORM}_${BROWSER}") {
                                  bat 'echo this is a Windows command'
                              }
                          }
                      ],
                      [
                          failFast: false,
                          axes: [
                              PLATFORM: ['linux', 'mac'],
                              BROWSER: ['chrome', 'firefox']
                          ],
                          actions: {
                              stage("${PLATFORM}_${BROWSER}") {
                                  sh 'echo this is a Unix-like command'
                              }
                          }
                      ]
                  ])
              }
          }
          

          Enjoy

          aabouzaid Ahmed AbouZaid added a comment - This issue making some problems in many pipelines I manage ... and till it's fixed in the upstream I've implemented a custom step dynamicMatrix . That custom step mimics Jenkins declarative matrix but with better visualization and extra customization options. It makes it easier to run a parallel matrix with different combinations and groups. Here is an example: stage( 'single_matrix' ) { steps { dynamicMatrix([ failFast: false , axes: [ PLATFORM: [ 'linux' , 'mac' , 'windows' ], BROWSER: [ 'chrome' , 'firefox' , 'safari' ] ], actions: { stage( "${PLATFORM}_${BROWSER}" ) { sh 'echo ${PLATFORM} - ${BROWSER}' sh 'echo ${MATRIX_STAGE_NAME}' } } ]) } } It also has 2 other varieties to work with different combinations and groups. If you have different axes combinations but the same stages then dynamicMatrixMultiCombinations could be used for that case. stage( 'multi_combinations_matrix' ) { steps { dynamicMatrixMultiCombinations([ failFast: false , axes: [ [ PLATFORM: [ 'linux' , 'mac' , 'windows' ], BROWSER: [ 'chrome' , 'firefox' ] ], [ PLATFORM: [ 'mac' , 'windows' ] BROWSER: [ 'safari' ], ], ], actions: { stage( "${PLATFORM}_${BROWSER}" ) { sh 'echo ${PLATFORM}, ${BROWSER}' sh 'echo ${MATRIX_STAGE_NAME}' } } ]) } } If you have different groups or different stages that should run in parallel at the same time, then dynamicMatrixMultiGroups could be used for that case. stage( 'multi_groups_matrix' ) { steps { dynamicMatrixMultiGroups([ [ failFast: false , axes: [ PLATFORM: [ 'windows' ] BROWSER: [ 'edge' , 'safari' ], ], actions: { stage( "${PLATFORM}_${BROWSER}" ) { bat 'echo this is a Windows command' } } ], [ failFast: false , axes: [ PLATFORM: [ 'linux' , 'mac' ], BROWSER: [ 'chrome' , 'firefox' ] ], actions: { stage( "${PLATFORM}_${BROWSER}" ) { sh 'echo this is a Unix-like command' } } ] ]) } } Enjoy

          aabouzaid looks the library still has issues 

           
          08:14:57 org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed:08:14:57 /jenkins/rocmhead-jenkins/jobs/maintenance/jobs/pipeline-syntax/builds/32/libs/c33ce574fce29a3d6c07d8cfea025972d7308787b91e72d78da8f772c7b63863/vars/dynamicMatrix.groovy: 109: expecting '}', found '' @ line 109, column 10.08:14:57 echo "${stepName} Invalid parameter '${parameter}'. Available parameter are: ${availableParameters}"

          omkar143 omkar kakarparthi added a comment - aabouzaid  looks the library still has issues    08:14:57 org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed: 08:14:57 /jenkins/rocmhead-jenkins/jobs/maintenance/jobs/pipeline-syntax/builds/32/libs/c33ce574fce29a3d6c07d8cfea025972d7308787b91e72d78da8f772c7b63863/vars/dynamicMatrix.groovy: 109: expecting '}', found '' @ line 109, column 10. 08:14:57 echo "${stepName} Invalid parameter '${parameter}'. Available parameter are: ${availableParameters}"

          omkar143 I've fixed it
          Please give it another try now.

          aabouzaid Ahmed AbouZaid added a comment - omkar143 I've fixed it Please give it another try now.
          kohenchia Kohen Chia added a comment -

          Upvoted. Please add this feature. I'm begging you.

          kohenchia Kohen Chia added a comment - Upvoted. Please add this feature. I'm begging you.
          risingfish Zachary Feuerborn made changes -
          Comment [ Would love to to see this added! ]
          nehamittal3010 Neha Mittal added a comment -

          Hi bitwiseman ,

          Are we getting  fix with this issue as we are blocked and cannot use this. Please let us know when is ETA planned.

          Regards,

          Neha. 

          nehamittal3010 Neha Mittal added a comment - Hi bitwiseman , Are we getting  fix with this issue as we are blocked and cannot use this. Please let us know when is ETA planned. Regards, Neha. 
          pfuntner John Pfuntner added a comment - - edited

          I'm dealing with a project with several Jenkins scripts - some of them appear to accept a variable in the stage.  I don't understand why but it seems that if the stage statement is inside a script block, you can specify a variable in the stage name:

          stages {
            stage('static name') {
              steps {
                script {
                  stage("dynamic name with ${ENV.name}") {
                    .
                    .
                    .
                  }
                }
              }
            }
          }
          
          pfuntner John Pfuntner added a comment - - edited I'm dealing with a project with several Jenkins scripts - some of them appear to accept a variable in the stage.  I don't understand why but it seems that if the stage statement is inside a script block, you can specify a variable in the stage name: stages { stage('static name') { steps { script { stage("dynamic name with ${ENV.name}") { . . . } } } } }
          rittneje Jesse Rittner added a comment -

          pfuntner What you are seeing is declarative vs. scripted pipelines. Generally declarative pipelines have more constraints, but you can use a script block as an escape hatch to go from declarative to scripted. The stage block in a declarative pipeline and the stage function in a scripted pipeline are analogous to each other but are not entirely equivalent.

          rittneje Jesse Rittner added a comment - pfuntner What you are seeing is declarative vs. scripted pipelines. Generally declarative pipelines have more constraints, but you can use a script block as an escape hatch to go from declarative to scripted. The stage block in a declarative pipeline and the stage function in a scripted pipeline are analogous to each other but are not entirely equivalent.

          People

            bitwiseman Liam Newman
            rittneje Jesse Rittner
            Votes:
            50 Vote for this issue
            Watchers:
            47 Start watching this issue

            Dates

              Created:
              Updated: