Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48980

Add ability to hide skipped stages/ignore stages altogether (support DRY)

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I've seen this mentioned several times in other tickets and websites, but I was unable to find a ticket that specifically addresses it.  (So hopefully I didn't just simply miss it)

       

      Problem: Given several different repositories, with different branches that do different stages, there is no simple way to modify a pipeline based on some configuration in order to keep declarative pipelines DRY.

       

      Simple example:

      • Branch 1 stages: A -> B -> C
      • Branch 2 stages: A -> B -> D
      • Branch 3 stages: A -> B -> C -> D

      Currently, that requires defining 3 different pipelines, which means redundant code exists in different files and in turn increases the potential for errors to occur.  In more complex situations, the ability to review changes to all the different combinations of pipelines is simply not possible.

      Using "when" in stages is a workaround to this problem, but that causes confusion for developers who see a "Deploy" stage in a branch build that should not be deploying, for example.  Also, for complicated scenarios, there could be many skipped stages that will always be skipped and simply add unnecessary verbosity to the visualization.

      The other workaround is to use scripted pipelines, but many of the visualization features provided by using declarative don't exist or don't work correctly all the time.  Also, since the list of stages are not determined up-front, the stage view always resets during a build.

       

      There are 2 solutions that I can think of:

      1. Provide the ability to modify the "when" behavior, so that blue ocean visualization completely ignores it.  The problem with this approach is that the "when" condition, I believe, is evaluated when executing the stage, but the list of stages is determined in the beginning at the time the pipeline is evaluated.  It's possible that this could actually be implemented though to simply hide the stage in the visualizations rather than ignore it completely.  For example: (This can probably be greatly improved upon, but it shows the point):
        config = [
            skipStage2 : true
        ]
        
        pipeline {
            agent any
            stages {
                stage("Stage 1") { ... }
                stage("Stage 2") {
                    when {
                         showOnlyWhen {
                             expression { return !config.skipStage2 } // evaluated after the stage is executed
                         }
                     }
                    ...
                }
            }
        }
      1.  Add another directive that is evaluated up-front to determine the stage list.  If the list of stages changes from build to build, it would be no different than if the entire declarative pipeline changed and different stages existed.  For example:
        config = [
            skipStage2 : true, // irrelevant when existsStage2 = false
            existsStage2 : false
        ]
        
        pipeline {
            agent any
            stages {
                stage("Stage 1") { ... }
                stage("Stage 2") {
                    when {
                        expression { return !config.skipStage2 } // evaluated after the stage is executed
                    }
                    exists {
                        expression { return config.existsStage2 } // Evaluated up-front to determine if this stage is even run
                    }
                    ...
                }
            }
        }

         

       

        Attachments

          Activity

          Hide
          abayer Andrew Bayer added a comment -

          I think the best place to discuss/address this would be in Blue Ocean - so I'm moving this to that component. I don't think we want to change when behavior in Declarative itself, so visualization changes are the best option. Declarative already provides information on why a stage was skipped (i..e, skipped due to earlier failure, skipped due to when condition, etc) that Blue Ocean can access.

          Show
          abayer Andrew Bayer added a comment - I think the best place to discuss/address this would be in Blue Ocean - so I'm moving this to that component. I don't think we want to change when behavior in Declarative itself, so visualization changes are the best option. Declarative already provides information on why a stage was skipped (i..e, skipped due to earlier failure, skipped due to when condition, etc) that Blue Ocean can access.
          Hide
          pixman20 pixman20 added a comment - - edited

          Andrew Bayer, Is there already information provided by declarative so that Blue Ocean would know when to suppress showing the stage (rather than just marking it skipped)?
          Also, I was thinking that it would need to be added to declarative in order to hide the stages from the stage view on the main job page.

          Show
          pixman20 pixman20 added a comment - - edited Andrew Bayer , Is there already information provided by declarative so that Blue Ocean would know when to suppress showing the stage (rather than just marking it skipped)? Also, I was thinking that it would need to be added to declarative in order to hide the stages from the stage view on the main job page.
          Hide
          michaelneale Michael Neale added a comment -

          The problem is - before skipped were show, people would complain about not showing them (assuming going with the single pipeline). Adding more config/complexity for this case.. maybe the multiple pipeline is the way if the developers really don't want to know about deploy? not keen on this as it is at all based on past history. 

          Show
          michaelneale Michael Neale added a comment - The problem is - before skipped were show, people would complain about not showing them (assuming going with the single pipeline). Adding more config/complexity for this case.. maybe the multiple pipeline is the way if the developers really don't want to know about deploy? not keen on this as it is at all based on past history. 
          Hide
          erthad Timur Batyrshin added a comment - - edited

          Andrew Bayer, this could be done as an additional modificator to the pipeline, something like this:

          pipeline {
            stages {
              stage('hide me') {
                when { expression { false } }
                hide true
                steps { ... }
              }
              stage('skip me but show me') {
                when { expression { false } }
          //    hide false // default
                steps { ... }
              }
              stage('run me and show me') {
                when { expression { true } }
                hide true // hiding doesn't make sense when stage is not skipped
                steps { ... }
              }
            }
          }
          
          

          My use case is a pipeline library consisting of dozen steps which is shared across dozens different projects.

          Some steps are skipped based on branch name and showing them up is ok.

          Some steps are skipped based on project type and I would want to hide them.

           

          I could keep a copy of the pipeline for every project type but that doesn't look like very much viable option as the pipeline is 700+ LOC and I don't see much options of reducing it below 400-500 LOC which is quite much too.

          Copying becomes error prone and increases maintenance when doing changes to the pipeline stages common for all pipelines.

          So I see 2 viable options for improvement:

          Any thoughts?

           

           

          Show
          erthad Timur Batyrshin added a comment - - edited Andrew Bayer , this could be done as an additional modificator to the pipeline, something like this: pipeline { stages { stage( 'hide me' ) { when { expression { false } } hide true steps { ... } } stage( 'skip me but show me' ) { when { expression { false } } // hide false // default steps { ... } } stage( 'run me and show me' ) { when { expression { true } } hide true // hiding doesn't make sense when stage is not skipped steps { ... } } } } My use case is a pipeline library consisting of dozen steps which is shared across dozens different projects. Some steps are skipped based on branch name and showing them up is ok. Some steps are skipped based on project type and I would want to hide them.   I could keep a copy of the pipeline for every project type but that doesn't look like very much viable option as the pipeline is 700+ LOC and I don't see much options of reducing it below 400-500 LOC which is quite much too. Copying becomes error prone and increases maintenance when doing changes to the pipeline stages common for all pipelines. So I see 2 viable options for improvement: add the ability to hide some changes add the ability to offload stages to external files ( https://issues.jenkins-ci.org/browse/JENKINS-42224 ) Any thoughts?    
          Hide
          tomar99 Romain Marteau added a comment -

          I would also find it very useful. I like the syntax proposed by Timur Batyrshin.

          My use case is the same. A default pipeline for many projects. They build the code the same way, but some projects are only pushing artifacts, other projects are building docker image.

          Show
          tomar99 Romain Marteau added a comment - I would also find it very useful. I like the syntax proposed by Timur Batyrshin . My use case is the same. A default pipeline for many projects. They build the code the same way, but some projects are only pushing artifacts, other projects are building docker image.
          Hide
          miketd Mike Dukes added a comment -

          +1 for this request.

          My scenario is that we have different pipeline behaviour based on our branching strategy, and visually it makes developers confused when they see more steps, especially when they see "Production".

          Example:

          • master branch pipeline: Build > Test > Deploy to X > promote to Release branch
          • release branch pipeline: Build > Test > Deploy to Prod
          • whats seen: Build > Test > Deploy to X > Promote to Release > Deploy to Prod even though not all jobs are run

          I like Build and Test to be written once so we keep consistency, so control this with when statements. In my case I'm happy to hide all steps that aren't run, so the simplest feature for me would be a tickbox in the job configuration e.g. "Blue Ocean hide jobs which aren't run", but understand this might not be sufficient based on the other comments! So would be happy with whatever ability here.

          Show
          miketd Mike Dukes added a comment - +1 for this request. My scenario is that we have different pipeline behaviour based on our branching strategy, and visually it makes developers confused when they see more steps, especially when they see "Production". Example: master branch pipeline: Build > Test > Deploy to X > promote to Release branch release branch pipeline: Build > Test > Deploy to Prod whats seen: Build > Test > Deploy to X > Promote to Release > Deploy to Prod even though not all jobs are run I like Build and Test to be written once so we keep consistency, so control this with when statements. In my case I'm happy to hide all steps that aren't run, so the simplest feature for me would be a tickbox in the job configuration e.g. "Blue Ocean hide jobs which aren't run", but understand this might not be sufficient based on the other comments! So would be happy with whatever ability here.

            People

            Assignee:
            olamy Olivier Lamy
            Reporter:
            pixman20 pixman20
            Votes:
            21 Vote for this issue
            Watchers:
            22 Start watching this issue

              Dates

              Created:
              Updated: