Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42224

Need ability to create reusable chunks of Declarative Pipeline

      There is no way to write reusable chunks of Declarative Pipeline configuration.
      Jenkinsfile has no facility for this, and
      shared libraries support only scripted groovy (classes, steps, globals).

          [JENKINS-42224] Need ability to create reusable chunks of Declarative Pipeline

          Andrew Bayer added a comment -

          This is an interesting and legitimately challenging problem - needs thinking.

          Andrew Bayer added a comment - This is an interesting and legitimately challenging problem - needs thinking.

          Andrew Bayer added a comment -

          Ok, been thinking about this some. =) I think we could avoid some of the hassles of inserting GroovyShellDecoratorImpl all over the place for compile-time validating etc by instead putting the Declarative chunks in the resources directory of libraries and using the libraryResource step to get them as strings - then we can use the same logic as in the validateDeclarativePipeline step to do the validation without worrying about breaking other things outside of the Declarative scope.

          That may or may not make sense. =)

          The next challenge is figuring out how to insert the chunk into the runtime model. Still working on that.

          Andrew Bayer added a comment - Ok, been thinking about this some. =) I think we could avoid some of the hassles of inserting GroovyShellDecoratorImpl all over the place for compile-time validating etc by instead putting the Declarative chunks in the resources directory of libraries and using the libraryResource step to get them as strings - then we can use the same logic as in the validateDeclarativePipeline step to do the validation without worrying about breaking other things outside of the Declarative scope. That may or may not make sense. =) The next challenge is figuring out how to insert the chunk into the runtime model. Still working on that.

          Actually we can put some declarative code inside shared libraries, please don't break what is actually working.

           

          Thanks,

          Nico

          Nico Navarrete added a comment - Actually we can put some declarative code inside shared libraries, please don't break what is actually working.   Thanks, Nico

          Liam Newman added a comment -

          nicolasus
          Could you describe what you have working or provide a link? That would be very useful information.

          Liam Newman added a comment - nicolasus Could you describe what you have working or provide a link? That would be very useful information.

          Andrew Bayer added a comment -

          So, FYI, I'm beginning to think about some aspects of this - first and foremost the ability to programmatically add stages to a Declarative Pipeline via something coming from a shared library. There are a bunch of technical challenges/questions I need to figure out before this could possibly become reality. We do a lot of things to the pipeline block and its contents at compile-time, so both, say, explicit stage blocks or programmatic generation of a stage block in a shared library would present some real difficulties while using the Declarative syntax.

          What I'm vaguely kicking around now is creating an API with a new syntax specifically for use in shared libraries, which would allow adding stages (at least to start) programmatically without having to deal with the special hells of compile-time transformation. So...we'll see.

          Andrew Bayer added a comment - So, FYI, I'm beginning to think about some aspects of this - first and foremost the ability to programmatically add stages to a Declarative Pipeline via something coming from a shared library. There are a bunch of technical challenges/questions I need to figure out before this could possibly become reality. We do a lot of things to the pipeline block and its contents at compile-time, so both, say, explicit stage blocks or programmatic generation of a stage block in a shared library would present some real difficulties while using the Declarative syntax. What I'm vaguely kicking around now is creating an API with a new syntax specifically for use in shared libraries, which would allow adding stages (at least to start) programmatically without having to deal with the special hells of compile-time transformation. So...we'll see.

          Tim Downey added a comment -

          Hi abayer, thanks for looking at this.  Not sure if this helps provide context or not, but here's the use case that I'm dealing with.

          Basically, declarative pipeline is great if you have to define a single (or few) build.  In my case, I'm basically trying to set some standards that will apply across dozens or maybe hundreds of builds where I don't want to copy/paste 100+ lines of a declarative Jenkinsfile across projects.

          I'm looking for stuff like this:

          def config = [foo: 'foo", bar: 'bar']
          buildNpmApp(config)

          or

          def config = [foo: 'foo", bar: 'bar']
          buildSpringBootAppWithDocker(config)
          

          Now, I know that I can do that using global pipeline libraries now, but I'd like to be able to use the clean, simplified nature of declarative in the global pipeline library.  The notation is hard to follow for folks when using the more raw nature of the global pipeline libs (even when using var)

          So, I'm sort of thinking that in my case, even if I could stuff a whole pipeline {...} inside of a shared lib, I'd be in good shape.  Being able to mix and match steps and stages would be even better, but a whole pipeline would probably suffice.

          Tim

          Tim Downey added a comment - Hi abayer , thanks for looking at this.  Not sure if this helps provide context or not, but here's the use case that I'm dealing with. Basically, declarative pipeline is great if you have to define a single (or few) build.  In my case, I'm basically trying to set some standards that will apply across dozens or maybe hundreds of builds where I don't want to copy/paste 100+ lines of a declarative Jenkinsfile across projects. I'm looking for stuff like this: def config = [foo: 'foo", bar: ' bar'] buildNpmApp(config) or def config = [foo: 'foo", bar: ' bar'] buildSpringBootAppWithDocker(config) Now, I know that I can do that using global pipeline libraries now, but I'd like to be able to use the clean, simplified nature of declarative in the global pipeline library.  The notation is hard to follow for folks when using the more raw nature of the global pipeline libs (even when using var) So, I'm sort of thinking that in my case, even if I could stuff a whole pipeline {...} inside of a shared lib, I'd be in good shape.  Being able to mix and match steps and stages would be even better, but a whole pipeline would probably suffice. Tim

          Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.

           

          pipeline {
            libraries {
              lib("mylib@master")
            }
          
            ...
          
            stages {
              stage('Checkout') {
                steps {
                  gitCheckout()
                }
              }
          
              stage('Build & Verify') {
                when {
                   anyOf {
                     branch 'master'
                     branch 'PR**'
                     branch 'feature/**'
                  }
                }
                steps {
                   mavenVerify('-Pprod')
                }
              }
          
              stage('Build & Deploy') {
                when {
                  branch 'develop'
                }
                steps {
                  mavenDeploy('-Pprod')
                }
              }
          
              stage('Build & Release') {
                when {
                  branch 'release/**'
                }
                steps {
                  mavenRelease('-Pprod,release')
                }
              }
          
              stage('Sonar Analysis') {
                when {
                  anyOf {
                    branch 'master'
                    branch 'develop'
                    branch 'PR**'
                  }
                }
                steps {
                  sonar()
                }
              }
          
              stage("Sonar Quality Gate"){
                agent none
                when {
                  anyOf {
                    branch 'master'
                    branch 'develop'
                  }
                }
                steps {
                  timeout(time: 1, unit: 'HOURS') {
                    script {
                      def qg = waitForQualityGate()
                      echo "Sonar quality gate status: ${qg.status}"
                    }
                  }
                }  
              }
          
              stage('Post Analysis') {
                steps {
                  script {
                    echo "Collecting data..."
                  }
                }
                post {
                  always {
                    script {
                      collectData()
                    }
                  }
                }
              }
            }
          
            post {
              changed {
                script {
                  sendBuildStatusChangedNotifications()
                }
              }
            }
          }
          

           

          In an ideal world, it would become something like this.

           

          pipeline {
            libraries {
              lib("mylib@master")
            }
          
            ...
          
            stages {
              stage gitCheckout()
              stage mavenVerify('-Pprod')
              stage mavenDeploy('-Pprod')
              stage mavenRelease('-Pprod,release')
              stage sonar()
              stage sonarQualityGate()
              stage collectData()
            }
          
            // or better :)
            stages myCustomMavenBuild(config)
            stages myCustomNpmBuild(config)
          
            post sendBuildStatusChangedNotifications()
          }
          

           

           

          Peter Leibiger added a comment - Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage( 'Checkout' ) { steps { gitCheckout() } } stage( 'Build & Verify' ) { when { anyOf { branch 'master' branch 'PR**' branch 'feature/**' } } steps { mavenVerify( '-Pprod' ) } } stage( 'Build & Deploy' ) { when { branch 'develop' } steps { mavenDeploy( '-Pprod' ) } } stage( 'Build & Release' ) { when { branch 'release/**' } steps { mavenRelease( '-Pprod,release' ) } } stage( 'Sonar Analysis' ) { when { anyOf { branch 'master' branch 'develop' branch 'PR**' } } steps { sonar() } } stage( "Sonar Quality Gate" ){ agent none when { anyOf { branch 'master' branch 'develop' } } steps { timeout(time: 1, unit: 'HOURS' ) { script { def qg = waitForQualityGate() echo "Sonar quality gate status: ${qg.status}" } } } } stage( 'Post Analysis' ) { steps { script { echo "Collecting data..." } } post { always { script { collectData() } } } } } post { changed { script { sendBuildStatusChangedNotifications() } } } }   In an ideal world, it would become something like this.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage gitCheckout() stage mavenVerify( '-Pprod' ) stage mavenDeploy( '-Pprod' ) stage mavenRelease( '-Pprod,release' ) stage sonar() stage sonarQualityGate() stage collectData() } // or better :) stages myCustomMavenBuild(config) stages myCustomNpmBuild(config) post sendBuildStatusChangedNotifications() }    

          Hi, 

           

          actually we're reusing bits of declarative pipelines doing like this:

           

          • In a shared library, inside the vars dir web declare the common code, ReleaseBoot.groovy:
          import x.y.z.jenkins.spring.boot.DeployConfig;
          
          def call(body) {
          
              def mail = new x.y.z.jenkins.notifications.MailSender(this)
          
              // evaluate the body block, and collect configuration into the object
              def config = [:]
              body.resolveStrategy = Closure.DELEGATE_FIRST
              body.delegate = config
              body()
          
              pipeline {
                  options{   
                      skipDefaultCheckout()   
                  }
                  agent none
                  // node
                  stages {
                      stage('Input Parameters'){
                          agent any
                          steps{
                              timeout(time:1, unit:'DAYS') {
                                  script {
                                      def inputVar = input id: 'RELEASE_INPUT', message: '¿Release new version?', ok: 'Yes', parameters: [
                                          string(defaultValue: '', description: 'Release version:', name: 'release'),
                                          string(defaultValue: '', description: 'Development version:', name: 'dev'),
                                          string(defaultValue: '', description: 'Commit:', name: 'commit')
                                      ], submitterParameter: 'user'
                                      env.RELEASE_VERSION = inputVar['release']
                                      env.BUILD_COMMIT     = inputVar['commit']
                                      env.DEV_VERSION = inputVar['dev']
                                  }
                              }
                          }
                      }
                      stage('scm') {
                          agent any
                          steps {
                              echo "Skip scm is ${config.SKIP.SCM}"
                              deleteDir()
                              checkout scm
                              script {
                                  ........
                              }
                              
                              
                          }
                      }
          
                      
                      stage('Release') {
                          agent any
                          steps {
                               withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: "${env.GIT_CREDENTIALS}",
                                      usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                                  ............
                                  stash name:"jars", includes:"**/target/*${env.RELEASE_VERSION}*.jar"
                                  
                                  script{........
                                  }
                              }
                              
                          
                          
                              
                          }
                      }
                  }
                  post {
                      failure {
                          script{ mail.send(config.MAILS)}
                      }
                      unstable {
                          script{ mail.send(config.MAILS) }
                      }
                  }
              }
          }
          • In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy

           

          def call(body) {
          
          
              env.DEPLOY_ENV            = "test"
              
              // evaluate the body block, and collect configuration into the object
              def config = [:]
              body.resolveStrategy = Closure.DELEGATE_FIRST
              body.delegate = config
              body()
              
              if(env.JOB_NAME.contains('deploy')) {
                  ........
              }else if(env.JOB_NAME.contains('release')) {
                  // here we call the previous defined 'shared' pipeline
                  ReleaseBoot {
                      MAILS     = config.MAILS
                      DEPLOYMENTS = config.DEPLOYMENTS
                  }
              } else {
                 .............
              }
              
          }
          
          • And  in each project's Jekinsfile:
          #!groovy
          @Library('xyz-pipelines') _
          def OPTIONAL_MAILS = []
          def deployTest     = [user:"targetuser", host:"host", credentialsId:"jboss-dev",
                             // config values
                             checkUrl:"http://xyz:8096/health", infoUrl:"http://xyz:8096/info"]
          // defined in shared library vars dir            
          BootPipeline{
              MAILS = OPTIONAL_MAILS
              DEPLOYMENTS = deployTest
          }        
             
          

          We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working

           

          Please keep this working

           

           NIco

           

           

           

           

          Nico Navarrete added a comment - Hi,    actually we're reusing bits of declarative pipelines doing like this:   In a shared library, inside the vars dir web declare the common code, ReleaseBoot .groovy: import x.y.z.jenkins.spring.boot.DeployConfig; def call(body) {     def mail = new x.y.z.jenkins.notifications.MailSender( this )      // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()     pipeline {         options{                skipDefaultCheckout()            }         agent none          // node         stages {             stage( 'Input Parameters' ){                 agent any                 steps{                     timeout(time:1, unit: 'DAYS' ) {                         script {                             def inputVar = input id: 'RELEASE_INPUT' , message: '¿Release new version?' , ok: 'Yes' , parameters: [                                 string(defaultValue: '', description: ' Release version: ', name: ' release'),                                 string(defaultValue: '', description: ' Development version: ', name: ' dev'),                                 string(defaultValue: '', description: ' Commit: ', name: ' commit')                             ], submitterParameter: 'user'                             env.RELEASE_VERSION = inputVar[ 'release' ]                             env.BUILD_COMMIT     = inputVar[ 'commit' ]                             env.DEV_VERSION = inputVar[ 'dev' ]                         }                     }                 }             }             stage( 'scm' ) {                 agent any                 steps {                     echo "Skip scm is ${config.SKIP.SCM}"                     deleteDir()                     checkout scm                     script {                         ........                     }                                                           }             }                          stage( 'Release' ) {                 agent any                 steps {                      withCredentials([[$class: 'UsernamePasswordMultiBinding' , credentialsId: "${env.GIT_CREDENTIALS}" ,                             usernameVariable: 'USERNAME' , passwordVariable: 'PASSWORD' ]]) {                         ............                         stash name: "jars" , includes: "**/target/*${env.RELEASE_VERSION}*.jar"                                                  script{........                         }                     }                                                                                             }             }         }         post {             failure {                 script{ mail.send(config.MAILS)}             }             unstable {                 script{ mail.send(config.MAILS) }             }         }     } } In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy   def call(body) {     env.DEPLOY_ENV            = "test"           // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()           if (env.JOB_NAME.contains( 'deploy' )) {         ........     } else if (env.JOB_NAME.contains( 'release' )) { // here we call the previous defined 'shared' pipeline         ReleaseBoot {             MAILS     = config.MAILS             DEPLOYMENTS = config.DEPLOYMENTS         }     } else {        .............     }      } And  in each project's Jekinsfile: #!groovy @Library( 'xyz-pipelines' ) _ def OPTIONAL_MAILS = [] def deployTest     = [user: "targetuser" , host: "host" , credentialsId: "jboss-dev" ,                    // config values                    checkUrl: "http: //xyz:8096/health" , infoUrl: "http://xyz:8096/info" ] // defined in shared library vars dir             BootPipeline{     MAILS = OPTIONAL_MAILS     DEPLOYMENTS = deployTest }             We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working   Please keep this working    NIco        

          Andrew Bayer added a comment -

          nicolasus - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547, FYI.

          Andrew Bayer added a comment - nicolasus - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547 , FYI.

          Felix Neumann added a comment - - edited

          abayer: -JENKINS-46547- was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp.

          Therefore I'd like to see some kind of composable pipelines, where one could have a basic

          pipeline("my-shared-pipeline") {
            agent { /* ... */ }
            tools { /* ... */ }
            options { /* ... */ }
            stages {
              stage('Compile and Test') { /* ... */ }
              stage('Deploy') { /* ... */ }
            }
            post {
              failure { /* ... */ }
              alway { /* ... */ }
            }
          }
          

          and then extend it, using some mechanism to make up the final order of stages:

          pipeline(extends: "my-shared-pipeline") {
            stages {
              stage('Integration Test', after: 'Compile and Test') { /* ... */ }
            }
          }
          

          This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds.

          Btw., loading chunks from resources (as proposed above) also sounds like a good solution.

          Just my 5 Cents – thanks for making all this possible!

          Felix Neumann added a comment - - edited abayer : - JENKINS-46547 - was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp. Therefore I'd like to see some kind of composable pipelines, where one could have a basic pipeline( "my-shared-pipeline" ) { agent { /* ... */ } tools { /* ... */ } options { /* ... */ } stages { stage( 'Compile and Test' ) { /* ... */ } stage( 'Deploy' ) { /* ... */ } } post { failure { /* ... */ } alway { /* ... */ } } } and then extend it, using some mechanism to make up the final order of stages: pipeline( extends : "my-shared-pipeline" ) { stages { stage( 'Integration Test' , after: 'Compile and Test' ) { /* ... */ } } } This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds. Btw., loading chunks from resources (as proposed above) also sounds like a good solution. Just my 5 Cents – thanks for making all this possible!

          I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548 for defining reusable stages, maybe it is interesting for you as well.

          Tobias Larscheid added a comment - I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548  for defining reusable stages, maybe it is interesting for you as well.

          I would like this as well. But also, it seems like we can make some of those blocks one line, and we could use a preprocessor, if necessary. Just starting to get into this, but seems like groovy should support this already.

          Nick Roosevelt added a comment - I would like this as well. But also, it seems like we can make some of those blocks one line, and we could use a preprocessor, if necessary. Just starting to get into this, but seems like groovy should support this already.

          Bassam Khouri added a comment -

          I would like this as well. In my case, I have a single pipeline with multiple parallel stages. Each parallel stages have 90%+ of identical code and I would like to programatically defines different stages.

          for example, my pipeline looks like this

          pipeline {
              stages {
                  stage("stage1") {
                      agent {
                          // different agent based on stage
                      }
                      environment (
                          BUILD_OPTIONS = "different build options per stage"
                      )
                      steps {
                          deleteDir()
                          echo 'Building..'
                          build(BUILD_OPTIONS)
                      }
                      post {
                          // has some slight differences here depending on the stage
                      }
                  }
                  stage("stage2") {
                      agent {
                          // different agent based on stage
                      }
                      environment (
                          BUILD_OPTIONS = "another  build options per stage"
                      )
                      steps {
                          deleteDir()
                          echo 'Building..'
                          build(BUILD_OPTIONS)
                      }
                      post {
                          // has some slight differences here depending on the stage
                      }
                  }
                  ...
                  stage("stageN") {
                      agent {
                          // different agent based on stage
                      }
                      environment (
                          BUILD_OPTIONS = "build options for stage N"
                      )
                      steps {
                          deleteDir()
                          echo 'Building..'
                          build(BUILD_OPTIONS)
                      }
                      post {
                          // has some slight differences here depending on the stage
                      }
                  }
          
              }
          }
          

          Personally, I would like to do something like this

          def myStage(Map args) {
              // define the "dynamic" stage that uses the 'args' parameter
              if (args.specialAgent) {
                  agent { docker {...} }
              } else {
                  agent { label "SpecialAgent" } 
              }
              environment {
                 BUILD_OPTIONS = " ".join(args.buildOptions)
              }
              steps {
                 ...
              }
              post {
                  always {
                      if {args.UploadLogs) {
                          // call an "action"
                      }
                      // more actions
                  }
              }
          }
          
          pipeline {
              stages {
                  stage1 myStage([buildOptions: ["build", "option", "stage1"], arg1: "something"])
                  stage2 myStage([buildOptions: ["stage2"], arg1: "something"])
                  ...
                  stageN myStage([buildOptions: [STAGE_NAME], arg1: "something"0)
              }
          }
          

          Bassam Khouri added a comment - I would like this as well. In my case, I have a single pipeline with multiple parallel stages. Each parallel stages have 90%+ of identical code and I would like to programatically defines different stages. for example, my pipeline looks like this pipeline { stages { stage( "stage1" ) { agent { // different agent based on stage } environment ( BUILD_OPTIONS = "different build options per stage" ) steps { deleteDir() echo 'Building..' build(BUILD_OPTIONS) } post { // has some slight differences here depending on the stage } } stage( "stage2" ) { agent { // different agent based on stage } environment ( BUILD_OPTIONS = "another build options per stage" ) steps { deleteDir() echo 'Building..' build(BUILD_OPTIONS) } post { // has some slight differences here depending on the stage } } ... stage( "stageN" ) { agent { // different agent based on stage } environment ( BUILD_OPTIONS = "build options for stage N" ) steps { deleteDir() echo 'Building..' build(BUILD_OPTIONS) } post { // has some slight differences here depending on the stage } } } } Personally, I would like to do something like this def myStage(Map args) { // define the "dynamic" stage that uses the 'args' parameter if (args.specialAgent) { agent { docker {...} } } else { agent { label "SpecialAgent" } } environment { BUILD_OPTIONS = " " .join(args.buildOptions) } steps { ... } post { always { if {args.UploadLogs) { // call an "action" } // more actions } } } pipeline { stages { stage1 myStage([buildOptions: [ "build" , "option" , "stage1" ], arg1: "something" ]) stage2 myStage([buildOptions: [ "stage2" ], arg1: "something" ]) ... stageN myStage([buildOptions: [STAGE_NAME], arg1: "something" 0) } }

          Any update here?

          Michael Hüttermann added a comment - Any update here?

          Jean-Luc Pé added a comment -

          Please, could we know if this ticket is actually treated ?

          Jean-Luc Pé added a comment - Please, could we know if this ticket is actually treated ?

            Unassigned Unassigned
            bitwiseman Liam Newman
            Votes:
            89 Vote for this issue
            Watchers:
            102 Start watching this issue

              Created:
              Updated: