Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42224

Need ability to create reusable chunks of Declarative Pipeline

      There is no way to write reusable chunks of Declarative Pipeline configuration.
      Jenkinsfile has no facility for this, and
      shared libraries support only scripted groovy (classes, steps, globals).

          [JENKINS-42224] Need ability to create reusable chunks of Declarative Pipeline

          Tim Downey added a comment -

          Hi abayer, thanks for looking at this.  Not sure if this helps provide context or not, but here's the use case that I'm dealing with.

          Basically, declarative pipeline is great if you have to define a single (or few) build.  In my case, I'm basically trying to set some standards that will apply across dozens or maybe hundreds of builds where I don't want to copy/paste 100+ lines of a declarative Jenkinsfile across projects.

          I'm looking for stuff like this:

          def config = [foo: 'foo", bar: 'bar']
          buildNpmApp(config)

          or

          def config = [foo: 'foo", bar: 'bar']
          buildSpringBootAppWithDocker(config)
          

          Now, I know that I can do that using global pipeline libraries now, but I'd like to be able to use the clean, simplified nature of declarative in the global pipeline library.  The notation is hard to follow for folks when using the more raw nature of the global pipeline libs (even when using var)

          So, I'm sort of thinking that in my case, even if I could stuff a whole pipeline {...} inside of a shared lib, I'd be in good shape.  Being able to mix and match steps and stages would be even better, but a whole pipeline would probably suffice.

          Tim

          Tim Downey added a comment - Hi abayer , thanks for looking at this.  Not sure if this helps provide context or not, but here's the use case that I'm dealing with. Basically, declarative pipeline is great if you have to define a single (or few) build.  In my case, I'm basically trying to set some standards that will apply across dozens or maybe hundreds of builds where I don't want to copy/paste 100+ lines of a declarative Jenkinsfile across projects. I'm looking for stuff like this: def config = [foo: 'foo", bar: ' bar'] buildNpmApp(config) or def config = [foo: 'foo", bar: ' bar'] buildSpringBootAppWithDocker(config) Now, I know that I can do that using global pipeline libraries now, but I'd like to be able to use the clean, simplified nature of declarative in the global pipeline library.  The notation is hard to follow for folks when using the more raw nature of the global pipeline libs (even when using var) So, I'm sort of thinking that in my case, even if I could stuff a whole pipeline {...} inside of a shared lib, I'd be in good shape.  Being able to mix and match steps and stages would be even better, but a whole pipeline would probably suffice. Tim

          Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.

           

          pipeline {
            libraries {
              lib("mylib@master")
            }
          
            ...
          
            stages {
              stage('Checkout') {
                steps {
                  gitCheckout()
                }
              }
          
              stage('Build & Verify') {
                when {
                   anyOf {
                     branch 'master'
                     branch 'PR**'
                     branch 'feature/**'
                  }
                }
                steps {
                   mavenVerify('-Pprod')
                }
              }
          
              stage('Build & Deploy') {
                when {
                  branch 'develop'
                }
                steps {
                  mavenDeploy('-Pprod')
                }
              }
          
              stage('Build & Release') {
                when {
                  branch 'release/**'
                }
                steps {
                  mavenRelease('-Pprod,release')
                }
              }
          
              stage('Sonar Analysis') {
                when {
                  anyOf {
                    branch 'master'
                    branch 'develop'
                    branch 'PR**'
                  }
                }
                steps {
                  sonar()
                }
              }
          
              stage("Sonar Quality Gate"){
                agent none
                when {
                  anyOf {
                    branch 'master'
                    branch 'develop'
                  }
                }
                steps {
                  timeout(time: 1, unit: 'HOURS') {
                    script {
                      def qg = waitForQualityGate()
                      echo "Sonar quality gate status: ${qg.status}"
                    }
                  }
                }  
              }
          
              stage('Post Analysis') {
                steps {
                  script {
                    echo "Collecting data..."
                  }
                }
                post {
                  always {
                    script {
                      collectData()
                    }
                  }
                }
              }
            }
          
            post {
              changed {
                script {
                  sendBuildStatusChangedNotifications()
                }
              }
            }
          }
          

           

          In an ideal world, it would become something like this.

           

          pipeline {
            libraries {
              lib("mylib@master")
            }
          
            ...
          
            stages {
              stage gitCheckout()
              stage mavenVerify('-Pprod')
              stage mavenDeploy('-Pprod')
              stage mavenRelease('-Pprod,release')
              stage sonar()
              stage sonarQualityGate()
              stage collectData()
            }
          
            // or better :)
            stages myCustomMavenBuild(config)
            stages myCustomNpmBuild(config)
          
            post sendBuildStatusChangedNotifications()
          }
          

           

           

          Peter Leibiger added a comment - Basically most of my Jenkinsfiles look like this, using a custom library, but still have to copy it every time and just change some properties/configs.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage( 'Checkout' ) { steps { gitCheckout() } } stage( 'Build & Verify' ) { when { anyOf { branch 'master' branch 'PR**' branch 'feature/**' } } steps { mavenVerify( '-Pprod' ) } } stage( 'Build & Deploy' ) { when { branch 'develop' } steps { mavenDeploy( '-Pprod' ) } } stage( 'Build & Release' ) { when { branch 'release/**' } steps { mavenRelease( '-Pprod,release' ) } } stage( 'Sonar Analysis' ) { when { anyOf { branch 'master' branch 'develop' branch 'PR**' } } steps { sonar() } } stage( "Sonar Quality Gate" ){ agent none when { anyOf { branch 'master' branch 'develop' } } steps { timeout(time: 1, unit: 'HOURS' ) { script { def qg = waitForQualityGate() echo "Sonar quality gate status: ${qg.status}" } } } } stage( 'Post Analysis' ) { steps { script { echo "Collecting data..." } } post { always { script { collectData() } } } } } post { changed { script { sendBuildStatusChangedNotifications() } } } }   In an ideal world, it would become something like this.   pipeline { libraries { lib( "mylib@master" ) } ... stages { stage gitCheckout() stage mavenVerify( '-Pprod' ) stage mavenDeploy( '-Pprod' ) stage mavenRelease( '-Pprod,release' ) stage sonar() stage sonarQualityGate() stage collectData() } // or better :) stages myCustomMavenBuild(config) stages myCustomNpmBuild(config) post sendBuildStatusChangedNotifications() }    

          Hi, 

           

          actually we're reusing bits of declarative pipelines doing like this:

           

          • In a shared library, inside the vars dir web declare the common code, ReleaseBoot.groovy:
          import x.y.z.jenkins.spring.boot.DeployConfig;
          
          def call(body) {
          
              def mail = new x.y.z.jenkins.notifications.MailSender(this)
          
              // evaluate the body block, and collect configuration into the object
              def config = [:]
              body.resolveStrategy = Closure.DELEGATE_FIRST
              body.delegate = config
              body()
          
              pipeline {
                  options{   
                      skipDefaultCheckout()   
                  }
                  agent none
                  // node
                  stages {
                      stage('Input Parameters'){
                          agent any
                          steps{
                              timeout(time:1, unit:'DAYS') {
                                  script {
                                      def inputVar = input id: 'RELEASE_INPUT', message: '¿Release new version?', ok: 'Yes', parameters: [
                                          string(defaultValue: '', description: 'Release version:', name: 'release'),
                                          string(defaultValue: '', description: 'Development version:', name: 'dev'),
                                          string(defaultValue: '', description: 'Commit:', name: 'commit')
                                      ], submitterParameter: 'user'
                                      env.RELEASE_VERSION = inputVar['release']
                                      env.BUILD_COMMIT     = inputVar['commit']
                                      env.DEV_VERSION = inputVar['dev']
                                  }
                              }
                          }
                      }
                      stage('scm') {
                          agent any
                          steps {
                              echo "Skip scm is ${config.SKIP.SCM}"
                              deleteDir()
                              checkout scm
                              script {
                                  ........
                              }
                              
                              
                          }
                      }
          
                      
                      stage('Release') {
                          agent any
                          steps {
                               withCredentials([[$class: 'UsernamePasswordMultiBinding', credentialsId: "${env.GIT_CREDENTIALS}",
                                      usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD']]) {
                                  ............
                                  stash name:"jars", includes:"**/target/*${env.RELEASE_VERSION}*.jar"
                                  
                                  script{........
                                  }
                              }
                              
                          
                          
                              
                          }
                      }
                  }
                  post {
                      failure {
                          script{ mail.send(config.MAILS)}
                      }
                      unstable {
                          script{ mail.send(config.MAILS) }
                      }
                  }
              }
          }
          • In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy

           

          def call(body) {
          
          
              env.DEPLOY_ENV            = "test"
              
              // evaluate the body block, and collect configuration into the object
              def config = [:]
              body.resolveStrategy = Closure.DELEGATE_FIRST
              body.delegate = config
              body()
              
              if(env.JOB_NAME.contains('deploy')) {
                  ........
              }else if(env.JOB_NAME.contains('release')) {
                  // here we call the previous defined 'shared' pipeline
                  ReleaseBoot {
                      MAILS     = config.MAILS
                      DEPLOYMENTS = config.DEPLOYMENTS
                  }
              } else {
                 .............
              }
              
          }
          
          • And  in each project's Jekinsfile:
          #!groovy
          @Library('xyz-pipelines') _
          def OPTIONAL_MAILS = []
          def deployTest     = [user:"targetuser", host:"host", credentialsId:"jboss-dev",
                             // config values
                             checkUrl:"http://xyz:8096/health", infoUrl:"http://xyz:8096/info"]
          // defined in shared library vars dir            
          BootPipeline{
              MAILS = OPTIONAL_MAILS
              DEPLOYMENTS = deployTest
          }        
             
          

          We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working

           

          Please keep this working

           

           NIco

           

           

           

           

          Nico Navarrete added a comment - Hi,    actually we're reusing bits of declarative pipelines doing like this:   In a shared library, inside the vars dir web declare the common code, ReleaseBoot .groovy: import x.y.z.jenkins.spring.boot.DeployConfig; def call(body) {     def mail = new x.y.z.jenkins.notifications.MailSender( this )      // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()     pipeline {         options{                skipDefaultCheckout()            }         agent none          // node         stages {             stage( 'Input Parameters' ){                 agent any                 steps{                     timeout(time:1, unit: 'DAYS' ) {                         script {                             def inputVar = input id: 'RELEASE_INPUT' , message: '¿Release new version?' , ok: 'Yes' , parameters: [                                 string(defaultValue: '', description: ' Release version: ', name: ' release'),                                 string(defaultValue: '', description: ' Development version: ', name: ' dev'),                                 string(defaultValue: '', description: ' Commit: ', name: ' commit')                             ], submitterParameter: 'user'                             env.RELEASE_VERSION = inputVar[ 'release' ]                             env.BUILD_COMMIT     = inputVar[ 'commit' ]                             env.DEV_VERSION = inputVar[ 'dev' ]                         }                     }                 }             }             stage( 'scm' ) {                 agent any                 steps {                     echo "Skip scm is ${config.SKIP.SCM}"                     deleteDir()                     checkout scm                     script {                         ........                     }                                                           }             }                          stage( 'Release' ) {                 agent any                 steps {                      withCredentials([[$class: 'UsernamePasswordMultiBinding' , credentialsId: "${env.GIT_CREDENTIALS}" ,                             usernameVariable: 'USERNAME' , passwordVariable: 'PASSWORD' ]]) {                         ............                         stash name: "jars" , includes: "**/target/*${env.RELEASE_VERSION}*.jar"                                                  script{........                         }                     }                                                                                             }             }         }         post {             failure {                 script{ mail.send(config.MAILS)}             }             unstable {                 script{ mail.send(config.MAILS) }             }         }     } } In our case we declarr also a wraper that choose between  differents reusable pipelines , BootPipeline.groovy   def call(body) {     env.DEPLOY_ENV            = "test"           // evaluate the body block, and collect configuration into the object     def config = [:]     body.resolveStrategy = Closure.DELEGATE_FIRST     body.delegate = config     body()           if (env.JOB_NAME.contains( 'deploy' )) {         ........     } else if (env.JOB_NAME.contains( 'release' )) { // here we call the previous defined 'shared' pipeline         ReleaseBoot {             MAILS     = config.MAILS             DEPLOYMENTS = config.DEPLOYMENTS         }     } else {        .............     }      } And  in each project's Jekinsfile: #!groovy @Library( 'xyz-pipelines' ) _ def OPTIONAL_MAILS = [] def deployTest     = [user: "targetuser" , host: "host" , credentialsId: "jboss-dev" ,                    // config values                    checkUrl: "http: //xyz:8096/health" , infoUrl: "http://xyz:8096/info" ] // defined in shared library vars dir             BootPipeline{     MAILS = OPTIONAL_MAILS     DEPLOYMENTS = deployTest }             We've got the info from shared libraries doc, not everythig is working ( stage execution expresions are ignored ) not  but mainly stages, steps, options , post are working   Please keep this working    NIco        

          Andrew Bayer added a comment -

          nicolasus - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547, FYI.

          Andrew Bayer added a comment - nicolasus - your particular use case is exactly what I'm now experimenting with over in JENKINS-46547 , FYI.

          Felix Neumann added a comment - - edited

          abayer: -JENKINS-46547- was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp.

          Therefore I'd like to see some kind of composable pipelines, where one could have a basic

          pipeline("my-shared-pipeline") {
            agent { /* ... */ }
            tools { /* ... */ }
            options { /* ... */ }
            stages {
              stage('Compile and Test') { /* ... */ }
              stage('Deploy') { /* ... */ }
            }
            post {
              failure { /* ... */ }
              alway { /* ... */ }
            }
          }
          

          and then extend it, using some mechanism to make up the final order of stages:

          pipeline(extends: "my-shared-pipeline") {
            stages {
              stage('Integration Test', after: 'Compile and Test') { /* ... */ }
            }
          }
          

          This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds.

          Btw., loading chunks from resources (as proposed above) also sounds like a good solution.

          Just my 5 Cents – thanks for making all this possible!

          Felix Neumann added a comment - - edited abayer : - JENKINS-46547 - was a great step forward. From my point of view, the biggest remaining problem is that one often copy-and-pastes whole pipeline to have different variations, e.g. with/without a "Integration Test" stage, with/without special Post-Steps, with/without special tools etc.pp. Therefore I'd like to see some kind of composable pipelines, where one could have a basic pipeline( "my-shared-pipeline" ) { agent { /* ... */ } tools { /* ... */ } options { /* ... */ } stages { stage( 'Compile and Test' ) { /* ... */ } stage( 'Deploy' ) { /* ... */ } } post { failure { /* ... */ } alway { /* ... */ } } } and then extend it, using some mechanism to make up the final order of stages: pipeline( extends : "my-shared-pipeline" ) { stages { stage( 'Integration Test' , after: 'Compile and Test' ) { /* ... */ } } } This way one would neither need to copy-and-paste the whole pipeline, nor need to introduce empty, never executed stages into builds. Btw., loading chunks from resources (as proposed above) also sounds like a good solution. Just my 5 Cents – thanks for making all this possible!

          I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548 for defining reusable stages, maybe it is interesting for you as well.

          Tobias Larscheid added a comment - I just opened https://issues.jenkins-ci.org/browse/JENKINS-50548  for defining reusable stages, maybe it is interesting for you as well.

          I would like this as well. But also, it seems like we can make some of those blocks one line, and we could use a preprocessor, if necessary. Just starting to get into this, but seems like groovy should support this already.

          Nick Roosevelt added a comment - I would like this as well. But also, it seems like we can make some of those blocks one line, and we could use a preprocessor, if necessary. Just starting to get into this, but seems like groovy should support this already.

          Bassam Khouri added a comment -

          I would like this as well. In my case, I have a single pipeline with multiple parallel stages. Each parallel stages have 90%+ of identical code and I would like to programatically defines different stages.

          for example, my pipeline looks like this

          pipeline {
              stages {
                  stage("stage1") {
                      agent {
                          // different agent based on stage
                      }
                      environment (
                          BUILD_OPTIONS = "different build options per stage"
                      )
                      steps {
                          deleteDir()
                          echo 'Building..'
                          build(BUILD_OPTIONS)
                      }
                      post {
                          // has some slight differences here depending on the stage
                      }
                  }
                  stage("stage2") {
                      agent {
                          // different agent based on stage
                      }
                      environment (
                          BUILD_OPTIONS = "another  build options per stage"
                      )
                      steps {
                          deleteDir()
                          echo 'Building..'
                          build(BUILD_OPTIONS)
                      }
                      post {
                          // has some slight differences here depending on the stage
                      }
                  }
                  ...
                  stage("stageN") {
                      agent {
                          // different agent based on stage
                      }
                      environment (
                          BUILD_OPTIONS = "build options for stage N"
                      )
                      steps {
                          deleteDir()
                          echo 'Building..'
                          build(BUILD_OPTIONS)
                      }
                      post {
                          // has some slight differences here depending on the stage
                      }
                  }
          
              }
          }
          

          Personally, I would like to do something like this

          def myStage(Map args) {
              // define the "dynamic" stage that uses the 'args' parameter
              if (args.specialAgent) {
                  agent { docker {...} }
              } else {
                  agent { label "SpecialAgent" } 
              }
              environment {
                 BUILD_OPTIONS = " ".join(args.buildOptions)
              }
              steps {
                 ...
              }
              post {
                  always {
                      if {args.UploadLogs) {
                          // call an "action"
                      }
                      // more actions
                  }
              }
          }
          
          pipeline {
              stages {
                  stage1 myStage([buildOptions: ["build", "option", "stage1"], arg1: "something"])
                  stage2 myStage([buildOptions: ["stage2"], arg1: "something"])
                  ...
                  stageN myStage([buildOptions: [STAGE_NAME], arg1: "something"0)
              }
          }
          

          Bassam Khouri added a comment - I would like this as well. In my case, I have a single pipeline with multiple parallel stages. Each parallel stages have 90%+ of identical code and I would like to programatically defines different stages. for example, my pipeline looks like this pipeline { stages { stage( "stage1" ) { agent { // different agent based on stage } environment ( BUILD_OPTIONS = "different build options per stage" ) steps { deleteDir() echo 'Building..' build(BUILD_OPTIONS) } post { // has some slight differences here depending on the stage } } stage( "stage2" ) { agent { // different agent based on stage } environment ( BUILD_OPTIONS = "another build options per stage" ) steps { deleteDir() echo 'Building..' build(BUILD_OPTIONS) } post { // has some slight differences here depending on the stage } } ... stage( "stageN" ) { agent { // different agent based on stage } environment ( BUILD_OPTIONS = "build options for stage N" ) steps { deleteDir() echo 'Building..' build(BUILD_OPTIONS) } post { // has some slight differences here depending on the stage } } } } Personally, I would like to do something like this def myStage(Map args) { // define the "dynamic" stage that uses the 'args' parameter if (args.specialAgent) { agent { docker {...} } } else { agent { label "SpecialAgent" } } environment { BUILD_OPTIONS = " " .join(args.buildOptions) } steps { ... } post { always { if {args.UploadLogs) { // call an "action" } // more actions } } } pipeline { stages { stage1 myStage([buildOptions: [ "build" , "option" , "stage1" ], arg1: "something" ]) stage2 myStage([buildOptions: [ "stage2" ], arg1: "something" ]) ... stageN myStage([buildOptions: [STAGE_NAME], arg1: "something" 0) } }

          Any update here?

          Michael Hüttermann added a comment - Any update here?

          Jean-Luc Pé added a comment -

          Please, could we know if this ticket is actually treated ?

          Jean-Luc Pé added a comment - Please, could we know if this ticket is actually treated ?

            Unassigned Unassigned
            bitwiseman Liam Newman
            Votes:
            89 Vote for this issue
            Watchers:
            102 Start watching this issue

              Created:
              Updated: