Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41396

Add method definition section to root pipeline block

      Had a discussion about this on #jenkins IRC with abayer and rtyler.

      Summary
      Currently, there are two ways to declare helper methods to use in declarative pipeline: importing a shared library or putting Groovy method definitions before of the pipeline block.

      This JIRA is aimed at the later case. I understand that we do not want to encourage people to use script blocks in Declarative Pipeline, but there is a difference between providing guidance that says, "Avoid use script blocks, consider using shared libraries, here's how you migrate from script to library" and requiring users to go outside of Declarative Pipeline entirely to do something (but saying they can do it). Further by doing this we make it harder for users to leverage code reuse, one of the key advantages of Pipeline as code.

      This is a akin to the exposing of currentBuild.rawBuild. We tell people they can use it but probably shouldn't, while at the same time not exposing basic build information such as build causes anywhere but on currentBuild.rawBuild.

      We should provide a clear progression for users from raw steps, to helper methods, to shared library. Right now, Scripted Pipeline makes that second step easy, while Declarative Pipeline make it harder and ugly. I think it would be better to have the Declarative model include a section that supports that second step, rather than something that people have to do outside of the model.

      As part of the model, we can reason over it and provide better feedback. For that matter we could even enforce rules on it such as "only declare variables and methods in this section".

      The basic idea would be to have method (or similarly named) section added directly under pipeline.

      At it's most basic, it would look like this:

      pipeline {
          agent any
          stages {
              stage ('Example') {
                  steps {
                      myHelper()
                  }
              }
          }
      
          method {
              def myHelper() {
                  echo "Some complex set of steps"
              }
          }
      }
      

      Or consider this Scripted Pipeline: https://github.com/bitwiseman/hermann/blob/blog/notifications/Jenkinsfile

      stage ('Build') {
      
        node {
          try {
          notifyBuild('STARTED')
      
            // Checkout
            checkout scm
      
            // install required bundles
            sh 'bundle install'
      
            // build and run tests with coverage
            sh 'bundle exec rake build spec'
      
            // Archive the built artifacts
            archive (includes: 'pkg/*.gem')
      
            // publish html
            publishHTML ([
                allowMissing: false,
                alwaysLinkToLastBuild: false,
                keepAll: true,
                reportDir: 'coverage',
                reportFiles: 'index.html',
                reportName: "RCov Report"
              ])
          } catch (Exception e) {
              currentBuild.result = "FAILED"
              throw e
          } finally {
            notifyBuild(currentBuild.result)
          }
        }
      }
      
      def notifyBuild(String buildStatus = 'STARTED') {
        // build status of null means successful
        buildStatus =  buildStatus ?: 'SUCCESSFUL'
      
        // Default values
        def colorName = 'RED'
        def colorCode = '#FF0000'
        def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
        def summary = "${subject} (${env.BUILD_URL})"
        def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
          <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>"""
      
        // Override default values based on build status
        if (buildStatus == 'STARTED') {
          color = 'YELLOW'
          colorCode = '#FFFF00'
        } else if (buildStatus == 'SUCCESSFUL') {
          color = 'GREEN'
          colorCode = '#00FF00'
        } else {
          color = 'RED'
          colorCode = '#FF0000'
        }
      
        // Send notifications
        slackSend (color: colorCode, message: summary)
      
        hipchatSend (color: color, notify: true, message: summary)
      
        emailext (
            to: 'bitwiseman@bitwiseman.com',
            subject: subject,
            body: details,
            recipientProviders: [[$class: 'DevelopersRecipientProvider']]
          )
      }
      

      The current guidance for migrating this to Declarative would be to keep notifyBuild outside of pipeline. This is not bad, but to anyone paying attention is a glaring gap in the Declarative model. We support script blocks under steps, but for cross-stage helpers we have to switch to shared libraries. I'll add more detail to this example shortly.

          [JENKINS-41396] Add method definition section to root pipeline block

          Liam Newman created issue -
          Liam Newman made changes -
          Link New: This issue relates to JENKINS-38110 [ JENKINS-38110 ]

          Liam Newman added a comment - - edited

          Thinking about this a bit more, it might make sense to do this as a section, which behaves similar to environment. In the environment section, you are only allowed to assign values to variables. Maybe have this root section be similar - a place where you can declare methods and variables, but not perform actions.

          We could even call it something other than script. Maybe library or methods.

          Liam Newman added a comment - - edited Thinking about this a bit more, it might make sense to do this as a section, which behaves similar to environment . In the environment section, you are only allowed to assign values to variables. Maybe have this root section be similar - a place where you can declare methods and variables, but not perform actions. We could even call it something other than script. Maybe library or methods .
          R. Tyler Croy made changes -
          Description Original: Had a discussion about this on #jenkins IRC with [~abayer] and [~201604291_tyler].

          Summary
          Currently, there are two ways to declare helper methods to use in declarative pipeline: importing a shared library or putting Groovy method definitions before of the {{pipeline}} block.

          This JIRA is aimed at the later case. I understand that we do not want to encourage people to use script blocks in Declarative Pipeline, but there is a difference between providing guidance that says, "Avoid use script blocks, consider using shared libraries, here's how you migrate from script to library" and requiring users to go outside of Declarative Pipeline entirely to do something (but saying they can do it). Further by doing this we make it harder for users to leverage code reuse, one of the key advantages of Pipeline as code.

          This is a akin to the exposing of {{currentBuild.rawBuild}}. We tell people they can use it but probably shouldn't, while at the same time not exposing basic build information such as build causes anywhere but on {{currentBuild.rawBuild}}.

          We should provide a clear progression for users from raw steps, to helper methods, to shared library. Right now, Scripted Pipeline makes that second step easy, while Declarative Pipeline make it harder and ugly. I think it would be better to have the Declarative model include a section that supports that second step, rather than something that people have to do outside of the model.

          As part of the model, we can reason over it and provide better feedback. For that matter we could even enforce rules on it such as "only declare variables and methods in this section".

          The basic idea would be to have {{script}} (or similarly named) section added directly under {{pipeline}}.

          At it's most basic, it would look like this:
          {code}
          pipeline {
              agent any
              stages {
                  stage ('Example') {
                      steps {
                          myHelper()
                      }
                  }
              }

              script {
                  def myHelper() {
                      echo "Some complex set of steps"
                  }
              }
          }
          {code}

          Or consider this Scripted Pipeline: https://github.com/bitwiseman/hermann/blob/blog/notifications/Jenkinsfile

          {code}
          stage ('Build') {

            node {
              try {
              notifyBuild('STARTED')

                // Checkout
                checkout scm

                // install required bundles
                sh 'bundle install'

                // build and run tests with coverage
                sh 'bundle exec rake build spec'

                // Archive the built artifacts
                archive (includes: 'pkg/*.gem')

                // publish html
                publishHTML ([
                    allowMissing: false,
                    alwaysLinkToLastBuild: false,
                    keepAll: true,
                    reportDir: 'coverage',
                    reportFiles: 'index.html',
                    reportName: "RCov Report"
                  ])
              } catch (Exception e) {
                  currentBuild.result = "FAILED"
                  throw e
              } finally {
                notifyBuild(currentBuild.result)
              }
            }
          }

          def notifyBuild(String buildStatus = 'STARTED') {
            // build status of null means successful
            buildStatus = buildStatus ?: 'SUCCESSFUL'

            // Default values
            def colorName = 'RED'
            def colorCode = '#FF0000'
            def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
            def summary = "${subject} (${env.BUILD_URL})"
            def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
              <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>"""

            // Override default values based on build status
            if (buildStatus == 'STARTED') {
              color = 'YELLOW'
              colorCode = '#FFFF00'
            } else if (buildStatus == 'SUCCESSFUL') {
              color = 'GREEN'
              colorCode = '#00FF00'
            } else {
              color = 'RED'
              colorCode = '#FF0000'
            }

            // Send notifications
            slackSend (color: colorCode, message: summary)

            hipchatSend (color: color, notify: true, message: summary)

            emailext (
                to: 'bitwiseman@bitwiseman.com',
                subject: subject,
                body: details,
                recipientProviders: [[$class: 'DevelopersRecipientProvider']]
              )
          }
          {code}

          The current guidance for migrating this to Declarative would be to keep {{notifyBuild}} outside of {{pipeline}}. This is not bad, but to anyone paying attention is a glaring gap in the Declarative model. We support {{script}} blocks under steps, but for cross-stage helpers we have to switch to shared libraries. I'll add more detail to this example shortly.


          New: Had a discussion about this on #jenkins IRC with [~abayer] and [~rtyler].

          Summary
          Currently, there are two ways to declare helper methods to use in declarative pipeline: importing a shared library or putting Groovy method definitions before of the {{pipeline}} block.

          This JIRA is aimed at the later case. I understand that we do not want to encourage people to use script blocks in Declarative Pipeline, but there is a difference between providing guidance that says, "Avoid use script blocks, consider using shared libraries, here's how you migrate from script to library" and requiring users to go outside of Declarative Pipeline entirely to do something (but saying they can do it). Further by doing this we make it harder for users to leverage code reuse, one of the key advantages of Pipeline as code.

          This is a akin to the exposing of {{currentBuild.rawBuild}}. We tell people they can use it but probably shouldn't, while at the same time not exposing basic build information such as build causes anywhere but on {{currentBuild.rawBuild}}.

          We should provide a clear progression for users from raw steps, to helper methods, to shared library. Right now, Scripted Pipeline makes that second step easy, while Declarative Pipeline make it harder and ugly. I think it would be better to have the Declarative model include a section that supports that second step, rather than something that people have to do outside of the model.

          As part of the model, we can reason over it and provide better feedback. For that matter we could even enforce rules on it such as "only declare variables and methods in this section".

          The basic idea would be to have {{script}} (or similarly named) section added directly under {{pipeline}}.

          At it's most basic, it would look like this:
          {code}
          pipeline {
              agent any
              stages {
                  stage ('Example') {
                      steps {
                          myHelper()
                      }
                  }
              }

              script {
                  def myHelper() {
                      echo "Some complex set of steps"
                  }
              }
          }
          {code}

          Or consider this Scripted Pipeline: https://github.com/bitwiseman/hermann/blob/blog/notifications/Jenkinsfile

          {code}
          stage ('Build') {

            node {
              try {
              notifyBuild('STARTED')

                // Checkout
                checkout scm

                // install required bundles
                sh 'bundle install'

                // build and run tests with coverage
                sh 'bundle exec rake build spec'

                // Archive the built artifacts
                archive (includes: 'pkg/*.gem')

                // publish html
                publishHTML ([
                    allowMissing: false,
                    alwaysLinkToLastBuild: false,
                    keepAll: true,
                    reportDir: 'coverage',
                    reportFiles: 'index.html',
                    reportName: "RCov Report"
                  ])
              } catch (Exception e) {
                  currentBuild.result = "FAILED"
                  throw e
              } finally {
                notifyBuild(currentBuild.result)
              }
            }
          }

          def notifyBuild(String buildStatus = 'STARTED') {
            // build status of null means successful
            buildStatus = buildStatus ?: 'SUCCESSFUL'

            // Default values
            def colorName = 'RED'
            def colorCode = '#FF0000'
            def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
            def summary = "${subject} (${env.BUILD_URL})"
            def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
              <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>"""

            // Override default values based on build status
            if (buildStatus == 'STARTED') {
              color = 'YELLOW'
              colorCode = '#FFFF00'
            } else if (buildStatus == 'SUCCESSFUL') {
              color = 'GREEN'
              colorCode = '#00FF00'
            } else {
              color = 'RED'
              colorCode = '#FF0000'
            }

            // Send notifications
            slackSend (color: colorCode, message: summary)

            hipchatSend (color: color, notify: true, message: summary)

            emailext (
                to: 'bitwiseman@bitwiseman.com',
                subject: subject,
                body: details,
                recipientProviders: [[$class: 'DevelopersRecipientProvider']]
              )
          }
          {code}

          The current guidance for migrating this to Declarative would be to keep {{notifyBuild}} outside of {{pipeline}}. This is not bad, but to anyone paying attention is a glaring gap in the Declarative model. We support {{script}} blocks under steps, but for cross-stage helpers we have to switch to shared libraries. I'll add more detail to this example shortly.


          Liam Newman added a comment -

          abayer
          These two are similar in idea. If the

          {define}

          block suggested in JENKINS-41335 could accept methods, it would resolve this issue as well.

          Liam Newman added a comment - abayer These two are similar in idea. If the {define} block suggested in JENKINS-41335 could accept methods, it would resolve this issue as well.
          Liam Newman made changes -
          Link New: This issue relates to JENKINS-41335 [ JENKINS-41335 ]
          Liam Newman made changes -
          Summary Original: Declarative: Add "script" section to root pipeline block New: Declarative: Add script definition section to root pipeline block

          Andrew Bayer added a comment -

          Yup, folding this into JENKINS-41335.

          Andrew Bayer added a comment - Yup, folding this into JENKINS-41335 .
          Andrew Bayer made changes -
          Link New: This issue duplicates JENKINS-41335 [ JENKINS-41335 ]
          Andrew Bayer made changes -
          Resolution New: Duplicate [ 3 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

            abayer Andrew Bayer
            bitwiseman Liam Newman
            Votes:
            12 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: