Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41396

Add method definition section to root pipeline block

      Had a discussion about this on #jenkins IRC with abayer and rtyler.

      Summary
      Currently, there are two ways to declare helper methods to use in declarative pipeline: importing a shared library or putting Groovy method definitions before of the pipeline block.

      This JIRA is aimed at the later case. I understand that we do not want to encourage people to use script blocks in Declarative Pipeline, but there is a difference between providing guidance that says, "Avoid use script blocks, consider using shared libraries, here's how you migrate from script to library" and requiring users to go outside of Declarative Pipeline entirely to do something (but saying they can do it). Further by doing this we make it harder for users to leverage code reuse, one of the key advantages of Pipeline as code.

      This is a akin to the exposing of currentBuild.rawBuild. We tell people they can use it but probably shouldn't, while at the same time not exposing basic build information such as build causes anywhere but on currentBuild.rawBuild.

      We should provide a clear progression for users from raw steps, to helper methods, to shared library. Right now, Scripted Pipeline makes that second step easy, while Declarative Pipeline make it harder and ugly. I think it would be better to have the Declarative model include a section that supports that second step, rather than something that people have to do outside of the model.

      As part of the model, we can reason over it and provide better feedback. For that matter we could even enforce rules on it such as "only declare variables and methods in this section".

      The basic idea would be to have method (or similarly named) section added directly under pipeline.

      At it's most basic, it would look like this:

      pipeline {
          agent any
          stages {
              stage ('Example') {
                  steps {
                      myHelper()
                  }
              }
          }
      
          method {
              def myHelper() {
                  echo "Some complex set of steps"
              }
          }
      }
      

      Or consider this Scripted Pipeline: https://github.com/bitwiseman/hermann/blob/blog/notifications/Jenkinsfile

      stage ('Build') {
      
        node {
          try {
          notifyBuild('STARTED')
      
            // Checkout
            checkout scm
      
            // install required bundles
            sh 'bundle install'
      
            // build and run tests with coverage
            sh 'bundle exec rake build spec'
      
            // Archive the built artifacts
            archive (includes: 'pkg/*.gem')
      
            // publish html
            publishHTML ([
                allowMissing: false,
                alwaysLinkToLastBuild: false,
                keepAll: true,
                reportDir: 'coverage',
                reportFiles: 'index.html',
                reportName: "RCov Report"
              ])
          } catch (Exception e) {
              currentBuild.result = "FAILED"
              throw e
          } finally {
            notifyBuild(currentBuild.result)
          }
        }
      }
      
      def notifyBuild(String buildStatus = 'STARTED') {
        // build status of null means successful
        buildStatus =  buildStatus ?: 'SUCCESSFUL'
      
        // Default values
        def colorName = 'RED'
        def colorCode = '#FF0000'
        def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
        def summary = "${subject} (${env.BUILD_URL})"
        def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
          <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>"""
      
        // Override default values based on build status
        if (buildStatus == 'STARTED') {
          color = 'YELLOW'
          colorCode = '#FFFF00'
        } else if (buildStatus == 'SUCCESSFUL') {
          color = 'GREEN'
          colorCode = '#00FF00'
        } else {
          color = 'RED'
          colorCode = '#FF0000'
        }
      
        // Send notifications
        slackSend (color: colorCode, message: summary)
      
        hipchatSend (color: color, notify: true, message: summary)
      
        emailext (
            to: 'bitwiseman@bitwiseman.com',
            subject: subject,
            body: details,
            recipientProviders: [[$class: 'DevelopersRecipientProvider']]
          )
      }
      

      The current guidance for migrating this to Declarative would be to keep notifyBuild outside of pipeline. This is not bad, but to anyone paying attention is a glaring gap in the Declarative model. We support script blocks under steps, but for cross-stage helpers we have to switch to shared libraries. I'll add more detail to this example shortly.

          [JENKINS-41396] Add method definition section to root pipeline block

          Liam Newman added a comment - - edited

          Thinking about this a bit more, it might make sense to do this as a section, which behaves similar to environment. In the environment section, you are only allowed to assign values to variables. Maybe have this root section be similar - a place where you can declare methods and variables, but not perform actions.

          We could even call it something other than script. Maybe library or methods.

          Liam Newman added a comment - - edited Thinking about this a bit more, it might make sense to do this as a section, which behaves similar to environment . In the environment section, you are only allowed to assign values to variables. Maybe have this root section be similar - a place where you can declare methods and variables, but not perform actions. We could even call it something other than script. Maybe library or methods .

          Liam Newman added a comment -

          abayer
          These two are similar in idea. If the

          {define}

          block suggested in JENKINS-41335 could accept methods, it would resolve this issue as well.

          Liam Newman added a comment - abayer These two are similar in idea. If the {define} block suggested in JENKINS-41335 could accept methods, it would resolve this issue as well.

          Andrew Bayer added a comment -

          Yup, folding this into JENKINS-41335.

          Andrew Bayer added a comment - Yup, folding this into JENKINS-41335 .

          Andrew Bayer added a comment -

          Reopening since this is a smaller scope than JENKINS-41335.

          Andrew Bayer added a comment - Reopening since this is a smaller scope than JENKINS-41335 .

          Liam Newman added a comment -

          My latest blog post covers exactly this scenario: https://jenkins.io/blog/2017/02/15/declarative-notifications/

          Liam Newman added a comment - My latest blog post covers exactly this scenario: https://jenkins.io/blog/2017/02/15/declarative-notifications/

          Andrew Bayer added a comment -

          So for reasons I'm fairly sheepish about (I needed a break from the Really Hard Problems of JENKINS-42829JENKINS-42753JENKINS-42748, and all their demented friends), I just spent a few hours on this, only to discover that there's a biiiiig problem. You simply can't define a method anywhere in Groovy except right inside a class, so 

          methods {
            def foo() {
              return "BANANAS ARE TASTY SOMETIMES"
            }
          }

          won't get past the second phase in Groovy compilation (i.e., conversion) and that's something we just plain can't get around, at least not without doing some truly twisted magic in the parse phase of compilation in certain weird-ass edge cases - and that's ANTLR territory, so, well, I am not gonna mess with that.

          So, alternatives - we could do something like:

          methods {
            define {
              name someMethodName
              returnType Foo.class     // optional, defaulting to dynamic, a.k.a. "def"
              parameter {              // optional, multiple parameter { } blocks allowed
                name someParamName
                paramType String.class // optional, dunno how we'd do List<String> and friends anyway
                defaultValue "foo"     // optional again
              }
              nonCPS true              // optional
              body {
                return someParamName
              }
            }
          }

          But that...looks awful. Just godawful.

          I'm considering whether I can transform the following into a proper method behind the scenes:

          methods {
            someMethodName { String someParam ->
              return someParam
            }
          }

          I think I can? But there are still some caveats - no way to define return type, so the transformed method in the shared library (oh yeah, I figured out how to do a dynamically generated non-source-control backed shared library) would be def someMethodName(String someParam) ... but I think that's ok. Not sure how to mark @NonCPS - the annotation can only go on methods, so we can't just toss it on the method call/closure above, so we'd theoretically need some other way to record that. And beyond that, it's not exactly a comfortable syntax for non-Groovy people, is it? Grr.

          Andrew Bayer added a comment - So for reasons I'm fairly sheepish about (I needed a break from the Really Hard Problems of  JENKINS-42829 ,  JENKINS-42753 ,  JENKINS-42748 , and all their demented friends), I just spent a few hours on this, only to discover that there's a biiiiig problem. You simply can't define a method anywhere in Groovy except right inside a class, so  methods { def foo() { return "BANANAS ARE TASTY SOMETIMES" } } won't get past the second phase in Groovy compilation (i.e., conversion) and that's something we just plain can't get around, at least not without doing some truly twisted magic in the parse phase of compilation in certain weird-ass edge cases - and that's ANTLR territory, so, well, I am not gonna mess with that . So, alternatives - we could do something like: methods { define { name someMethodName returnType Foo.class // optional, defaulting to dynamic, a.k.a. "def" parameter { // optional, multiple parameter { } blocks allowed name someParamName paramType String .class // optional, dunno how we'd do List< String > and friends anyway defaultValue "foo" // optional again } nonCPS true // optional body { return someParamName } } } But that...looks awful . Just godawful. I'm considering whether I can transform the following into a proper method behind the scenes: methods { someMethodName { String someParam -> return someParam } } I think I can? But there are still some caveats - no way to define return type, so the transformed method in the shared library (oh yeah, I figured out how to do a dynamically generated non-source-control backed shared library) would be def someMethodName(String someParam) ... but I think that's ok. Not sure how to mark @NonCPS - the annotation can only go on methods, so we can't just toss it on the method call/closure above, so we'd theoretically need some other way to record that. And beyond that, it's not exactly a comfortable syntax for non-Groovy people, is it? Grr.

          Andrew Bayer added a comment -

          Yeah, I don't think there's a good answer for this one, so I'm closing it.

          Andrew Bayer added a comment - Yeah, I don't think there's a good answer for this one, so I'm closing it.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

            abayer Andrew Bayer
            bitwiseman Liam Newman
            Votes:
            12 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: