Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49135

Support for reusable sharing Declarative directives

      I'd like to be able to share recommended defaults for options directives. My initial use case was sharing steps through shared libraries.

       

      // vars/defaultOptionsConfig.groovy
      call() {
          timestamps()
      }
      
      options {
         defaultOptionsConfig()
         ansiColor('xterm')
      }

       

      At the same time, It'd also be nice to be able to be able to shared directives generally, so I didn't want to scope the request down too far.

       

      // vars/defaultOptions.groovy
      call() {
         options {
             timestamps()
         }
      }
      
      pipeline {
      ...
         defaultOptions()
         stages {
             ...
         }
      }

       

          [JENKINS-49135] Support for reusable sharing Declarative directives

          Andrew Bayer added a comment -

          Oh, and as I mention in the PR, this definitely won't ever support all Declarative sections or directives. For example, anything that contains an actual step execution (so steps, post, when (in part because of when expression and in part because it'd be a pain in the ass), stage, and stages (because stage contains steps and stages contains stage) would not be supported. Nor would environment be supported. Main reason for that is because of how we handle steps and environment resolution in the AST transformation, plus that pushes even farther past what I'm comfortable with syntactically/complexity-wise than just things like options, triggers, parameters, tools, or agent.

          Andrew Bayer added a comment - Oh, and as I mention in the PR, this definitely won't ever support all Declarative sections or directives. For example, anything that contains an actual step execution (so steps , post , when (in part because of when expression and in part because it'd be a pain in the ass), stage , and stages (because stage contains steps and stages contains stage ) would not be supported. Nor would environment be supported. Main reason for that is because of how we handle steps and environment resolution in the AST transformation, plus that pushes even farther past what I'm comfortable with syntactically/complexity-wise than just things like options , triggers , parameters , tools , or agent .

          Cool! That looks like a reasonable approach. I hope it lands eventually!

          Leandro Lucarella added a comment - Cool! That looks like a reasonable approach. I hope it lands eventually!

          Hi abayer, can you elaborate a little more on why it'd be a pain in the ass to support stages? Because I feel like you wouldn't be very happy about JENKINS-50548 then...

          Tobias Larscheid added a comment - Hi abayer , can you elaborate a little more on why it'd be a pain in the ass to support stages? Because I feel like you wouldn't be very happy about JENKINS-50548  then...

          Andrew Bayer added a comment -

          tobilarscheid - parsing hell, largely. Declarative is all a bundle of fun Groovy AST manipulation at its core. Initially, that was just for compile-time validation and for transformation into an intermediate JSON format for use with the Blue Ocean Pipeline Editor, but Pipeline's CPS transformations made a traditional Groovy closure-based DSL interpretation approach too painful after a while, so now we actually parse the pipeline block into an internal representation at compile time, use that internal representation for validation, and then transform that internal representation into the actual runtime model that tells us what to execute when, etc, via some janky Groovy AST manipulation.

          Figuring out how to mash together the parsing, the validation, the runtime transformation, and the ability to actually specify parts of the pipeline block elsewhere and consume them as method calls/objects within the pipeline block is something I have not managed to achieve yet. I'm not averse to the idea, but I am wary of making this already overly-complex tooling even more complex, making maintainability even harder, potentially breaking existing things, etc.

          Feel free to take a look at the code - https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy is probably the place to start looking. That gets called at the time that the Jenkinsfile is compiled - it does the parsing from the raw Groovy AST into the internal representation, calls the validation, and finally transforms the Groovy AST into various constructor calls so that the pipeline block actually ends up returning a https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy instance to the runtime interpreter for actual execution. It's messy. I apologize.

          Andrew Bayer added a comment - tobilarscheid - parsing hell, largely. Declarative is all a bundle of fun Groovy AST manipulation at its core. Initially, that was just for compile-time validation and for transformation into an intermediate JSON format for use with the Blue Ocean Pipeline Editor, but Pipeline's CPS transformations made a traditional Groovy closure-based DSL interpretation approach too painful after a while, so now we actually parse the pipeline block into an internal representation at compile time, use that internal representation for validation, and then transform that internal representation into the actual runtime model that tells us what to execute when, etc, via some janky Groovy AST manipulation. Figuring out how to mash together the parsing, the validation, the runtime transformation, and the ability to actually specify parts of the pipeline block elsewhere and consume them as method calls/objects within the pipeline block is something I have not managed to achieve yet. I'm not averse to the idea , but I am wary of making this already overly-complex tooling even more complex, making maintainability even harder, potentially breaking existing things, etc. Feel free to take a look at the code - https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy is probably the place to start looking. That gets called at the time that the Jenkinsfile is compiled - it does the parsing from the raw Groovy AST into the internal representation, calls the validation, and finally transforms the Groovy AST into various constructor calls so that the pipeline block actually ends up returning a https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy instance to the runtime interpreter for actual execution. It's messy. I apologize.

          woops, what a can of worms I opened here  thanks for your detailed explanation, if anything I at least understood it's way more complex than I was hoping. If I find the time I will give the code a look, happy to year that you are not averse to the idea itself. Thank you!

          Tobias Larscheid added a comment - woops, what a can of worms I opened here  thanks for your detailed explanation, if anything I at least understood it's way more complex than I was hoping. If I find the time I will give the code a look, happy to year that you are not averse to the idea itself. Thank you!

          Andrew Bayer added a comment -

          So fwiw, https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/260 may end up being the beginning of a way to kinda do this. I'm doing it for a more complex proprietary use case, and it does require an actual Jenkins plugin to implement a new directive to do the replacement, but I suppose it's theoretically possible to build something off the new extension point there to provide a new directive like, say, stageFromLibrary library: "some-lib@master", method: "someMethodInLib" that would (through some more complex tooling than is available currently) go get and run that method and insert the result into the model before validation and transformation occur... We'll see. This is very early prototype stuff at this point - I threw the PR up mainly because I like to have a PR to point to when I'm doing downstream prototyping work, but I think this would open options for someone to implement tooling to satisfy the requests behind this ticket.

          Andrew Bayer added a comment - So fwiw, https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/260 may end up being the beginning of a way to kinda do this. I'm doing it for a more complex proprietary use case, and it does require an actual Jenkins plugin to implement a new directive to do the replacement, but I suppose it's theoretically possible to build something off the new extension point there to provide a new directive like, say, stageFromLibrary library: "some-lib@master", method: "someMethodInLib" that would (through some more complex tooling than is available currently) go get and run that method and insert the result into the model before validation and transformation occur... We'll see. This is very early prototype stuff at this point - I threw the PR up mainly because I like to have a PR to point to when I'm doing downstream prototyping work, but I think this would open options for someone to implement tooling to satisfy the requests behind this ticket.

          Dominik Heim added a comment -

          This would help absolutely to clean up Jenkinsfile and make also common code possible outside of steps within stages. Really looking forward to this. 

          Dominik Heim added a comment - This would help absolutely to clean up Jenkinsfile and make also common code possible outside of steps within stages. Really looking forward to this. 

          The linked PR is closed since Nov 2018. Have there been any other ideas or actions on this ticket?

          Daniel Geißler added a comment - The linked PR is closed since Nov 2018. Have there been any other ideas or actions on this ticket?

          Tavin Cole added a comment -

          Well if you want to mess around with Groovy AST, it does work to write something like this:

          pipeline {
              @MyAnnotation options
              ...
          }
          

          You can define @MyAnnotation just before calling the pipeline, e.g.:

          @org.codehaus.groovy.transform.GroovyASTTransformationClass(['mypkg.MyTransformation'])
          @interface MyAnnotation {}
          

          MyTransformation will need to be in a global shared library, loaded automatically or with @Library. Then you have to replace the annotated statement with valid pipeline syntax. If the transformation is written in Groovy, it must be @NonCPS, and I've noticed that any errors don't seem to be logged – Jenkins just goes on and tries to execute the untransformed pipeline, which then fails syntax validation.

          Tavin Cole added a comment - Well if you want to mess around with Groovy AST, it does work to write something like this: pipeline { @MyAnnotation options ... } You can define @MyAnnotation just before calling the pipeline, e.g.: @org.codehaus.groovy.transform.GroovyASTTransformationClass([ 'mypkg.MyTransformation' ]) @ interface MyAnnotation {} MyTransformation will need to be in a global shared library, loaded automatically or with @Library . Then you have to replace the annotated statement with valid pipeline syntax. If the transformation is written in Groovy, it must be @NonCPS , and I've noticed that any errors don't seem to be logged – Jenkins just goes on and tries to execute the untransformed pipeline, which then fails syntax validation.

          Tavin Cole added a comment -

          Here is a test case which demonstrates the technique: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/387

          Tavin Cole added a comment - Here is a test case which demonstrates the technique:  https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/387

            Unassigned Unassigned
            rpocase Robby Pocase
            Votes:
            27 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated: