Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45901

Allow common groovy annotation in Pipeline Shared Library Classes

      Currently the following code fails with a cryptic error message:

       

      package org.test.;
      import groovy.transform.ToString
      @ToString(includeNames=true)
      class X implements Serializable {
       def script
       def A, B
      Promotemodul(script, m) {
       this.script = script
       this.A = m.A
       this.B = m.B
       }
      }
      

      It would be great if some common annotations (ToString, EqualsAndHashCode...) would be supported in Pipeline Shared Libraries. 

       

       

          [JENKINS-45901] Allow common groovy annotation in Pipeline Shared Library Classes

          Dee Kryvenko added a comment - - edited

          jglick it has been been 3 years since your last comment, may I inquire what were you referring to when you said "allowing external process execution, not wasting any more time on the CPS engine" and is this something available to use? Is there any documentation on that alternative mechanism?

          Unless I am missing something, I see no real alternative at the moment after 3 years of waiting. I would imagine Groovy being chosen to be the engine for its high level abstractions, meta programming features and ease one could create their own DSL. But when so fundamental features such as commonly used annotations ToString, EqualsAndHashCode, Delegate as well as Groovy Traits are not available - it is kind of defeats the purpose, doesn't it? Sometimes it is so frustrating to work on complex libraries as you may never know if your code will actually work in the CPS or not.

          If not fix it, why not extend NonCPS annotation to support classes or maybe even allow users to disable it altogether for a given libraries? In certain use cases - the library forcibly injected into the pipeline and it is only limited to generating the pipeline and is not used in runtime - meaning all of the library classes are trusted and they require no survivability as the actual generated pipeline will be executed separately - in which case CPS is nothing but trouble, introducing artificial limitations for no practical gain. In my pretty complex library I have 99% of my methods annotated with NonCPS (except for the ones that are exposed to be called from Jenkinsfile context), meaning - CPS is actually only 1% efficient for my use case.

          I would argue this issue has to be revisited and acted on. Unless, of course, there is a viable alternative that I am not aware of.

          Dee Kryvenko added a comment - - edited jglick  it has been been 3 years since your last comment, may I inquire what were you referring to when you said "allowing external process execution, not wasting any more time on the CPS engine" and is this something available to use? Is there any documentation on that alternative mechanism? Unless I am missing something, I see no real alternative at the moment after 3 years of waiting. I would imagine Groovy being chosen to be the engine for its high level abstractions, meta programming features and ease one could create their own DSL. But when so fundamental features such as commonly used annotations ToString, EqualsAndHashCode, Delegate as well as Groovy Traits are not available - it is kind of defeats the purpose, doesn't it? Sometimes it is so frustrating to work on complex libraries as you may never know if your code will actually work in the CPS or not. If not fix it, why not extend NonCPS annotation to support classes or maybe even allow users to disable it altogether for a given libraries? In certain use cases - the library forcibly injected into the pipeline and it is only limited to generating the pipeline and is not used in runtime - meaning all of the library classes are trusted and they require no survivability as the actual generated pipeline will be executed separately - in which case CPS is nothing but trouble, introducing artificial limitations for no practical gain. In my pretty complex library I have 99% of my methods annotated with NonCPS (except for the ones that are exposed to be called from Jenkinsfile context), meaning - CPS is actually only 1% efficient for my use case. I would argue this issue has to be revisited and acted on. Unless, of course, there is a viable alternative that I am not aware of.

          Jesse Glick added a comment -

          is this something available to use?

          No.

          the library […] is only limited to generating the pipeline and is not used in runtime - meaning all of the library classes are trusted and they require no survivability as the actual generated pipeline will be executed separately

          Not exactly sure I am following the use case you are describing here. A bunch of complex plain old Groovy code stuff generating (Pipeline) Groovy source code and then calling evaluate? If so, I would suggest using JENKINS-26635, at least if an agent is available briefly for this setup phase.

          Jesse Glick added a comment - is this something available to use? No. the library […] is only limited to generating the pipeline and is not used in runtime - meaning all of the library classes are trusted and they require no survivability as the actual generated pipeline will be executed separately Not exactly sure I am following the use case you are describing here. A bunch of complex plain old Groovy code stuff generating (Pipeline) Groovy source code and then calling evaluate ? If so, I would suggest using JENKINS-26635 , at least if an agent is available briefly for this setup phase.

          Dee Kryvenko added a comment - - edited

          jglick are you talking about this? https://www.jenkins.io/doc/pipeline/steps/groovy/

          That might be just what I was looking for - thanks for the tip, going to try that.

          Let me try to explain my use case in a little more details.

          My users have read-only access to my Jenkins. My automation provisions a GitHub Org Folder for their orgs, and in the folder job config I make use of the following two plugins:

          https://plugins.jenkins.io/pipeline-multibranch-defaults/
          https://plugins.jenkins.io/inline-pipeline/

          With the help of these two plugins - for the Org Folder there is a custom marker/recognizer file instead of the Jenkinsfile (for the sake of the argument let's say it is an arbitrary yaml configuration file) along with a default hard-coded inline Jenkinsfile definition that the users cannot change. The hardcoded Jenkinsfile fetches the yaml configuration file and delegates to the global shared library. The library in turn reads the configuration file and produces a Jenkinsfile DSL text that is then being evaluated in the aforementioned hard-coded Jenkinsfile. The DSL generator library is just that - a text producing template engine. It doesn't even define steps (other than the entry point step to be called from that hardcoded Jenkinsfile). All the real steps are in other libraries, and they are not called from this DSL generator library, and they all are CPS-friendly.

          As you can see - there is zero untrusted DSL code that is coming from the users. 100% of the code is trusted. Neither does this code requires CPS survivability as it is an idempotent text producer. Its execution time is seconds and even if it gets interrupted for whatever reason, it can safely be called again with the same input.

          In such a setup having to deal with CPS woodoo is just that what you said 3 years ago - a waste of everyone's time. You could argue this particular shared library could be actually a plugin, which is absolute truth, but keeping it as a library adds a benefit for our users to specify a library version per repository/branch - which adds tremendous level of stability, isolation and a blast radius damage control. If it was a plugin - it would be a version shared globally across all my users.

          With https://www.jenkins.io/doc/pipeline/steps/groovy/ - I wonder if all I have to do is simply wrap either call to the library step in my hardcoded Jenkisfile with that or wrap a call to the library classes from the actual library step. I am definitely going to try that.

          Dee Kryvenko added a comment - - edited jglick  are you talking about this? https://www.jenkins.io/doc/pipeline/steps/groovy/ That might be just what I was looking for - thanks for the tip, going to try that. Let me try to explain my use case in a little more details. My users have read-only access to my Jenkins. My automation provisions a GitHub Org Folder for their orgs, and in the folder job config I make use of the following two plugins: https://plugins.jenkins.io/pipeline-multibranch-defaults/ https://plugins.jenkins.io/inline-pipeline/ With the help of these two plugins - for the Org Folder there is a custom marker/recognizer file instead of the Jenkinsfile (for the sake of the argument let's say it is an arbitrary yaml configuration file) along with a default hard-coded inline Jenkinsfile definition that the users cannot change. The hardcoded Jenkinsfile fetches the yaml configuration file and delegates to the global shared library. The library in turn reads the configuration file and produces a Jenkinsfile DSL text that is then being evaluated in the aforementioned hard-coded Jenkinsfile. The DSL generator library is just that - a text producing template engine. It doesn't even define steps (other than the entry point step to be called from that hardcoded Jenkinsfile). All the real steps are in other libraries, and they are not called from this DSL generator library, and they all are CPS-friendly. As you can see - there is zero untrusted DSL code that is coming from the users. 100% of the code is trusted. Neither does this code requires CPS survivability as it is an idempotent text producer. Its execution time is seconds and even if it gets interrupted for whatever reason, it can safely be called again with the same input. In such a setup having to deal with CPS woodoo is just that what you said 3 years ago - a waste of everyone's time. You could argue this particular shared library could be actually a plugin, which is absolute truth, but keeping it as a library adds a benefit for our users to specify a library version per repository/branch - which adds tremendous level of stability, isolation and a blast radius damage control. If it was a plugin - it would be a version shared globally across all my users. With  https://www.jenkins.io/doc/pipeline/steps/groovy/  - I wonder if all I have to do is simply wrap either call to the library step in my hardcoded Jenkisfile with that or wrap a call to the library classes from the actual library step. I am definitely going to try that.

          Jesse Glick added a comment - - edited

          Yes that was the step I meant. May or may not be helpful depending on your use case. If you have efficient access to a node before the “real” build begins—and in your case it sounds safe to enable on-controller executors—then yes, your default Jenkinsfile could look like

          evaluate(library('xxx').generateScript(someParam: 'some-value'))
          

          where vars/generateScript.groovy would be like

          def call(Map params) {
            node('master') {
              writeFile file: "$WORKSPACE_TMP/lib.groovy", text: libraryResource('lib.groovy')
              withGroovy(input: params) {
                sh 'groovy "$WORKSPACE_TMP/lib.groovy"'
              }
            }
          }
          

          and resources/lib.groovy:

          String generatePipelineScript(Map params) {
            // go wild, this is stock Groovy running in its own process
          }
          Pipeline.output(generatePipelineScript(Pipeline.input()))
          

          The above scenario is not tested, just off the top of my head. You do not really need a library at all, BTW, if you are using a hard-coded Jenkinsfile—can just inline the withGroovy idiom, and get your lib.groovy or whatever from a Git checkout of some tool repo, etc.

          Jesse Glick added a comment - - edited Yes that was the step I meant. May or may not be helpful depending on your use case. If you have efficient access to a node before the “real” build begins—and in your case it sounds safe to enable on-controller executors—then yes, your default Jenkinsfile could look like evaluate(library( 'xxx' ).generateScript(someParam: 'some-value' )) where vars/generateScript.groovy would be like def call(Map params) { node( 'master' ) { writeFile file: "$WORKSPACE_TMP/lib.groovy" , text: libraryResource( 'lib.groovy' ) withGroovy(input: params) { sh 'groovy "$WORKSPACE_TMP/lib.groovy" ' } } } and resources/lib.groovy : String generatePipelineScript(Map params) { // go wild, this is stock Groovy running in its own process } Pipeline.output(generatePipelineScript(Pipeline.input())) The above scenario is not tested, just off the top of my head. You do not really need a library at all, BTW, if you are using a hard-coded Jenkinsfile —can just inline the withGroovy idiom, and get your lib.groovy or whatever from a Git checkout of some tool repo, etc.

          Dee Kryvenko added a comment -

          Amazing! This is very cool. Yeah I do not currently have a node available in this setup stage - but it wouldn't be a problem to schedule a build pod to run this. Last question though - can I pass into the `input` some context objects like `env` or `params`? I suppose for the logging purposes since it would be a standalone process now - I can just spit out everything to stdout, but the input would be the last piece of the puzzle.

          No.

          Lastly, why did you lie to me? The answer is - yes!

          You do not really need a library at all

          In this case it sounds like - I probably do not need Groovy at all! My generator could be anything really.

          Dee Kryvenko added a comment - Amazing! This is very cool. Yeah I do not currently have a node available in this setup stage - but it wouldn't be a problem to schedule a build pod to run this. Last question though - can I pass into the `input` some context objects like `env` or `params`? I suppose for the logging purposes since it would be a standalone process now - I can just spit out everything to stdout, but the input would be the last piece of the puzzle. No. Lastly, why did you lie to me? The answer is - yes! You do not really need a library at all In this case it sounds like - I probably do not need Groovy at all! My generator could be anything really.

          Jesse Glick added a comment -

          can I pass into the `input` some context objects like `env` or `params`?

          Check step docs. Can basically be any plain old Java/Groovy structures like lists and maps and the like. No Jenkins-defined objects, no user-defined types.

          why did you lie to me? The answer is - yes!

          Well, for your tightly constrained case there is a fairly easy resolution. This changes nothing about how the “real” Pipeline is executed.

          I probably do not need Groovy at all! My generator could be anything really.

          Sure, it could. withGroovy is most helpful if you had a lot of Groovy code you do not want to rewrite, or just happen to love (real) Groovy. And even before this step, you could of course run

          def result
          withEnv(["ONE_PARAM=$params.one", "TWO_PARAM=$params.two"]) {
           result = sh script: 'groovy lib.groovy', returnStdout: true
          }
          

          but there would be hassles that might tempt you to skip it and keep on doing everything inside Jenkins: you would have to make sure Groovy (and a JRE) were installed, structured parameters would have to be manually flattened or formatted at JSON, the output could get corrupted by stray println’s, etc.

          Jesse Glick added a comment - can I pass into the `input` some context objects like `env` or `params`? Check step docs. Can basically be any plain old Java/Groovy structures like lists and maps and the like. No Jenkins-defined objects, no user-defined types. why did you lie to me? The answer is - yes! Well, for your tightly constrained case there is a fairly easy resolution. This changes nothing about how the “real” Pipeline is executed. I probably do not need Groovy at all! My generator could be anything really. Sure, it could. withGroovy is most helpful if you had a lot of Groovy code you do not want to rewrite, or just happen to love (real) Groovy. And even before this step, you could of course run def result withEnv([ "ONE_PARAM=$params.one" , "TWO_PARAM=$params.two" ]) { result = sh script: 'groovy lib.groovy' , returnStdout: true } but there would be hassles that might tempt you to skip it and keep on doing everything inside Jenkins: you would have to make sure Groovy (and a JRE) were installed, structured parameters would have to be manually flattened or formatted at JSON, the output could get corrupted by stray println ’s, etc.

          Dee Kryvenko added a comment -

          Thanks a lot! Yeah indeed it does not solve the original issue which was about the limits CPS have over normal Groovy... that unless we want to say the solution is - stop using shared libraries. Which happens to work in my specific case for one specific library. But, seriously, you have no idea how easier you just made my life going forward. You have a great weekend sir!

          Dee Kryvenko added a comment - Thanks a lot! Yeah indeed it does not solve the original issue which was about the limits CPS have over normal Groovy... that unless we want to say the solution is - stop using shared libraries. Which happens to work in my specific case for one specific library. But, seriously, you have no idea how easier you just made my life going forward. You have a great weekend sir!

          red der added a comment -

          I feel like this really should be documented. Maybe doesn't need to be exhaustively documented, but I wasted some time.

          I spent some time refactoring some code locally and DRYed it up with `@MapConstructor` which I was excited to use, tested it, etc.

          Only to get to the point where I ran an actual build I saw this:
          unable to resolve class groovy.transform.MapConstructor
          @ line 3, column 1.
          import groovy.transform.MapConstructor
          ^
           

          red der added a comment - I feel like this really should be documented. Maybe doesn't need to be exhaustively documented, but I wasted some time. I spent some time refactoring some code locally and DRYed it up with `@MapConstructor` which I was excited to use, tested it, etc. Only to get to the point where I ran an actual build I saw this: unable to resolve class groovy.transform.MapConstructor @ line 3, column 1. import groovy.transform.MapConstructor ^  

          John Hancock added a comment -

          We use Jacoco to generate code coverage numbers on our shared Jenkins library, but trying to use the `@Generated` annotation to exclude a method from code coverage causes an actual Jenkins pipeline to fail.

          John Hancock added a comment - We use Jacoco to generate code coverage numbers on our shared Jenkins library, but trying to use the `@Generated` annotation to exclude a method from code coverage causes an actual Jenkins pipeline to fail.

          Zaitcev Peter added a comment -

          Are there any updates on this issue?

          Zaitcev Peter added a comment - Are there any updates on this issue?

            Unassigned Unassigned
            pmr Philipp Moeller
            Votes:
            15 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated: