Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26192

Add support for Grape (@Grab) in workflow scripts

      The workflow plugin and its groovy DSL should support the use of Grape and the @Grab attribute to allow the creation of workflows that depend on external libraries such as HTTP Builder or jYaml.

      This issue seems closely related to JENKINS-18349 and other issues referenced in the comments of that issue.

          [JENKINS-26192] Add support for Grape (@Grab) in workflow scripts

          Kenneth Baltrinic created issue -

          Jesse Glick added a comment -

          (I could have sworn this was already filed, but I cannot find it now. The idea was brought up during the Workflow Summit a couple months ago but perhaps I failed to record it.)

          There are two major reasons why Grape support could be problematic here. First, binary dependencies (compiled to *.class) cannot participate in CPS transformation, and thus calls to libraries would be considered “native” methods which cannot survive a Jenkins restart. If the call does significant I/O, especially network I/O, this would make the flow less robust. (This consideration shows why this issue differs from JENKINS-18349; the plain Groovy plugin just makes a call to the standard embedded Groovy runtime, whereas Workflow uses its own CPS-transformed interpreter.)

          Second, when running in Groovy sandbox mode, the normal case when running a secured Jenkins instance with a distinction between administrators and job maintainers, any native calls would have to whitelisted, which assumes they are unconditionally safe—which might not be true if, say, a URL handling library permits file-protocol URLs. This consideration is in common with other plugins allowing use of Groovy in Jenkins.

          Both these considerations would also apply if the workflow definition were to support the optional binary classpath feature in the Script Security plugin (currently it does not).

          While it is obviously not practical to do so for every useful Groovy library, in general it is best to create custom workflow steps for important operations. These can then be made to survive restarts (or slave disconnections) if necessary, and can perform any necessary input validation and/or access control checks. Where applicable the step can also interact with files in a remote (slave) workspace, which would be impossible if using a non-Jenkins-specific library directly.

          If you need to perform some complex or expensive tasks with unrestricted Groovy physically running on a slave, it may be simplest and most effective to simply write that code in a *.groovy file in your workspace (for example, in an SCM checkout) and then use tool and sh/bat to run Groovy as an external process; or even put this stuff into a Gradle script, Groovy Maven plugin execution, etc. The workflow script itself should be limited to simple and extremely lightweight logical operations focused on orchestrating the overall flow of control and interacting with other Jenkins features—slave allocation, user input, and the like.

          Jesse Glick added a comment - (I could have sworn this was already filed, but I cannot find it now. The idea was brought up during the Workflow Summit a couple months ago but perhaps I failed to record it.) There are two major reasons why Grape support could be problematic here. First, binary dependencies (compiled to *.class ) cannot participate in CPS transformation, and thus calls to libraries would be considered “native” methods which cannot survive a Jenkins restart. If the call does significant I/O, especially network I/O, this would make the flow less robust. (This consideration shows why this issue differs from JENKINS-18349 ; the plain Groovy plugin just makes a call to the standard embedded Groovy runtime, whereas Workflow uses its own CPS-transformed interpreter.) Second, when running in Groovy sandbox mode, the normal case when running a secured Jenkins instance with a distinction between administrators and job maintainers, any native calls would have to whitelisted, which assumes they are unconditionally safe—which might not be true if, say, a URL handling library permits file -protocol URLs. This consideration is in common with other plugins allowing use of Groovy in Jenkins. Both these considerations would also apply if the workflow definition were to support the optional binary classpath feature in the Script Security plugin (currently it does not). While it is obviously not practical to do so for every useful Groovy library, in general it is best to create custom workflow steps for important operations. These can then be made to survive restarts (or slave disconnections) if necessary, and can perform any necessary input validation and/or access control checks. Where applicable the step can also interact with files in a remote (slave) workspace, which would be impossible if using a non-Jenkins-specific library directly. If you need to perform some complex or expensive tasks with unrestricted Groovy physically running on a slave, it may be simplest and most effective to simply write that code in a *.groovy file in your workspace (for example, in an SCM checkout) and then use tool and sh / bat to run Groovy as an external process; or even put this stuff into a Gradle script, Groovy Maven plugin execution, etc. The workflow script itself should be limited to simple and extremely lightweight logical operations focused on orchestrating the overall flow of control and interacting with other Jenkins features—slave allocation, user input, and the like.

          Jesse Glick that seems like good advice, but there are a number of use cases where external libraries provide enormous value and don't involve significant I/O or computation.

          For me, the most important are web service client libraries, which would be terrific to be able to conveniently use from a workflow script to coordinate all the busy work that surrounds building and deployment.

          After a brief fit of frustration trying to add support for @Grab to the cps/workflow-cps-plugin, I ended up writing/re-writing a half-dozen REST client libraries in Groovy scripts so that our build and deployment process could coordinate with JIRA, HipChat, GitHub, and release notes website. Using only 'sh' step, I also finally have great 'library' for managing deploys to AppEngine!

          The result is actually quite nice, but now I have, for example, the source for a GitHub client code locked into a git repo on our jenkins server – and no clear way to contribute this back to other users.

          If not Grape, we need some way to package and reuse workflow logic, and the jenkins plugin mechanism feels too heavy after getting used to a rapid change, push, and build 'workflow'.

          Do you guys have a general direction in mind for this kind of code reuse?

          Alexander Bertram added a comment - Jesse Glick that seems like good advice, but there are a number of use cases where external libraries provide enormous value and don't involve significant I/O or computation. For me, the most important are web service client libraries, which would be terrific to be able to conveniently use from a workflow script to coordinate all the busy work that surrounds building and deployment. After a brief fit of frustration trying to add support for @Grab to the cps/workflow-cps-plugin, I ended up writing/re-writing a half-dozen REST client libraries in Groovy scripts so that our build and deployment process could coordinate with JIRA, HipChat, GitHub, and release notes website. Using only 'sh' step, I also finally have great 'library' for managing deploys to AppEngine! The result is actually quite nice, but now I have, for example, the source for a GitHub client code locked into a git repo on our jenkins server – and no clear way to contribute this back to other users. If not Grape, we need some way to package and reuse workflow logic, and the jenkins plugin mechanism feels too heavy after getting used to a rapid change, push, and build 'workflow'. Do you guys have a general direction in mind for this kind of code reuse?

          Jesse Glick added a comment -

          For the case of web service client libraries, again I do not think you want to run these as plain blocking Java calls (for example using an existing grape) because they involve network I/O which would block the CPS VM thread and not survive interruption. You could write a Step using asynchronous I/O. Or you could use DurableTaskStep, such as simply via sh, or perhaps we need a variant that runs the local JRE/bin/java with a main class, or runs a forked groovy interpreter, etc.

          I had no particular plans for cross-server code reuse other than via plugins with steps. The GroovyShellDecorator extension point allows other options. If Grape supports Groovy-source-based libraries then supporting it from Workflow would presumably work via this extension point.

          Jesse Glick added a comment - For the case of web service client libraries, again I do not think you want to run these as plain blocking Java calls (for example using an existing grape) because they involve network I/O which would block the CPS VM thread and not survive interruption. You could write a Step using asynchronous I/O. Or you could use DurableTaskStep , such as simply via sh , or perhaps we need a variant that runs the local JRE/bin/java with a main class, or runs a forked groovy interpreter, etc. I had no particular plans for cross-server code reuse other than via plugins with steps. The GroovyShellDecorator extension point allows other options. If Grape supports Groovy-source-based libraries then supporting it from Workflow would presumably work via this extension point.

          CPS was designed for writing build definitions, and I can see why one should avoid turning it into a general data processing language.

          In my use case, data processing (e.g. communicating with a web service) could as well be done with a beefed-up Groovy build step (-> JENKINS-26635)

          Thomas Goeppel added a comment - CPS was designed for writing build definitions, and I can see why one should avoid turning it into a general data processing language. In my use case, data processing (e.g. communicating with a web service) could as well be done with a beefed-up Groovy build step (-> JENKINS-26635 )
          Jesse Glick made changes -
          Link New: This issue is related to JENKINS-26635 [ JENKINS-26635 ]

          Jesse Glick added a comment -

          This RFE goes against the intended usage of Workflow. JENKINS-26635 would be easier to do and have a much more effective result.

          Jesse Glick added a comment - This RFE goes against the intended usage of Workflow. JENKINS-26635 would be easier to do and have a much more effective result.
          Jesse Glick made changes -
          Resolution New: Won't Do [ 10001 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Jesse Glick made changes -
          Link New: This issue is related to JENKINS-31155 [ JENKINS-31155 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 160183 ] New: JNJira + In-Review [ 196355 ]

            jglick Jesse Glick
            kbaltrinic Kenneth Baltrinic
            Votes:
            7 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: