-
New Feature
-
Resolution: Fixed
-
Major
-
None
The workflow plugin and its groovy DSL should support the use of Grape and the @Grab attribute to allow the creation of workflows that depend on external libraries such as HTTP Builder or jYaml.
This issue seems closely related to JENKINS-18349 and other issues referenced in the comments of that issue.
- depends on
-
-
- Open
-
- is related to
-
-
- Resolved
-
-
-
- Open
-
-
JENKINS-31155 Workflow shared library improvements
-
- Closed
-
- links to
(1 links to)
[JENKINS-26192] Add support for Grape (@Grab) in workflow scripts
Alexander Bertram
added a comment - Jesse Glick that seems like good advice, but there are a number of use cases where external libraries provide enormous value and don't involve significant I/O or computation.
For me, the most important are web service client libraries, which would be terrific to be able to conveniently use from a workflow script to coordinate all the busy work that surrounds building and deployment.
After a brief fit of frustration trying to add support for @Grab to the cps/workflow-cps-plugin, I ended up writing/re-writing a half-dozen REST client libraries in Groovy scripts so that our build and deployment process could coordinate with JIRA, HipChat, GitHub, and release notes website. Using only 'sh' step, I also finally have great 'library' for managing deploys to AppEngine!
The result is actually quite nice, but now I have, for example, the source for a GitHub client code locked into a git repo on our jenkins server – and no clear way to contribute this back to other users.
If not Grape, we need some way to package and reuse workflow logic, and the jenkins plugin mechanism feels too heavy after getting used to a rapid change, push, and build 'workflow'.
Do you guys have a general direction in mind for this kind of code reuse?
Alexander Bertram
added a comment - Jesse Glick that seems like good advice, but there are a number of use cases where external libraries provide enormous value and don't involve significant I/O or computation.
For me, the most important are web service client libraries, which would be terrific to be able to conveniently use from a workflow script to coordinate all the busy work that surrounds building and deployment.
After a brief fit of frustration trying to add support for @Grab to the cps/workflow-cps-plugin, I ended up writing/re-writing a half-dozen REST client libraries in Groovy scripts so that our build and deployment process could coordinate with JIRA, HipChat, GitHub, and release notes website. Using only 'sh' step, I also finally have great 'library' for managing deploys to AppEngine!
The result is actually quite nice, but now I have, for example, the source for a GitHub client code locked into a git repo on our jenkins server – and no clear way to contribute this back to other users.
If not Grape, we need some way to package and reuse workflow logic, and the jenkins plugin mechanism feels too heavy after getting used to a rapid change, push, and build 'workflow'.
Do you guys have a general direction in mind for this kind of code reuse? For the case of web service client libraries, again I do not think you want to run these as plain blocking Java calls (for example using an existing grape) because they involve network I/O which would block the CPS VM thread and not survive interruption. You could write a Step using asynchronous I/O. Or you could use DurableTaskStep, such as simply via sh, or perhaps we need a variant that runs the local JRE/bin/java with a main class, or runs a forked groovy interpreter, etc.
I had no particular plans for cross-server code reuse other than via plugins with steps. The GroovyShellDecorator extension point allows other options. If Grape supports Groovy-source-based libraries then supporting it from Workflow would presumably work via this extension point.
Thomas Goeppel
added a comment - CPS was designed for writing build definitions, and I can see why one should avoid turning it into a general data processing language.
In my use case, data processing (e.g. communicating with a web service) could as well be done with a beefed-up Groovy build step (-> JENKINS-26635)
Thomas Goeppel
added a comment - CPS was designed for writing build definitions, and I can see why one should avoid turning it into a general data processing language.
In my use case, data processing (e.g. communicating with a web service) could as well be done with a beefed-up Groovy build step (-> JENKINS-26635 ) This RFE goes against the intended usage of Workflow. JENKINS-26635 would be easier to do and have a much more effective result.
While I continue to maintain that for many purposes this would be a bad idea, there are some legitimate use cases, where the library is really focused on (near-instantaneous) computation. In that case the CPS transformation is unnecessary. I found that it is possible to make global libraries load binary Java libraries via Grape. (The sandbox issue is no longer an issue in that case due to JENKINS-34650.)
It is also apparently possible to load CPS-transformed Groovy libraries via Grape in Jenkins 2 (does not work in Jenkins 1).
Built and installed the 2.3-SNAPSHOT version from the branch on jglick's fork of the repo.
When the global lib is loaded, before the build begins stepping, I get:
Started by user G. Richard Bellamy Replayed #48 Connecting to https://api.github.com using terradatum-automation/****** (Terradatum Automation GitHub credentials) [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s GitHub has been notified of this commit’s build result java.lang.NoClassDefFoundError: org/apache/ivy/core/settings/IvySettings at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetPublicMethods(Class.java:2902) at java.lang.Class.getMethods(Class.java:1615) at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1336) at java.beans.Introspector.getTargetMethodInfo(Introspector.java:1197) at java.beans.Introspector.getBeanInfo(Introspector.java:426) at java.beans.Introspector.getBeanInfo(Introspector.java:173) at groovy.lang.MetaClassImpl$15.run(MetaClassImpl.java:3290) at java.security.AccessController.doPrivileged(Native Method) at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3288) at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3265) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:254) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:285) at groovy.grape.GrapeIvy.$getStaticMetaClass(GrapeIvy.groovy) at groovy.grape.GrapeIvy.<init>(GrapeIvy.groovy:81) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at groovy.grape.Grape.getInstance(Grape.java:121) at groovy.grape.Grape.grab(Grape.java:159) at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:378) at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:545) at org.codehaus.groovy.control.ClassNodeResolver.tryAsLoaderClassOrScript(ClassNodeResolver.java:185) at org.codehaus.groovy.control.ClassNodeResolver.findClassNode(ClassNodeResolver.java:170) at org.codehaus.groovy.control.ClassNodeResolver.resolveName(ClassNodeResolver.java:126) at org.codehaus.groovy.control.ResolveVisitor.resolveToOuter(ResolveVisitor.java:676) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308) at org.codehaus.groovy.control.ResolveVisitor.resolveFromModule(ResolveVisitor.java:638) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:308) at org.codehaus.groovy.control.ResolveVisitor.resolve(ResolveVisitor.java:276) at org.codehaus.groovy.control.ResolveVisitor.resolveOrFail(ResolveVisitor.java:260) at org.codehaus.groovy.control.ResolveVisitor.resolveOrFail(ResolveVisitor.java:272) at org.codehaus.groovy.control.ResolveVisitor.transformConstructorCallExpression(ResolveVisitor.java:1047) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:706) at org.codehaus.groovy.control.ResolveVisitor.transformDeclarationExpression(ResolveVisitor.java:1088) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:698) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1318) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.control.ResolveVisitor.transformClosureExpression(ResolveVisitor.java:1040) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:704) at org.codehaus.groovy.ast.expr.Expression.transformExpressions(Expression.java:51) at org.codehaus.groovy.ast.expr.ArgumentListExpression.transformExpression(ArgumentListExpression.java:69) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:711) at org.codehaus.groovy.control.ResolveVisitor.transformMethodCallExpression(ResolveVisitor.java:1061) at org.codehaus.groovy.control.ResolveVisitor.transform(ResolveVisitor.java:702) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitExpressionStatement(ClassCodeExpressionTransformer.java:142) at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:42) at org.codehaus.groovy.ast.CodeVisitorSupport.visitBlockStatement(CodeVisitorSupport.java:37) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitBlockStatement(ClassCodeVisitorSupport.java:166) at org.codehaus.groovy.control.ResolveVisitor.visitBlockStatement(ResolveVisitor.java:1318) at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:71) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClassCodeContainer(ClassCodeVisitorSupport.java:104) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitConstructorOrMethod(ClassCodeVisitorSupport.java:115) at org.codehaus.groovy.ast.ClassCodeExpressionTransformer.visitConstructorOrMethod(ClassCodeExpressionTransformer.java:53) at org.codehaus.groovy.control.ResolveVisitor.visitConstructorOrMethod(ResolveVisitor.java:201) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitMethod(ClassCodeVisitorSupport.java:126) at org.codehaus.groovy.ast.ClassNode.visitContents(ClassNode.java:1078) at org.codehaus.groovy.ast.ClassCodeVisitorSupport.visitClass(ClassCodeVisitorSupport.java:53) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1261) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:176) at org.codehaus.groovy.control.CompilationUnit$11.call(CompilationUnit.java:651) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:542) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268) at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688) at groovy.lang.GroovyShell.parse(GroovyShell.java:700) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:67) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:410) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:373) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:213) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:404) Caused by: java.lang.ClassNotFoundException: org.apache.ivy.core.settings.IvySettings at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:450) at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:403) ... 94 more Finished: FAILURE
Since this code does not permit GrapeIvy to be loaded from an alternate class loader, it can only come from wherever Groovy is defined, which is currently in Jenkins core.
Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/plugins/WorkflowPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/5e60b00fd4750fb0dd6c0ec28516f6ee57b88d90
Log:
JENKINS-26192 Demonstrating @Grab support in a realistic class loader environment.
Code changed in jenkins
User: Oliver Gondža
Path:
src/test/java/plugins/WorkflowPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/cf30b67dada2829f67da3a882f711640f018dbb7
Log:
Merge pull request #176 from jglick/Grape-JENKINS-26192
JENKINS-26192 Demonstrating @Grab support in a realistic class loader environment
Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/6a9c8497af42...cf30b67dada2
Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/plugins/WorkflowPluginTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/0a43987a7dfad296b0521a7e242bcb6e090dc0f1
Log:
JENKINS-26192 workflow-cps-global-lib 2.3 was released.
Amends #176.
(I could have sworn this was already filed, but I cannot find it now. The idea was brought up during the Workflow Summit a couple months ago but perhaps I failed to record it.)
There are two major reasons why Grape support could be problematic here. First, binary dependencies (compiled to *.class) cannot participate in CPS transformation, and thus calls to libraries would be considered “native” methods which cannot survive a Jenkins restart. If the call does significant I/O, especially network I/O, this would make the flow less robust. (This consideration shows why this issue differs from JENKINS-18349; the plain Groovy plugin just makes a call to the standard embedded Groovy runtime, whereas Workflow uses its own CPS-transformed interpreter.)
Second, when running in Groovy sandbox mode, the normal case when running a secured Jenkins instance with a distinction between administrators and job maintainers, any native calls would have to whitelisted, which assumes they are unconditionally safe—which might not be true if, say, a URL handling library permits file-protocol URLs. This consideration is in common with other plugins allowing use of Groovy in Jenkins.
Both these considerations would also apply if the workflow definition were to support the optional binary classpath feature in the Script Security plugin (currently it does not).
While it is obviously not practical to do so for every useful Groovy library, in general it is best to create custom workflow steps for important operations. These can then be made to survive restarts (or slave disconnections) if necessary, and can perform any necessary input validation and/or access control checks. Where applicable the step can also interact with files in a remote (slave) workspace, which would be impossible if using a non-Jenkins-specific library directly.
If you need to perform some complex or expensive tasks with unrestricted Groovy physically running on a slave, it may be simplest and most effective to simply write that code in a *.groovy file in your workspace (for example, in an SCM checkout) and then use tool and sh/bat to run Groovy as an external process; or even put this stuff into a Gradle script, Groovy Maven plugin execution, etc. The workflow script itself should be limited to simple and extremely lightweight logical operations focused on orchestrating the overall flow of control and interacting with other Jenkins features—slave allocation, user input, and the like.