Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49904

workflow-cps groovy engine hijacks closure.rehydrate()

      workflow-cps-plugin's CpsScript breaks a fundamental Groovy language feature: closure.rehydrate().
       
      In this short snippet of pipeline code we expect hydratedClosure() to run foo() successfully (which it does in a regular GroovyShell or Jenkins script console):
       

      class TasksSpec implements Serializable {
          TasksSpec() {
          }
      
          def foo() {
              println this
          }
      
          def runClosure(closure) {
              def hydratedClosure = closure.rehydrate(this, this, this)
              hydratedClosure.resolveStrategy = Closure.DELEGATE_ONLY
              hydratedClosure()
           }
      }
      
      def tasksSpec = new TasksSpec()
      tasksSpec.runClosure() {
          foo()
      }
      

       
      However, the Groovy Cps implementation seems to totally disregard the fact that we carry out a rehydrate() with the intention to execute the closure in the context of the tasksSpec object. Instead the closure is executed in the context of the WorkflowScript and fails with:
       

      Java.lang.NoSuchMethodError: No such DSL method 'foo' found among steps [archive, bat, catchError, checkout, deleteDir, dir, echo, error, fileExists, getContext, git, isUnix, library, libraryResource, load, mail, node, parallel, pwd, readFile, retry, sh, sleep, stash, step, svn, timeout, tool, unarchive, unstash, waitUntil, withContext, withEnv, wrap, writeFile, ws] or symbols [all, always, apiToken, architecture, archiveArtifacts, artifactManager, batchFile, booleanParam, buildButton, buildDiscarder, caseInsensitive, caseSensitive, choice, choiceParam, clock, cloud, command, credentials, cron, crumb, defaultView, demand, disableConcurrentBuilds, downloadSettings, downstream, dumb, envVars, file, fileParam, filePath, fingerprint, frameOptions, freeStyle, freeStyleJob, fromScm, fromSource, git, headRegexFilter, headWildcardFilter, hyperlink, hyperlinkToModels, installSource, jdk, jdkInstaller, jgit, jgitapache, jnlp, jobName, junit, lastDuration, lastFailure, lastGrantedAuthorities, lastStable, lastSuccess, legacy, legacySCM, list, local, location, logRotator, loggedInUsersCanDoAnything, masterBuild, maven, maven3Mojos, mavenErrors, mavenMojos, mavenWarnings, modernSCM, myView, nodeProperties, nonStoredPasswordParam, none, paneStatus, parameters, password, pattern, pipelineTriggers, plainText, plugin, projectNamingStrategy, proxy, queueItemAuthenticator, quietPeriod, run, runParam, schedule, scm, scmRetryCount, search, security, shell, slave, sourceRegexFilter, sourceWildcardFilter, stackTrace, standard, status, string, stringParam, swapSpace, text, textParam, tmpSpace, toolLocation, unsecured, upstream, viewsTabBar, weather, zfs, zip] or globals [currentBuild, env, params]
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
      	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1280)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1174)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
      	at WorkflowScript.run(WorkflowScript:23)
      	at TasksSpec.runClosure(WorkflowScript:14)
      	at WorkflowScript.run(WorkflowScript:21)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:77)
      	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
      	at java.util.concurrent.FutureTask.run(FutureTask.java)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call$$$capture(Executors.java:511)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java)
      	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
      	at java.util.concurrent.FutureTask.run(FutureTask.java)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

       
      It seems that if we try to carry out a rehydrate(), the closure ends up being always executed through CpsScripts invokeMethod():

          /**
           * We use DSL here to try invoke the step implementation, if there is Step implementation found it's handled or
           * it's an error.
           *
           * <p>
           * sandbox security execution relies on the assumption that CpsScript.invokeMethod() is safe for sandboxed code.
           * That means we cannot let user-written script override this method, hence the final.
           */
          @Override
          public final Object invokeMethod(String name, Object args) {
              // TODO probably better to call super method and only proceed here incase of MissingMethodException:
              // if global variables are defined by that name, try to call it.
              // the 'call' convention comes from Closure
              GlobalVariable v = GlobalVariable.byName(name, $buildNoException());
              if (v != null) {
                  try {
                      Object o = v.getValue(this);
                      return InvokerHelper.getMetaClass(o).invokeMethod(o, "call", args);
                  } catch (Exception x) {
                      throw new InvokerInvocationException(x);
                  }
              }
      
              // otherwise try Step impls.
              DSL dsl = (DSL) getBinding().getVariable(STEPS_VAR);
              return dsl.invokeMethod(name,args);
          }
      

      Instead of running tasksSpec.foo() when hydratedClosure() is executed, invokeMethod() tries to find/execute the call()-method on the (non-existing) global variable foo or if that fails, continue by searching through the pipeline build steps for a match!
       
      This has some major annoying consequences when trying to reuse (or design) a groovy framework for building pipelines.

          [JENKINS-49904] workflow-cps groovy engine hijacks closure.rehydrate()

          Gustaf Lundh added a comment -

          jglick: Sorry for pinging you, but we are currently building a groovy framework/backend for pipeline jobs. The framework would allow our end users to setup and manage extremely complex pipelines, consisting of 100's of projects (and allow them to this in an reasonable easy way).

          Our original vision was very dependent on a working rehydrate() functionality. So before we restart the design work, I would like to know if you think this issue will be fixed within a reasonable timeframe (or if Cloudbees was already aware of the issue). Or perhaps this behaviour is by design?

          The Groovy and Cps implementation is still a bit too alien to allow me to fix the issue myself within a sensible timeframe. 

          Gustaf Lundh added a comment - jglick : Sorry for pinging you, but we are currently building a groovy framework/backend for pipeline jobs. The framework would allow our end users to setup and manage extremely complex pipelines, consisting of 100's of projects (and allow them to this in an reasonable easy way). Our original vision was very dependent on a working rehydrate() functionality. So before we restart the design work, I would like to know if you think this issue will be fixed within a reasonable timeframe (or if Cloudbees was already aware of the issue). Or perhaps this behaviour is by design? The Groovy and Cps implementation is still a bit too alien to allow me to fix the issue myself within a sensible timeframe. 

          Andrew Bayer added a comment -

          gustafl - so from a preliminary investigation, the underlying problem here is something with Closure#clone(). If you just set the delegate on the closure (and the resolveStrategy, it works, but cloning somehow gets things stuck.

          Andrew Bayer added a comment - gustafl - so from a preliminary investigation, the underlying problem here is something with Closure#clone() . If you just set the delegate on the closure (and the resolveStrategy , it works, but cloning somehow gets things stuck.

          Logan Mzz added a comment -

          I have same needs for same reason: having both a delegation DSL and access to script/definition-time scope.

           

          mylib.pipeline { // <- call it 'pipeline' closure, only delegate to 'PipelineDSL'
              init { // <- call it 'pipeline.init' closure, delegate to 'StageInitDSL'
                  // delegate == StageInitDSL@xxxxxx
                  // owner == pipeline
                  // this == WorkflowScript@xxxxxxx
                  
                  echo 'blablabla...' // <- Not possible because, it's only accessible from 'script' scope
                  this.echo 'blablabla' // OK   
              }
          }
          

          Solution would be to set init.owner from pipeline.thisObject but it currently doesn't work:

          class MyLib {
              def pipeline(declaration) {
                  def stages = [init: {}]
                  def pipelineDSL = new PipelineDSL()
                  def stageDSL    = new StageDSL()
                  pipelineDsl.init = {  // it.delegate == declaration
                                        // it.owner    == declaration
                                        // it.this     == script
                      def fn = it.rehydrate(stageDSL, it.thisObject, it.thisObject)
                      fn.resolveStrategy = Closure.DELEGATE_FIRST
                      stages['init'] = fn  // fn.delegate == stageDSL
                                           // fn.owner    == script
                                           // fn.this     == script
                  }
                  declaration.delegate = pipelineDSL
                  declaration.resolveStrategy = Closure.DELEGATE_ONLY
                  declaration()
          
                  stages.each { name, fn -> fn() }
              }
          }
          
          def mylib = new MyLib()
          
          mylib.pipeline {
              init {
                  delegate  // == declaration/pipeline  <>  script
                  owner     // == declaration/pipeline  <>  script
                  this      // == script
          
                  echo       //  Unknown property
                  mylib      //  Unknown property
                  this.echo  //  Ok
                  this.mylib //  Ok
              }
          }
          

           

          Logan Mzz added a comment - I have same needs for same reason: having both a delegation DSL and access to script/definition-time scope.   mylib.pipeline { // <- call it 'pipeline' closure, only delegate to 'PipelineDSL' init { // <- call it 'pipeline.init' closure, delegate to 'StageInitDSL' // delegate == StageInitDSL@xxxxxx // owner == pipeline // this == WorkflowScript@xxxxxxx echo 'blablabla...' // <- Not possible because, it 's only accessible from ' script' scope this .echo 'blablabla' // OK } } Solution would be to set init.owner from pipeline.thisObject but it currently doesn't work: class MyLib { def pipeline(declaration) { def stages = [init: {}] def pipelineDSL = new PipelineDSL() def stageDSL = new StageDSL() pipelineDsl.init = { // it.delegate == declaration // it.owner == declaration // it. this == script def fn = it.rehydrate(stageDSL, it.thisObject, it.thisObject) fn.resolveStrategy = Closure.DELEGATE_FIRST stages[ 'init' ] = fn // fn.delegate == stageDSL // fn.owner == script // fn. this == script } declaration.delegate = pipelineDSL declaration.resolveStrategy = Closure.DELEGATE_ONLY declaration() stages.each { name, fn -> fn() } } } def mylib = new MyLib() mylib.pipeline { init { delegate // == declaration/pipeline <> script owner // == declaration/pipeline <> script this // == script echo // Unknown property mylib // Unknown property this .echo // Ok this .mylib // Ok } }  

          Has anyone managed to get through this problem OR is there any fix for it ?

          Benjamin Boyer added a comment - Has anyone managed to get through this problem OR is there any fix for it ?

          Ivan Martinez added a comment -

          Any news on this issue? Is there any workaround to rehydrate a Closure without using the rehydrate method from the Closure class itself?

          Ivan Martinez added a comment - Any news on this issue? Is there any workaround to rehydrate a Closure without using the rehydrate method from the Closure class itself?

          Jesse Glick added a comment -

          I would not expect support for exotic language features like Closure.rehydrate or .resolveStrategy to ever be implemented. Switch to something more straightforward.

          Jesse Glick added a comment - I would not expect support for exotic language features like Closure.rehydrate or .resolveStrategy to ever be implemented. Switch to something more straightforward.

          Logan Mzz added a comment -

          'exotic', this is some "basics" Groovy feature. It is part of what Groovy shines to DSL support.

          But as I tend to say, Jenkins pipeline scripting language is not Groovy.

          Logan Mzz added a comment - 'exotic', this is some "basics" Groovy feature. It is part of what Groovy shines to DSL support. But as I tend to say, Jenkins pipeline scripting language is not Groovy.

            Unassigned Unassigned
            gustafl Gustaf Lundh
            Votes:
            7 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: