• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • workflow-cps-plugin
    • None

      Hi,

      First of all - I will describe what I'm trying to do:

      I have multiple "library" pipeline scripts. I'm using the pipeline Load step to load them.

      Some scripts need to use methods in other scripts - think of the following scenario:
      pipeline Job loads script1.groovy and script2.groovy , script1.groovy loads script2.groovy as well. This results in script2.groovy being loaded twice for no good reason.

      I want to achieve lazy loading, meaning if the same script has already been loaded - I want to reuse it and not load it again, meaning script1.groovy should use script2.groovy which was loaded by the job itself.

      I tried to do it with the following code - in this example I do want the 2nd script to load the file on its own:

      job code:

      timestamps {
      
          node('some node') {
       
              //some code to get the files to the workspace - not relevant to the situation
              def test = load("./src/jenkins/pipeline/utils/test.groovy")
      
              test.test()
      
          }
      }
      

       

      test.groovy

      /* ***** lazy loading - BEGIN ***** */
      this._scripts = [:]
      
      def getScript(def scriptName) {
          // lazy loading
          if (this._scripts."${scriptName}" == null) {
              this._scripts."${scriptName}" = load("${WORKSPACE}/src/jenkins/pipeline/utils/${scriptName}.groovy")
              this._scripts."${scriptName}".init(this._scripts)
          }
          return this._scripts."${scriptName}"
      }
      
      def init(def scripts) {
          if (scripts)
              this._scripts = scripts
      }
      
      /* ***** lazy loading - END ***** */
      
      def test() {
          getScript("command_runner").runHostCommand("echo 'hello world'")
      }
      
      return this
      

      command_runner.groovy

      /* ***** lazy loading - BEGIN ***** */
      this._scripts = [:]
      
      def getScript(def scriptName) {
          // lazy loading
          if (this._scripts."${scriptName}" == null) {
              this._scripts."${scriptName}" = load("${WORKSPACE}/src/jenkins/pipeline/utils/${scriptName}.groovy")
              this._scripts."${scriptName}".init(this._scripts)
          }
          return this._scripts."${scriptName}"
      }
      
      def init(def scripts) {
          if (scripts)
              this._scripts = scripts
      }
      
      /* ***** lazy loading - END ***** */
      
      
      /**
       * Runs a command and returns the exit code. assumes windows [for now]
       * @param command
       * @return
       */
      def runHostCommand(def command) {
          int exitCode
          exitCode = bat script: command, returnStatus: true
      
          return exitCode
      }
      
      def runHostCommandWithOutput(def command) {
          currentDir = pwd()
          println "${currentDir}>${command}"
          def output = bat (script: command, returnStdout: true).trim()
          //output contains the call the command itself so we should remove it when printing
          output = output.substring(output.indexOf('\n')+1);
          println output
          return output
      }
      
      /* *** mandatory to be able to call from another bootstrap script *** */
      return this
      

      When running it - I get the following exception

      ...
      [Pipeline] load
      [Pipeline] { (./src/jenkins/pipeline/utils/test.groovy)
      [Pipeline] }
      [Pipeline] // load
      [Pipeline] load
      [Pipeline] { (F:\jenkins-remote\workspace\Mor\pipeline_tests/src/jenkins/pipeline/utils/command_runner.groovy)
      [Pipeline] }
      [Pipeline] // load
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      java.lang.NullPointerException: Cannot invoke method init() on null object
      	at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
      	at Script1.getScript(Script1.groovy:13)
      	at Script1.test(Script1.groovy:30)
      	at WorkflowScript.run(WorkflowScript:26)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:76)
      	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
      	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66)
      	at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      Finished: FAILURE
      

      According to the log, the Load command within test.groovy succeeded, but for some reason returned null.

       

      so I'm looking for either a solution to the problem, or a better way to achieve lazy loading. And yes - I am familiar with Shared Pipeline Libraries plugin which is not suitable for my case at this stage.

       

      Thanks in advance,

      Mor

          [JENKINS-44841] Load step within loaded script returns null

          Mor L added a comment -

          It looks as though if I replace the groovy property notation

           

          this._scripts."${scriptName}"
          

          with standard map put/get calls - it works (at least it doesn't fail with the same error for the exact same scenario - I haven't tested all scenarios yet)

           

          def getScript(def scriptName) {
              // lazy loading
              if (!this._scripts.get(scriptName)) {
                  this._scripts.put(scriptName, load("${WORKSPACE}/src/jenkins/pipeline/utils/${scriptName}.groovy"))
                  this._scripts.get(scriptName).init(this._scripts)
              }
              return this._scripts.get(scriptName)
          }
          

          So this must be a pipeline bug...

          If it works for all scenarios - then I have my lazy loading working which is great

           

          Mor L added a comment - It looks as though if I replace the groovy property notation   this ._scripts. "${scriptName}" with standard map put/get calls - it works (at least it doesn't fail with the same error for the exact same scenario - I haven't tested all scenarios yet)   def getScript(def scriptName) { // lazy loading if (! this ._scripts.get(scriptName)) { this ._scripts.put(scriptName, load( "${WORKSPACE}/src/jenkins/pipeline/utils/${scriptName}.groovy" )) this ._scripts.get(scriptName).init( this ._scripts) } return this ._scripts.get(scriptName) } So this must be a pipeline bug... If it works for all scenarios - then I have my lazy loading working which is great  

          Jesse Glick added a comment -

          Is

          this._scripts."${scriptName}"

          even valid Groovy syntax? I have never seen that before; looks like it should be a compiler error to me. If you have a dynamic map key, the sugar is

          this._scripts[scriptName]
          

          Jesse Glick added a comment - Is this ._scripts. "${scriptName}" even valid Groovy syntax? I have never seen that before; looks like it should be a compiler error to me. If you have a dynamic map key, the sugar is this ._scripts[scriptName]

          Mor L added a comment - - edited

          using CodingGround 's groovy console for instance - the following works (they use groovy 2.4.4):

           

          def map = [:]
          map."hello" = "world"
          
          def key = "hello"
          
          assert map[key] == "world"
          assert map."${key}" == "world"
          

           But I do agree the [] notation is prettier. I will switch to that and continue testing

          Mor L added a comment - - edited using CodingGround 's groovy console for instance - the following works (they use groovy 2.4.4):   def map = [:] map. "hello" = "world" def key = "hello" assert map[key] == "world" assert map. "${key}" == "world"  But I do agree the [] notation is prettier. I will switch to that and continue testing

          Jesse Glick added a comment -

          It is possible the dot notation calls something different internally which is not yet supported by either groovy-sandbox or groovy-cps. I am guessing it has never been tested.

          Jesse Glick added a comment - It is possible the dot notation calls something different internally which is not yet supported by either groovy-sandbox or groovy-cps . I am guessing it has never been tested.

          Mor L added a comment -

          Do you agree its a bug then?
          The title needs to be changed though to reflect the actual issue.
          But, I do have a workaround so the priority can be lowered

          Mor L added a comment - Do you agree its a bug then? The title needs to be changed though to reflect the actual issue. But, I do have a workaround so the priority can be lowered

          Jesse Glick added a comment -

          Sounds like a bug. Do you know of some simpler way to reproduce that would clarify the issue?

          Jesse Glick added a comment - Sounds like a bug. Do you know of some simpler way to reproduce that would clarify the issue?

          Mor L added a comment -

          I will try to reproduce with minimal code tomorrow, hopefully without loading other scripts.

          Mor L added a comment - I will try to reproduce with minimal code tomorrow, hopefully without loading other scripts.

          Mor L added a comment -

          Unfortunately the same error reproduces even when using the [] notation.

          This happens when a script loads a script which loads another script - I get the same 
          java.lang.NullPointerException: Cannot invoke method init() on null object
          So I'm not entirely sure the problem is just with the dot notation.

          My current workaround is to use pre-loading (load all needed scripts in the first script and then init all other scripts).

           

          Mor L added a comment - Unfortunately the same error reproduces even when using the [] notation. This happens when a script loads a script which loads another script - I get the same  java.lang.NullPointerException: Cannot invoke method init() on null object So I'm not entirely sure the problem is just with the dot notation. My current workaround is to use pre-loading (load all needed scripts in the first script and then init all other scripts).  

          Jesse Glick added a comment -

          Will leave this open, but am not likely to spend time on it unless there is a minimal test case.

          Jesse Glick added a comment - Will leave this open, but am not likely to spend time on it unless there is a minimal test case.

          Mor L added a comment -

          I understand, I have more urgent issues to attend to but will do my best to have a minimal case sometime soon. Regardless, thanks for taking the time to reply

          Mor L added a comment - I understand, I have more urgent issues to attend to but will do my best to have a minimal case sometime soon. Regardless, thanks for taking the time to reply

            Unassigned Unassigned
            pyrocks Mor L
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: