Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27421

Unserializable iterator & entry classes from Java Collections

      Not currently possible to use a Java 5-style for loop on an ArrayList (for example) from within CPS-transformed code, since its Iterator implementation is not marked Serializable.

          [JENKINS-27421] Unserializable iterator & entry classes from Java Collections

          Owen Mehegan added a comment -

          Being able to write:

          node('linux_slave', 'windows_slave') {
          // some steps to run in parallel on both slaves
          }

          would be fantastic.

          Owen Mehegan added a comment - Being able to write: node('linux_slave', 'windows_slave') { // some steps to run in parallel on both slaves } would be fantastic.

          Jesse Glick added a comment -

          owenmehegan you can already do that using

          def labels = ['linux', 'windows']
          for (int i = 0; i < labels.size(); i++) {
            node(labels.get(i)) {
              // something, though you presumably need to select sh vs. bat here
            }
          }
          

          Jesse Glick added a comment - owenmehegan you can already do that using def labels = [ 'linux' , 'windows' ] for ( int i = 0; i < labels.size(); i++) { node(labels.get(i)) { // something, though you presumably need to select sh vs. bat here } }

          Code changed in jenkins
          User: Jesse Glick
          Path:
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7
          Log:
          JENKINS-27421 Reproduced in test.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7 Log: JENKINS-27421 Reproduced in test.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
          http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e
          Log:
          Merge pull request #206 from jglick/Itr-JENKINS-27421

          JENKINS-27421 Investigating ArrayList.Itr problem

          Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e Log: Merge pull request #206 from jglick/Itr- JENKINS-27421 JENKINS-27421 Investigating ArrayList.Itr problem Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238

          steve sides added a comment - - edited

          Here's another context where moving away from foreach was not effective.
          I was reading some mercurial repositories with source and subdir (to check out to) from a json file:

          for(repomap in jsonrepos.configs.repos) {
              for ( key in repomap.keySet() ){
                  source = repomap.get(key)['source']
                  subdir = repomap.get(key)['subdir']
                  checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
             }
          

          This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here):

          def slurper = new JsonSlurper()
          def jsonText = readFile 'flow-configs/prebuild.json'
          jsonrepos = slurper.parseText( jsonText )
          repoCount = jsonrepos.configs.repos.size
          println "repositories: ${repoCount}"
          for ( i = 0 ; i < repoCount ; i++ ) {
              rmap = jsonrepos.configs.repos[i]
              key = rmap.keySet().toList().getAt(0)
              source = rmap.get(key)['source']
              subdir = rmap.get(key)['subdir']
              println("  source: ${source}  subdir: ${subdir}")
              checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
          }
          

          It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious).
          This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug.

          java.io.NotSerializableException: groovy.json.JsonSlurper
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
          at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
          at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
          at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
          at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
          at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
          at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
          at java.util.HashMap.writeObject(HashMap.java:1354)
          at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
          at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
          at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
          at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
          at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
          at java.util.TreeMap.writeObject(TreeMap.java:2434)
          at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
          at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
          at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
          at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
          at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178)
          at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
          at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: an exception which occurred:
          in field locals
          in field parent
          in field parent
          in field caller
          in field e
          in field program
          in field threads
          in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4
          Finished: FAILURE

          steve sides added a comment - - edited Here's another context where moving away from foreach was not effective. I was reading some mercurial repositories with source and subdir (to check out to) from a json file: for (repomap in jsonrepos.configs.repos) { for ( key in repomap.keySet() ){ source = repomap.get(key)[ 'source' ] subdir = repomap.get(key)[ 'subdir' ] checkout([$class: 'MercurialSCM' , credentialsId: '', installation: ' (Default)', source: "${source}" , subdir: "${subdir}" ]) } This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here): def slurper = new JsonSlurper() def jsonText = readFile 'flow-configs/prebuild.json' jsonrepos = slurper.parseText( jsonText ) repoCount = jsonrepos.configs.repos.size println "repositories: ${repoCount}" for ( i = 0 ; i < repoCount ; i++ ) { rmap = jsonrepos.configs.repos[i] key = rmap.keySet().toList().getAt(0) source = rmap.get(key)[ 'source' ] subdir = rmap.get(key)[ 'subdir' ] println( " source: ${source} subdir: ${subdir}" ) checkout([$class: 'MercurialSCM' , credentialsId: '', installation: ' (Default)', source: "${source}" , subdir: "${subdir}" ]) } It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious). This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug. java.io.NotSerializableException: groovy.json.JsonSlurper at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.HashMap.internalWriteEntries(HashMap.java:1777) at java.util.HashMap.writeObject(HashMap.java:1354) at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65) at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56) at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50) at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.TreeMap.writeObject(TreeMap.java:2434) at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032) at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: an exception which occurred: in field locals in field parent in field parent in field caller in field e in field program in field threads in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4 Finished: FAILURE

          Jesse Glick added a comment -

          ssides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS. See the tutorial.

          Jesse Glick added a comment - ssides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS . See the tutorial.

          Sam Van Oort added a comment - - edited

          jglick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works:

          import com.cloudbees.groovy.cps.NonCPS
          
          @NonCPS
          List<Map.Entry> get_map_entries(map) {
              // This is harder than it seems, toArray doesn't work as expected and there are other gotchas to know
              // Also set iterators are forbidden, so you need an indexed collection
              def myarray = []
              myarray.addAll(map.entrySet())
              return myarray
          }
          
          node {
            def test_envs = [:]
             test_envs["debian:wheezy"] = ["echo goober"]
             def entries = get_map_entries(test_envs)
             
             for (int i=0; i<entries.size(); i++){
                 String key = entries.get(i).key
                 String value =  entries.get(i).value
                 echo "Key $key and value $value"
             }
          }
          

          Sam Van Oort added a comment - - edited jglick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works: import com.cloudbees.groovy.cps.NonCPS @NonCPS List<Map.Entry> get_map_entries(map) { // This is harder than it seems, toArray doesn't work as expected and there are other gotchas to know // Also set iterators are forbidden, so you need an indexed collection def myarray = [] myarray.addAll(map.entrySet()) return myarray } node { def test_envs = [:] test_envs[ "debian:wheezy" ] = [ "echo goober" ] def entries = get_map_entries(test_envs) for ( int i=0; i<entries.size(); i++){ String key = entries.get(i).key String value = entries.get(i).value echo "Key $key and value $value" } }

          Sam Van Oort added a comment -

          Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so:

          def myarray = map.entrySet().toArray()
          

          Will give this error:
          hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0]
          Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange)
          at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
          at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
          at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15)
          at WorkflowScript.run(WorkflowScript:31)
          at Unknown.Unknown(Unknown)
          at __cps.transform__(Native Method)
          at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69)
          at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106)
          at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79)
          at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
          at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
          at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
          at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
          at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
          at com.cloudbees.groovy.cps.Next.step(Next.java:58)
          at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145)
          at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183)
          at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181)
          at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
          at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Finished: FAILURE

          Sam Van Oort added a comment - Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so: def myarray = map.entrySet().toArray() Will give this error: hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0] Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55) at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15) at WorkflowScript.run(WorkflowScript:31) at Unknown.Unknown(Unknown) at __ cps.transform __(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79) at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22) at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55) at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16) at com.cloudbees.groovy.cps.Next.step(Next.java:58) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Finished: FAILURE

          Jesse Glick added a comment -

          The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i).

          Jesse Glick added a comment - The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i) .

          Jesse Glick added a comment -

          NonCPS need not be imported—Workflow scripts get it by default.

          Jesse Glick added a comment - NonCPS need not be imported—Workflow scripts get it by default.

          Jesse Glick added a comment -

          If you use an asynchronous step

          sh "echo Key $key and value $value"
          

          you will see that your workaround does not really work:

          Started by user anonymous
          [Workflow] Allocate node : Start
          Running on master in …
          [Workflow] node {
          [Workflow] sh
          [flow] Running shell script
          [Workflow] } //node
          [Workflow] Allocate node : End
          [Workflow] End of Workflow
          java.io.NotSerializableException: java.util.LinkedHashMap$Entry
          	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
          	at …
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
          	at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
          	at java.util.HashMap.writeObject(HashMap.java:1354)
          	at …
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
          	at java.util.TreeMap.writeObject(TreeMap.java:2434)
          	at …
          	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347)
          	at …
          Caused by: an exception which occurred:
          	in field locals
          	in field parent
          	in field parent
          	in field caller
          	in field e
          	in field program
          	in field threads
          	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287
          Finished: FAILURE
          

          Here is a working script:

          @NonCPS
          List<List<Object>> get_map_entries(map) {
              map.collect {k, v -> [k, v]}
          }
          node {
             def mymap = [a: 1, b: 2]
             def entries = get_map_entries(mymap)
             for (int i=0; i<entries.size(); i++){
                 String key = entries[i][0]
                 String value =  entries[i][1]
                 sh "echo Key $key and value $value"
             }
          }
          

          Jesse Glick added a comment - If you use an asynchronous step sh "echo Key $key and value $value" you will see that your workaround does not really work: Started by user anonymous [Workflow] Allocate node : Start Running on master in … [Workflow] node { [Workflow] sh [flow] Running shell script [Workflow] } //node [Workflow] Allocate node : End [Workflow] End of Workflow java.io.NotSerializableException: java.util.LinkedHashMap$Entry at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860) at … at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.HashMap.internalWriteEntries(HashMap.java:1777) at java.util.HashMap.writeObject(HashMap.java:1354) at … at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.TreeMap.writeObject(TreeMap.java:2434) at … at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347) at … Caused by: an exception which occurred: in field locals in field parent in field parent in field caller in field e in field program in field threads in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287 Finished: FAILURE Here is a working script: @NonCPS List<List< Object >> get_map_entries(map) { map.collect {k, v -> [k, v]} } node { def mymap = [a: 1, b: 2] def entries = get_map_entries(mymap) for ( int i=0; i<entries.size(); i++){ String key = entries[i][0] String value = entries[i][1] sh "echo Key $key and value $value" } }

          Sam Van Oort added a comment - - edited

          Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM:

          org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure
          at

          Also probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.

          Sam Van Oort added a comment - - edited Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM: org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure at Also probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.

          Jesse Glick added a comment -

          That method is fine to whitelist.

          Jesse Glick added a comment - That method is fine to whitelist.

          Sam Van Oort added a comment -

          Yes, with note that it is safe because the passed closure is subject to the security rules.

          Sam Van Oort added a comment - Yes, with note that it is safe because the passed closure is subject to the security rules.

          Andrew Bayer added a comment -

          Is this related to JENKINS-26481 or is it a separate cause?

          Andrew Bayer added a comment - Is this related to JENKINS-26481 or is it a separate cause?

          Jesse Glick added a comment -

          abayer an unrelated issue.

          Jesse Glick added a comment - abayer an unrelated issue.

          Jesse Glick added a comment -

          Blocking at least the initially attempted use case from JENKINS-26481.

          Jesse Glick added a comment - Blocking at least the initially attempted use case from JENKINS-26481 .

          Code changed in jenkins
          User: Jesse Glick
          Path:
          cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629
          Log:
          [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629 Log: [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,

          Jesse Glick added a comment -

          Not in master yet, be patient JIRA link daemon…

          Jesse Glick added a comment - Not in master yet, be patient JIRA link daemon…

          Code changed in jenkins
          User: Jesse Glick
          Path:
          cps/pom.xml
          cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90
          Log:
          Merge pull request #372 from jenkinsci/eachClosure-JENKINS-26481

          JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure)

          Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cps/pom.xml cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90 Log: Merge pull request #372 from jenkinsci/eachClosure- JENKINS-26481 JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure) Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674

          Code changed in jenkins
          User: Jesse Glick
          Path:
          aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5
          Log:
          JENKINS-27421 Reproduced in test.
          Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5 Log: JENKINS-27421 Reproduced in test. Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7

          Code changed in jenkins
          User: Jesse Glick
          Path:
          cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505
          Log:
          [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,
          Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505 Log: [FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr, Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629

          Jesse Glick added a comment -

          Best to reopen since the original fix covered only ArrayList, not other collections.

          Jesse Glick added a comment - Best to reopen since the original fix covered only ArrayList , not other collections.

          Hey jglick

          I saw you link this issue to a forum question regarding this error: java.io.NotSerializableException: java.util.HashMap$Entry

          After updating (today) to jenkins v2.10 and updating all Pipeline plugins to the latest as well, I'm seeing a similar error: java.io.NotSerializableException: groovy.json.internal.LazyMap. If this is not the correct place for this, I can create a separate issue. The following script is producing the error... As you can see, I'm not creating a map, so I assume it would be in the jsonSlurper code?

          import groovy.json.JsonSlurper;
          
          /**
           * Determine if there is a build for a specific branch
           * currently running.  There is no jenkins api to determine
           * this.
           *
           * @return isBranchBuilding {Boolean}
           */
          def getIsBranchBuilding()
          {
              sh "curl ${env.JOB_URL}api/json?pretty=true > api.json"
          
              def json = readFile("api.json")
              def result = getApi( json )
              def previousBuild = result.lastBuild.number - 1;
          
              for( def i = previousBuild; i > 1; i-- )
              {
                  sh "curl ${env.JOB_URL}${i}/api/json?pretty=true > api2.json"
          
                  def buildApi = readFile( "api2.json" )
                  def buildInfo = getApi( buildApi )
          
                  if( buildInfo.building ) // we've found a build already in progress
                      return true;
                  else if( buildInfo.duration > 5000 ) // we've found the most recent build that has actually built and is complete
                      break;
              }
          
              return false;
          }
          
          @NonCPS
          def getApi( jsonString )
          {
              def slurper = new JsonSlurper()
              return slurper.parseText( jsonString );
          }
          

          Michael Scharp added a comment - Hey jglick I saw you link this issue to a forum question regarding this error: java.io.NotSerializableException: java.util.HashMap$Entry After updating (today) to jenkins v2.10 and updating all Pipeline plugins to the latest as well, I'm seeing a similar error: java.io.NotSerializableException: groovy.json.internal.LazyMap . If this is not the correct place for this, I can create a separate issue. The following script is producing the error... As you can see, I'm not creating a map, so I assume it would be in the jsonSlurper code? import groovy.json.JsonSlurper; /** * Determine if there is a build for a specific branch * currently running. There is no jenkins api to determine * this . * * @ return isBranchBuilding { Boolean } */ def getIsBranchBuilding() { sh "curl ${env.JOB_URL}api/json?pretty= true > api.json" def json = readFile( "api.json" ) def result = getApi( json ) def previousBuild = result.lastBuild.number - 1; for ( def i = previousBuild; i > 1; i-- ) { sh "curl ${env.JOB_URL}${i}/api/json?pretty= true > api2.json" def buildApi = readFile( "api2.json" ) def buildInfo = getApi( buildApi ) if ( buildInfo.building ) // we've found a build already in progress return true ; else if ( buildInfo.duration > 5000 ) // we've found the most recent build that has actually built and is complete break ; } return false ; } @NonCPS def getApi( jsonString ) { def slurper = new JsonSlurper() return slurper.parseText( jsonString ); }

          Jesse Glick added a comment -

          mscharp Yes whatever parseText is returning. Try pushing the .lastBuild.number calls down into the @NonCPS method.

          Jesse Glick added a comment - mscharp Yes whatever parseText is returning. Try pushing the .lastBuild.number calls down into the @NonCPS method.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          docs/BEST_PRACTICES.md
          http://jenkins-ci.org/commit/pipeline-examples/cff723de24a9a74fb4be2cafde2564755131b247
          Log:
          Merge pull request #48 from jenkinsci/map-entries

          JENKINS-27421 Documenting standard workaround for iterating Map.entrySet

          Compare: https://github.com/jenkinsci/pipeline-examples/compare/cbf9bb17f752...cff723de24a9

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: docs/BEST_PRACTICES.md http://jenkins-ci.org/commit/pipeline-examples/cff723de24a9a74fb4be2cafde2564755131b247 Log: Merge pull request #48 from jenkinsci/map-entries JENKINS-27421 Documenting standard workaround for iterating Map.entrySet Compare: https://github.com/jenkinsci/pipeline-examples/compare/cbf9bb17f752...cff723de24a9

          Max Wahler added a comment -

          I just wanted to say that I started to get confronted with this bug only since my latest Jenkins update today. Before that, a

          for (String a : items) {}

          worked perfectly, now I get a

          java.io.NotSerializableException: java.util.AbstractList$Itr

          Max Wahler added a comment - I just wanted to say that I started to get confronted with this bug only since my latest Jenkins update today. Before that, a for (String a : items) {} worked perfectly, now I get a java.io.NotSerializableException: java.util.AbstractList$Itr

          Filip Pytloun added a comment - - edited

          This seems to be totally broken in workflow-cps version 2.17 - when I updated, every map throw NotSerializableException. I had to update back to 2.13 to make this work again.

          Filip Pytloun added a comment - - edited This seems to be totally broken in workflow-cps version 2.17 - when I updated, every map throw NotSerializableException. I had to update back to 2.13 to make this work again.

          Johan Piet added a comment -

          I have the same issue as genunix is mentioning.

          Johan Piet added a comment - I have the same issue as genunix is mentioning.

          Anton B added a comment -

          Curious if there has been any progress on this. Really simple map iteration seems to be broken right now.

          def testMap = ["test1" : "test1v", "test2" : "test2v"]
          testMap.each { k, v -> echo k }
          

          prints only the first item in cps scripts

          Anton B added a comment - Curious if there has been any progress on this. Really simple map iteration seems to be broken right now. def testMap = [ "test1" : "test1v" , "test2" : "test2v" ] testMap.each { k, v -> echo k } prints only the first item in cps scripts

          aburdukovskiy It hasn't been working for a long time already, see JENKINS-26481

          Nickolay Rumyantsev added a comment - aburdukovskiy It hasn't been working for a long time already, see JENKINS-26481

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/4a65e417fe403ea4be8f48be2031bfb5d808a540
          Log:
          Simpler workaround for JENKINS-27421.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/4a65e417fe403ea4be8f48be2031bfb5d808a540 Log: Simpler workaround for JENKINS-27421 .

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/32c98dbbe54172b342f790443e9fec820fc125e3
          Log:
          Merge pull request #67 from jglick/test-update

          Simpler workaround for map iteration JENKINS-27421

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d35a7e0c47e2...32c98dbbe541

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/32c98dbbe54172b342f790443e9fec820fc125e3 Log: Merge pull request #67 from jglick/test-update Simpler workaround for map iteration JENKINS-27421 Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d35a7e0c47e2...32c98dbbe541

          Jesse Glick added a comment -

          The supported way to iterate a Map is to define a helper function:

          @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
          

          and then to call it like:

          for (def e in entries(map)) {
            echo "got ${e[0]} → ${e[1]}"
          }
          

          Jesse Glick added a comment - The supported way to iterate a Map is to define a helper function: @NonCPS def entries(m) {m.collect {k, v -> [k, v]}} and then to call it like: for (def e in entries(map)) { echo "got ${e[0]} → ${e[1]}" }

          Ben Dean added a comment -

          jglick, is there going to be a better way to do that? Because it seems a little silly to have my Jenkinsfile scripts littered with stuff like

          @NonCPS def safeArray(c) {
              c.collect { v -> v }
          }
          
          // ...
          def someMap = [a: 1, b:2]
          for (def thing in safeArray(someMap.values())) {
            echo "do something with $thing"
          }
          

          in fact, why bother with .collect, it just needs to be an ArrayList

          def someMap = [a: 1, b:2]
          for (def thing in new ArrayList(someMap.values())) {
            echo "do something with $thing"
          }
          

          which needs script security approval, But that's basically what .collect is doing. I guess I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList.

          Ben Dean added a comment - jglick , is there going to be a better way to do that? Because it seems a little silly to have my Jenkinsfile scripts littered with stuff like @NonCPS def safeArray(c) { c.collect { v -> v } } // ... def someMap = [a: 1, b:2] for (def thing in safeArray(someMap.values())) { echo " do something with $thing" } in fact, why bother with .collect , it just needs to be an ArrayList def someMap = [a: 1, b:2] for (def thing in new ArrayList(someMap.values())) { echo " do something with $thing" } which needs script security approval, But that's basically what .collect is doing . I guess I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList .

          Jesse Glick added a comment -

          is there going to be a better way to do that?

          If and when I can figure out how to resolve this issue.

          I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList

          Every usage of an intermediary value which does not implement java.io.Serializable must be encapsulated in a method marked with the @NonCPS annotation in order to run in Pipeline Script. Your safeArray example will not work, since it is both receiving and returning nonserializable values. The point of entries is that it takes a serializable Map, and returns a serializable List<List>. You could probably also have something like (untested):

          @NonCPS def entrySet(m) {m.collect {k, v -> [key: k, value: v]}}
          for (def e in entrySet(map)) {
            echo "got ${e.key} → ${e.value}"
          }
          

          which at least looks more like the standard looping using

          for (def e in map.entrySet()) {
            echo "got ${e.key} → ${e.value}"
          }
          

          Jesse Glick added a comment - is there going to be a better way to do that? If and when I can figure out how to resolve this issue. I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList Every usage of an intermediary value which does not implement java.io.Serializable must be encapsulated in a method marked with the @NonCPS annotation in order to run in Pipeline Script. Your safeArray example will not work, since it is both receiving and returning nonserializable values. The point of entries is that it takes a serializable Map , and returns a serializable List<List> . You could probably also have something like (untested): @NonCPS def entrySet(m) {m.collect {k, v -> [key: k, value: v]}} for (def e in entrySet(map)) { echo "got ${e.key} → ${e.value}" } which at least looks more like the standard looping using for (def e in map.entrySet()) { echo "got ${e.key} → ${e.value}" }

          A little addition to jglick's comment: the pain point about looping over Map is that standard Map.entrySet() returns list of non-serializable entry objects, i.e. LinkedHashMap.Entry. Despite the Map itself is indeed Serializable.

          If talking about values() then I am not so sure because new ArrayList(someMap.values()) is Serializable whereas someMap.values() is not.
          jglick, am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

          Nickolay Rumyantsev added a comment - A little addition to jglick 's comment: the pain point about looping over Map is that standard Map.entrySet() returns list of non-serializable entry objects, i.e. LinkedHashMap.Entry. Despite the Map itself is indeed Serializable. If talking about values() then I am not so sure because new ArrayList(someMap.values()) is Serializable whereas someMap.values() is not. jglick , am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

          Max Wahler added a comment -

          Oh dear, all these complications just for the pipelines to be resumable? Is there any other reason for this?

          jglick: I really appreciate your work, but me (and I guess 99% of all users) would be fine with a pipeline that's not resumable but that supports all that groovy Groovy stuff without any headaches. It's annoying if you have a huge pipeline that takes dozens of minutes to build just to end up in a NotSerializableException. What about introducing a switch where the user can decide if a pipeline is resumable or not?

          Max Wahler added a comment - Oh dear, all these complications just for the pipelines to be resumable? Is there any other reason for this? jglick : I really appreciate your work, but me (and I guess 99% of all users) would be fine with a pipeline that's not resumable but that supports all that groovy Groovy stuff without any headaches. It's annoying if you have a huge pipeline that takes dozens of minutes to build just to end up in a NotSerializableException. What about introducing a switch where the user can decide if a pipeline is resumable or not?

          Joerg Schwaerzler added a comment - - edited

          +1
          Already some time ago when we tried to seriously use pipeline jobs I had the same thoughts.
          For us the resume feature currently is of no use.
          However is case you once are able to model the full build pipeline in one job it might be useful.
          I still doubt it. Since we still need to be able to trigger parts of the pipeline from repository changes.

          Joerg Schwaerzler added a comment - - edited +1 Already some time ago when we tried to seriously use pipeline jobs I had the same thoughts. For us the resume feature currently is of no use. However is case you once are able to model the full build pipeline in one job it might be useful. I still doubt it. Since we still need to be able to trigger parts of the pipeline from repository changes.

          Jesse Glick added a comment -

          am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

          Yes, declaring a local variable is just for human readability; the same code is run either way. This has to be done inside @NonCPS.

          introducing a switch where the user can decide if a pipeline is resumable or not?

          Not really possible, though there may be ways of just suppressing attempts to save the state. TBD. Do not discuss here please. Save it for JENKINS-33761. This issue is about fixing expressions involving iterating over collection classes. Unless you have a code-level proposal for a fix (unlikely, given the nature of the problem), or are offering a novel workaround in user scripts, please do not add further comments.

          we still need to be able to trigger parts of the pipeline from repository changes.

          This is actually possible, though awkward; please do not discuss it here as it is off topic.

          Jesse Glick added a comment - am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor? Yes, declaring a local variable is just for human readability; the same code is run either way. This has to be done inside @NonCPS . introducing a switch where the user can decide if a pipeline is resumable or not? Not really possible, though there may be ways of just suppressing attempts to save the state. TBD. Do not discuss here please. Save it for JENKINS-33761 . This issue is about fixing expressions involving iterating over collection classes. Unless you have a code-level proposal for a fix (unlikely, given the nature of the problem), or are offering a novel workaround in user scripts, please do not add further comments. we still need to be able to trigger parts of the pipeline from repository changes. This is actually possible, though awkward; please do not discuss it here as it is off topic.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/7ff4ffef84f27e6b989f9534b696026f5b94d585
          Log:
          JENKINS-27421 Improve behavior of map iteration workaround a bit.
          · Pick up new default whitelist entries.
          · Use a revised workaround that more closely matches the original syntax.
          · Fix the engine to cleanly fail the build with the serialization error, rather than throwing an assertion and hanging.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/7ff4ffef84f27e6b989f9534b696026f5b94d585 Log: JENKINS-27421 Improve behavior of map iteration workaround a bit. · Pick up new default whitelist entries. · Use a revised workaround that more closely matches the original syntax. · Fix the engine to cleanly fail the build with the serialization error, rather than throwing an assertion and hanging.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/a115a540ed39437d5612d2368db3b0921c863d36
          Log:
          Merge pull request #77 from jglick/mapIterator-JENKINS-27421

          JENKINS-27421 Improve behavior of map iteration workaround a bit

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/bee2879e1e13...a115a540ed39

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/a115a540ed39437d5612d2368db3b0921c863d36 Log: Merge pull request #77 from jglick/mapIterator- JENKINS-27421 JENKINS-27421 Improve behavior of map iteration workaround a bit Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/bee2879e1e13...a115a540ed39

          Ben Dean added a comment -

          jglick, the reason I mentioned ArrayList and my safeArray method is that based on the earlier entries method you mentioned, it seems like .collect is fine and it returns an ArrayList.

          Also I actually tried my safeArray thing and it worked to iterate over that in CPS code.

          Ben Dean added a comment - jglick , the reason I mentioned ArrayList and my safeArray method is that based on the earlier entries method you mentioned, it seems like .collect is fine and it returns an ArrayList . Also I actually tried my safeArray thing and it worked to iterate over that in CPS code.

          Sam Van Oort added a comment -

          jglick I see this one is still in progress since October - has it been completed or is it still open and impacting users?

          Sam Van Oort added a comment - jglick I see this one is still in progress since October - has it been completed or is it still open and impacting users?

          Jesse Glick added a comment -

          It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.

          Jesse Glick added a comment - It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.

          Harry G. added a comment -

          I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.

          Harry G. added a comment - I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.

          Jesse Glick added a comment -

          Yes this issue is well known. To recap:

          • some kinds of collections, such as ArrayList, have a fix
          • for other collections, you can either
            • use a pre-Java-5-style for loop with an index
            • or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body
            • convert to an ArrayList, either via its constructor for another List or Set, or for a Map via the aforementioned helper method:
          @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
          

          Jesse Glick added a comment - Yes this issue is well known. To recap: some kinds of collections, such as ArrayList , have a fix for other collections, you can either use a pre-Java-5-style for loop with an index or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body convert to an ArrayList , either via its constructor for another List or Set , or for a Map via the aforementioned helper method: @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3
          Log:
          JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3 Log: JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5
          Log:
          Merge pull request #124 from jglick/GroovyCategorySupport-JENKINS-26481

          JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5 Log: Merge pull request #124 from jglick/GroovyCategorySupport- JENKINS-26481 JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca

          Great to see that a solution is implemented.

          I've got one question, though:

          Will the solution work for any collections and allow all kind of different possibilities to iterate through them?

          Joerg Schwaerzler added a comment - Great to see that a solution is implemented. I've got one question, though: Will the solution work for any collections and allow all kind of different possibilities to iterate through them?

          Jesse Glick added a comment -

          The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.

          Jesse Glick added a comment - The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.

          This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.

          autarch princeps added a comment - This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.

          Robby Pocase added a comment -

          For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.

          Robby Pocase added a comment - For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.

          Jesse Glick added a comment -

          autarchprinceps / rpocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.

          Jesse Glick added a comment - autarchprinceps / rpocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            38 Vote for this issue
            Watchers:
            59 Start watching this issue

              Created:
              Updated:
              Resolved: