bump The inability to use collection iterators at all is a very significant limitation.

Workarounds:

· Use a plain C-style for-loop.
· Wrap the code in a method marked @NonCps, assuming it does not invoke anything interruptible (incl. calling Workflow steps).

Being able to write:

node('linux_slave', 'windows_slave') {
// some steps to run in parallel on both slaves
}

would be fantastic.

owenmehegan you can already do that using

def labels = ['linux', 'windows']
for (int i = 0; i < labels.size(); i++) {
  node(labels.get(i)) {
    // something, though you presumably need to select sh vs. bat here
  }
}

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7
Log:
JENKINS-27421 Reproduced in test.

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e
Log:
Merge pull request #206 from jglick/Itr-JENKINS-27421

JENKINS-27421 Investigating ArrayList.Itr problem

Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238

Here's another context where moving away from foreach was not effective.
I was reading some mercurial repositories with source and subdir (to check out to) from a json file:

for(repomap in jsonrepos.configs.repos) {
    for ( key in repomap.keySet() ){
        source = repomap.get(key)['source']
        subdir = repomap.get(key)['subdir']
        checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
   }

This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here):

def slurper = new JsonSlurper()
def jsonText = readFile 'flow-configs/prebuild.json'
jsonrepos = slurper.parseText( jsonText )
repoCount = jsonrepos.configs.repos.size
println "repositories: ${repoCount}"
for ( i = 0 ; i < repoCount ; i++ ) {
    rmap = jsonrepos.configs.repos[i]
    key = rmap.keySet().toList().getAt(0)
    source = rmap.get(key)['source']
    subdir = rmap.get(key)['subdir']
    println("  source: ${source}  subdir: ${subdir}")
    checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"])
}

It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious).
This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug.

java.io.NotSerializableException: groovy.json.JsonSlurper
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
at java.util.HashMap.writeObject(HashMap.java:1354)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.TreeMap.writeObject(TreeMap.java:2434)
at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: an exception which occurred:
in field locals
in field parent
in field parent
in field caller
in field e
in field program
in field threads
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4
Finished: FAILURE

ssides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS. See the tutorial.

jglick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works:

import com.cloudbees.groovy.cps.NonCPS

@NonCPS
List<Map.Entry> get_map_entries(map) {
    // This is harder than it seems, toArray doesn't work as expected and there are other gotchas to know
    // Also set iterators are forbidden, so you need an indexed collection
    def myarray = []
    myarray.addAll(map.entrySet())
    return myarray
}

node {
  def test_envs = [:]
   test_envs["debian:wheezy"] = ["echo goober"]
   def entries = get_map_entries(test_envs)
   
   for (int i=0; i<entries.size(); i++){
       String key = entries.get(i).key
       String value =  entries.get(i).value
       echo "Key $key and value $value"
   }
}

Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so:

def myarray = map.entrySet().toArray()

Will give this error:
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0]
Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15)
at WorkflowScript.run(WorkflowScript:31)
at Unknown.Unknown(Unknown)
at __cps.transform__(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
at com.cloudbees.groovy.cps.Next.step(Next.java:58)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE

The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i).

NonCPS need not be imported—Workflow scripts get it by default.

If you use an asynchronous step

sh "echo Key $key and value $value"

you will see that your workaround does not really work:

Started by user anonymous
[Workflow] Allocate node : Start
Running on master in …
[Workflow] node {
[Workflow] sh
[flow] Running shell script
[Workflow] } //node
[Workflow] Allocate node : End
[Workflow] End of Workflow
java.io.NotSerializableException: java.util.LinkedHashMap$Entry
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
	at …
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
	at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
	at java.util.HashMap.writeObject(HashMap.java:1354)
	at …
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
	at java.util.TreeMap.writeObject(TreeMap.java:2434)
	at …
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347)
	at …
Caused by: an exception which occurred:
	in field locals
	in field parent
	in field parent
	in field caller
	in field e
	in field program
	in field threads
	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287
Finished: FAILURE

Here is a working script:

@NonCPS
List<List<Object>> get_map_entries(map) {
    map.collect {k, v -> [k, v]}
}
node {
   def mymap = [a: 1, b: 2]
   def entries = get_map_entries(mymap)
   for (int i=0; i<entries.size(); i++){
       String key = entries[i][0]
       String value =  entries[i][1]
       sh "echo Key $key and value $value"
   }
}

Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM:

org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure
at

Also probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.

That method is fine to whitelist.

Yes, with note that it is safe because the passed closure is subject to the security rules.

Is this related to JENKINS-26481 or is it a separate cause?

abayer an unrelated issue.

Blocking at least the initially attempted use case from JENKINS-26481.

Code changed in jenkins
User: Jesse Glick
Path:
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629
Log:
[FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,

Not in master yet, be patient JIRA link daemon…

Code changed in jenkins
User: Jesse Glick
Path:
cps/pom.xml
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90
Log:
Merge pull request #372 from jenkinsci/eachClosure-JENKINS-26481

JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure)

Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674

Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5
Log:
JENKINS-27421 Reproduced in test.
Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7

Code changed in jenkins
User: Jesse Glick
Path:
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505
Log:
[FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,
Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629

Best to reopen since the original fix covered only ArrayList, not other collections.

Hey jglick

I saw you link this issue to a forum question regarding this error: java.io.NotSerializableException: java.util.HashMap$Entry

After updating (today) to jenkins v2.10 and updating all Pipeline plugins to the latest as well, I'm seeing a similar error: java.io.NotSerializableException: groovy.json.internal.LazyMap. If this is not the correct place for this, I can create a separate issue. The following script is producing the error... As you can see, I'm not creating a map, so I assume it would be in the jsonSlurper code?

import groovy.json.JsonSlurper;

/**
 * Determine if there is a build for a specific branch
 * currently running.  There is no jenkins api to determine
 * this.
 *
 * @return isBranchBuilding {Boolean}
 */
def getIsBranchBuilding()
{
    sh "curl ${env.JOB_URL}api/json?pretty=true > api.json"

    def json = readFile("api.json")
    def result = getApi( json )
    def previousBuild = result.lastBuild.number - 1;

    for( def i = previousBuild; i > 1; i-- )
    {
        sh "curl ${env.JOB_URL}${i}/api/json?pretty=true > api2.json"

        def buildApi = readFile( "api2.json" )
        def buildInfo = getApi( buildApi )

        if( buildInfo.building ) // we've found a build already in progress
            return true;
        else if( buildInfo.duration > 5000 ) // we've found the most recent build that has actually built and is complete
            break;
    }

    return false;
}

@NonCPS
def getApi( jsonString )
{
    def slurper = new JsonSlurper()
    return slurper.parseText( jsonString );
}

mscharp Yes whatever parseText is returning. Try pushing the .lastBuild.number calls down into the @NonCPS method.

Code changed in jenkins
User: Andrew Bayer
Path:
docs/BEST_PRACTICES.md
http://jenkins-ci.org/commit/pipeline-examples/cff723de24a9a74fb4be2cafde2564755131b247
Log:
Merge pull request #48 from jenkinsci/map-entries

JENKINS-27421 Documenting standard workaround for iterating Map.entrySet

Compare: https://github.com/jenkinsci/pipeline-examples/compare/cbf9bb17f752...cff723de24a9

I just wanted to say that I started to get confronted with this bug only since my latest Jenkins update today. Before that, a

for (String a : items) {}

worked perfectly, now I get a

java.io.NotSerializableException: java.util.AbstractList$Itr

This seems to be totally broken in workflow-cps version 2.17 - when I updated, every map throw NotSerializableException. I had to update back to 2.13 to make this work again.

I have the same issue as genunix is mentioning.

Curious if there has been any progress on this. Really simple map iteration seems to be broken right now.

def testMap = ["test1" : "test1v", "test2" : "test2v"]
testMap.each { k, v -> echo k }

prints only the first item in cps scripts

aburdukovskiy It hasn't been working for a long time already, see JENKINS-26481

Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/4a65e417fe403ea4be8f48be2031bfb5d808a540
Log:
Simpler workaround for JENKINS-27421.

Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/32c98dbbe54172b342f790443e9fec820fc125e3
Log:
Merge pull request #67 from jglick/test-update

Simpler workaround for map iteration JENKINS-27421

Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d35a7e0c47e2...32c98dbbe541

The supported way to iterate a Map is to define a helper function:

@NonCPS def entries(m) {m.collect {k, v -> [k, v]}}

and then to call it like:

for (def e in entries(map)) {
  echo "got ${e[0]} → ${e[1]}"
}

jglick, is there going to be a better way to do that? Because it seems a little silly to have my Jenkinsfile scripts littered with stuff like

@NonCPS def safeArray(c) {
    c.collect { v -> v }
}

// ...
def someMap = [a: 1, b:2]
for (def thing in safeArray(someMap.values())) {
  echo "do something with $thing"
}

in fact, why bother with .collect, it just needs to be an ArrayList

def someMap = [a: 1, b:2]
for (def thing in new ArrayList(someMap.values())) {
  echo "do something with $thing"
}

which needs script security approval, But that's basically what .collect is doing. I guess I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList.

is there going to be a better way to do that?

If and when I can figure out how to resolve this issue.

I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList

Every usage of an intermediary value which does not implement java.io.Serializable must be encapsulated in a method marked with the @NonCPS annotation in order to run in Pipeline Script. Your safeArray example will not work, since it is both receiving and returning nonserializable values. The point of entries is that it takes a serializable Map, and returns a serializable List<List>. You could probably also have something like (untested):

@NonCPS def entrySet(m) {m.collect {k, v -> [key: k, value: v]}}
for (def e in entrySet(map)) {
  echo "got ${e.key} → ${e.value}"
}

which at least looks more like the standard looping using

for (def e in map.entrySet()) {
  echo "got ${e.key} → ${e.value}"
}

A little addition to jglick's comment: the pain point about looping over Map is that standard Map.entrySet() returns list of non-serializable entry objects, i.e. LinkedHashMap.Entry. Despite the Map itself is indeed Serializable.

If talking about values() then I am not so sure because new ArrayList(someMap.values()) is Serializable whereas someMap.values() is not.
jglick, am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

Oh dear, all these complications just for the pipelines to be resumable? Is there any other reason for this?

jglick: I really appreciate your work, but me (and I guess 99% of all users) would be fine with a pipeline that's not resumable but that supports all that groovy Groovy stuff without any headaches. It's annoying if you have a huge pipeline that takes dozens of minutes to build just to end up in a NotSerializableException. What about introducing a switch where the user can decide if a pipeline is resumable or not?

+1
Already some time ago when we tried to seriously use pipeline jobs I had the same thoughts.
For us the resume feature currently is of no use.
However is case you once are able to model the full build pipeline in one job it might be useful.
I still doubt it. Since we still need to be able to trigger parts of the pipeline from repository changes.

am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?

Yes, declaring a local variable is just for human readability; the same code is run either way. This has to be done inside @NonCPS.

introducing a switch where the user can decide if a pipeline is resumable or not?

Not really possible, though there may be ways of just suppressing attempts to save the state. TBD. Do not discuss here please. Save it for JENKINS-33761. This issue is about fixing expressions involving iterating over collection classes. Unless you have a code-level proposal for a fix (unlikely, given the nature of the problem), or are offering a novel workaround in user scripts, please do not add further comments.

we still need to be able to trigger parts of the pipeline from repository changes.

This is actually possible, though awkward; please do not discuss it here as it is off topic.

Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/7ff4ffef84f27e6b989f9534b696026f5b94d585
Log:
JENKINS-27421 Improve behavior of map iteration workaround a bit.
· Pick up new default whitelist entries.
· Use a revised workaround that more closely matches the original syntax.
· Fix the engine to cleanly fail the build with the serialization error, rather than throwing an assertion and hanging.

Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/a115a540ed39437d5612d2368db3b0921c863d36
Log:
Merge pull request #77 from jglick/mapIterator-JENKINS-27421

JENKINS-27421 Improve behavior of map iteration workaround a bit

Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/bee2879e1e13...a115a540ed39

jglick, the reason I mentioned ArrayList and my safeArray method is that based on the earlier entries method you mentioned, it seems like .collect is fine and it returns an ArrayList.

Also I actually tried my safeArray thing and it worked to iterate over that in CPS code.

jglick I see this one is still in progress since October - has it been completed or is it still open and impacting users?

It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.

I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.

Yes this issue is well known. To recap:

• some kinds of collections, such as ArrayList, have a fix
• for other collections, you can either
  • use a pre-Java-5-style for loop with an index
  • or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body
  • convert to an ArrayList, either via its constructor for another List or Set, or for a Map via the aforementioned helper method:

@NonCPS def entries(m) {m.collect {k, v -> [k, v]}}

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3
Log:
JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.

Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java
src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java
src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java
src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5
Log:
Merge pull request #124 from jglick/GroovyCategorySupport-JENKINS-26481

JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack

Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca

Great to see that a solution is implemented.

I've got one question, though:

Will the solution work for any collections and allow all kind of different possibilities to iterate through them?

The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.

This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.

For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.

autarchprinceps / rpocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.