-
Bug
-
Resolution: Fixed
-
Major
-
Powered by SuggestiMate
Not currently possible to use a Java 5-style for loop on an ArrayList (for example) from within CPS-transformed code, since its Iterator implementation is not marked Serializable.
- is blocking
-
JENKINS-26481 Mishandling of binary methods accepting Closure
-
- Resolved
-
- is duplicated by
-
JENKINS-34645 for in loop over result of String.split() gives NotSerializableException
-
- Resolved
-
- is related to
-
JENKINS-25925 More polite reporting of NotSerializableException
-
- Open
-
- links to
[JENKINS-27421] Unserializable iterator & entry classes from Java Collections
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/e916b5e9f35b00db679360dc29e42658d566efb7
Log:
JENKINS-27421 Reproduced in test.
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceFailureTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStep.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/PersistenceProblemStepExecution.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
http://jenkins-ci.org/commit/workflow-plugin/61b7d70b1238d9d66dea22f497cac26debaaf88e
Log:
Merge pull request #206 from jglick/Itr-JENKINS-27421
JENKINS-27421 Investigating ArrayList.Itr problem
Compare: https://github.com/jenkinsci/workflow-plugin/compare/a24f13091cf8...61b7d70b1238
Here's another context where moving away from foreach was not effective.
I was reading some mercurial repositories with source and subdir (to check out to) from a json file:
for(repomap in jsonrepos.configs.repos) { for ( key in repomap.keySet() ){ source = repomap.get(key)['source'] subdir = repomap.get(key)['subdir'] checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"]) }
This looked nice and elegant, but I hit this exception noted in this bug, so I removed the foreach loops (I include the json steps here):
def slurper = new JsonSlurper() def jsonText = readFile 'flow-configs/prebuild.json' jsonrepos = slurper.parseText( jsonText ) repoCount = jsonrepos.configs.repos.size println "repositories: ${repoCount}" for ( i = 0 ; i < repoCount ; i++ ) { rmap = jsonrepos.configs.repos[i] key = rmap.keySet().toList().getAt(0) source = rmap.get(key)['source'] subdir = rmap.get(key)['subdir'] println(" source: ${source} subdir: ${subdir}") checkout([$class: 'MercurialSCM', credentialsId: '', installation: '(Default)', source: "${source}", subdir: "${subdir}"]) }
It's fine up to the println statement. If you comment out the checkout line, it's all good and prints all the repsitories and subdir info, but if I uncomment out the checkout line I get this exception, which is similar to the rest reported except for the slurper part (which is curious).
This may be a different bug(???), but the stack trace seems to be the same (except the json.JsonSlurper) as other bugs referred to as another instance of this bug.
java.io.NotSerializableException: groovy.json.JsonSlurper
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.HashMap.internalWriteEntries(HashMap.java:1777)
at java.util.HashMap.writeObject(HashMap.java:1354)
at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.TreeMap.writeObject(TreeMap.java:2434)
at sun.reflect.GeneratedMethodAccessor206.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:344)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:303)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:71)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:180)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:178)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: an exception which occurred:
in field locals
in field parent
in field parent
in field caller
in field e
in field program
in field threads
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@509e61d4
Finished: FAILURE
ssides unrelated user error. You cannot use an unserializable class such as JsonSlurper from a flow script unless it is contained entirely within a method marked @NonCPS. See the tutorial.
jglick I've got a workaround for how to handle map iteration with Workflow DSL here. A bit ugly but it works:
import com.cloudbees.groovy.cps.NonCPS @NonCPS List<Map.Entry> get_map_entries(map) { // This is harder than it seems, toArray doesn't work as expected and there are other gotchas to know // Also set iterators are forbidden, so you need an indexed collection def myarray = [] myarray.addAll(map.entrySet()) return myarray } node { def test_envs = [:] test_envs["debian:wheezy"] = ["echo goober"] def entries = get_map_entries(test_envs) for (int i=0; i<entries.size(); i++){ String key = entries.get(i).key String value = entries.get(i).value echo "Key $key and value $value" } }
Worth noting (may be logged elsewhere or user error), but trying to do toArray on the entrySet to get an array output easily like so:
def myarray = map.entrySet().toArray()
Will give this error:
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: [Ljava.lang.Object;.get() is applicable for argument types: (java.lang.Integer) values: [0]
Possible solutions: getAt(java.lang.Integer), grep(), getAt(java.lang.String), grep(java.lang.Object), getAt(java.util.Collection), getAt(groovy.lang.ObjectRange)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:15)
at WorkflowScript.run(WorkflowScript:31)
at Unknown.Unknown(Unknown)
at __cps.transform__(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:69)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:106)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:79)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:33)
at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:22)
at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
at com.cloudbees.groovy.cps.Next.step(Next.java:58)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE
The MissingMethodException is probably fixable by using entries[i] rather than entries.get(i).
If you use an asynchronous step
sh "echo Key $key and value $value"
you will see that your workaround does not really work:
Started by user anonymous [Workflow] Allocate node : Start Running on master in … [Workflow] node { [Workflow] sh [flow] Running shell script [Workflow] } //node [Workflow] Allocate node : End [Workflow] End of Workflow java.io.NotSerializableException: java.util.LinkedHashMap$Entry at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860) at … at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.HashMap.internalWriteEntries(HashMap.java:1777) at java.util.HashMap.writeObject(HashMap.java:1354) at … at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344) at java.util.TreeMap.writeObject(TreeMap.java:2434) at … at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:132) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:347) at … Caused by: an exception which occurred: in field locals in field parent in field parent in field caller in field e in field program in field threads in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@3e45e287 Finished: FAILURE
Here is a working script:
@NonCPS List<List<Object>> get_map_entries(map) { map.collect {k, v -> [k, v]} } node { def mymap = [a: 1, b: 2] def entries = get_map_entries(mymap) for (int i=0; i<entries.size(); i++){ String key = entries[i][0] String value = entries[i][1] sh "echo Key $key and value $value" } }
Notable limitation: the closure-based solution can trigger script-security violation if you're running workflow scripts from SCM:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods collect java.util.Map groovy.lang.Closure
at
Also probably completely doable to find a non-closure approach to this though, just don't have one in hand right now.
Yes, with note that it is safe because the passed closure is subject to the security rules.
Blocking at least the initially attempted use case from JENKINS-26481.
Code changed in jenkins
User: Jesse Glick
Path:
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/a990468a828e64961eda24332673c9cb8835c629
Log:
[FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,
Code changed in jenkins
User: Jesse Glick
Path:
cps/pom.xml
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-plugin/5b102a5b06745b1eeeb7c1305096bf80e37c1a90
Log:
Merge pull request #372 from jenkinsci/eachClosure-JENKINS-26481
JENKINS-26481 JENKINS-27421 Fix ArrayList$Itr, and integration test for Object.each(Closure)
Compare: https://github.com/jenkinsci/workflow-plugin/compare/e3906483924d...5b102a5b0674
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/0ceee9b7cb7bded004c2222d93e2927230b5d9c5
Log:
JENKINS-27421 Reproduced in test.
Originally-Committed-As: e916b5e9f35b00db679360dc29e42658d566efb7
Code changed in jenkins
User: Jesse Glick
Path:
cps/src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
cps/src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/4ab54f1ad2ef9685e66804dc3d3de7add29fc505
Log:
[FIXED JENKINS-27421] Producing a safe serialization replacement for ArrayList$Itr,
Originally-Committed-As: a990468a828e64961eda24332673c9cb8835c629
Best to reopen since the original fix covered only ArrayList, not other collections.
Hey jglick
I saw you link this issue to a forum question regarding this error: java.io.NotSerializableException: java.util.HashMap$Entry
After updating (today) to jenkins v2.10 and updating all Pipeline plugins to the latest as well, I'm seeing a similar error: java.io.NotSerializableException: groovy.json.internal.LazyMap. If this is not the correct place for this, I can create a separate issue. The following script is producing the error... As you can see, I'm not creating a map, so I assume it would be in the jsonSlurper code?
import groovy.json.JsonSlurper; /** * Determine if there is a build for a specific branch * currently running. There is no jenkins api to determine * this. * * @return isBranchBuilding {Boolean} */ def getIsBranchBuilding() { sh "curl ${env.JOB_URL}api/json?pretty=true > api.json" def json = readFile("api.json") def result = getApi( json ) def previousBuild = result.lastBuild.number - 1; for( def i = previousBuild; i > 1; i-- ) { sh "curl ${env.JOB_URL}${i}/api/json?pretty=true > api2.json" def buildApi = readFile( "api2.json" ) def buildInfo = getApi( buildApi ) if( buildInfo.building ) // we've found a build already in progress return true; else if( buildInfo.duration > 5000 ) // we've found the most recent build that has actually built and is complete break; } return false; } @NonCPS def getApi( jsonString ) { def slurper = new JsonSlurper() return slurper.parseText( jsonString ); }
mscharp Yes whatever parseText is returning. Try pushing the .lastBuild.number calls down into the @NonCPS method.
Code changed in jenkins
User: Andrew Bayer
Path:
docs/BEST_PRACTICES.md
http://jenkins-ci.org/commit/pipeline-examples/cff723de24a9a74fb4be2cafde2564755131b247
Log:
Merge pull request #48 from jenkinsci/map-entries
JENKINS-27421 Documenting standard workaround for iterating Map.entrySet
Compare: https://github.com/jenkinsci/pipeline-examples/compare/cbf9bb17f752...cff723de24a9
I just wanted to say that I started to get confronted with this bug only since my latest Jenkins update today. Before that, a
for (String a : items) {}
worked perfectly, now I get a
java.io.NotSerializableException: java.util.AbstractList$Itr
This seems to be totally broken in workflow-cps version 2.17 - when I updated, every map throw NotSerializableException. I had to update back to 2.13 to make this work again.
Curious if there has been any progress on this. Really simple map iteration seems to be broken right now.
def testMap = ["test1" : "test1v", "test2" : "test2v"] testMap.each { k, v -> echo k }
prints only the first item in cps scripts
aburdukovskiy It hasn't been working for a long time already, see JENKINS-26481
Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/4a65e417fe403ea4be8f48be2031bfb5d808a540
Log:
Simpler workaround for JENKINS-27421.
Code changed in jenkins
User: Jesse Glick
Path:
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/32c98dbbe54172b342f790443e9fec820fc125e3
Log:
Merge pull request #67 from jglick/test-update
Simpler workaround for map iteration JENKINS-27421
Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d35a7e0c47e2...32c98dbbe541
The supported way to iterate a Map is to define a helper function:
@NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
and then to call it like:
for (def e in entries(map)) { echo "got ${e[0]} → ${e[1]}" }
jglick, is there going to be a better way to do that? Because it seems a little silly to have my Jenkinsfile scripts littered with stuff like
@NonCPS def safeArray(c) { c.collect { v -> v } } // ... def someMap = [a: 1, b:2] for (def thing in safeArray(someMap.values())) { echo "do something with $thing" }
in fact, why bother with .collect, it just needs to be an ArrayList
def someMap = [a: 1, b:2] for (def thing in new ArrayList(someMap.values())) { echo "do something with $thing" }
which needs script security approval, But that's basically what .collect is doing. I guess I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList.
is there going to be a better way to do that?
If and when I can figure out how to resolve this issue.
I'm not seeing the point of us having the have a bunch of helper methods to turn all Collection into ArrayList
Every usage of an intermediary value which does not implement java.io.Serializable must be encapsulated in a method marked with the @NonCPS annotation in order to run in Pipeline Script. Your safeArray example will not work, since it is both receiving and returning nonserializable values. The point of entries is that it takes a serializable Map, and returns a serializable List<List>. You could probably also have something like (untested):
@NonCPS def entrySet(m) {m.collect {k, v -> [key: k, value: v]}} for (def e in entrySet(map)) { echo "got ${e.key} → ${e.value}" }
which at least looks more like the standard looping using
for (def e in map.entrySet()) { echo "got ${e.key} → ${e.value}" }
A little addition to jglick's comment: the pain point about looping over Map is that standard Map.entrySet() returns list of non-serializable entry objects, i.e. LinkedHashMap.Entry. Despite the Map itself is indeed Serializable.
If talking about values() then I am not so sure because new ArrayList(someMap.values()) is Serializable whereas someMap.values() is not.
jglick, am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?
Oh dear, all these complications just for the pipelines to be resumable? Is there any other reason for this?
jglick: I really appreciate your work, but me (and I guess 99% of all users) would be fine with a pipeline that's not resumable but that supports all that groovy Groovy stuff without any headaches. It's annoying if you have a huge pipeline that takes dozens of minutes to build just to end up in a NotSerializableException. What about introducing a switch where the user can decide if a pipeline is resumable or not?
+1
Already some time ago when we tried to seriously use pipeline jobs I had the same thoughts.
For us the resume feature currently is of no use.
However is case you once are able to model the full build pipeline in one job it might be useful.
I still doubt it. Since we still need to be able to trigger parts of the pipeline from repository changes.
am I in danger of getting NonSerializable Exception if I don't use any intermediate variable to store someMap.values() and immediately pass it into ArrayList constructor?
Yes, declaring a local variable is just for human readability; the same code is run either way. This has to be done inside @NonCPS.
introducing a switch where the user can decide if a pipeline is resumable or not?
Not really possible, though there may be ways of just suppressing attempts to save the state. TBD. Do not discuss here please. Save it for JENKINS-33761. This issue is about fixing expressions involving iterating over collection classes. Unless you have a code-level proposal for a fix (unlikely, given the nature of the problem), or are offering a novel workaround in user scripts, please do not add further comments.
we still need to be able to trigger parts of the pipeline from repository changes.
This is actually possible, though awkward; please do not discuss it here as it is off topic.
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/7ff4ffef84f27e6b989f9534b696026f5b94d585
Log:
JENKINS-27421 Improve behavior of map iteration workaround a bit.
· Pick up new default whitelist entries.
· Use a revised workaround that more closely matches the original syntax.
· Fix the engine to cleanly fail the build with the serialization error, rather than throwing an assertion and hanging.
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/a115a540ed39437d5612d2368db3b0921c863d36
Log:
Merge pull request #77 from jglick/mapIterator-JENKINS-27421
JENKINS-27421 Improve behavior of map iteration workaround a bit
Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/bee2879e1e13...a115a540ed39
jglick, the reason I mentioned ArrayList and my safeArray method is that based on the earlier entries method you mentioned, it seems like .collect is fine and it returns an ArrayList.
Also I actually tried my safeArray thing and it worked to iterate over that in CPS code.
jglick I see this one is still in progress since October - has it been completed or is it still open and impacting users?
It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.
I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.
Yes this issue is well known. To recap:
- some kinds of collections, such as ArrayList, have a fix
- for other collections, you can either
- use a pre-Java-5-style for loop with an index
- or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body
- convert to an ArrayList, either via its constructor for another List or Set, or for a Map via the aforementioned helper method:
@NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3
Log:
JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.
Code changed in jenkins
User: Jesse Glick
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java
src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java
src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java
src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5
Log:
Merge pull request #124 from jglick/GroovyCategorySupport-JENKINS-26481
JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack
Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca
Great to see that a solution is implemented.
I've got one question, though:
Will the solution work for any collections and allow all kind of different possibilities to iterate through them?
The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.
This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.
For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.
autarchprinceps / rpocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.
owenmehegan you can already do that using