Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27421

Unserializable iterator & entry classes from Java Collections

      Not currently possible to use a Java 5-style for loop on an ArrayList (for example) from within CPS-transformed code, since its Iterator implementation is not marked Serializable.

          [JENKINS-27421] Unserializable iterator & entry classes from Java Collections

          Jesse Glick added a comment -

          It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.

          Jesse Glick added a comment - It is still open and affecting users. There is a fix for some common cases but I have not managed to either generalize it or extend it to some other important cases.

          Harry G. added a comment -

          I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.

          Harry G. added a comment - I can also confirm that this is still affecting us. There is some workaround as discussed above, but no clean solution.

          Jesse Glick added a comment -

          Yes this issue is well known. To recap:

          • some kinds of collections, such as ArrayList, have a fix
          • for other collections, you can either
            • use a pre-Java-5-style for loop with an index
            • or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body
            • convert to an ArrayList, either via its constructor for another List or Set, or for a Map via the aforementioned helper method:
          @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}
          

          Jesse Glick added a comment - Yes this issue is well known. To recap: some kinds of collections, such as ArrayList , have a fix for other collections, you can either use a pre-Java-5-style for loop with an index or wrap your code logic in a @NonCPS block, assuming you are not calling any steps or other CPS-transformed code from inside the loop body convert to an ArrayList , either via its constructor for another List or Set , or for a Map via the aforementioned helper method: @NonCPS def entries(m) {m.collect {k, v -> [k, v]}}

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3
          Log:
          JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/ee845109789193b4fed6640e8ec1977238e976f3 Log: JENKINS-27421 Categories also seem to provide a far simpler way to work around unserializable iterators, map entries, etc.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          pom.xml
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java
          src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java
          src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java
          src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java
          http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5
          Log:
          Merge pull request #124 from jglick/GroovyCategorySupport-JENKINS-26481

          JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack

          Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/GroovyClassLoaderWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/SandboxContinuable.java src/main/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHack.java src/test/java/org/jenkinsci/plugins/workflow/SerializationTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/SnippetizerTest.java src/test/java/org/jenkinsci/plugins/workflow/cps/persistence/IteratorHackTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/3ec591db4bca053eb70534a92583dcd5b52bf6e5 Log: Merge pull request #124 from jglick/GroovyCategorySupport- JENKINS-26481 JENKINS-26481 JENKINS-27421 Use GroovyCategorySupport to invoke CpsDefaultGroovyMethods (w/o DGMPatcher) & IteratorHack Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/b48af161645f...3ec591db4bca

          Great to see that a solution is implemented.

          I've got one question, though:

          Will the solution work for any collections and allow all kind of different possibilities to iterate through them?

          Joerg Schwaerzler added a comment - Great to see that a solution is implemented. I've got one question, though: Will the solution work for any collections and allow all kind of different possibilities to iterate through them?

          Jesse Glick added a comment -

          The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.

          Jesse Glick added a comment - The fix is not sensitive to the implementation class of the collection but it is specific to the interface method being used to produce an iterator. Commonly used methods should be covered but there are surely some things missing. You can review IteratorHack and its test for details.

          This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.

          autarch princeps added a comment - This is still broken, being able to explain why isn't the same as fixing it. And it definitively is a bug. The same code works fine in raw Groovy.

          Robby Pocase added a comment -

          For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.

          Robby Pocase added a comment - For me, this bug is really inconsistent. I've hit this in production pipelines but then be unable to generate a minimal reproduction. I'm going to get a sample job as I have time.

          Jesse Glick added a comment -

          autarchprinceps / rpocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.

          Jesse Glick added a comment - autarchprinceps / rpocase please do not reopen. If you continue to have issues, file fresh bug reports with complete steps to reproduce from scratch and link them to this one.

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            38 Vote for this issue
            Watchers:
            59 Start watching this issue

              Created:
              Updated:
              Resolved: