Reporting this as I expect failure like this could start affecting more users.

And we are not sure yet how to solve this for our workflows.

We do regular upgrades of jenkins plugins and versions and deploy, deploying with this version upgrade looks like it could break workflow in jenkins containers and we will have failures on every site, not sure if in addition every site would have a broken jenkins.

Workflow run after Pipeline upgrades from 2.2221 to 2.2234 failed with java stack-dump in field groovy.lang.Closure.delegate, .. yadda ..,  in object of type org.jenkinsci.plugins.workflow.cps.CpsThreadGroup
Caused: java.io.InvalidClassException: org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AnyScript; Class does not extend stream superclass .. 

The workflow failed after a 1 minute sleep  at end of a pipeline stage or at the start of a new pipeline stage. Apologies I am unfamiliar with digging in more to the java stack-dump, full dump below. 

We run jenkins in a docker container and upgrades are done by stopping docker container and starting new with new image with updated scripts, jenkins plugins, etc ... The XXX-ku-upgrade.yml task does this (by spawning an upgrade container to manage switch-over).

stage('ku upgrade') {
         steps {
            script {
                def inventory = readYaml file: 'inventory.yml'

                if (inventory.all.children.ku.hosts.ku_container.latest_version == inventory.all.children.ku.hosts.ku_container.current_version)

                ansiblePlaybook inventory: 'inventory.d/', playbook: '/ansible_playbooks/kuku/XXX-ku-upgrade.yml'
            }
            sleep 60
         }
      }
      stage('Send deploy event') {
          steps {
              sleep 60
              script {
                    def inventory = readYaml file: 'inventory.yml'

                    if (inventory.all.children.ku.hosts.ku_container.latest_version == inventory.all.children.ku.hosts.ku_container.current_version)

                    timeout(time: 60, unit: 'MINUTES')

                }
            }
        }

   }

We can see warning about docker-workflow plugin (in release 2.2221) but we do not use that.

"Warning: this is a breaking change for docker-workflow. Please update to https://github.com/jenkinsci/docker-workflow-plugin/releases/tag/599.v76126c79a_a_2d or newer.

Also, Declarative builds running across the upgrade may fail. New builds should work."

Workflow failure on upgrade, from workflow job/console:

No debug.log produced

[Pipeline] }
[Pipeline] // script
[Pipeline] sleep
Sleeping for 1 min 0 sec
an exception which occurred:
    in field groovy.lang.Closure.delegate
    in object org.jenkinsci.plugins.workflow.cps.CpsClosure2@1c5abcac
    in object of type org.jenkinsci.plugins.workflow.cps.CpsClosure2
    in field groovy.lang.Closure.delegate
    in object org.jenkinsci.plugins.workflow.cps.CpsClosure2@3ed08b09
    in object of type org.jenkinsci.plugins.workflow.cps.CpsClosure2
    in object of type java.util.HashMap
    in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.closures
    in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7726ffa3
    in object of type org.jenkinsci.plugins.workflow.cps.CpsThreadGroup
Caused: java.io.InvalidClassException: org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AnyScript; Class does not extend stream superclass
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1122)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1431)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:259)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1978)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1891)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:259)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1978)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1891)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:500)
    at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467)
    at java.base/java.util.HashMap.readObject(HashMap.java:1552)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.reflect.SerMethods.callReadObject(SerMethods.java:83)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:252)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1868)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:259)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1978)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1891)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:244)
    at PluginClassLoader for workflow-support//org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
    at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:331)
    at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.sandbox(RiverReader.java:250)
    at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.readObject(RiverReader.java:263)
    at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:879)
    at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:872)
    at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:147)
    at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.DirectExecutor.execute(DirectExecutor.java:33)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1307)
    at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:772)
    at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:157)
    at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:97)
    at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:869)
    at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:836)
    at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:733)
    at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:597)
    at hudson.model.RunMap.retrieve(RunMap.java:273)
    at hudson.model.RunMap.retrieve(RunMap.java:65)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:703)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:685)
    at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:579)
    at hudson.model.RunMap.getById(RunMap.java:237)
    at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:979)
    at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:991)
    at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$DefaultStorage.resume(FlowExecutionList.java:252)
    at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList.resume(FlowExecutionList.java:127)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
ERROR: Failed to load program
Finished: FAILURE