Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50020

CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major

      This happens to multiple jobs in our queues, but the logs are related to one job. the job stop processing and the master logs contains the following 

       

      Mar 07, 2018 4:51:52 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      INFO: Running CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript #2290]] unresponsive for 5 sec
      -- SNIP --
      Mar 07, 2018 4:55:18 PM org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0
      INFO: Running CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript #2290]] unresponsive for 3 min 30 sec
      

      Shortly thereafter we see an exception.

      Mar 07, 2018 4:55:22 PM
      org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem
      WARNING: Unexpected exception in CPS VM thread:
      CpsFlowExecution[Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript
      #2290]]
      java.lang.RuntimeException: Failed to serialize
      org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions
      for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
              at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
              at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
              at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
              at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
              at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
              at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
              at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
              at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
              at hudson.XmlFile.write(XmlFile.java:181)
              at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.writeNode(SimpleXStreamFlowNodeStorage.java:182)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.flushNode(SimpleXStreamFlowNodeStorage.java:151)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.autopersist(SimpleXStreamFlowNodeStorage.java:136)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.autopersist(CpsFlowExecution.java:1691)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.maybeAutoPersistNode(CpsFlowExecution.java:1149)
              at org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:120)
              at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.addBodyEndFlowNode(CpsBodyExecution.java:405)
              at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.access$800(CpsBodyExecution.java:64)
              at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:343)
              at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:157)
              at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:49)
              at com.cloudbees.groovy.cps.Next.step(Next.java:83)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.RuntimeException: Failed to serialize
      org.jenkinsci.plugins.workflow.actions.ErrorAction#error for class
      org.jenkinsci.plugins.workflow.actions.ErrorAction
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
              at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
              at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
              at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
              at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
              at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
              at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
              at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              ... 50 more
      Caused by: java.lang.RuntimeException: Failed to serialize
      java.lang.Throwable#cause for class java.lang.RuntimeException
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
              at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
              at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
              at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
              at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
              at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
              at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              ... 65 more
      Caused by: java.lang.RuntimeException: Failed to serialize
      java.lang.Throwable#stackTrace for class
      com.thoughtworks.xstream.io.StreamException
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
              at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
              at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
              at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
              at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
              at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
              at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              ... 75 more
      Caused by: com.thoughtworks.xstream.io.StreamException:  : null
              at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
              at com.thoughtworks.xstream.core.util.QuickWriter.write(QuickWriter.java:49)
              at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.writeText(PrettyPrintWriter.java:279)
              at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.writeText(PrettyPrintWriter.java:231)
              at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.setValue(PrettyPrintWriter.java:214)
              at com.thoughtworks.xstream.io.WriterWrapper.setValue(WriterWrapper.java:45)
              at com.thoughtworks.xstream.converters.SingleValueConverterWrapper.marshal(SingleValueConverterWrapper.java:45)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
              at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
              at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
              at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              ... 85 more
      Caused by: java.nio.channels.ClosedByInterruptException
              at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
              at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
              at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
              at java.nio.channels.Channels.writeFully(Channels.java:101)
              at java.nio.channels.Channels.access$000(Channels.java:61)
              at java.nio.channels.Channels$1.write(Channels.java:174)
              at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
              at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
              at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
              at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
              at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
              at java.io.BufferedWriter.flush(BufferedWriter.java:254)
              at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
              at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
              ... 102 more
      

      And then some more exceptions

      Mar 07, 2018 5:03:41 PM
      org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
      WARNING: already completed
      CpsStepContext[74:node]:Owner[Language-Tests/JavaScript/2290:Language-Tests/JavaScript
      #2290]
      java.lang.IllegalStateException: delivered here
              at org.jenkinsci.plugins.workflow.cps.CpsStepContext.completed(CpsStepContext.java:353)
              at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:329)
              at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask$PlaceholderExecutable.run(ExecutorStepExecution.java:705)
              at hudson.model.ResourceController.execute(ResourceController.java:97)
              at hudson.model.Executor.run(Executor.java:429)
      

      and finally. 

      Mar 07, 2018 5:03:41 PM
      org.jenkinsci.plugins.workflow.cps.CpsStepContext completed
      INFO: earlier failure
      java.lang.RuntimeException: Failed to serialize
      org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions
      for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
              at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
              at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
              at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
              at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
              at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
              at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
              at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
              at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
              at hudson.XmlFile.write(XmlFile.java:181)
              at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.writeNode(SimpleXStreamFlowNodeStorage.java:182)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:193)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.saveActions(CpsFlowExecution.java:1702)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:1129)
              at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:446)
              at org.jenkinsci.plugins.workflow.graph.FlowNode.persistSafe(FlowNode.java:452)
              at org.jenkinsci.plugins.workflow.graph.FlowNode.access$100(FlowNode.java:61)
              at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:411)
              at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:401)
              at java.util.AbstractList.add(AbstractList.java:108)
              at hudson.model.Actionable.addAction(Actionable.java:155)
              at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution.start(ExecutorStepExecution.java:104)
              at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
              at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
              at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
              at companyNode.call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/company-utils/vars/companyNode.groovy:4)
              at companyJob.call(/mnt/jenkins-home/jobs/Language-Tests/jobs/JavaScript/builds/2290/libs/company-utils/vars/companyJob.groovy:88)
              at ___cps.transform___(Native Method)
              at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
              at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
              at sun.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
              at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
              at com.cloudbees.groovy.cps.Next.step(Next.java:83)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
              at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
              at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
              at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
              at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
              at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
              at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
              at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
              at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
              at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: com.thoughtworks.xstream.io.StreamException:  : null
              at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
              at com.thoughtworks.xstream.core.util.QuickWriter.write(QuickWriter.java:37)
              at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:317)
              at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
              at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
              at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:65)
              at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
              at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
              at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
              at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
              at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
              at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
              at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
              at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
              at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
              at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
              at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
              at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
              at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
              at hudson.XmlFile.write(XmlFile.java:181)
              at org.jenkinsci.plugins.workflow.support.PipelineIOUtils.writeByXStream(PipelineIOUtils.java:30)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.writeNode(SimpleXStreamFlowNodeStorage.java:182)
              at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:193)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.saveActions(CpsFlowExecution.java:1702)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:1129)
              at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:446)
              at org.jenkinsci.plugins.workflow.graph.FlowNode.persistSafe(FlowNode.java:452)
              at org.jenkinsci.plugins.workflow.graph.FlowNode.access$100(FlowNode.java:61)
              at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:411)
              at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:401)
              at java.util.AbstractList.add(AbstractList.java:108)
              at hudson.model.Actionable.addAction(Actionable.java:155)
              at org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution.start(ExecutorStepExecution.java:104)
              at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
              at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
              at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
              at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
              at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
              at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
              at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
              ... 33 more
      Caused by: java.nio.channels.ClosedByInterruptException
              at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
              at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
              at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
              at java.nio.channels.Channels.writeFully(Channels.java:101)
              at java.nio.channels.Channels.access$000(Channels.java:61)
              at java.nio.channels.Channels$1.write(Channels.java:174)
              at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
              at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
              at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
              at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
              at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
              at java.io.BufferedWriter.flush(BufferedWriter.java:254)
              at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
              at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
              ... 77 more

      Once the job has hung, we need to kill the job. 

      Any assistance would be welcomed in trying to find the root cause and solving this problem. 

       az

          [JENKINS-50020] CpsFlowExecution unresponsive and Failed to serialize SimpleXStreamFlowNodeStorage

          Andrew Bayer added a comment -

          Alrightie, I’ll dig tomorrow.

          Andrew Bayer added a comment - Alrightie, I’ll dig tomorrow.

          Thanks for taking a look! It's possibly worth noting that after this happens a lot we also see a completely unresponsive web interface which forces us to kill jenkins and start up again, accompanied by these log messages:

          Mar 01, 2018 1:02:56 PM jenkins.metrics.api.Metrics$HealthChecker execute
          WARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [Handling POST /ajaxExecutors from 10.42.2.5 : RequestHandlerThread[#8880] View/ajaxExecutors.jelly locked on java.util.concurrent.locks.ReentrantReadWriteLock$No
          nfairSync@3acfc25d (owned by Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=Cp
          sStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null}):
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
          at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
          at hudson.model.Executor.getAsynchronousExecution(Executor.java:646)
          at hudson.model.Executor.isDisplayCell(Executor.java:666)
          at hudson.model.Computer.getDisplayExecutors(Computer.java:991)
          at sun.reflect.GeneratedMethodAccessor1217.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
          at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
          at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
          at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
          at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
          at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
          at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
          at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
          at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
          at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
          at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
          at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
          at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
          at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)
          at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
          at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
          at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
          at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
          at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
          at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
          at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
          at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
          at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
          at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:97)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:782)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
          at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
          , Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=CpsStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null} locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@34082b74 (owned by jenkins.util.Timer [#7]):
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
          at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
          at hudson.model.Queue._withLock(Queue.java:1430)
          at hudson.model.Queue.withLock(Queue.java:1293)
          at jenkins.model.Nodes.updateNode(Nodes.java:178)
          at jenkins.model.Jenkins.updateNode(Jenkins.java:2115)
          at hudson.model.Node.save(Node.java:136)
          at hudson.model.Node.setTemporaryOfflineCause(Node.java:271)
          at hudson.model.Computer.setTemporarilyOffline(Computer.java:714)
          at com.microsoft.azure.vmagent.AzureVMAgent.setCleanUpAction(AzureVMAgent.java:326)
          at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.done(AzureVMCloudOnceRetentionStrategy.java:94)
          at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.done(AzureVMCloudOnceRetentionStrategy.java:69)
          at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.taskCompleted(AzureVMCloudOnceRetentionStrategy.java:56)
          at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:343)
          at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140)
          at hudson.model.Executor.finish1(Executor.java:477)
          at hudson.model.Executor.completedAsynchronous(Executor.java:499)
          at jenkins.model.queue.AsynchronousExecution.setExecutor(AsynchronousExecution.java:115)
          at hudson.model.Executor.run(Executor.java:434)
          , jenkins.util.Timer [#7] locked on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3acfc25d (owned by Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=CpsStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null}):
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
          at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
          at hudson.model.Executor.isParking(Executor.java:674)
          at hudson.model.Queue.maintain(Queue.java:1473)
          at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          ]]

          Dominic Hargreaves added a comment - Thanks for taking a look! It's possibly worth noting that after this happens a lot we also see a completely unresponsive web interface which forces us to kill jenkins and start up again, accompanied by these log messages: Mar 01, 2018 1:02:56 PM jenkins.metrics.api.Metrics$HealthChecker execute WARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [Handling POST /ajaxExecutors from 10.42.2.5 : RequestHandlerThread[#8880] View/ajaxExecutors.jelly locked on java.util.concurrent.locks.ReentrantReadWriteLock$No nfairSync@3acfc25d (owned by Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=Cp sStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null}): at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at hudson.model.Executor.getAsynchronousExecution(Executor.java:646) at hudson.model.Executor.isDisplayCell(Executor.java:666) at hudson.model.Computer.getDisplayExecutors(Computer.java:991) at sun.reflect.GeneratedMethodAccessor1217.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:97) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:782) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) , Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=CpsStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null} locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@34082b74 (owned by jenkins.util.Timer [#7]): at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1430) at hudson.model.Queue.withLock(Queue.java:1293) at jenkins.model.Nodes.updateNode(Nodes.java:178) at jenkins.model.Jenkins.updateNode(Jenkins.java:2115) at hudson.model.Node.save(Node.java:136) at hudson.model.Node.setTemporaryOfflineCause(Node.java:271) at hudson.model.Computer.setTemporarilyOffline(Computer.java:714) at com.microsoft.azure.vmagent.AzureVMAgent.setCleanUpAction(AzureVMAgent.java:326) at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.done(AzureVMCloudOnceRetentionStrategy.java:94) at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.done(AzureVMCloudOnceRetentionStrategy.java:69) at com.microsoft.azure.vmagent.AzureVMCloudOnceRetentionStrategy.taskCompleted(AzureVMCloudOnceRetentionStrategy.java:56) at hudson.slaves.SlaveComputer.taskCompleted(SlaveComputer.java:343) at hudson.model.queue.WorkUnitContext.synchronizeEnd(WorkUnitContext.java:140) at hudson.model.Executor.finish1(Executor.java:477) at hudson.model.Executor.completedAsynchronous(Executor.java:499) at jenkins.model.queue.AsynchronousExecution.setExecutor(AsynchronousExecution.java:115) at hudson.model.Executor.run(Executor.java:434) , jenkins.util.Timer [#7] locked on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3acfc25d (owned by Executor #0 for jenkins-linux-build-slaveb4d1b0 : executing PlaceholderExecutable:ExecutorStepExecution.PlaceholderTask{runId=JOBNAME#1247,label=(linux||linux-fallback||linux-scalable),context=CpsStepContext[10:node]:Owner[JOBNAME/1247:JOBNAME #1247],cookie=null,auth=null}): at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at hudson.model.Executor.isParking(Executor.java:674) at hudson.model.Queue.maintain(Queue.java:1473) at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) ]]

          Sam Van Oort added a comment -

          So, there's three defects here:

          1. The latter defect is probably root-caused with a Pipeline internal issue – something with dangling tasks that aren't properly killed, and it will probably be resolved once I can reliably reproduce an issue we've been seeing lately and issue a fix.
          2. The Queue is having some sort of locking issue with the PlaceholderTask for Pipeline – no clue here, but it shouldn't be able to hang. The actual trigger for this is trying to display the Executors in use which interacts with both the Executor and the Queue.
          3. The actual failure to serialize a trivial FlowNode, along with an associated InterruptedException (a timeout?).

          hentis Do you have any idea how to regularly reproduce this with a simpler job?

          Sam Van Oort added a comment - So, there's three defects here: 1. The latter defect is probably root-caused with a Pipeline internal issue – something with dangling tasks that aren't properly killed, and it will probably be resolved once I can reliably reproduce an issue we've been seeing lately and issue a fix. 2. The Queue is having some sort of locking issue with the PlaceholderTask for Pipeline – no clue here, but it shouldn't be able to hang. The actual trigger for this is trying to display the Executors in use which interacts with both the Executor and the Queue. 3. The actual failure to serialize a trivial FlowNode, along with an associated InterruptedException (a timeout?). hentis Do you have any idea how to regularly reproduce this with a simpler job?

          Henti Smith added a comment -

          svanoort

          Our build is pretty simple. 

          @Library('semmle-utils') _
          
          semmleJob {
          semmleNode(getLabelExpr('linux')) {
          stage('Checkout') {
          semmleCheckout(gitRepo, gitBranch, gitSha)
          }
          stage('Test') {
          sh "./clean"
          sh "./build target/test/javascript-extractor-tests/results.xml"
          sh "./build target/test/javascript/results.xml"
          sh "./build target/test/common-javascript/results.xml"
          junit 'target/test/*/*.xml'
          }
          }
          }
          

          semmleJob checks to see if the  job is mergable or out of date

          semmleNode checks to see if the last build on the node failed and if so clean up first otherwise periodic cleanups are triggered.

          semmleCheckout dis sane git checkouts with submodules etc based on a sha

          I suspect it might be related to the Azure VM agent plugin, as can be seen in Dominic's post as we had a similar deadlock this morning. 

          Henti Smith added a comment - svanoort Our build is pretty simple.  @Library( 'semmle-utils' ) _ semmleJob { semmleNode(getLabelExpr( 'linux' )) { stage( 'Checkout' ) { semmleCheckout(gitRepo, gitBranch, gitSha) } stage( 'Test' ) { sh "./clean" sh "./build target/test/javascript-extractor-tests/results.xml" sh "./build target/test/javascript/results.xml" sh "./build target/test/common-javascript/results.xml" junit 'target/test/*/*.xml' } } } semmleJob checks to see if the  job is mergable or out of date semmleNode checks to see if the last build on the node failed and if so clean up first otherwise periodic cleanups are triggered. semmleCheckout dis sane git checkouts with submodules etc based on a sha I suspect it might be related to the Azure VM agent plugin, as can be seen in Dominic's post as we had a similar deadlock this morning. 

          2. The Queue is having some sort of locking issue with the PlaceholderTask for Pipeline – no clue here, but it shouldn't be able to hang. The actual trigger for this is trying to display the Executors in use which interacts with both the Executor and the Queue.

          I believe PR#3354 fixes/works around the deadlock issue (which is very similar to JENKINS-46248). The serialization issues discussed here are separate.

          Pavel Avgustinov added a comment - 2. The Queue is having some sort of locking issue with the PlaceholderTask for Pipeline – no clue here, but it shouldn't be able to hang. The actual trigger for this is trying to display the Executors in use which interacts with both the Executor and the Queue. I believe PR#3354  fixes/works around the deadlock issue (which is very similar to JENKINS-46248 ). The serialization issues discussed here are separate.

          Sam Van Oort added a comment -

          I suspect the pipeline-internal issue is at least partially addressed in the latest workflow-cps and workflow-job releases, and issues with ErrorAction are handled in the latest workflow-api releases.

          Sam Van Oort added a comment - I suspect the pipeline-internal issue is at least partially addressed in the latest workflow-cps and workflow-job releases, and issues with ErrorAction are handled in the latest workflow-api releases.

          Sam Van Oort added a comment -

          The fixes to JENKINS-50752 should handle the serialization issues.

          Sam Van Oort added a comment - The fixes to JENKINS-50752 should handle the serialization issues.

          We're still seeing this problem with the following versions:

          Jenkins core 2.107.3

          workflow-cps 2.53

          workflow-job 2.21

          Once the CpsFlowExecution unresponsive errors start, they continue until I restart the server. The Web UI becomes unresponsive within minutes of the errors starting (possibly immediately, I don't have sufficiently accurate monitoring to be sure).

          Does the core need to be updated as well (there isn't an LTS release with the fix in yet) or should the plugin update be enough?

          Philip Douglas added a comment - We're still seeing this problem with the following versions: Jenkins core 2.107.3 workflow-cps 2.53 workflow-job 2.21 Once the CpsFlowExecution unresponsive errors start, they continue until I restart the server. The Web UI becomes unresponsive within minutes of the errors starting (possibly immediately, I don't have sufficiently accurate monitoring to be sure). Does the core need to be updated as well (there isn't an LTS release with the fix in yet) or should the plugin update be enough?

          Sam Van Oort added a comment -

          pdouglas I'm sorry to hear that.    This fix does not require a core update, but since the cause here is resolved I'm assuming your issues represent a different problem generating similar results.  So that we can investigate,  please open a new ticket up, and include all of the Pipeline plugin versions (one of the fixes here was in workflow-api plugin)  and the actual stack trace of the error.

          Sam Van Oort added a comment - pdouglas  I'm sorry to hear that.    This fix does not require a core update, but since the cause here is resolved I'm assuming your issues represent a different problem generating similar results.  So that we can investigate,  please open a new ticket up, and include all of the Pipeline plugin versions (one of the fixes here was in workflow-api plugin)  and the actual stack trace of the error.

          Ok, thanks. I've created a new ticket here: https://issues.jenkins-ci.org/browse/JENKINS-52362

          Philip Douglas added a comment - Ok, thanks. I've created a new ticket here: https://issues.jenkins-ci.org/browse/JENKINS-52362

            abayer Andrew Bayer
            hentis Henti Smith
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: