-
Bug
-
Resolution: Fixed
-
Minor
-
-
Pipeline - April 2018
From the logs of ci.jenkins.io
Dec 11, 2017 7:16:02 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Infra/plugin-site-api/generate-data/14477:Infra/plugin-site-api/generate-data #14477]] java.lang.NullPointerException at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:342) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Dec 11, 2017 7:16:02 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Infra/plugin-site-api/generate-data/14476:Infra/plugin-site-api/generate-data #14476]] java.lang.NullPointerException at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:342) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Dec 11, 2017 7:16:02 PM org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService reportProblem WARNING: Unexpected exception in CPS VM thread: CpsFlowExecution[Owner[Infra/plugin-site-api/generate-data/14475:Infra/plugin-site-api/generate-data #14475]] java.lang.NullPointerException at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:342) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
[JENKINS-48491] NPE in CpsThreadGroup.run & IOOBE in CpsThreadGroup.readResolve
Looking at the specific runs that are referenced here, I see this in the last FlowNode XML it generated.
<error class="java.io.IOException"> <detailMessage>Failed to load build state</detailMessage> <cause class="java.lang.IndexOutOfBoundsException"> <detailMessage>Index: 0, Size: 0</detailMessage> <cause class="org.jboss.marshalling.TraceInformation" plugin="workflow-support@2.16"> <stackTrace/> <suppressedExceptions class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList"> <c class="list"/> <list reference="../c"/> </suppressedExceptions> <info class="org.jboss.marshalling.TraceInformation$IncompleteObjectInfo"> <targetClassName>org.jenkinsci.plugins.workflow.cps.CpsThreadGroup</targetClassName> </info> </cause> <stackTrace> <trace>java.util.ArrayList.rangeCheck(ArrayList.java:653)</trace> <trace>java.util.ArrayList.get(ArrayList.java:429)</trace> <trace>org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.readResolve(CpsThreadGroup.java:157)</trace> <trace>sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)</trace> <trace>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)</trace> <trace>java.lang.reflect.Method.invoke(Method.java:498)</trace> <trace>org.jboss.marshalling.reflect.SerializableClass.callReadResolve(SerializableClass.java:415)</trace> <trace>org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1293)</trace> <trace>org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)</trace> <trace>org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)</trace> <trace>org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)</trace> <trace>org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:627)</trace> <trace>org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:620)</trace> <trace>org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:150)</trace> <trace>com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)</trace> <trace>com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)</trace> <trace>com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)</trace> <trace>com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:155)</trace> <trace>org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:160)</trace> <trace>org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:90)</trace> <trace>org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:617)</trace> <trace>org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:589)</trace> <trace>org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:613)</trace> <trace>hudson.model.RunMap.retrieve(RunMap.java:225)</trace> <trace>hudson.model.RunMap.retrieve(RunMap.java:57)</trace> ...
The wonderfully weird thing about that is that right before line 157, where it does call scripts.get(0), is if (!scripts.isEmpty()). Whaaaat.
Note that all the builds with this problem seem to be almost four months old and attempted to resume a few weeks ago (according to the timestamp on the 2.xml and 3.xml files). They still have program.dat files, which date to four days ago, and their log files date to the last restart of ci.jenkins.io.
Obviously, the logspam can be eliminated by just deleting the weirdo builds, but there's at least a couple levels of weirdness here - the IndexOutOfBoundsException in CpsThreadGroup#readResolve, and above that, the NullPointerException in CpsThreadGroup#run. The NPE is probably easily worked around as jglick laid out, but I'm legitimately curious as to how the IndexOutOfBoundsException can actually happen, given that we just verified that the list in question isn't empty. Good times?
A little more detail - I cut off the stack trace from the FlowNode before I should have. It follows with...
<trace>hudson.model.RunMap.retrieve(RunMap.java:225)</trace>
<trace>hudson.model.RunMap.retrieve(RunMap.java:57)</trace>
<trace>jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:500)</trace>
<trace>jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:482)</trace>
<trace>jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:380)</trace>
<trace>jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)</trace>
<trace>org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:228)</trace>
<trace>io.jenkins.blueocean.rest.impl.pipeline.PipelineRunImpl.getCommitUrl(PipelineRunImpl.java:231)</trace>
<trace>io.jenkins.blueocean.commons.stapler.export.MethodProperty.getValue(MethodProperty.java:72)</trace>
<trace>io.jenkins.blueocean.commons.stapler.export.ExportInterceptor$1.getValue(ExportInterceptor.java:43)</trace>
<trace>io.jenkins.blueocean.commons.stapler.Export$BlueOceanExportInterceptor.getValue(Export.java:167)</trace>
<trace>io.jenkins.blueocean.commons.stapler.export.Property.writeTo(Property.java:136)</trace>
<trace>io.jenkins.blueocean.commons.stapler.export.Model.writeNestedObjectTo(Model.java:228)</trace>
<trace>io.jenkins.blueocean.commons.stapler.export.Model.writeTo(Model.java:199)</trace>
<trace>io.jenkins.blueocean.commons.stapler.Export.writeOne(Export.java:148)</trace>
<trace>io.jenkins.blueocean.commons.stapler.Export.serveExposedBean(Export.java:139)</trace>
<trace>io.jenkins.blueocean.commons.stapler.Export.doJson(Export.java:79)</trace>
<trace>io.jenkins.blueocean.commons.stapler.TreeResponse$Processor$1.generateResponse(TreeResponse.java:48)</trace>
And then a bunch more stapler stuff. Smells to me like Blue Ocean is trying to do something with the run at the same time it's loading, and things are colliding badly.
Moving to Blue Ocean since this is being caused by something Blue Ocean is doing.
The stack frames from Blue Ocean look harmless to me. It is loading build records, which is not great, but hardly unusual.
As to the IOOBE, note that this is a mutable field, so it is conceivable that the scripts.clear() call is somehow running concurrently with readResolve, for example (though I cannot imagine how that would happen). At any rate, it would be prudent for readResolve to at least make a local copy. Or simply ignore this IOOBE for now, under the assumption it is unreproxucible, and focus on fixing the NPE so that the build at least terminates with a visible stack trace.
Sam Van Oort
added a comment - I suspect that the megafix I have been working on will resolve several of the weirdnesses here (finalized and in review now).
Sam Van Oort
added a comment - I suspect that the megafix I have been working on will resolve several of the weirdnesses here (finalized and in review now). Sam Van Oort
added a comment - rtyler I believe this should be resolved as of workflow-cps 2.47 and workflow-job 2.18 – can you give the updates a try and let us know if comes up after the update? Thanks!
Sam Van Oort
added a comment - rtyler I believe this should be resolved as of workflow-cps 2.47 and workflow-job 2.18 – can you give the updates a try and let us know if comes up after the update? Thanks! 
Jenn Briden
added a comment - svanoort, was this fixed as part of https://github.com/jenkinsci/workflow-cps-plugin/pull/216?
Jenn Briden
added a comment - svanoort , was this fixed as part of https://github.com/jenkinsci/workflow-cps-plugin/pull/216? Sam Van Oort
added a comment - I'm going to assume this was fixed since rtyler never replied back and we've addressed this with code changes.
Sam Van Oort
added a comment - I'm going to assume this was fixed since rtyler never replied back and we've addressed this with code changes. 
Somehow an exception is being thrown and then there is no active thread corresponding to it. No clue how this could happen, but anyway this is probably a build-ending event so if something here is unexpectedly null, log what it is and skip the ErrorAction for now. I do not really follow the logic here and how it interacts with exception handling in parallel blocks.