Details
-
Bug
-
Status: Resolved (View Workflow)
-
Blocker
-
Resolution: Fixed
Description
Just upgraded to 1.561, and I'm happy to see JENKINS-20769 fixed, but now I'm seeing these instead:
SEVERE: Trying to unexport an object that's already unexported java.lang.IllegalStateException: Invalid object ID 6 iota=36 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:300) at hudson.remoting.Channel.unexport(Channel.java:600) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38) at hudson.remoting.Channel$2.handle(Channel.java:475) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: java.lang.Exception: Object was recently deallocated #6 (ref.0) : hudson.remoting.FastPipedOutputStream Created at Fri May 02 09:50:35 PDT 2014 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:86) at hudson.remoting.ExportTable.export(ExportTable.java:239) at hudson.remoting.Channel.export(Channel.java:592) at hudson.remoting.Pipe.writeObject(Pipe.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at hudson.remoting.UserRequest._serialize(UserRequest.java:155) at hudson.remoting.UserRequest.serialize(UserRequest.java:164) at hudson.remoting.UserRequest.<init>(UserRequest.java:62) at hudson.remoting.Channel.callAsync(Channel.java:766) at hudson.FilePath.read(FilePath.java:1640) at hudson.FilePath.readToString(FilePath.java:1662) at hudson.tools.JDKInstaller.performInstallation(JDKInstaller.java:126) at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:61) at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:107) at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:204) at hudson.model.JDK.forNode(JDK.java:126) at org.jenkinsci.plugins.envinject.service.EnvInjectVariableGetter.getBuildVariables(EnvInjectVariableGetter.java:79) at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironmentWithoutJobPropertyObject(EnvInjectListener.java:231) at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironment(EnvInjectListener.java:46) at hudson.model.AbstractBuild$AbstractBuildExecution.createLauncher(AbstractBuild.java:570) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:476) at hudson.model.Run.execute(Run.java:1709) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Released at Fri May 02 09:50:35 PDT 2014 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303) at hudson.remoting.Channel.unexport(Channel.java:600) at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:384) at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270) ... 5 more Caused by: Released at Fri May 02 09:50:35 PDT 2014 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303) at hudson.remoting.Channel.unexport(Channel.java:600) at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:384) at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)
Attachments
Issue Links
- is related to
-
JENKINS-23271 Intermittent Invalid Object ID in remoting module
-
- Resolved
-
-
JENKINS-18781 Configurable channel timeout for slaves
-
- Open
-
- links to
From what I see, the issue happens because Jenkins tries to unexport ProxyInputStream objects from two sides.
My repro steps:
1) Create a test slave with 20 executors
2) Run the Pipeline script below
3) Interrupt build once it starts performing short stashes
Side 1. Remote call releases exports at the end of the call. E.g. if it gets interrupted
Side 2. When ProxyInputStream gets closed, it sends asynchronous EOF command to the remote side. This EOF also tries to release the allocated object (FileInputStream in my case). By the time the request gets to the remote executor, UserRequest.releaseExports() unexports the object.
ProxyInputStream is nested into RemoteInputStream, which is actually being used for many cases like stdin/stdout propagation in remote calls. So technically any remote call operation may overlap with UserRequest termination handler and cause an overlap in such case.
So the solution would be to either make ProxyInputStream EOF command synchronous or to make it tolerant against race conditions.
Reproducing the issue in tests