-
Bug
-
Resolution: Fixed
-
Blocker
-
1.544.2, 1.563, 1.625.1 - Windows or Linux Master, various slaves
-
Powered by SuggestiMate
For the last two weeks, I have been seeing an intermittent failure during the clearcase stage of our builds. It throws a Java exception along the lines of
FATAL: Invalid object ID 18649 iuota=18470
java.lang.IllegalStateException: Invalid object ID 18469 iota=18470
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
etc...
It always happens during the Clearcase segment of the build, but on a different step each time. Sometimes after lsview, sometimes after lshistory, sometimes after mkview ...
It happens a very low percentage of the time, and it happens on a different project each time. I have around 300 projects running between a number of matrix jobs- we are seeing one or two failures on each runthrough of this, always on different jobs and on different slaves. It started a couple weeks ago- our clearcase server crashed, and jenkins was having some trouble interacting with the server. I upgraded to 1.563 to try and solve that problem (I wound up having to manually delete all the views that had been created in the period between the backup and the crash). Once that issue had been resolved though, we started seeing this problem.
I upgraded the clearcase plugin to 1.5, and then tried downgrading Jenkins to 1.554.2. I cannot swear to it, but I thought I was on 1.558 before the first problem popped up, and had been very stable for a long time with all build failures being attributed to bad code, so suddenly having Jenkins fail some is kind of jarring.
At this point, I am not sure what I changed to cause the problem. I remember I had to upgrade slave.jar to version 2.41 from whatever it was before when I did the first upgrade to 1.563. Other then that, nothing else substantial has changed. The Jenkins is on a restricted network with no internet access- I am going to type the stack trace below, so forgive any typos.
FATAL: Invalid object ID 18649 iuota=18470
java.lang.IllegalStateException: Invalid object ID 18469 iota=18470
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
at hudson.remoting.ExportTable.get(ExportTable.java:261)
at hudson.remoting.Channel.getExportedObject(Channel.java:596)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:292
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:281)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:240)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(UNKNOWN SOURCE)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Cased by: java.lang.Exception: Object was recently deallocated
#18649 (ref.0) : hudson.Launcher$RemoteLaunchCallable$1
created at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:86)
at hudson.remoting.ExportTable.export(ExportTable.java:239)
at hudson.remoting.Channel.export(Channel.java:592)
at hudson.remoting.Channel.export(Channel.java:583)
at hudson.remoting.Channel.export(Channel.java:553)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1025)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:990)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:63)
at java.lang.Thread.run(Unknown Source)
Released at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
at hudson.remoting.Channel.unexport(Channel.java:600)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270)
... 15 more
Caused by: Released at Fri May 30 19:27:39 EDT 2014
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
at hudson.remoting.Channel.unexport(Channel.java:600)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
at hudson.remoting.Channel$2.handle(Channel.java:475)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
- is duplicated by
-
JENKINS-16393 "Unable to call join" build failures
-
- Resolved
-
- is related to
-
JENKINS-47161 Maven builds hang intermittently with a remoting error
-
- Open
-
-
JENKINS-22853 SEVERE: Trying to unexport an object that's already unexported
-
- Resolved
-
-
JENKINS-42533 Trying to unexport an object that's already unexported
-
- Resolved
-
- links to
[JENKINS-23271] Intermittent Invalid Object ID in remoting module
I've got a reproduction case using Workflow with parallel nodes "unstash"ing some contents from the same "stash" in Workflow:
Using this Jenkinsfile running on a Jenkins master with the Docker plugin. The docker slave image is evarga/jenkins-slave with the following additional system packages installed: git, zlibc, ruby.
jenkins.log from the master:
Dec 09, 2015 2:52:58 PM org.jenkinsci.plugins.workflow.cps.CpsStepContext onFailure WARNING: already completed CpsStepContext[21]:Owner[jenkins-infra/staging/25:jenkins-infra/staging #25] java.lang.IllegalStateException: java.lang.InterruptedException at org.jenkinsci.plugins.workflow.cps.CpsStepContext.onFailure(CpsStepContext.java:314) at org.jenkinsci.plugins.workflow.cps.CpsBodySubContext.onFailure(CpsBodySubContext.java:63) at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onFailure(BodyExecutionCallback.java:124) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:294) at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:58) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:147) at hudson.remoting.Channel.call(Channel.java:780) at hudson.FilePath.act(FilePath.java:979) at hudson.FilePath.act(FilePath.java:968) at hudson.FilePath.untarFrom(FilePath.java:718) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:111) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:54) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) ... 5 more Dec 09, 2015 2:52:58 PM hudson.remoting.Channel$1 handle SEVERE: Failed to execute command EOF(46) (channel Docker-d8dcccb80239) java.lang.IllegalStateException: Invalid object ID 46 iota=47 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) at hudson.remoting.ExportTable.get(ExportTable.java:330) at hudson.remoting.Channel.getExportedObject(Channel.java:633) at hudson.remoting.ProxyInputStream$EOF.execute(ProxyInputStream.java:151) at hudson.remoting.Channel$1.handle(Channel.java:501) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: java.lang.Exception: Object was recently deallocated #46 (ref.0) : object=null type=java.io.FileInputStream interfaces=[java.io.InputStream] Created at Wed Dec 09 14:52:58 PST 2015 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99) at hudson.remoting.ExportTable.export(ExportTable.java:305) at hudson.remoting.Channel.internalExport(Channel.java:629) at hudson.remoting.RemoteInputStream.writeObject(RemoteInputStream.java:168) at sun.reflect.GeneratedMethodAccessor589.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.UserRequest._serialize(UserRequest.java:157) at hudson.remoting.UserRequest.serialize(UserRequest.java:166) at hudson.remoting.UserRequest.<init>(UserRequest.java:62) at hudson.remoting.Channel.call(Channel.java:779) at hudson.FilePath.act(FilePath.java:979) at hudson.FilePath.act(FilePath.java:968) at hudson.FilePath.untarFrom(FilePath.java:718) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:111) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:54) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Released at Wed Dec 09 14:52:58 PST 2015 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:192) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:188) at hudson.remoting.Channel.call(Channel.java:798) at hudson.FilePath.act(FilePath.java:979) at hudson.FilePath.act(FilePath.java:968) at hudson.FilePath.untarFrom(FilePath.java:718) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:111) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:54) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379) ... 5 more Caused by: Released at Wed Dec 09 14:52:58 PST 2015 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:192) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:188) at hudson.remoting.Channel.call(Channel.java:798) at hudson.FilePath.act(FilePath.java:979) at hudson.FilePath.act(FilePath.java:968) at hudson.FilePath.untarFrom(FilePath.java:718) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:111) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:54) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Dec 09, 2015 2:52:58 PM hudson.remoting.Channel$1 handle SEVERE: This command is created here Command EOF(46) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyInputStream$EOF.<init>(ProxyInputStream.java:145) at hudson.remoting.ProxyInputStream.close(ProxyInputStream.java:95) at hudson.remoting.ProxyInputStream.read(ProxyInputStream.java:87) at hudson.remoting.RemoteInputStream.read(RemoteInputStream.java:271) at hudson.util.HeadBufferingStream.read(HeadBufferingStream.java:61) at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:175) at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at hudson.util.IOUtils.copy(IOUtils.java:40) at hudson.FilePath.readFromTar(FilePath.java:2290) at hudson.FilePath.access$400(FilePath.java:190) at hudson.FilePath$10.invoke(FilePath.java:720) at hudson.FilePath$10.invoke(FilePath.java:718) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Dec 09, 2015 2:52:59 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: jenkins-infra/staging #25 completed: FAILURE Dec 09, 2015 2:54:24 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Disconnected computer Dec 09, 2015 2:54:24 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Stopped container 95a0292213952bd7e963e7bf264d0ba4adb9c2ca39dc63374bc4368b63258949 Dec 09, 2015 2:54:25 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Removed container 95a0292213952bd7e963e7bf264d0ba4adb9c2ca39dc63374bc4368b63258949 Dec 09, 2015 2:54:30 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Disconnected computer Dec 09, 2015 2:54:30 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Docker-aeb26c5042fb java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Dec 09, 2015 2:54:30 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Stopped container aeb26c5042fbd4ae7a41d4e22282c6a0835b9f62043e98e58f34870c6bd0c618 Dec 09, 2015 2:54:31 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Removed container aeb26c5042fbd4ae7a41d4e22282c6a0835b9f62043e98e58f34870c6bd0c618 Dec 09, 2015 2:54:34 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Disconnected computer Dec 09, 2015 2:54:34 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel Docker-d8dcccb80239 java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Dec 09, 2015 2:54:34 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Stopped container d8dcccb802393b4d82aad2bd176f33c557e8ae7f530cd57698a0bb402e581ad1 Dec 09, 2015 2:54:35 PM com.nirima.jenkins.plugins.docker.DockerSlave _terminate INFO: Removed container d8dcccb802393b4d82aad2bd176f33c557e8ae7f530cd57698a0bb402e581ad1
console output from the executing workflow:
[Workflow] stash Stashed 16495 file(s) [Workflow] Execute sub-workflows in parallel : Start [Workflow] [linting] parallel {: Parallel branch: linting [Workflow] [rspec] parallel {: Parallel branch: rspec [Workflow] [linting] Allocate node : Start [linting] Running on Docker-d8dcccb80239 in /home/jenkins/workspace/jenkins-infra/staging [Workflow] [rspec] Allocate node : Start [rspec] Running on Docker-95a029221395 in /home/jenkins/workspace/jenkins-infra/staging [Workflow] [linting] node { [Workflow] [rspec] node { [Workflow] [linting] sh [linting] [staging] Running shell script [Workflow] [rspec] sh [rspec] [staging] Running shell script [linting] + ls [linting] Gemfile [linting] Gemfile.lock [linting] Jenkinsfile [linting] Puppetfile [linting] README.md [linting] Rakefile [linting] Vagrantfile [linting] ci [linting] dist [linting] hieradata [linting] manifests [linting] spec [linting] vagrant-bootstrap [linting] vendor [linting] + pwd [linting] /home/jenkins/workspace/jenkins-infra/staging [Workflow] [linting] unstash [rspec] + ls [rspec] Gemfile [rspec] Gemfile.lock [rspec] Jenkinsfile [rspec] Puppetfile [rspec] README.md [rspec] Rakefile [rspec] Vagrantfile [rspec] ci [rspec] dist [rspec] hieradata [rspec] manifests [rspec] spec [rspec] vagrant-bootstrap [rspec] vendor [rspec] + pwd [rspec] /home/jenkins/workspace/jenkins-infra/staging [Workflow] [rspec] unstash [Workflow] } //node [Workflow] Allocate node : End [Workflow] } //parallel [Workflow] } //node [Workflow] Allocate node : End [Workflow] } //parallel [Workflow] Execute sub-workflows in parallel : End [Workflow] } //node [Workflow] Allocate node : End [Workflow] End of Workflow org.jenkinsci.plugins.workflow.cps.steps.ParallelStepException: Parallel step rspec failed at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.checkAllDone(ParallelStep.java:153) at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onFailure(ParallelStep.java:119) at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:294) at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:58) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:145) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:106) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:164) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:274) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$000(CpsThreadGroup.java:74) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:183) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:181) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: remote file operation failed: /home/jenkins/workspace/jenkins-infra/staging at hudson.remoting.Channel@515af5c8:Docker-95a029221395: java.io.IOException: Failed to extract input stream at hudson.FilePath.act(FilePath.java:986) at hudson.FilePath.act(FilePath.java:968) at hudson.FilePath.untarFrom(FilePath.java:718) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:111) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:64) at org.jenkinsci.plugins.workflow.support.steps.stash.UnstashStep$Execution.run(UnstashStep.java:54) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:49) at hudson.security.ACL.impersonate(ACL.java:213) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:47) ... 5 more Caused by: java.io.IOException: Failed to extract input stream at hudson.FilePath.readFromTar(FilePath.java:2300) at hudson.FilePath.access$400(FilePath.java:190) at hudson.FilePath$10.invoke(FilePath.java:720) at hudson.FilePath$10.invoke(FilePath.java:718) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to Docker-95a029221395(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.UserResponse.retrieve(UserRequest.java:220) at hudson.remoting.Channel.call(Channel.java:781) at hudson.FilePath.act(FilePath.java:979) ... 13 more Caused by: java.io.FileNotFoundException: /home/jenkins/workspace/jenkins-infra/staging/vendor/gems/ruby/1.9.1/cache/bundler/git/puppet-lint-6195c688e04f7521049d25a60ac88d23bc4387ef/objects/pack/pack-a6d1070252234bdef67cf729aab5d2ebec395149.idx (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:171) at hudson.util.IOUtils.copy(IOUtils.java:38) at hudson.FilePath.readFromTar(FilePath.java:2290) at hudson.FilePath.access$400(FilePath.java:190) at hudson.FilePath$10.invoke(FilePath.java:720) at hudson.FilePath$10.invoke(FilePath.java:718) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2719) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more Finished: FAILURE
oleg_nenashev: thanks for the feedback. With this reproduction scenario, given that you are the assignee, does this mean you are currently looking at this? Have you been able to zoom in on the problem?
patrickdepinguin Yes I'm looking at the issue, but currently there is no ETA. We firstly want to get a scenario with a high reproduction rate.
oleg_nenashev Understood, thanks. I'm looking forward to more details when you have them.
So we've been plagued by this issue a long time. And have some good tips how to reproduce it .. Get a matrix job with 10+ jobs .. and make each one copy an artifact like a big jar (hundreds of megs) from another job in jenkins. The issue seems to happen when all the matrix jobs at the same time pull the artifact. We reduce the problem by putting some sleeps in the matrix jobs to make sure it doesnt run at the same time.
Have same problem
- Jenkins 1.6.3.5
- Master and Slave running in docker container
java.lang.IllegalStateException: Invalid object ID 17979 iota=17980 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) at hudson.remoting.ExportTable.get(ExportTable.java:330) at hudson.remoting.Channel.getExportedObject(Channel.java:630) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at ......remote call to deimsv0302(rel)(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1413) at hudson.remoting.UserResponse.retrieve(UserRequest.java:221) at hudson.remoting.Channel.call(Channel.java:778) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) at com.sun.proxy.$Proxy47.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:992) at hudson.Launcher$ProcStarter.join(Launcher.java:389) at hudson.tools.AbstractCommandInstaller.performInstallation(AbstractCommandInstaller.java:77) at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:68) at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:108) at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:206) at hudson.model.JDK.forNode(JDK.java:143) at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:358) at hudson.model.Run.getEnvironment(Run.java:2232) at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:941) at hudson.model.AbstractBuild$AbstractBuildExecution.decideWorkspace(AbstractBuild.java:481) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Caused by: java.lang.Exception: Object was recently deallocated #17979 (ref.0) : object=null type=hudson.Launcher$RemoteLaunchCallable$1 interfaces=[hudson.Launcher$RemoteProcess] Created at Tue Dec 15 22:43:06 CET 2015 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99) at hudson.remoting.ExportTable.export(ExportTable.java:305) at hudson.remoting.Channel.internalExport(Channel.java:626) at hudson.remoting.Channel.export(Channel.java:617) at hudson.remoting.Channel.export(Channel.java:587) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1151) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1114) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Released at Tue Dec 15 22:43:06 CET 2015 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) at hudson.remoting.Channel.unexport(Channel.java:638) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) at hudson.remoting.Channel$1.handle(Channel.java:498) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: Command hudson.remoting.UnexportCommand@1ecf5d78 created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319) at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@463b was created for interface hudson.Launcher$RemoteProcess at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137) at hudson.remoting.Channel.export(Channel.java:618) at hudson.remoting.Channel.export(Channel.java:587) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1151) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1114) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379) at hudson.remoting.ExportTable.get(ExportTable.java:330) at hudson.remoting.Channel.getExportedObject(Channel.java:630) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: Released at Tue Dec 15 22:43:06 CET 2015 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) at hudson.remoting.Channel.unexport(Channel.java:638) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) at hudson.remoting.Channel$1.handle(Channel.java:498) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: Command hudson.remoting.UnexportCommand@4fbc561 created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319) at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@463b was created for interface hudson.Launcher$RemoteProcess at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137) at hudson.remoting.Channel.export(Channel.java:618) at hudson.remoting.Channel.export(Channel.java:587) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1151) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1114) at hudson.remoting.UserRequest.perform(UserRequest.java:121) at hudson.remoting.UserRequest.perform(UserRequest.java:49) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more
For my use-case I'm going to go ahead and remove my use of parallel { }}} in my {{Jenkinsfile so I can be unblocked, but I'm looking forward to this one-day being fixed
I'm blocked by this issue as well. Seeing this in matrix jobs and copying artifacts. Please let us know how you proceed with this issue.
Current state:
- I have a reproducible scenario for this issue
- I've spent some time on it's debugging, but unfortunately I had not enough spare time to finish it and to draft a fix
- The issue is in my short-list, so I'm going to handle it once I finish the Ownership plugin release or when my company assigns me to work on it during the working time
We just starting run into this with using the CloudBees Docker Custom Build Environment Plugin to run our jobs inside of a docker container.
10:50:53 ERROR: Build step failed with exception 10:50:53 java.lang.IllegalStateException: Invalid object ID 125 iota=126 10:50:53 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) 10:50:53 at hudson.remoting.ExportTable.get(ExportTable.java:330) 10:50:53 at hudson.remoting.Channel.getExportedObject(Channel.java:633) 10:50:53 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599) 10:50:53 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) 10:50:53 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:120) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 10:50:53 at hudson.remoting.Request$2.run(Request.java:326) 10:50:53 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 10:50:53 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 10:50:53 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 10:50:53 at java.lang.Thread.run(Thread.java:745) 10:50:53 at ......remote call to docker 1.9.1 ami (i-07e75b8e)(Native Method) 10:50:53 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) 10:50:53 at hudson.remoting.UserResponse.retrieve(UserRequest.java:220) 10:50:53 at hudson.remoting.Channel.call(Channel.java:781) 10:50:53 at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) 10:50:53 at com.sun.proxy.$Proxy104.join(Unknown Source) 10:50:53 at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:992) 10:50:53 at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137) 10:50:53 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97) 10:50:53 at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) 10:50:53 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 10:50:53 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785) 10:50:53 at hudson.model.Build$BuildExecution.build(Build.java:205) 10:50:53 at hudson.model.Build$BuildExecution.doRun(Build.java:162) 10:50:53 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537) 10:50:53 at hudson.model.Run.execute(Run.java:1741) 10:50:53 at hudson.matrix.MatrixRun.run(MatrixRun.java:146) 10:50:53 at hudson.model.ResourceController.execute(ResourceController.java:98) 10:50:53 at hudson.model.Executor.run(Executor.java:408) 10:50:53 Caused by: java.lang.Exception: Object was recently deallocated 10:50:53 #125 (ref.0) : object=null type=hudson.Launcher$RemoteLaunchCallable$1 interfaces=[hudson.Launcher$RemoteProcess] 10:50:53 Created at Mon Jan 18 18:50:52 UTC 2016 10:50:53 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99) 10:50:53 at hudson.remoting.ExportTable.export(ExportTable.java:305) 10:50:53 at hudson.remoting.Channel.internalExport(Channel.java:629) 10:50:53 at hudson.remoting.Channel.export(Channel.java:620) 10:50:53 at hudson.remoting.Channel.export(Channel.java:590) 10:50:53 at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1151) 10:50:53 at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1114) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:120) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 10:50:53 at hudson.remoting.Request$2.run(Request.java:326) 10:50:53 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 10:50:53 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 10:50:53 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 10:50:53 at java.lang.Thread.run(Thread.java:745) 10:50:53 Released at Mon Jan 18 18:50:52 UTC 2016 10:50:53 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) 10:50:53 at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) 10:50:53 at hudson.remoting.Channel.unexport(Channel.java:641) 10:50:53 at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) 10:50:53 at hudson.remoting.Channel$1.handle(Channel.java:501) 10:50:53 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) 10:50:53 Caused by: Command hudson.remoting.UnexportCommand@730d4d70 created at 10:50:53 at hudson.remoting.Command.<init>(Command.java:67) 10:50:53 at hudson.remoting.Command.<init>(Command.java:50) 10:50:53 at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33) 10:50:53 at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360) 10:50:53 at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319) 10:50:53 at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420) 10:50:53 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 10:50:53 at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) 10:50:53 at java.lang.Thread.run(Thread.java:745) 10:50:53 Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@7d was created for interface hudson.Launcher$RemoteProcess 10:50:53 at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125) 10:50:53 at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137) 10:50:53 at hudson.remoting.Channel.export(Channel.java:621) 10:50:53 at hudson.remoting.Channel.export(Channel.java:590) 10:50:53 at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1151) 10:50:53 at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1114) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:120) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 10:50:53 at hudson.remoting.Request$2.run(Request.java:326) 10:50:53 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 10:50:53 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 10:50:53 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 10:50:53 ... 1 more 10:50:53 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379) 10:50:53 at hudson.remoting.ExportTable.get(ExportTable.java:330) 10:50:53 at hudson.remoting.Channel.getExportedObject(Channel.java:633) 10:50:53 at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599) 10:50:53 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) 10:50:53 at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:120) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 10:50:53 at hudson.remoting.Request$2.run(Request.java:326) 10:50:53 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 10:50:53 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 10:50:53 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 10:50:53 at java.lang.Thread.run(Thread.java:745) 10:50:53 Caused by: Released at Mon Jan 18 10:50:52 PST 2016 10:50:53 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) 10:50:53 at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) 10:50:53 at hudson.remoting.Channel.unexport(Channel.java:641) 10:50:53 at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) 10:50:53 at hudson.remoting.Channel$1.handle(Channel.java:501) 10:50:53 at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) 10:50:53 Caused by: Command hudson.remoting.UnexportCommand@2aad54cc created at 10:50:53 at hudson.remoting.Command.<init>(Command.java:67) 10:50:53 at hudson.remoting.Command.<init>(Command.java:50) 10:50:53 at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33) 10:50:53 at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360) 10:50:53 at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319) 10:50:53 at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420) 10:50:53 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 10:50:53 at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) 10:50:53 at java.lang.Thread.run(Thread.java:745) 10:50:53 Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@7d was created for interface hudson.Launcher$RemoteProcess 10:50:53 at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125) 10:50:53 at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137) 10:50:53 at hudson.remoting.Channel.export(Channel.java:621) 10:50:53 at hudson.remoting.Channel.export(Channel.java:590) 10:50:53 at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1151) 10:50:53 at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1114) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:120) 10:50:53 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 10:50:53 at hudson.remoting.Request$2.run(Request.java:326) 10:50:53 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) 10:50:53 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 10:50:53 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 10:50:53 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 10:50:53 ... 1 more
Hi Christoph,
My apologies for the delay, but I still have no time to work on it.
I'll update the status when I find a timeslot. Currently it's not "in progress".
Can you possibly share your steps for the reproducible scenario, so maybe someone else can fix it?
Thanks.
jwhitcraft
My reproduction steps are based on the closed-source plugins from CloudBees Jenkins Platform. But the case from z1lv1n4s also has a good reproduction rate
We are also seeing this issue, but we didn't change Jenkins versions, just after updated to Oracle Java 1.8.
We started to see this issue the first time.
We have a matrix job with 130 variants and the problem happened in post build when it's evaluating whether the current build status is success before triggering other things.
eb 09, 2016 7:12:00 AM org.jenkins_ci.plugins.run_condition.BuildStepRunner logEvaluateException WARNING: Exception caught evaluating condition: [java.lang.IllegalStateException: Invalid object ID 883 iota=884], action = [Fail the build] java.lang.IllegalStateException: Invalid object ID 883 iota=884 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) at hudson.remoting.ExportTable.get(ExportTable.java:330) at hudson.remoting.Channel.getExportedObject(Channel.java:633) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at ......remote call to jenkins-slave4(192.168.100.153)(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.UserResponse.retrieve(UserRequest.java:220) at hudson.remoting.Channel.call(Channel.java:781) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250) at com.sun.proxy.$Proxy50.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:991) at hudson.Launcher$ProcStarter.join(Launcher.java:388) at org.jenkins_ci.plugins.run_condition.contributed.CommandInterperterCondition.runPerform(CommandInterperterCondition.java:89) at org.jenkins_ci.plugins.run_condition.BuildStepRunner$2.evaluate(BuildStepRunner.java:107) at org.jenkins_ci.plugins.run_condition.BuildStepRunner$Fail.conditionalRun(BuildStepRunner.java:147) at org.jenkins_ci.plugins.run_condition.BuildStepRunner.perform(BuildStepRunner.java:105) at org.jenkinsci.plugins.conditionalbuildstep.ConditionalBuilder.perform(ConditionalBuilder.java:133) at org.jenkinsci.plugins.postbuildscript.PostBuildScript.processBuildSteps(PostBuildScript.java:204) at org.jenkinsci.plugins.postbuildscript.PostBuildScript.processScripts(PostBuildScript.java:143) at org.jenkinsci.plugins.postbuildscript.PostBuildScript._perform(PostBuildScript.java:105) at org.jenkinsci.plugins.postbuildscript.PostBuildScript.perform(PostBuildScript.java:82) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:723) at hudson.model.Build$BuildExecution.post2(Build.java:185) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:668) at hudson.model.Run.execute(Run.java:1763) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Caused by: java.lang.Exception: Object was recently deallocated #883 (ref.0) : object=null type=hudson.Launcher$RemoteLaunchCallable$1 interfaces=[hudson.Launcher$RemoteProcess] Created at Tue Feb 09 07:12:00 PST 2016 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99) at hudson.remoting.ExportTable.export(ExportTable.java:305) at hudson.remoting.Channel.internalExport(Channel.java:629) at hudson.remoting.Channel.export(Channel.java:620) at hudson.remoting.Channel.export(Channel.java:590) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1150) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Released at Tue Feb 09 07:12:00 PST 2016 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) at hudson.remoting.Channel$1.handle(Channel.java:501) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: Command hudson.remoting.UnexportCommand@53e0ac9e created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319) at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@373 was created for interface hudson.Launcher$RemoteProcess at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137) at hudson.remoting.Channel.export(Channel.java:621) at hudson.remoting.Channel.export(Channel.java:590) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1150) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379) at hudson.remoting.ExportTable.get(ExportTable.java:330) at hudson.remoting.Channel.getExportedObject(Channel.java:633) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: Released at Tue Feb 09 07:12:00 PST 2016 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) at hudson.remoting.Channel$1.handle(Channel.java:501) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: Command hudson.remoting.UnexportCommand@54ba65e8 created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360) at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319) at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@373 was created for interface hudson.Launcher$RemoteProcess at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137) at hudson.remoting.Channel.export(Channel.java:621) at hudson.remoting.Channel.export(Channel.java:590) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1150) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113) at hudson.remoting.UserRequest.perform(UserRequest.java:120) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more
Just an update:
After we downgraded from java8 to java7, we no longer see the issue.
We were running into this issue after we upgraded to Java 1.8.60 in January with Jenkins 1.632. We downgraded to Java on the master to 1.7.80 and the "iota" errors went away.
Oleg, is there a plan when this issue will be fixed? I have the feeling that many users are hit by this problem. Please give us a short update.
As previously, I have no spare personal time to work on this issue. Sorry about that, but I assure you there are serious reasons for that. When I agreed to take the ownership of this library, I explicitly notified the previous maintainer that I won't be able to dedicate my personal time.
My work plans are being determined by the management. The issue requires several days without distraction, so it's hard to get it scheduled. Actually, I have already made an attempt to fix it, but I was not very successful. The "good" news is that it's high in the backlogs, so it may be taken in the short-term.
I ran in same issue:
jenkins 1.652
linux derived from ubuntu trusty
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) Server VM (build 25.51-b03, mixed mode)
I also see it only like 1 of 1000 builds for matrix projects.
ERROR: Build step failed with exception
java.lang.IllegalStateException: Invalid object ID 31032 iota=31033
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386)
at hudson.remoting.ExportTable.get(ExportTable.java:330)
at hudson.remoting.Channel.getExportedObject(Channel.java:633)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to PEARF_172.24.200.196(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
at com.sun.proxy.$Proxy49.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:991)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:135)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:95)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:64)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782)
at hudson.model.Build$BuildExecution.build(Build.java:205)
at hudson.model.Build$BuildExecution.doRun(Build.java:162)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1738)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: java.lang.Exception: Object was recently deallocated
#31032 (ref.0) : object=null type=hudson.Launcher$RemoteLaunchCallable$1 interfaces=[hudson.Launcher$RemoteProcess]
Created at Mon Apr 25 18:57:07 UTC 2016
at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99)
at hudson.remoting.ExportTable.export(ExportTable.java:305)
at hudson.remoting.Channel.internalExport(Channel.java:629)
at hudson.remoting.Channel.export(Channel.java:620)
at hudson.remoting.Channel.export(Channel.java:590)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1150)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Released at Mon Apr 25 18:57:07 UTC 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414)
at hudson.remoting.Channel.unexport(Channel.java:641)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
at hudson.remoting.Channel$1.handle(Channel.java:501)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Caused by: Command hudson.remoting.UnexportCommand@142e5b6 created at
at hudson.remoting.Command.<init>(Command.java:67)
at hudson.remoting.Command.<init>(Command.java:50)
at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@7938 was created for interface hudson.Launcher$RemoteProcess
at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125)
at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137)
at hudson.remoting.Channel.export(Channel.java:621)
at hudson.remoting.Channel.export(Channel.java:590)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1150)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379)
at hudson.remoting.ExportTable.get(ExportTable.java:330)
at hudson.remoting.Channel.getExportedObject(Channel.java:633)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:599)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: Released at Mon Apr 25 18:57:07 UTC 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414)
at hudson.remoting.Channel.unexport(Channel.java:641)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
at hudson.remoting.Channel$1.handle(Channel.java:501)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Caused by: Command hudson.remoting.UnexportCommand@a3c9f created at
at hudson.remoting.Command.<init>(Command.java:67)
at hudson.remoting.Command.<init>(Command.java:50)
at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:360)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$700(RemoteInvocationHandler.java:319)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:420)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@7938 was created for interface hudson.Launcher$RemoteProcess
at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:125)
at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:137)
at hudson.remoting.Channel.export(Channel.java:621)
at hudson.remoting.Channel.export(Channel.java:590)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1150)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1113)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Fyi, the workaround asking the java downgrade to version 7 works well.
As a summary:
- Master and slaves running with Java 8 => some builds fails with such exception:
10:14:43 ERROR: Build step failed with exception
10:14:43 java.lang.IllegalStateException (http://stacktrace.jenkins-ci.org/search?query=java.lang.IllegalStateException) : Invalid object ID 52 iota=53 - Upgrading them to the latest Java 8 release (here 1.8.0_92) does not fix the issue.
- Downgrading Master and slaves to Java 7 (here 1.7.0_79) solved the problem.
Has anyone time to test with the new version of the swarm plugin (2.1)? https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin
I'm currently digging into the issue.
Some analysis is available here (in progress): https://docs.google.com/document/d/1nu7jD2bNqEzD7kOM1R0VUpExycPNUq8wOTTLrr9xvOY/edit
I suspect an automatic object unexporter glitch.
Major changes in it:
- Jun 10, 2015: New implementation of Unexporter
- https://github.com/jenkinsci/remoting/commit/7e858e3651f1e535d289eebaa92084c31a4b610b
- Released at: remoting-2.52 (jenkins-1.621+, LTS - 1.609.3)
- The issue is reported before and after this change
- Apr 20, 2016 - Ensure that the unexporter cleans up whatever шt can each sweep
- https://github.com/jenkinsci/remoting/commit/f4d287652e09da627f51779bec1c3ca7f1d46e70
- Released at: remoting-2.58 (jenkins-2.4,LTS- 2.7.1+)
2.58 may have occasionally fixed the issue, I'm investigating it now.
Has anybody seen the issue on Jenkins 2.4+ including 2.7.1 or 2.7.2?
> Has anybody seen the issue on Jenkins 2.4+ including 2.7.1 or 2.7.2?
NVM, the bug still exist there
Created https://github.com/jenkinsci/remoting/pull/99 and https://github.com/jenkinsci/jenkins/pull/2503, which seem to solve/suppress issue in my case. Need a code review BTW
Not correct. Maybe I'll apply a hack in Jenkins core, which should at least reduce the impact
It would appear as though we're hitting this too and it would appear that the errors are correlated to a recent Java update on our build farm as well, but we updated from an older build of 1.7 (update 40 iirc) to a newer build of 1.7 (update 50 iirc) and at no point were using a 1.8 version, so maybe some change between update 40 and 50 is somehow contributing to the problem. Not sure if this helps further isolate the issue or not.
Also, in our case we see the errors happen when running builds within docker containers, and the errors occur as the docker image is being downloaded to the agent and / or when the container is being launched (ie: before any of the actual build operations take place).
So, given the fact that this issue has been outstanding for years and appears (based on a review of the comment thread) that it is not being considered priority for anyone to fix any time soon, are there any suggestions to mitigate, workaround or otherwise avoid this issue in the meantime?
leedega, I noted in one environment that issue tends to happen more often if you have more simultaneous builds/configurations. So reducing from few dozens concurrent to about a dozen helped a bit.
It was like 3-5 failed builds of 100 and lowered to 1 per 200-300.
We haven't seen the issue on any of our 10+ jenkins severs since we started running Java 1.7.80.
Would be interesting if you see the problem under that version of java.
Code changed in jenkins
User: Bertrand Roussel
Path:
client/pom.xml
http://jenkins-ci.org/commit/swarm-plugin/8ad209aaafad5ec6256aa547702870da083ff6cd
Log:
Upgrade remoting API to 2.62
Version 2.62 contains a fix for the following bugs:
- https://issues.jenkins-ci.org/browse/JENKINS-22853
- https://issues.jenkins-ci.org/browse/JENKINS-23271 (possibly)
This occurs mostly with Java 8.
The remoting version matches with Jenkins 2.18/2.19/2.19.1/2.19.2.
Created a PR, which seems to be a plausible workaround.
https://github.com/jenkinsci/jenkins/pull/2635
Marking as Resolved for a while, it should become available in 2.32.1 LTS. I have found no evidence of other cases in the Jenkins core, which cause such object de-provisioning (but they may exist). Please create an issue if it happens in 2.31+ on your instances
Oleg, I am still seeing these issues with 2.34:
FATAL: java.util.concurrent.ExecutionException: Invalid object ID 811 iota=812
hudson.remoting.RemotingSystemException: java.util.concurrent.ExecutionException: Invalid object ID 811 iota=812
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
at com.sun.proxy.$Proxy85.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1009)
at hudson.plugins.clearcase.HudsonClearToolLauncher.run(HudsonClearToolLauncher.java:143)
at hudson.plugins.clearcase.ClearToolExec.describe(ClearToolExec.java:126)
at hudson.plugins.clearcase.base.BaseHistoryAction.parseUpdt(BaseHistoryAction.java:137)
at hudson.plugins.clearcase.base.BaseHistoryAction.runLsHistory(BaseHistoryAction.java:156)
at hudson.plugins.clearcase.history.AbstractHistoryAction.runAndFilterLsHistory(AbstractHistoryAction.java:259)
at hudson.plugins.clearcase.history.AbstractHistoryAction.getChanges(AbstractHistoryAction.java:81)
at hudson.plugins.clearcase.AbstractClearCaseScm.saveChangeLog(AbstractClearCaseScm.java:1106)
at hudson.plugins.clearcase.AbstractClearCaseScm.checkout(AbstractClearCaseScm.java:395)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1278)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1728)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:404)
Caused by: java.util.concurrent.ExecutionException: Invalid object ID 811 iota=812
at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:408)
at hudson.remoting.ExportTable.get(ExportTable.java:333)
at hudson.remoting.Channel.getExportedObject(Channel.java:660)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:885)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:869)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:828)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
at ......remote call to Channel to /10.0.50.50(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:822)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
... 18 more
Caused by: java.lang.Exception: Object was recently deallocated
#811 (ref.0) : object=null type=hudson.Launcher$RemoteLaunchCallable$1 interfaces=[hudson.Launcher$RemoteProcess]
Created at Mon Nov 28 20:32:47 CET 2016
at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:102)
at hudson.remoting.ExportTable.export(ExportTable.java:308)
at hudson.remoting.Channel.internalExport(Channel.java:656)
at hudson.remoting.Channel.export(Channel.java:647)
at hudson.remoting.Channel.export(Channel.java:617)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1168)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1131)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
Released at Mon Nov 28 20:32:49 CET 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:134)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:449)
at hudson.remoting.Channel.unexport(Channel.java:684)
at hudson.remoting.Channel.unexport(Channel.java:673)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command hudson.remoting.UnexportCommand@154ee63 created at
at hudson.remoting.Command.<init>(Command.java:70)
at hudson.remoting.Command.<init>(Command.java:53)
at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:366)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$900(RemoteInvocationHandler.java:325)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:583)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@32b was created for interface hudson.Launcher$RemoteProcess
at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:126)
at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:138)
at hudson.remoting.Channel.export(Channel.java:648)
at hudson.remoting.Channel.export(Channel.java:617)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1168)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1131)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:401)
at hudson.remoting.ExportTable.get(ExportTable.java:333)
at hudson.remoting.Channel.getExportedObject(Channel.java:660)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:885)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:869)
at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:828)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
Caused by: Released at Mon Nov 28 20:32:49 CET 2016
at hudson.remoting.ExportTable$Entry.release(ExportTable.java:134)
at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:449)
at hudson.remoting.Channel.unexport(Channel.java:684)
at hudson.remoting.Channel.unexport(Channel.java:673)
at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command hudson.remoting.UnexportCommand@33cebea8 created at
at hudson.remoting.Command.<init>(Command.java:70)
at hudson.remoting.Command.<init>(Command.java:53)
at hudson.remoting.UnexportCommand.<init>(UnexportCommand.java:33)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.cleanup(RemoteInvocationHandler.java:366)
at hudson.remoting.RemoteInvocationHandler$PhantomReferenceImpl.access$900(RemoteInvocationHandler.java:325)
at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:583)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Proxy hudson.remoting.RemoteInvocationHandler@32b was created for interface hudson.Launcher$RemoteProcess
at hudson.remoting.RemoteInvocationHandler.<init>(RemoteInvocationHandler.java:126)
at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:138)
at hudson.remoting.Channel.export(Channel.java:648)
at hudson.remoting.Channel.export(Channel.java:617)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1168)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1131)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:94)
at java.lang.Thread.run(Unknown Source)
I had this with three different builds last night. As these builds run only once per night they accumulate a lot of changes. And while ClearCase is accumulating these changes on the view update the job chokes with this error report. Or is this completely unrelated?
Yes, it is the same issue, but with another code. Likely we need a better solution
Could I help out with something where more logging is enabled? I have about 10 of these SonarQube Jobs running every night and at least one of them fails almost every night.
bruce Give me several hours. I will create a pull request with a wider solution. Then you will be able to take a PoC from the CI just to verify if it helps
We have been seeing this error periodically as well.
Just out of curiosity, is there any indication on which version(s) of Jenkins (ie: v1 / v2) the fix will be applied once one has been approved?
Jenkins 1.x is not supported by the community anymore (though I'm not a fan of the current release model).
The fix will be integrated into 2.35 and likely then backported to 2.32.1
Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/Launcher.java
core/src/main/java/hudson/Proc.java
http://jenkins-ci.org/commit/jenkins/29893354a52efac20efcb4e0e91723f3bd889059
Log:
JENKINS-23271 - Process statuses of Remote process join() operations directly inside methods (#2653)
JENKINS-23271- Process statuses of Remote process join() operations directly inside methods
JENKINS-23271- Also prevent the issue when the kill() command is the last call in the usage sequence
Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/Launcher.java
http://jenkins-ci.org/commit/jenkins/fcfd271ad5ad5947075052d37177249c760f6184
Log:
JENKINS-23271 - Prevent early deallocation of the Proc instance by GC in ProcStarter#join() (#2635)
JENKINS-23271- Prevent the prelimimary deallocation of the Proc instance by GC
It is a hackish way, which likely prevents a preliminary deallocation of the spawned RemoteProc instance, which we see in JENKINS-23271. Proc instance was not actually required in the original code since we were creating and using RemoteInvocationHandler wrapper only, and the theory discussed with @stephenc was that object gets removed by Java8 garbage collector before we get into join().
This fix enforces the persistency of ProcStarter#start() result by adding logging and the enforced volatile field (maybe the last one is not really required, but JIT compiler in Java implementations may be smart enough to skip unused loggers)
This is a pretty old fix from August, which has been soak tested on my instance for several weeks (mid-August => Jenkins World). On the reference instance (just a small Jenkins instance with 4 agents and very frequent builds with CommandInterpreter steps) I saw 2 failures over the period. On the fixed instance - 0. It does not proof anything, but at least the fix was soak tested a bit
JENKINS-23271- Get rid of the procHolderForJoin field
JENKINS-23271- Also put the check into the finally statement as @stephenc proposed
- Remove assert
(cherry picked from commit fd6c6aff929be9818f4eb4b84ed6b4593356853f)
Code changed in jenkins
User: Oleg Nenashev
Path:
core/src/main/java/hudson/Launcher.java
core/src/main/java/hudson/Proc.java
http://jenkins-ci.org/commit/jenkins/5500e5c00bc19e6c421c9aaa88f6dbb57c666db3
Log:
JENKINS-23271 - Process statuses of Remote process join() operations directly inside methods (#2653)
JENKINS-23271- Process statuses of Remote process join() operations directly inside methods
JENKINS-23271- Also prevent the issue when the kill() command is the last call in the usage sequence
(cherry picked from commit 29893354a52efac20efcb4e0e91723f3bd889059)
Compare: https://github.com/jenkinsci/jenkins/compare/dacbe03e0d4e...5500e5c00bc1
Code changed in jenkins
User: Oleg Nenashev
Path:
docs/configuration.md
src/main/java/hudson/remoting/Base64.java
src/main/java/hudson/remoting/Channel.java
src/main/java/hudson/remoting/ErrorPropagatingOutputStream.java
src/main/java/hudson/remoting/ExportTable.java
src/main/java/hudson/remoting/LocalChannel.java
src/main/java/hudson/remoting/RemoteInvocationHandler.java
src/main/java/hudson/remoting/VirtualChannel.java
http://jenkins-ci.org/commit/remoting/27bb51dd14db44952348539d15ee004c90c248b2
Log:
Document and annotate object Export/Unexport logic
During last releases we have fixed several issues related to the object export/unexport operations (JENKINS-23271, JENKINS-41852). I decided that it makes sense to improve the method documentation just to make Javadocs more explicit.
The change also adds missing documentation of hudson.remoting.ExportTable.unexportLogSize
rtyler managed to reproduce the issue using this WF script: https://github.com/jenkins-infra/jenkins-infra/blob/staging/Jenkinsfile . He used Docker cloud with nodes with a single executor.