-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Jenkins 2.89.3
Mesos-Plugin 0.15.0
SSH Slaves 1.25
-
-
ssh-slaves-1.30.0
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds.
We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: https://github.com/mesosphere/dcos-jenkins-dind-agent
The master would then get pegged @ 100% CPU usage and all builds would start to fail
===================== Errors
WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more
Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more
Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed
- is related to
-
-
- Closed
-
[JENKINS-49118] SSH Slaves 1.25 Breaks
| Description |
Original:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: https://github.com/mesosphere/dcos-jenkins-dind-agent The master would then get pegged @ 100% CPU usage {{WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more}} |
New:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage {{ WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space}} {{ java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:192)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{ Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273)}} {{ at com.sun.proxy.$Proxy6.fetch(Unknown Source)}} {{ at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}} {{ at java.lang.Class.getDeclaredMethods0(Native Method)}} {{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}} {{ at java.lang.Class.getDeclaredMethod(Class.java:2128)}} {{ at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)}} {{ at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)}} {{ at java.security.AccessController.doPrivileged(Native Method)}} {{ at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)}} {{ at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ Caused by: java.lang.InterruptedException}} {{ at java.lang.Object.wait(Native Method)}} {{ at hudson.remoting.Request.call(Request.java:169)}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)}} {{ ... 38 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{ WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space}} {{ java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} |
| Description |
Original:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage {{ WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space}} {{ java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:192)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{ Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273)}} {{ at com.sun.proxy.$Proxy6.fetch(Unknown Source)}} {{ at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}} {{ at java.lang.Class.getDeclaredMethods0(Native Method)}} {{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}} {{ at java.lang.Class.getDeclaredMethod(Class.java:2128)}} {{ at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)}} {{ at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)}} {{ at java.security.AccessController.doPrivileged(Native Method)}} {{ at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)}} {{ at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ Caused by: java.lang.InterruptedException}} {{ at java.lang.Object.wait(Native Method)}} {{ at hudson.remoting.Request.call(Request.java:169)}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)}} {{ ... 38 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{ WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space}} {{ java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} |
New:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage {{ WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space}} {{ java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:192)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{ Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273)}} {{ at com.sun.proxy.$Proxy6.fetch(Unknown Source)}} {{ at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}} {{ at java.lang.Class.getDeclaredMethods0(Native Method)}} {{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}} {{ at java.lang.Class.getDeclaredMethod(Class.java:2128)}} {{ at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)}} {{ at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)}} {{ at java.security.AccessController.doPrivileged(Native Method)}} {{ at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)}} {{ at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ Caused by: java.lang.InterruptedException}} {{ at java.lang.Object.wait(Native Method)}} {{ at hudson.remoting.Request.call(Request.java:169)}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)}} {{ ... 38 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{ WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space}} {{ java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{ }}}} |
| Description |
Original:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage {{ WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space}} {{ java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:192)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{ Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273)}} {{ at com.sun.proxy.$Proxy6.fetch(Unknown Source)}} {{ at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}} {{ at java.lang.Class.getDeclaredMethods0(Native Method)}} {{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}} {{ at java.lang.Class.getDeclaredMethod(Class.java:2128)}} {{ at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)}} {{ at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)}} {{ at java.security.AccessController.doPrivileged(Native Method)}} {{ at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)}} {{ at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ Caused by: java.lang.InterruptedException}} {{ at java.lang.Object.wait(Native Method)}} {{ at hudson.remoting.Request.call(Request.java:169)}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)}} {{ ... 38 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{ WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space}} {{ java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{ Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{ }}}} |
New:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage {{WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space}} {{java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{Caused by: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:192)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273)}} {{ at com.sun.proxy.$Proxy6.fetch(Unknown Source)}} {{ at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}} {{ at java.lang.Class.getDeclaredMethods0(Native Method)}} {{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}} {{ at java.lang.Class.getDeclaredMethod(Class.java:2128)}} {{ at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)}} {{ at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)}} {{ at java.security.AccessController.doPrivileged(Native Method)}} {{ at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)}} {{ at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{Caused by: java.lang.InterruptedException}} {{ at java.lang.Object.wait(Native Method)}} {{ at hudson.remoting.Request.call(Request.java:169)}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)}} {{ ... 38 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space}} {{java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space}} {{java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}}{{Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath}} {{WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed}} |
| Description |
Original:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage {{WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space}} {{java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{Caused by: java.lang.Error: Failed to deserialize the Callable object.}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:192)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}} {{Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273)}} {{ at com.sun.proxy.$Proxy6.fetch(Unknown Source)}} {{ at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:424)}} {{ at java.lang.ClassLoader.loadClass(ClassLoader.java:357)}} {{ at java.lang.Class.getDeclaredMethods0(Native Method)}} {{ at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)}} {{ at java.lang.Class.getDeclaredMethod(Class.java:2128)}} {{ at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475)}} {{ at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498)}} {{ at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)}} {{ at java.security.AccessController.doPrivileged(Native Method)}} {{ at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)}} {{ at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{Caused by: java.lang.InterruptedException}} {{ at java.lang.Object.wait(Native Method)}} {{ at hudson.remoting.Request.call(Request.java:169)}} {{ at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260)}} {{ ... 38 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space}} {{java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}}{{Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor}} {{WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space}} {{java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:943)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:938)}} {{ at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)}} {{ at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)}} {{ at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)}} {{Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594}} {{ at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)}} {{ at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)}} {{ at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)}} {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)}} {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)}} {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)}} {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)}} {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:275)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:186)}} {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}} {{ at hudson.remoting.Request$2.run(Request.java:360)}} {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)}} {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at hudson.remoting.Engine$1$1.run(Engine.java:98)}} {{ at java.lang.Thread.run(Thread.java:748)}} {{ at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method)}} {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654)}} {{ at hudson.remoting.UserResponse.retrieve(UserRequest.java:311)}} {{ at hudson.remoting.Channel$2.adapt(Channel.java:941)}} {{ ... 4 more}}{{Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath}} {{WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed}} |
New:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage ===================== Errors WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed |
| Description |
Original:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage ===================== Errors WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed |
New:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage and all builds would start to fail ===================== Errors WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed |
| Attachment | New: plugins.xml [ 41222 ] |
| Assignee | Original: Oleg Nenashev [ oleg_nenashev ] |
| Assignee | New: Ivan Fernandez Calvo [ ifernandezcalvo ] |
| Link |
New:
This issue is related to |
| Description |
Original:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage and all builds would start to fail ===================== Errors WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed |
New:
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds. We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: [https://github.com/mesosphere/dcos-jenkins-dind-agent] The master would then get pegged @ 100% CPU usage and all builds would start to fail ===================== Errors {code} WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more {code} {code} Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more {code} {code} Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed {code} |