-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Jenkins 2.89.3
Mesos-Plugin 0.15.0
SSH Slaves 1.25
-
-
ssh-slaves-1.30.0
After upgrading to SSH Slaves plugin 1.25 from 1.23 we started seeing these errors in the logs. Slaves started to report clock differences of 10 seconds and ping times upwards of 17 seconds.
We spin up our slaves using the mesos-plugin and mesosphere's docker-in-docker jenkins containers: https://github.com/mesosphere/dcos-jenkins-dind-agent
The master would then get pegged @ 100% CPU usage and all builds would start to fail
===================== Errors
WARNING: Failed to monitor mesos-jenkins-ad54e4db86ab493cbfcbf40fc586031d-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.lang.Error: Failed to deserialize the Callable object. at hudson.remoting.UserRequest.perform(UserRequest.java:192) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-141.us-west-2.compute.internal/10.89.142.141:48042(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:273) at com.sun.proxy.$Proxy6.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethod(Class.java:2128) at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1475) at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:498) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:598) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:169) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:260) ... 38 more
Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Disk Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more
Jan 22, 2018 7:41:37 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitor WARNING: Failed to monitor mesos-jenkins-9fe9204c80af4f5c98e67cf012e24a39-mesos for Free Temp Space java.util.concurrent.ExecutionException: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at hudson.remoting.Channel$2.adapt(Channel.java:943) at hudson.remoting.Channel$2.adapt(Channel.java:938) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96) at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305) Caused by: java.io.InvalidClassException: hudson.FilePath; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = -7135276226716035594 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.UserRequest.deserialize(UserRequest.java:275) at hudson.remoting.UserRequest.perform(UserRequest.java:186) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:360) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Thread.java:748) at ......remote call to JNLP4-connect connection from ip-10-89-142-71.us-west-2.compute.internal/10.89.142.71:35556(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1654) at hudson.remoting.UserResponse.retrieve(UserRequest.java:311) at hudson.remoting.Channel$2.adapt(Channel.java:941) ... 4 more Jan 22, 2018 7:41:38 PM org.jenkinsci.plugins.mesos.MesosSlave getRootPath WARNING: IO exception while absolutizing slave root path: java.io.IOException: remote file operation failed: jenkins at hudson.remoting.Channel@1e42b1f:JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/xxxxxx:48042: java.io.IOException: Remote call on JNLP4-connect connection from ip-xxxxxx.us-west-2.compute.internal/1xxxx:48042 failed
- is related to
-
JENKINS-49309 All SSH slaves unexpectedly disconnect when one job finishes
-
- Closed
-
Please provide your agent configuration. Do you ise custom remoting version on the remote side?