Details
-
Bug
-
Status: Resolved (View Workflow)
-
Blocker
-
Resolution: Fixed
-
jenkins-2.6+
Description
Credits to Vladimir Konnov for discovering it. Impact is unclear.
RingBufferLogHandler is beig used on the slaves side (LOG_HOLDER). SlaveInitializer writes to logs and invokes logic leading to SystemProperties usage. ServletContext class cannot be serialized at that point.
Cause:
private static final int DEFAULT_RING_BUFFER_SIZE = SystemProperties.getInteger(RingBufferLogHandler.class.getName() + ".defaultSize", Integer.valueOf(256)).intValue();
Log:
<===[JENKINS REMOTING CAPACITY]===>channel started Slave.jar version: 2.59 This is a Unix agent ERROR: Unable to launch the agent for jenkins-pipline-13 java.io.IOException: Remote call on jenkins-pipline-13 failed at hudson.remoting.Channel.call(Channel.java:789) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:536) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:381) at hudson.slaves.CommandLauncher.launch(CommandLauncher.java:131) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:253) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.LinkageError: Failed to load jenkins.util.SystemProperties at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:342) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:251) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.util.RingBufferLogHandler.<clinit>(RingBufferLogHandler.java:39) at hudson.slaves.SlaveComputer$LogHolder.<clinit>(SlaveComputer.java:799) at hudson.slaves.SlaveComputer$SlaveInitializer.call(SlaveComputer.java:808) at hudson.slaves.SlaveComputer$SlaveInitializer.call(SlaveComputer.java:802) at hudson.remoting.UserRequest.perform(UserRequest.java:152) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at ......remote call to jenkins-pipline-13(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.UserResponse.retrieve(UserRequest.java:252) at hudson.remoting.Channel.call(Channel.java:781) ... 9 more Caused by: java.lang.NoClassDefFoundError: javax/servlet/ServletContextListener at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:338) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:251) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.util.RingBufferLogHandler.<clinit>(RingBufferLogHandler.java:39) at hudson.slaves.SlaveComputer$LogHolder.<clinit>(SlaveComputer.java:799) at hudson.slaves.SlaveComputer$SlaveInitializer.call(SlaveComputer.java:808) at hudson.slaves.SlaveComputer$SlaveInitializer.call(SlaveComputer.java:802) at hudson.remoting.UserRequest.perform(UserRequest.java:152) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) ... 4 more Caused by: java.lang.ClassNotFoundException: javax.servlet.ServletContextListener at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch(RemoteClassLoader.java:724) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:884) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at ......remote call to channel(Native Method) at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) at hudson.remoting.Request.call(Request.java:172) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:252) at com.sun.proxy.$Proxy5.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:259) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 19 more ERROR: Connection terminated java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2353) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2822) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) ERROR: Process terminated with exit code 143
Attachments
Issue Links
- relates to
-
JENKINS-26677 Blacklist certain classes from loading in slaves
-
- Open
-
-
-
- Resolved
-
Activity
Yoann Dubreuil
added a comment - Yes, it is related. It seems that a plugin is trying to use SystemProperties from a slave. Unfortunately, the stacktrace doesn't contain enough information to pinpoint which plugin causes this.
To get you some context, the fix which went to 2.19.2 changes the way the slave initializer get the ringBufferSize property in order to not use SystemProperties from a slave, because it's not legal (which is why the patched also added OnMaster marker to SystemProperties class)
Yoann Dubreuil
added a comment - Yes, it is related. It seems that a plugin is trying to use SystemProperties from a slave. Unfortunately, the stacktrace doesn't contain enough information to pinpoint which plugin causes this.
To get you some context, the fix which went to 2.19.2 changes the way the slave initializer get the ringBufferSize property in order to not use SystemProperties from a slave, because it's not legal (which is why the patched also added OnMaster marker to SystemProperties class) 
Robert Klaus
added a comment - Thanks for the info. We also suspected a plugin but are a bit lost figuring out which one causes this. Is there a log where this might be found?
Robert Klaus
added a comment - Thanks for the info. We also suspected a plugin but are a bit lost figuring out which one causes this. Is there a log where this might be found? Yoann Dubreuil
added a comment - Not that I know of. To find which plugin it is, you will need to find the class which triggers the load on the slave. Turning classloading logs on a slave might help or remoting layer logs.
Yoann Dubreuil
added a comment - Not that I know of. To find which plugin it is, you will need to find the class which triggers the load on the slave. Turning classloading logs on a slave might help or remoting layer logs. This is the remoting channel Jenkins uses to attach to a Maven Project build.
Don't use the Maven project type. It's evil.
We have the same failure but on our master (so not on a slave). We did an update to 2.19.2 but that didn't help. The problem hits us daily on different buildjobs without a clear pattern. A Jenkins restart helps.
I'm not sure if this actually is an issue on its own, but given we have nearly the same stacktrace I guess it is related to this.