-
Bug
-
Resolution: Unresolved
-
Major
-
None
The remote Class Loading of a Jenkins agent may slow down step initialization and cause some delays depending on the environment (network latency).
Generally, this impact a particular step / feature the first time it is used. For example, the first execution of a checkout step using GitSCM on a newly started agent would need to load Git plugin and Git client plugin related classes. Then next execution of that step would not load those classes anymore, and that step executioon would generally be a few seconds faster (in some environment, I have seen a delay of actually almost 30s).
*BUT, if the agent process is restarted, then again the next execution will need to go through the class loading of that step. And that is regardless of whether the **jarCache* is pre-populated. So this is really about the performance of remoting Class loading.
This delay seems negligible, especially when using permanent agent. But for ephemeral agents (such as when using the Kubernetes plugin) the impact of the performance of the class loading is not negligible anymore as it happens almost all the time.
Some steps - like the git / checkout steps that probably have more class to load - causes a bigger delay than other.
Data
Thread dump on the agent side would show that we spend this time in:
"pool-1-thread-7 for test-agent id=7994 / waiting for test-agent id=382" #39 daemon prio=5 os_prio=31 cpu=692.03ms elapsed=95.84s tid=0x00007f91aaa8e800 nid=0x14003 in Object.wait() [0x000070001269f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.16.1/Native Method) - waiting on <no object reference available> at hudson.remoting.Request.call(Request.java:177) - waiting to re-lock in wait() <0x000000061bac6e08> (a hudson.remoting.RemoteInvocationHandler$RPCRequest) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288) at com.sun.proxy.$Proxy7.fetch3(Unknown Source) at hudson.remoting.RemoteClassLoader.prefetchClassReference(RemoteClassLoader.java:354) at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:259) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:229) at java.lang.ClassLoader.loadClass(java.base@11.0.16.1/ClassLoader.java:589) - locked <0x000000061da00000> (a hudson.remoting.RemoteClassLoader) at java.lang.ClassLoader.loadClass(java.base@11.0.16.1/ClassLoader.java:522) at com.thoughtworks.xstream.XStream.setupConverters(XStream.java:936) at hudson.util.XStream2.setupConverters(XStream2.java:286) at com.thoughtworks.xstream.XStream.<init>(XStream.java:548) at com.thoughtworks.xstream.XStream.<init>(XStream.java:476) at com.thoughtworks.xstream.XStream.<init>(XStream.java:450) at com.thoughtworks.xstream.XStream.<init>(XStream.java:403) at com.thoughtworks.xstream.XStream.<init>(XStream.java:377) at hudson.util.XStream2.<init>(XStream2.java:171) at hudson.ProxyConfiguration.<clinit>(ProxyConfiguration.java:506) at java.lang.Class.forName0(java.base@11.0.16.1/Native Method) at java.lang.Class.forName(java.base@11.0.16.1/Class.java:315) at com.sun.proxy.$Proxy10.<clinit>(Unknown Source) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@11.0.16.1/Native Method) at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@11.0.16.1/NativeConstructorAccessorImpl.java:62) at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@11.0.16.1/DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(java.base@11.0.16.1/Constructor.java:490) at java.lang.reflect.Proxy.newProxyInstance(java.base@11.0.16.1/Proxy.java:1022) at java.lang.reflect.Proxy.newProxyInstance(java.base@11.0.16.1/Proxy.java:1008) at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:168) at hudson.remoting.Channel.export(Channel.java:812) at hudson.remoting.Channel.export(Channel.java:775) at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:232) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.16.1/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.16.1/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.16.1/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.16.1/Method.java:566) at java.io.ObjectStreamClass.invokeWriteReplace(java.base@11.0.16.1/ObjectStreamClass.java:1106) at java.io.ObjectOutputStream.writeObject0(java.base@11.0.16.1/ObjectOutputStream.java:1127) at java.io.ObjectOutputStream.writeObject(java.base@11.0.16.1/ObjectOutputStream.java:345) at hudson.remoting.UserRequest._serialize(UserRequest.java:263) at hudson.remoting.UserRequest.serialize(UserRequest.java:272) at hudson.remoting.UserRequest.perform(UserRequest.java:222) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:377) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at hudson.remoting.InterceptingExecutorService$$Lambda$95/0x000000080025fc40.call(Unknown Source) at java.util.concurrent.FutureTask.run(java.base@11.0.16.1/FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16.1/ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16.1/ThreadPoolExecutor.java:628) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125) at hudson.remoting.Engine$1$$Lambda$96/0x000000080025f040.run(Unknown Source) at java.lang.Thread.run(java.base@11.0.16.1/Thread.java:829)
I collected FINE logs of the hudson.remoting.RemoteInvocationHandler and hudson.remoting.RemoteClassLoader when the agent perform the first checkout after being started. And also after being restarted.
Note: FINE logs for those classes don't show anything between checkout steps when the agent is not restarted. So no point collecting something in that case.
How to Reproduce
- Spin up Jenkins
- Connect a permanent Agent
- Create a simple pipeline like the following
pipeline {
agent {
label "test-agent"
}
options {
skipDefaultCheckout()
timestamps()
}
stages {
stage('Test') {
steps {
checkout(
changelog: false,
poll: false,
scm: [$class: 'GitSCM',
branches: [[name: '*/master']],
extensions: [cloneOption(honorRefspec: true, noTags: true, reference: '', shallow: false)],
userRemoteConfigs: [[
refspec: '+refs/heads/master:refs/remotes/origin/master',
credentialsId: 'test-githubapp',
url: 'https://github.com/jenkinsci/support-core-plugin'
]]
]
)
}
}
}
}
- Build the job
- Notice the delay when initializing the client:
17:15:48 The recommended git tool is: NONE 17:15:52 using credential test-githubapp
- Build again
- Notice that there is no delay anymore:
17:17:46 The recommended git tool is: NONE 17:17:46 using credential test-githubapp
- Restart the agent (kill the process, start it again
- Notice that the delay is back:
17:19:15 The recommended git tool is: NONE 17:19:19 using credential test-githubapp
