-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.204.2 (remoting 3.36.1)
Linux OpenJDK 1.8.0_242
I know this sound like a duplicate of some previously reported bugs, but I'm opening this new issue to have a single one I can refer to in a PR I'm about to open.
I'm trying to improve resilience of the agent's RemoteClassLoader when it receives a RemotingSystemException caused by an InterruptedException.
The overall issue is that the class loader can't really recover after an exception has been raised while some classes were being loaded / initialized; later attempts at using a class that failed being loaded/initialized will raise a LinkageError.
This is not something new, see JENKINS-36991 / PR #94. This one got fixed by adding some try/catch/retry logic to the affected part of the RemoteClassLoader#findClass code. This way, the interruption can be deferred until after the class has been successfully loaded.
But there are other parts of the RemoteClassLoader code which are not protected yet, despite being on the critical path of classes loading/initialization.
See for instance JENKINS-51854: the exception occurs in a call to RemoteClassLoader#getResource made during initialization of some Subversion plugin classes, and from there the plugin can't be used on this agent anymore.
I've reproduced something similar with the Maven plugin, simply by interrupting (from the GUI) a Maven job while it was starting on a freshly started agent.
- error (slave log) when this first Maven job execution gets interrupted:
Feb 10, 2020 5:51:39 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@38569db8 java.lang.ExceptionInInitializerError at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299) at com.sun.proxy.$Proxy5.getResource2(Unknown Source) at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426) at java.lang.ClassLoader.getResource(ClassLoader.java:1090) at java.lang.Class.getResource(Class.java:2267) at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53) at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45) at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107) at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:128) at org.jvnet.localizer.Localizable.toString(Localizable.java:67) at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64) at hudson.model.Run.<clinit>(Run.java:2535) ... 11 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:177) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) ... 22 more
see RemoteClassLoader.java:426
- error (slave log) when trying to run any other Maven job on the same agent after that:
Feb 10, 2020 5:53:38 PM hudson.remoting.UserRequest perform WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@5e66716c java.lang.NoClassDefFoundError: Could not initialize class hudson.maven.MavenModuleSetBuild at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
And with the same manual procedure, I also sometimes get the fatal exception in an other part of RemoteClassLoader#findClass which is not protected covered by the retry logic either (at least, not for RemotingSystemException):
- error (slave log) when this first Maven job gets interrupted:
2020-02-11T12:16:49.933+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@601f4e25 java.lang.ExceptionInInitializerError at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182) at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168) at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166) at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159) at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110) at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137) at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299) at com.sun.proxy.$Proxy5.fetch(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:89) at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:79) at org.eclipse.sisu.plexus.ClassRealmManager.<clinit>(ClassRealmManager.java:66) ... 18 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:177) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) ... 25 more
see RemoteClassLoader.java:315
- error (slave log) when trying to run any other Maven job on the same agent after that:
2020-02-11T12:17:41.347+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@69183d46 java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.sisu.plexus.ClassRealmManager at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182) at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168) at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166) at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159) at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110) at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137) at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Both these cases could be fixed by adding more "try / catch / retry with deferred interruption" logic on the relevant parts of the RemoteClassLoader code,
but instead I propose to do it at a slightly lower level, on methods of the IClassLoader proxy (that's the com.sun.proxy.$Proxy5.getResource2(Unknown Source)
and com.sun.proxy.$Proxy5.fetch(Unknown Source) in the above stack traces).
I will submit a PR implementing that (with the retry code in a dynamic proxy). With this change, when I try to interrupt the first execution of a Maven job, there aren't any LinkageError reported anymore. The build is still eventually interrupted of course, but then I can re-execute the job without triggering any NoClassDefFoundError, as one would expect.
- causes
-
JENKINS-56137 EC2-slave jar cache vs class initialization race.
- Open
- is related to
-
JENKINS-65621 Post-processing can be interrupted by the plugin when aborting running builds
- Open
- relates to
-
JENKINS-51854 LinkageError while performing UserRequest:hudson.scm.SubversionSCM
- Open
-
JENKINS-36991 Unable to load class once the loading was interrupted
- Resolved