Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61103

LinkageError on agent following RemotingSystemException(InterruptedException)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • maven-plugin, remoting
    • None
    • Jenkins 2.204.2 (remoting 3.36.1)
      Linux OpenJDK 1.8.0_242

      I know this sound like a duplicate of some previously reported bugs, but I'm opening this new issue to have a single one I can refer to in a PR I'm about to open.

      I'm trying to improve resilience of the agent's RemoteClassLoader when it receives a RemotingSystemException caused by an InterruptedException.

      The overall issue is that the class loader can't really recover after an exception has been raised while some classes were being loaded / initialized; later attempts at using a class that failed being loaded/initialized will raise a LinkageError.
      This is not something new, see JENKINS-36991 / PR #94. This one got fixed by adding some try/catch/retry logic to the affected part of the RemoteClassLoader#findClass code. This way, the interruption can be deferred until after the class has been successfully loaded.

      But there are other parts of the RemoteClassLoader code which are not protected yet, despite being on the critical path of classes loading/initialization.

      See for instance JENKINS-51854: the exception occurs in a call to RemoteClassLoader#getResource made during initialization of some Subversion plugin classes, and from there the plugin can't be used on this agent anymore.

      I've reproduced something similar with the Maven plugin, simply by interrupting (from the GUI) a Maven job while it was starting on a freshly started agent.

      • error (slave log) when this first Maven job execution gets interrupted:
        Feb 10, 2020 5:51:39 PM hudson.remoting.UserRequest perform
        WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@38569db8
        java.lang.ExceptionInInitializerError
        	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
        	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
        	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
        	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
        	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        	at hudson.remoting.Request$2.run(Request.java:369)
        	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        	at java.lang.Thread.run(Thread.java:748)
        Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
        	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
        	at com.sun.proxy.$Proxy5.getResource2(Unknown Source)
        	at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:426)
        	at java.lang.ClassLoader.getResource(ClassLoader.java:1090)
        	at java.lang.Class.getResource(Class.java:2267)
        	at org.jvnet.localizer.ResourceProvider$1.getResource(ResourceProvider.java:53)
        	at org.jvnet.localizer.ResourceProvider.findResource(ResourceProvider.java:45)
        	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:107)
        	at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:128)
        	at org.jvnet.localizer.Localizable.toString(Localizable.java:67)
        	at hudson.security.PermissionGroup.<init>(PermissionGroup.java:64)
        	at hudson.model.Run.<clinit>(Run.java:2535)
        	... 11 more
        Caused by: java.lang.InterruptedException
        	at java.lang.Object.wait(Native Method)
        	at hudson.remoting.Request.call(Request.java:177)
        	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
        	... 22 more
        

      see RemoteClassLoader.java:426

      • error (slave log) when trying to run any other Maven job on the same agent after that:
      Feb 10, 2020 5:53:38 PM hudson.remoting.UserRequest perform
      WARNING: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@5e66716c
      java.lang.NoClassDefFoundError: Could not initialize class hudson.maven.MavenModuleSetBuild
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1276)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      And with the same manual procedure, I also sometimes get the fatal exception in an other part of RemoteClassLoader#findClass which is not protected covered by the retry logic either (at least, not for RemotingSystemException):

      • error (slave log) when this first Maven job gets interrupted:
      2020-02-11T12:16:49.933+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@601f4e25
      java.lang.ExceptionInInitializerError
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
      	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: hudson.remoting.RemotingSystemException: java.lang.InterruptedException
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
      	at com.sun.proxy.$Proxy5.fetch(Unknown Source)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:315)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
      	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:89)
      	at org.eclipse.sisu.inject.Weak.concurrentKeys(Weak.java:79)
      	at org.eclipse.sisu.plexus.ClassRealmManager.<clinit>(ClassRealmManager.java:66)
      	... 18 more
      Caused by: java.lang.InterruptedException
      	at java.lang.Object.wait(Native Method)
      	at hudson.remoting.Request.call(Request.java:177)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
      	... 25 more
      

      see RemoteClassLoader.java:315

      • error (slave log) when trying to run any other Maven job on the same agent after that:
      2020-02-11T12:17:41.347+0100 WARNING hudson.remoting.UserRequest perform: LinkageError while performing UserRequest:hudson.maven.MavenModuleSetBuild$PomParser@69183d46
      java.lang.NoClassDefFoundError: Could not initialize class org.eclipse.sisu.plexus.ClassRealmManager
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:182)
      	at org.codehaus.plexus.DefaultPlexusContainer.<init>(DefaultPlexusContainer.java:168)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:166)
      	at hudson.maven.MavenEmbedderUtils.buildPlexusContainer(MavenEmbedderUtils.java:159)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:110)
      	at hudson.maven.MavenEmbedder.<init>(MavenEmbedder.java:137)
      	at hudson.maven.MavenUtil.createEmbedder(MavenUtil.java:211)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1323)
      	at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1126)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3071)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

       

      Both these cases could be fixed by adding more "try / catch / retry with deferred interruption" logic on the relevant parts of the RemoteClassLoader code,
      but instead I propose to do it at a slightly lower level, on methods of the IClassLoader proxy (that's the com.sun.proxy.$Proxy5.getResource2(Unknown Source)
      and com.sun.proxy.$Proxy5.fetch(Unknown Source) in the above stack traces).

      I will submit a PR implementing that (with the retry code in a dynamic proxy). With this change, when I try to interrupt the first execution of a Maven job, there aren't any LinkageError reported anymore. The build is still eventually interrupted of course, but then I can re-execute the job without triggering any NoClassDefFoundError, as one would expect.

            Unassigned Unassigned
            tom_gl Thomas de Grenier de Latour
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: