• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • warnings-ng-plugin
    • None

      Our Jenkins controller and agents are in 2 different clouds, with a latency of ~ 80ms between them, and the Warnings Next Generation plugin's Javac issue scanner takes about 7 minutes to parse our Javac warnings:

      00:40:07  [Java Compiler] -> found 366 issues (skipped 45 duplicates)
      00:47:09  [Java Compiler] Post processing issues on 'ci-6.localdomain' with source code encoding 'UTF-8'
      00:47:09  [Java Compiler] Skipping SCM blames as requested
      

      The timestamps are misleading, because the "Post processing issues" line is only logged after the processing is done.

      Most of the time is spent here:

      "pool-1-thread-23408 for JNLP4-connect connection to ci.org/12.23.34.45:8888 id=2470453 / waiting for JNLP4-connect connection to ci.org/12.23.34.45:8888 id=16923" #27074 daemon prio=5 os_prio=0 tid=0x00007ff4bc119800 nid=0x18d46f in Object.wait() [0x00007ff4d9bc9000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at hudson.remoting.Request.call(Request.java:177)
      	- locked <0x00000007461cb1b8> (a hudson.remoting.RemoteInvocationHandler$RPCRequest)
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:289)
      	at com.sun.proxy.$Proxy6.fetch3(Unknown Source)
      	at hudson.remoting.RemoteClassLoader.prefetchClassReference(RemoteClassLoader.java:348)
      	at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:253)
      	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:223)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
      	- locked <0x000000064b71b330> (a hudson.remoting.RemoteClassLoader)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
      	at org.apache.xerces.impl.dv.ObjectFactory.findProviderClass(Unknown Source)
      	at org.apache.xerces.impl.dv.ObjectFactory.newInstance(Unknown Source)
      	at org.apache.xerces.impl.dv.DTDDVFactory.getInstance(Unknown Source)
      	at org.apache.xerces.impl.dv.DTDDVFactory.getInstance(Unknown Source)
      	at org.apache.xerces.parsers.XML11Configuration.<init>(Unknown Source)
      	at org.apache.xerces.parsers.XIncludeAwareParserConfiguration.<init>(Unknown Source)
      	at org.apache.xerces.parsers.XIncludeAwareParserConfiguration.<init>(Unknown Source)
      	at sun.reflect.GeneratedConstructorAccessor95.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at java.lang.Class.newInstance(Class.java:442)
      	at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
      	at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
      	at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
      	at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
      	at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParserImpl(Unknown Source)
      	at org.apache.xerces.jaxp.SAXParserFactoryImpl.setFeature(Unknown Source)
      	at edu.hm.hafner.analysis.SecureXmlParserFactory.configureSaxParserFactory(SecureXmlParserFactory.java:121)
      	at edu.hm.hafner.analysis.SecureDigester.<init>(SecureDigester.java:27)
      	at edu.hm.hafner.analysis.ModuleDetector.parsePomAttribute(ModuleDetector.java:294)
      	at edu.hm.hafner.analysis.ModuleDetector.parsePom(ModuleDetector.java:288)
      	at edu.hm.hafner.analysis.ModuleDetector.collectMavenProjects(ModuleDetector.java:127)
      	at edu.hm.hafner.analysis.ModuleDetector.createFilesToModuleMapping(ModuleDetector.java:109)
      	at edu.hm.hafner.analysis.ModuleDetector.<init>(ModuleDetector.java:91)
      	at io.jenkins.plugins.analysis.core.steps.IssuesScanner$ReportPostProcessor.resolveModuleNames(IssuesScanner.java:290)
      	at io.jenkins.plugins.analysis.core.steps.IssuesScanner$ReportPostProcessor.invoke(IssuesScanner.java:249)
      	at io.jenkins.plugins.analysis.core.steps.IssuesScanner$ReportPostProcessor.invoke(IssuesScanner.java:223)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3331)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      

      I enabled all logs for hudson.remoting.RemoteClassLoader and it shows a lot of repetition in the remote class loading:

      • 3552 FINER hudson.remoting.RemoteClassLoader prefetchClassReference fetch3(org.apache.xerces.impl.dv.dtd.DTDDVFactoryImpl)
      • 3552 FINER hudson.remoting.RemoteClassLoader prefetchClassReference fetch3(org.apache.xerces.parsers.XIncludeAwareParserConfiguration)

      I have tried switching to -workDir, but it didn't make any difference.

          [JENKINS-66268] Repeated remote loading of Xerces classes

          Dan Berindei added a comment -

          I found a workaround: I have downloaded a copy of xercesImpl-2.12.1.jar and I put it on the classpath of the agent with -Xbootclasspath/a.

          I had to also switch the agents to Java 11 because they would fail with a NoClassDefFound on Java 8, but here are now 0 fetch3(org.apache.xerces.*) messages in the agent log, and the time to record issues dropped from 7 minutes to 43 seconds. It's still a bit longer than I'd like, but it's good enough.

          Dan Berindei added a comment - I found a workaround: I have downloaded a copy of xercesImpl-2.12.1.jar and I put it on the classpath of the agent with -Xbootclasspath/a . I had to also switch the agents to Java 11 because they would fail with a NoClassDefFound on Java 8, but here are now 0 fetch3(org.apache.xerces.*) messages in the agent log, and the time to record issues dropped from 7 minutes to 43 seconds. It's still a bit longer than I'd like, but it's good enough.

          thomas zoratto added a comment - - edited

          Hello ! As discussed by email, I'm reopening this issue.

          We can see the same behavior on our Jenkins installation and it forces us to keep using direct TCP connection (for inbound agents) instead of websockets (see below).

          When recordIssues is used, for some reason, 2 classes from Xerces (DTDDVFactoryImpl and XIncludeAwareParserConfiguration) are being loaded from the controller to the agent running the step through the RemoteClassLoader (hudson.remoting.RemoteClassLoader) because analysis-model needs them. While I understand why these 2 classes need to be downloaded, I don't understand why they are being downloaded so many times for a single call to recordIssues.

          All these round-trips between the agent and the controller to fetch these 2 classes dozens of times, have a terrible impact on the performance. Performing a thread dump every second when recordIssues is running (just before the "Post processing issues on XXX" is logged) is insightful, every single time there is a thread waiting for one of the Xerces class to be fetched from the controller.

          The performance issue is particularly noticeable when using websockets connection between the agents and the controller through NGINX (8-9 minutes per recordIssues call), but can still be confirmed with a direct TCP connection : it's quicker, but the thread dumps still indicate the same behavior and the duration can go as high as almost 3min per recordIssues call.

          It might be easier to reproduce this on a multi-module maven project (just a guess).

          The workaround mentioned by danberindei above only confirms that there is an issue with the class loading, it cannot be considered as a satisfying fix. 

          I'm pretty sure this issue is the same as in https://issues.jenkins.io/browse/JENKINS-69813

          thomas zoratto added a comment - - edited Hello ! As discussed by email, I'm reopening this issue. We can see the same behavior on our Jenkins installation and it forces us to keep using direct TCP connection (for inbound agents) instead of websockets (see below). When recordIssues is used, for some reason, 2 classes from Xerces (DTDDVFactoryImpl and XIncludeAwareParserConfiguration) are being loaded from the controller to the agent running the step through the RemoteClassLoader (hudson.remoting.RemoteClassLoader) because analysis-model needs them. While I understand why these 2 classes need to be downloaded, I don't understand why they are being downloaded so many times for a single call to recordIssues. All these round-trips between the agent and the controller to fetch these 2 classes dozens of times, have a terrible impact on the performance. Performing a thread dump every second when recordIssues is running (just before the "Post processing issues on XXX" is logged) is insightful, every single time there is a thread waiting for one of the Xerces class to be fetched from the controller. The performance issue is particularly noticeable when using websockets connection between the agents and the controller through NGINX (8-9 minutes per recordIssues call), but can still be confirmed with a direct TCP connection : it's quicker, but the thread dumps still indicate the same behavior and the duration can go as high as almost 3min per recordIssues call. It might be easier to reproduce this on a multi-module maven project (just a guess). The workaround mentioned by danberindei above only confirms that there is an issue with the class loading, it cannot be considered as a satisfying fix.  I'm pretty sure this issue is the same as in https://issues.jenkins.io/browse/JENKINS-69813

            drulli Ulli Hafner
            danberindei Dan Berindei
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: