Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9090

Too many directory scans in warnings, analysis-core plugins

XMLWordPrintable

      analysis-core 1.18
      warnings 3.15

      I have a fairly large C/C++ project, which in its current configuration, has about 200k files in the workspace. It takes about 20 minutes to build everything, but an hour and a half to scan the warnings (only ~1000 warnings, GCC4 parser).

      Looking at the threadDumps (one to follow), it's not the log parsing that's taking long, but it's spending enormous time in the DirectoryScanner, kicked off from the ModuleDetector class. An instance of analysis.util.ModuleDetector does two recursive scans: one to look for Ant build files, and one to look for Maven modules. I don't have either setup in a non-Java project and don't really care much about pretty printing modules anyway.

      To make matters worse, the WarningsPublisher class creates two instances of the ModuleDetector - once in WarningsPublisher::perform() and another indirectly through the use of an AnnotationsClassifier object. This results in 4x useless scans, and from what I've read, the Ant DirectoryScanner is slow enough already.

      I suspect there are other recursive scans going on too, as I have seen other stack traces trying to resolve relative paths, etc. These should also be considered for optimization.

      This poor of performance dwarfing my actual build times is making my continuous integration much less "continuous".
      Possible improvements:

      • Make Java module detection a configurable option of the plugin(s)!!!
      • Combine the scans of Ant and Maven files into a single recursive search
      • Cache the results in a ModuleDetector singleton.
      • Make the search path for resolving paths configurable (the warnings for the files I care about are only in a small section of the build tree)

      (Forgive me if I don't have a complete understanding of all the pieces involved... I was just browsing the GitHub repo).

      One snapshot of a stack trace

      "Executor #0 for master : executing DevBuild-CommSW_Jaguar_Dev #163" Id=63 Group=main RUNNABLE
	at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:64)
	-  locked org.apache.tools.ant.util.VectorSet@20ef8994
	at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:75)
	-  locked org.apache.tools.ant.util.VectorSet@20ef8994
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1236)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1184)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1146)
	at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:928)
	at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:882)
	-  locked org.apache.tools.ant.DirectoryScanner@cb91d2b
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:490)
	at hudson.plugins.analysis.util.FileFinder.find(FileFinder.java:76)
	at hudson.plugins.analysis.util.ModuleDetector.find(ModuleDetector.java:166)
	at hudson.plugins.analysis.util.ModuleDetector.findAntProjects(ModuleDetector.java:153)
	at hudson.plugins.analysis.util.ModuleDetector.createFilesToModuleMapping(ModuleDetector.java:103)
	at hudson.plugins.analysis.util.ModuleDetector.(ModuleDetector.java:69)
	at hudson.plugins.analysis.util.ModuleDetector.(ModuleDetector.java:56)
	at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:239)
	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:281)
	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:622)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:601)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:579)
	at hudson.model.Build$RunnerImpl.post2(Build.java:156)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:548)
	at hudson.model.Run.run(Run.java:1386)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:145)

            drulli Ulli Hafner
            mrmalek Mark Malek
            Votes:
            3 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: