[JENKINS-9090] Too many directory scans in warnings, analysis-core plugins - Jenkins Jira

XML

Word

Printable

Details

Type: Improvement
Status: Resolved (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: analysis-core-plugin, warnings-plugin
Labels:
None
Environment:
Linux, 16 core, 12GB RAM
Hudson 1.395

Similar Issues:

Show

Description

analysis-core 1.18
warnings 3.15

I have a fairly large C/C++ project, which in its current configuration, has about 200k files in the workspace. It takes about 20 minutes to build everything, but an hour and a half to scan the warnings (only ~1000 warnings, GCC4 parser).

Looking at the threadDumps (one to follow), it's not the log parsing that's taking long, but it's spending enormous time in the DirectoryScanner, kicked off from the ModuleDetector class. An instance of analysis.util.ModuleDetector does two recursive scans: one to look for Ant build files, and one to look for Maven modules. I don't have either setup in a non-Java project and don't really care much about pretty printing modules anyway.

To make matters worse, the WarningsPublisher class creates two instances of the ModuleDetector - once in WarningsPublisher::perform() and another indirectly through the use of an AnnotationsClassifier object. This results in 4x useless scans, and from what I've read, the Ant DirectoryScanner is slow enough already.

I suspect there are other recursive scans going on too, as I have seen other stack traces trying to resolve relative paths, etc. These should also be considered for optimization.

This poor of performance dwarfing my actual build times is making my continuous integration much less "continuous".
Possible improvements:

Make Java module detection a configurable option of the plugin(s)!!!
Combine the scans of Ant and Maven files into a single recursive search
Cache the results in a ModuleDetector singleton.
Make the search path for resolving paths configurable (the warnings for the files I care about are only in a small section of the build tree)

(Forgive me if I don't have a complete understanding of all the pieces involved... I was just browsing the GitHub repo).

One snapshot of a stack trace

"Executor #0 for master : executing DevBuild-CommSW_Jaguar_Dev #163" Id=63 Group=main RUNNABLE
	at org.apache.tools.ant.util.VectorSet.doAdd(VectorSet.java:64)
	-  locked org.apache.tools.ant.util.VectorSet@20ef8994
	at org.apache.tools.ant.util.VectorSet.addElement(VectorSet.java:75)
	-  locked org.apache.tools.ant.util.VectorSet@20ef8994
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1236)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1259)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1184)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1146)
	at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:928)
	at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:882)
	-  locked org.apache.tools.ant.DirectoryScanner@cb91d2b
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:490)
	at hudson.plugins.analysis.util.FileFinder.find(FileFinder.java:76)
	at hudson.plugins.analysis.util.ModuleDetector.find(ModuleDetector.java:166)
	at hudson.plugins.analysis.util.ModuleDetector.findAntProjects(ModuleDetector.java:153)
	at hudson.plugins.analysis.util.ModuleDetector.createFilesToModuleMapping(ModuleDetector.java:103)
	at hudson.plugins.analysis.util.ModuleDetector.(ModuleDetector.java:69)
	at hudson.plugins.analysis.util.ModuleDetector.(ModuleDetector.java:56)
	at hudson.plugins.warnings.WarningsPublisher.perform(WarningsPublisher.java:239)
	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:281)
	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:622)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:601)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:579)
	at hudson.model.Build$RunnerImpl.post2(Build.java:156)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:548)
	at hudson.model.Run.run(Run.java:1386)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:145)

Attachments

Issue Links

is duplicated by

JENKINS-8915 huge performance drop in post build actions

Resolved

is related to

JENKINS-14024 Disable resolving relative to absolute paths

Closed

Activity

Ascending order - Click to sort in descending order

Mark Malek created issue - 2011-03-17 22:01

Ulli Hafner made changes - 2011-03-17 22:53

Field	Original Value	New Value
Link		This issue is duplicated by ~~JENKINS-8915~~ [ ~~JENKINS-8915~~ ]

Ulli Hafner added a comment - 2011-03-17 22:56

This part of the code actually needs some improvements. Also when warnings in the log only have relative file names then the file names are evaluated for each warning with a single slave call.

Ulli Hafner added a comment - 2011-03-17 22:56 This part of the code actually needs some improvements. Also when warnings in the log only have relative file names then the file names are evaluated for each warning with a single slave call.

Sergey added a comment - 2011-03-31 08:32

Ullrich, are you planning modify \ improvements this code near future?

Sergey added a comment - 2011-03-31 08:32 Ullrich, are you planning modify \ improvements this code near future?

Ulli Hafner added a comment - 2011-03-31 08:52

Yes, I already have some local changes...

Ulli Hafner added a comment - 2011-03-31 08:52 Yes, I already have some local changes...

Ulli Hafner made changes - 2011-04-05 13:00

Status

Open [ 1 ]

In Progress [ 3 ]

SCM/JIRA link daemon added a comment - 2011-04-07 09:23

Code changed in jenkins
User: Ulli Hafner
Path:
go.sh
src/main/java/hudson/plugins/analysis/core/AnnotationsClassifier.java
src/main/java/hudson/plugins/analysis/core/FilesParser.java
src/main/java/hudson/plugins/analysis/core/HealthAwarePublisher.java
src/main/java/hudson/plugins/analysis/util/ModuleDetector.java
src/main/java/hudson/plugins/analysis/util/NullModuleDetector.java
src/main/resources/util/advanced.jelly
src/main/resources/util/advanced.properties
src/main/resources/util/advanced_de.properties
http://jenkins-ci.org/commit/analysis-core-plugin/0c4c162be8819627b0526bfd182eb60992fc0d98
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23 Code changed in jenkins User: Ulli Hafner Path: go.sh src/main/java/hudson/plugins/analysis/core/AnnotationsClassifier.java src/main/java/hudson/plugins/analysis/core/FilesParser.java src/main/java/hudson/plugins/analysis/core/HealthAwarePublisher.java src/main/java/hudson/plugins/analysis/util/ModuleDetector.java src/main/java/hudson/plugins/analysis/util/NullModuleDetector.java src/main/resources/util/advanced.jelly src/main/resources/util/advanced.properties src/main/resources/util/advanced_de.properties http://jenkins-ci.org/commit/analysis-core-plugin/0c4c162be8819627b0526bfd182eb60992fc0d98 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/checkstyle/CheckStylePublisher.java
http://jenkins-ci.org/commit/checkstyle-plugin/99ebcb8d4103e95869e3b0155e535c54d2b6370f
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23 Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/checkstyle/CheckStylePublisher.java http://jenkins-ci.org/commit/checkstyle-plugin/99ebcb8d4103e95869e3b0155e535c54d2b6370f Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/dry/DryPublisher.java
http://jenkins-ci.org/commit/dry-plugin/84bda2973ea009bcdc8036bfeea1089053d2167f
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23 Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/dry/DryPublisher.java http://jenkins-ci.org/commit/dry-plugin/84bda2973ea009bcdc8036bfeea1089053d2167f Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23

Code changed in jenkins
User: Ulli Hafner
Path:
plugin/src/main/java/hudson/plugins/findbugs/FindBugsPublisher.java
http://jenkins-ci.org/commit/findbugs-plugin/3578cec72b3e399dd29e6d245900ba97df16cc13
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:23 Code changed in jenkins User: Ulli Hafner Path: plugin/src/main/java/hudson/plugins/findbugs/FindBugsPublisher.java http://jenkins-ci.org/commit/findbugs-plugin/3578cec72b3e399dd29e6d245900ba97df16cc13 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:24

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/pmd/PmdPublisher.java
http://jenkins-ci.org/commit/pmd-plugin/25c44c7247932df1f4dff4ebf8c1de839baf2593
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:24 Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/pmd/PmdPublisher.java http://jenkins-ci.org/commit/pmd-plugin/25c44c7247932df1f4dff4ebf8c1de839baf2593 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:24

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/tasks/TasksPublisher.java
src/main/java/hudson/plugins/tasks/parser/WorkspaceScanner.java
http://jenkins-ci.org/commit/tasks-plugin/ebfde8b48b4fb9e1dd550217350df7f522553b29
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:24 Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/tasks/TasksPublisher.java src/main/java/hudson/plugins/tasks/parser/WorkspaceScanner.java http://jenkins-ci.org/commit/tasks-plugin/ebfde8b48b4fb9e1dd550217350df7f522553b29 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:24

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/warnings/WarningsPublisher.java
http://jenkins-ci.org/commit/warnings-plugin/5c6329d25f593f167e3d80de6fae7d25e92929d6
Log:
~~JENKINS-9090~~ Added project configuration option to activate
scanning of workspace for build.xml and pom.xml files to detect
the module names of warnings.

SCM/JIRA link daemon added a comment - 2011-04-07 09:24 Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/warnings/WarningsPublisher.java http://jenkins-ci.org/commit/warnings-plugin/5c6329d25f593f167e3d80de6fae7d25e92929d6 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.

Ulli Hafner added a comment - 2011-04-07 13:03

Added a configuration option to enable the scanning of ant/maven modules.

Ulli Hafner added a comment - 2011-04-07 13:03 Added a configuration option to enable the scanning of ant/maven modules.

Ulli Hafner made changes - 2011-04-07 13:03

Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Resolved [ 5 ]

SCM/JIRA link daemon added a comment - 2011-04-12 08:43

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/analysis/collector/AnalysisPublisher.java
http://jenkins-ci.org/commit/analysis-collector-plugin/769d1376930c92456eea8c05fe99b83611140bc9
Log:
~~JENKINS-9090~~ Added project configuration option to activate scanning
of workspace for build.xml and pom.xml files to detect the module
names of warnings.)

Compare: https://github.com/jenkinsci/analysis-collector-plugin/compare/f48b669...769d137

SCM/JIRA link daemon added a comment - 2011-04-12 08:43 Code changed in jenkins User: Ulli Hafner Path: src/main/java/hudson/plugins/analysis/collector/AnalysisPublisher.java http://jenkins-ci.org/commit/analysis-collector-plugin/769d1376930c92456eea8c05fe99b83611140bc9 Log: JENKINS-9090 Added project configuration option to activate scanning of workspace for build.xml and pom.xml files to detect the module names of warnings.) Compare: https://github.com/jenkinsci/analysis-collector-plugin/compare/f48b669...769d137

Kevin Neal made changes - 2012-06-05 22:01

Link

This issue is related to ~~JENKINS-14024~~ [ ~~JENKINS-14024~~ ]

R. Tyler Croy made changes - 2016-07-25 23:54

Workflow

JNJira [ 139246 ]

JNJira + In-Review [ 188407 ]

People

Assignee:: Ulli Hafner

Reporter:: Mark Malek

Votes:: 3 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 2011-03-17 22:01

Updated:: 2012-06-05 22:01

Resolved:: 2011-04-07 13:03