-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins 2.89.3
Performance Publisher plugin 8.05
JRE 1.8.0_144-b01
Windows 7
I have tried a lot of things, but whenever a single test fails, the performance publisher plugin does not create graphs for individual tests. Only the overview (sum) of all tests is created. If no test has failed, the graphs are created as expected.
Since I cannot think of any reason why someone would want this, I must assume that this is a bug.
My Jenkins job runs the tests with MSTest. Each test measures the execution time for various scenarios and each test creates a new xml file that is parsed by the performance publisher plugin. If a test reports an error (e.g. the execution time exceeds a certain limit), the job is marked as unstable. This is done automatically by the MSTest plugin.
The plugin works fine with multiple xml files, as long as all tests have passed.
[JENKINS-49528] Peformance graphs for individual tests are not created when a single test has failed
Tobias Lingemann
added a comment - I don't believe it is related to the number of XML files. I took a look at the source code and while I'm not sure, I believe the problem is in TestDetails.java.
The graph for execution time, compile time and performance is only created when the result of the abstract build is success.
if (!abstractBuild.isBuilding() && abstractBuild.getResult().isBetterOrEqualTo(Result.SUCCESS))
Curiously the graph for custom metrics is created if the build is unstable or better. I have updated my tests to use a custom metric in the meantime. I will check tomorrow if it worked.
Tobias Lingemann
added a comment - I don't believe it is related to the number of XML files. I took a look at the source code and while I'm not sure, I believe the problem is in TestDetails.java .
The graph for execution time, compile time and performance is only created when the result of the abstract build is success.
if (!abstractBuild.isBuilding() && abstractBuild.getResult().isBetterOrEqualTo(Result.SUCCESS))
Curiously the graph for custom metrics is created if the build is unstable or better. I have updated my tests to use a custom metric in the meantime. I will check tomorrow if it worked.
Tobias Lingemann
added a comment - As I suspected. The custom metric graph was created successfully.
Tobias Lingemann
added a comment - As I suspected. The custom metric graph was created successfully. Thanks, I see now.
So now
execution time chart is shown if the result is success
compile time chart is shown if the result is success
performance chart is shown is the result is success
metrics chart is shown if the result is unstable or success
It looks logical to change these conditions to:
execution time chart is shown if the result is unstable or success
compile time chart is shown if the result is failure, unstable or success (better than not built)
performance chart is shown is the result is unstable or success
metrics chart is shown if the result is unstable or success (do not change)
Agree?
Tobias Lingemann
added a comment - Agreed Code changed in jenkins
User: Eugene Schava
Path:
src/main/java/hudson/plugins/PerfPublisher/TestDetails.java
http://jenkins-ci.org/commit/perfpublisher-plugin/f49132726ea71fef2d0b8df91a33ba42c929ad16
Log:
JENKINS-49528 Peformance graphs for individual tests are not created when a single test has failed
I've committed and pushed this change to repository
But I think it's too minor change to create new release
Are you OK to build and update your Jenkins installation manually with the version from github repository?
Tobias Lingemann
added a comment - For now I simply create the graph via the custom metric. I will wait until a new release is published.
Tobias Lingemann
added a comment - For now I simply create the graph via the custom metric. I will wait until a new release is published. 
Sorry, I need more details
Is your issue related to multiple XML files only? I usually use only one XML report file and cannot confirm that some chart differs if there is no or only one failed test