Details
-
Type:
Improvement
-
Status: Resolved (View Workflow)
-
Priority:
Major
-
Resolution: Fixed
-
Component/s: testng-plugin
-
Labels:None
-
Similar Issues:
Description
There was an issue 'JENKINS-16832 Do not show failed builds in trend graph' which was fixed. Now, failed builds are not shown in trend graph.
We have test dashboard with trend graph. We run nightly regression each day. If build is failed then we don't see it in the graph and we can easily mislead that test build was fine, but it wasn't.
Is it possible to make an optional to show failed and aborted builds in trend graph? This option can be disabled by default.
Attachments
Issue Links
- is related to
-
JENKINS-20162 TestNG Results Trend doesn't show a failures
-
- Resolved
-
Activity
Ivan Sas
created issue -
Nalin Makar
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Link |
This issue is related to |
We use gradle build and have setIgnoreFailures set to true. We have test dashboard with aggregated embedded of multiple test results graphs for different products.
We have few thousands tests and it sometimes occurs that build is not created, for any reason.
For example some external xml schemas were not available during the build time, or build causes out of memory, or someone committed code which is not compiling. Another example which occurs last time. Someone has committed code which didn't close files. Tests opened so many files. We are doing some conversion of testNG xml reports at the end of test suite and then pass it to jenkins testng plugin. There were opened so many files, which causes that it was unable to create test XML report and the build was failed.
The purpose of this change is to know the state of tests even if build was not prepared for any reason. If someone asked about nightly tests result, I can say from the graph that we have no test result, because something goes wrong with the build.
Ivan Sas
added a comment - We use gradle build and have setIgnoreFailures set to true. We have test dashboard with aggregated embedded of multiple test results graphs for different products.
We have few thousands tests and it sometimes occurs that build is not created, for any reason.
For example some external xml schemas were not available during the build time, or build causes out of memory, or someone committed code which is not compiling. Another example which occurs last time. Someone has committed code which didn't close files. Tests opened so many files. We are doing some conversion of testNG xml reports at the end of test suite and then pass it to jenkins testng plugin. There were opened so many files, which causes that it was unable to create test XML report and the build was failed.
The purpose of this change is to know the state of tests even if build was not prepared for any reason. If someone asked about nightly tests result, I can say from the graph that we have no test result, because something goes wrong with the build. I'm sort of shocked this doesn't exist already.
There's really no way to know that 102 tests failed out of 10,222 yesterday, and today 105 are failing?
In my test environment I set up the Jenkins jobs to run my tests using a windows batch file. Any test failures results in a failed build and the test results are not then included in the graph. I can work around this by using EXIT /B 0 in my batch file but this then does not indicate whether there were any failures for this particular test run. I was relying on a failed test run/build to send me notification which no longer happens. I feel it would be better if including the results in the graph or not was set up as an available option.
Doug Stewart
added a comment - In my test environment I set up the Jenkins jobs to run my tests using a windows batch file. Any test failures results in a failed build and the test results are not then included in the graph. I can work around this by using EXIT /B 0 in my batch file but this then does not indicate whether there were any failures for this particular test run. I was relying on a failed test run/build to send me notification which no longer happens. I feel it would be better if including the results in the graph or not was set up as an available option. Nalin Makar
made changes -
| Status | Open [ 1 ] | In Progress [ 3 ] |
Code changed in jenkins
User: Nalin Makar
Path:
README
src/main/java/hudson/plugins/testng/Publisher.java
src/main/java/hudson/plugins/testng/TestNGProjectAction.java
src/main/resources/hudson/plugins/testng/Publisher/config.jelly
src/main/resources/hudson/plugins/testng/Publisher/help-showFailedBuilds.html
src/test/java/hudson/plugins/testng/PublisherTest.java
src/test/java/hudson/plugins/testng/TestNGProjectActionTest.java
src/test/java/hudson/plugins/testng/TestNGTestResultBuildActionTest.java
src/test/java/hudson/plugins/testng/results/ClassResultTest.java
src/test/java/hudson/plugins/testng/results/MethodResultTest.java
src/test/java/hudson/plugins/testng/results/PackageResultTest.java
http://jenkins-ci.org/commit/testng-plugin-plugin/f984cfafde186d3e1f3a779fccb8a2c61a20007d
Log:
Added: JENKINS-20438 - Make it an option to show failed builds in trend graph
Compare: https://github.com/jenkinsci/testng-plugin-plugin/compare/b03c0ad77827...f984cfafde18
Fixed. option will be available in next release.
Nalin Makar
added a comment - Fixed. option will be available in next release. Nalin Makar
made changes -
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
| Workflow | JNJira [ 151963 ] | JNJira + In-Review [ 194156 ] |
Nalin Makar i have toggle ON the option "Show failed builds in trend graph", still not able to see the failed build in TestNG result Trend graph
ankush goyal
added a comment - Nalin Makar i have toggle ON the option "Show failed builds in trend graph", still not able to see the failed build in TestNG result Trend graph 
Why does the build fail? If this is a maven build, use "-Dmaven.test.failure.ignore=true" to ensure that the build is not marked as failed even on test failures. This way you'll correctly see that the build is unstable.