-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
Jenkins 1.480
Please also show the total number of tests that have executed as part of the test columns. A statement "0 failed tests" is misleading when the test suite crashes and only e.g. 1 out of 50 tests executed (and that 1 test succeeded).
[JENKINS-15155] Show total number of tests in test column
Sebastian Schuberth
added a comment - I'm running tests on Android, so yes, it's basically JUnit. The application under test contains native C++ code that's called via JNI. If there's an uncaught exception in native code, that will take down the Java process, too, and in fact even restart the VM. So there's actually no Java exception thrown in that case that JUnit could catch and report that test as failed. It simply terminates as if the whole test suite ran through.
I was also hoping this could be done without modifications to Jenkins core. On the other hand, I'm also missing the number of total tests on the job's project page ("Latest Test Result (3 failures / +2)"). Would it appear there, too, if you modified the Jelly script? If so, that would be a great solution solving to issues at a time.
A tooltip would be better than nothing, but as you've guessed the whole purpose of this feature is to see at a first glance whether all tests have been executed. If you're thinking about adding a separate columns for that, how about having separate columns for all of succeeded / failed / diff / total tests?
Sebastian Schuberth
added a comment - I'm running tests on Android, so yes, it's basically JUnit. The application under test contains native C++ code that's called via JNI. If there's an uncaught exception in native code, that will take down the Java process, too, and in fact even restart the VM. So there's actually no Java exception thrown in that case that JUnit could catch and report that test as failed. It simply terminates as if the whole test suite ran through.
I was also hoping this could be done without modifications to Jenkins core. On the other hand, I'm also missing the number of total tests on the job's project page ("Latest Test Result (3 failures / +2)"). Would it appear there, too, if you modified the Jelly script? If so, that would be a great solution solving to issues at a time.
A tooltip would be better than nothing, but as you've guessed the whole purpose of this feature is to see at a first glance whether all tests have been executed. If you're thinking about adding a separate columns for that, how about having separate columns for all of succeeded / failed / diff / total tests? 
Fred G
added a comment - How about adding a little check after your tests, that compares the number of executed tests with the total number of tests and fails if it the numbers don't match?
Yup, if I change the jelly code in core it also shows up on the job page. But... core changes would only affect newer versions of Jenkins, so users stuck on older version would not see it.
Therefore changing code in the plug-in would be backwards-compatible and preferable.
Maybe we can do both.
I'm not sure about the right format of the string yet. How about "(3 failures/10 tests/ +2)"?
Any suggestions?
As horizontal space is always limited I'd always prefer one column that shows multiple values.
I'll see if I can let the user configure it.
Fred G
added a comment - How about adding a little check after your tests, that compares the number of executed tests with the total number of tests and fails if it the numbers don't match?
Yup, if I change the jelly code in core it also shows up on the job page. But... core changes would only affect newer versions of Jenkins, so users stuck on older version would not see it.
Therefore changing code in the plug-in would be backwards-compatible and preferable.
Maybe we can do both.
I'm not sure about the right format of the string yet. How about "(3 failures/10 tests/ +2)"?
Any suggestions?
As horizontal space is always limited I'd always prefer one column that shows multiple values.
I'll see if I can let the user configure it. Sebastian Schuberth
added a comment - Adding a check for the number of test on my side would only fix the problem for me, whereas fixing the issues in Jenkins helps others, too. So I prefer the latter.
I usually update Jenkins regularly, so I would be OK with a core change as I would get that change soon anyway. I'm not sure how many users are stuck on an older Jenkins version but are able to update the plugin, thus I'm also not sure whether changing the code in the plugin really helps those users.
Regarding the format, I'd prefer a slightly more human readable string like "3 of 10 failed (+2)". I'd also say "0 of 10 failed (±0)" instead of "no failures" if there were no failures.
Sebastian Schuberth
added a comment - Adding a check for the number of test on my side would only fix the problem for me, whereas fixing the issues in Jenkins helps others, too. So I prefer the latter.
I usually update Jenkins regularly, so I would be OK with a core change as I would get that change soon anyway. I'm not sure how many users are stuck on an older Jenkins version but are able to update the plugin, thus I'm also not sure whether changing the code in the plugin really helps those users.
Regarding the format, I'd prefer a slightly more human readable string like "3 of 10 failed (+2)". I'd also say "0 of 10 failed (±0)" instead of "no failures" if there were no failures. Code changed in jenkins
User: Fred G
Path:
src/main/java/jenkins/plugins/extracolumns/TestResultColumn.java
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/column.jelly
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/column.properties
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/columnHeader.jelly
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/columnHeader_de.properties
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/column_de.properties
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/config.jelly
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/config.properties
src/main/resources/jenkins/plugins/extracolumns/TestResultColumn/config_de.properties
src/main/webapp/help-testresult-column.html
src/main/webapp/help-testresult-column_de.html
http://jenkins-ci.org/commit/extra-columns-plugin/68238105632298a3532fd6aacc9112958a443498
Log:
[FIXED JENKINS-15155], [FIXED JENKINS-13702] Test column can now be
configured
Sorry, I'm a bit confused about the use-case. If a test suite crashes I'd expect it to fail the build/test job, not carry on like nothing happened.
What kind of test suite are you using? JUnit?
Apart from that, I currently re-use the same Jelly-script that shows the test-result on the job page.
So changing the output would require changes in Jenkins core. I'd like to avoid that, if possible.
Would a tooltip showing the total number of tests on the test result column be sufficient or do you
want to see the total number at a glance?
Alternatively I could add another column that shows the total number of tests, but it seems a bit over the top.
I'm open for other suggestions.