Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Minor
-
Resolution: Fixed
-
Component/s: junit-plugin
-
Labels:
-
Environment:Hudson 1.381
-
Similar Issues:
Description
When all JUnit tests are skipped, the JUnit result archiver considers the build to have failed, since pass count and fail count are both zero.
I believe that the attached patch would fix the problem, though I haven't succeeded to build it and test it.
Attachments
Issue Links
- is related to
-
JENKINS-22747 Add Job Configuration Option to NOT fail JUnit Report publish when no xml report files
-
- Resolved
-
Activity
Got it built and tested now. I confirm that the patch fixes the problem.
mhenoch
added a comment - Got it built and tested now. I confirm that the patch fixes the problem. Hi, is there a plan when this bug will be corrected? Thanks.
Zoltan Krakko
added a comment - Hi, is there a plan when this bug will be corrected? Thanks. I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between:
(1) Unstable - if any test fails.
(2) Not executed/abort/skipped - if none of the test is actually executed.
My GUI-automated tests require some complicated setup procedures,
which are not very robust. Without proper setup, the "real/interesting
part" of test cannot be executed, but I don't want them to be
considered "failed/red" because the "test" is actually not be able to
be performed.
Tzuchien
added a comment - I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between:
(1) Unstable - if any test fails.
(2) Not executed/abort/skipped - if none of the test is actually executed.
My GUI-automated tests require some complicated setup procedures,
which are not very robust. Without proper setup, the "real/interesting
part" of test cannot be executed, but I don't want them to be
considered "failed/red" because the "test" is actually not be able to
be performed. Quotation from Testing in Python mailing list:
On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote:
> When we last discussed this kohusuke felt it was a good backstop
> against broken configurations (no tests actually run is more likely to
> indicate a broken setup than a successful test run on a brand new
> project).
Good point. However, if each test case is executed in a separated
downstream job of a matrix job, the build result of the parent matrix
job will be considered "failed" if any of the downstream job/test
case is skipped. This is actually what is happening to me.
Tzuchien
added a comment - Quotation from Testing in Python mailing list:
On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote:
> When we last discussed this kohusuke felt it was a good backstop
> against broken configurations (no tests actually run is more likely to
> indicate a broken setup than a successful test run on a brand new
> project).
Good point. However, if each test case is executed in a separated
downstream job of a matrix job, the build result of the parent matrix
job will be considered "failed" if any of the downstream job/test
case is skipped. This is actually what is happening to me. Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped?
Here's an example use case where the project should not fail if there are no tests.
I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.
Richard Bondi
added a comment - Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped?
Here's an example use case where the project should not fail if there are no tests.
I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true. Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.
Dave Hunt
added a comment - Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is. I would also like to see this patch merged.
Todd Edwards
added a comment - I would also like to see this patch merged. Agree, it's absolutely reasonable to expect the build to only fail if there are really no test results - i.e. also no skipped tests.
kutzi
added a comment - Agree, it's absolutely reasonable to expect the build to only fail if there are really no test results - i.e. also no skipped tests. Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
core/src/main/java/hudson/tasks/junit/TestResult.java
http://jenkins-ci.org/commit/jenkins/9bc7e21dc0bd679e7b6506f264079630493237d4
Log:
[FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests.
Integrated in 
jenkins_main_trunk #2427
[FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests. (Revision 9bc7e21dc0bd679e7b6506f264079630493237d4)
Result = SUCCESS
Christoph Kutzinski : 9bc7e21dc0bd679e7b6506f264079630493237d4
Files :
- changelog.html
- core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
- core/src/main/java/hudson/tasks/junit/TestResult.java
Thank you so much!
Todd Edwards
added a comment - Thank you so much! Any estimate as to when this will make it into a release package? I don't see it in the changelog yet.
Joel Beaudoin
added a comment - Any estimate as to when this will make it into a release package? I don't see it in the changelog yet. Will be in 1.511. Look under 'upcoming changes': http://jenkins-ci.org/changelog
kutzi
added a comment - - edited Will be in 1.511. Look under 'upcoming changes': http://jenkins-ci.org/changelog Thanks mate. I missed the 'upcoming changes' link.
Joel Beaudoin
added a comment - Thanks mate. I missed the 'upcoming changes' link. 
(Hm, how did this get set to "major" priority? I should pay more attention...)