Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7970

If all JUnit tests are skipped, the build fails

      When all JUnit tests are skipped, the JUnit result archiver considers the build to have failed, since pass count and fail count are both zero.

      I believe that the attached patch would fix the problem, though I haven't succeeded to build it and test it.

          [JENKINS-7970] If all JUnit tests are skipped, the build fails

          mhenoch added a comment -

          (Hm, how did this get set to "major" priority? I should pay more attention...)

          mhenoch added a comment - (Hm, how did this get set to "major" priority? I should pay more attention...)

          mhenoch added a comment -

          Got it built and tested now. I confirm that the patch fixes the problem.

          mhenoch added a comment - Got it built and tested now. I confirm that the patch fixes the problem.

          Zoltan Krakko added a comment -

          Hi, is there a plan when this bug will be corrected? Thanks.

          Zoltan Krakko added a comment - Hi, is there a plan when this bug will be corrected? Thanks.

          Tzuchien added a comment -

          I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between:

          (1) Unstable - if any test fails.
          (2) Not executed/abort/skipped - if none of the test is actually executed.

          My GUI-automated tests require some complicated setup procedures,
          which are not very robust. Without proper setup, the "real/interesting
          part" of test cannot be executed, but I don't want them to be
          considered "failed/red" because the "test" is actually not be able to
          be performed.

          Tzuchien added a comment - I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between: (1) Unstable - if any test fails. (2) Not executed/abort/skipped - if none of the test is actually executed. My GUI-automated tests require some complicated setup procedures, which are not very robust. Without proper setup, the "real/interesting part" of test cannot be executed, but I don't want them to be considered "failed/red" because the "test" is actually not be able to be performed.

          Tzuchien added a comment -

          Quotation from Testing in Python mailing list:

          On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote:
          > When we last discussed this kohusuke felt it was a good backstop
          > against broken configurations (no tests actually run is more likely to
          > indicate a broken setup than a successful test run on a brand new
          > project).

          Good point. However, if each test case is executed in a separated
          downstream job of a matrix job, the build result of the parent matrix
          job will be considered "failed" if any of the downstream job/test
          case is skipped. This is actually what is happening to me.

          Tzuchien added a comment - Quotation from Testing in Python mailing list: On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote: > When we last discussed this kohusuke felt it was a good backstop > against broken configurations (no tests actually run is more likely to > indicate a broken setup than a successful test run on a brand new > project). Good point. However, if each test case is executed in a separated downstream job of a matrix job, the build result of the parent matrix job will be considered "failed" if any of the downstream job/test case is skipped. This is actually what is happening to me.

          Richard Bondi added a comment -

          Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped?

          Here's an example use case where the project should not fail if there are no tests.

          I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.

          Richard Bondi added a comment - Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped? Here's an example use case where the project should not fail if there are no tests. I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.

          Dave Hunt added a comment -

          Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.

          Dave Hunt added a comment - Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.

          Todd Edwards added a comment -

          I would also like to see this patch merged.

          Todd Edwards added a comment - I would also like to see this patch merged.

          kutzi added a comment -

          Agree, it's absolutely reasonable to expect the build to only fail if there are really no test results - i.e. also no skipped tests.

          kutzi added a comment - Agree, it's absolutely reasonable to expect the build to only fail if there are really no test results - i.e. also no skipped tests.

          Code changed in jenkins
          User: Christoph Kutzinski
          Path:
          changelog.html
          core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
          core/src/main/java/hudson/tasks/junit/TestResult.java
          http://jenkins-ci.org/commit/jenkins/9bc7e21dc0bd679e7b6506f264079630493237d4
          Log:
          [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java core/src/main/java/hudson/tasks/junit/TestResult.java http://jenkins-ci.org/commit/jenkins/9bc7e21dc0bd679e7b6506f264079630493237d4 Log: [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2427
          [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests. (Revision 9bc7e21dc0bd679e7b6506f264079630493237d4)

          Result = SUCCESS
          Christoph Kutzinski : 9bc7e21dc0bd679e7b6506f264079630493237d4
          Files :

          • changelog.html
          • core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
          • core/src/main/java/hudson/tasks/junit/TestResult.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2427 [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests. (Revision 9bc7e21dc0bd679e7b6506f264079630493237d4) Result = SUCCESS Christoph Kutzinski : 9bc7e21dc0bd679e7b6506f264079630493237d4 Files : changelog.html core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java core/src/main/java/hudson/tasks/junit/TestResult.java

          Todd Edwards added a comment -

          Thank you so much!

          Todd Edwards added a comment - Thank you so much!

          Joel Beaudoin added a comment -

          Any estimate as to when this will make it into a release package? I don't see it in the changelog yet.

          Joel Beaudoin added a comment - Any estimate as to when this will make it into a release package? I don't see it in the changelog yet.

          kutzi added a comment - - edited

          Will be in 1.511. Look under 'upcoming changes': http://jenkins-ci.org/changelog

          kutzi added a comment - - edited Will be in 1.511. Look under 'upcoming changes': http://jenkins-ci.org/changelog

          Joel Beaudoin added a comment -

          Thanks mate. I missed the 'upcoming changes' link.

          Joel Beaudoin added a comment - Thanks mate. I missed the 'upcoming changes' link.

            kutzi kutzi
            mhenoch mhenoch
            Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: