Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7970

If all JUnit tests are skipped, the build fails

      When all JUnit tests are skipped, the JUnit result archiver considers the build to have failed, since pass count and fail count are both zero.

      I believe that the attached patch would fix the problem, though I haven't succeeded to build it and test it.

          [JENKINS-7970] If all JUnit tests are skipped, the build fails

          mhenoch created issue -

          mhenoch added a comment -

          (Hm, how did this get set to "major" priority? I should pay more attention...)

          mhenoch added a comment - (Hm, how did this get set to "major" priority? I should pay more attention...)
          mhenoch made changes -
          Priority Original: Major [ 3 ] New: Minor [ 4 ]

          mhenoch added a comment -

          Got it built and tested now. I confirm that the patch fixes the problem.

          mhenoch added a comment - Got it built and tested now. I confirm that the patch fixes the problem.

          Zoltan Krakko added a comment -

          Hi, is there a plan when this bug will be corrected? Thanks.

          Zoltan Krakko added a comment - Hi, is there a plan when this bug will be corrected? Thanks.

          Tzuchien added a comment -

          I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between:

          (1) Unstable - if any test fails.
          (2) Not executed/abort/skipped - if none of the test is actually executed.

          My GUI-automated tests require some complicated setup procedures,
          which are not very robust. Without proper setup, the "real/interesting
          part" of test cannot be executed, but I don't want them to be
          considered "failed/red" because the "test" is actually not be able to
          be performed.

          Tzuchien added a comment - I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between: (1) Unstable - if any test fails. (2) Not executed/abort/skipped - if none of the test is actually executed. My GUI-automated tests require some complicated setup procedures, which are not very robust. Without proper setup, the "real/interesting part" of test cannot be executed, but I don't want them to be considered "failed/red" because the "test" is actually not be able to be performed.

          Tzuchien added a comment -

          Quotation from Testing in Python mailing list:

          On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote:
          > When we last discussed this kohusuke felt it was a good backstop
          > against broken configurations (no tests actually run is more likely to
          > indicate a broken setup than a successful test run on a brand new
          > project).

          Good point. However, if each test case is executed in a separated
          downstream job of a matrix job, the build result of the parent matrix
          job will be considered "failed" if any of the downstream job/test
          case is skipped. This is actually what is happening to me.

          Tzuchien added a comment - Quotation from Testing in Python mailing list: On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote: > When we last discussed this kohusuke felt it was a good backstop > against broken configurations (no tests actually run is more likely to > indicate a broken setup than a successful test run on a brand new > project). Good point. However, if each test case is executed in a separated downstream job of a matrix job, the build result of the parent matrix job will be considered "failed" if any of the downstream job/test case is skipped. This is actually what is happening to me.

          Richard Bondi added a comment -

          Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped?

          Here's an example use case where the project should not fail if there are no tests.

          I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.

          Richard Bondi added a comment - Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped? Here's an example use case where the project should not fail if there are no tests. I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.

          Dave Hunt added a comment -

          Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.

          Dave Hunt added a comment - Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.

          Todd Edwards added a comment -

          I would also like to see this patch merged.

          Todd Edwards added a comment - I would also like to see this patch merged.

            kutzi kutzi
            mhenoch mhenoch
            Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: