Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7970

If all JUnit tests are skipped, the build fails

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      When all JUnit tests are skipped, the JUnit result archiver considers the build to have failed, since pass count and fail count are both zero.

      I believe that the attached patch would fix the problem, though I haven't succeeded to build it and test it.

        Attachments

          Issue Links

            Activity

            mhenoch mhenoch created issue -
            Hide
            mhenoch mhenoch added a comment -

            (Hm, how did this get set to "major" priority? I should pay more attention...)

            Show
            mhenoch mhenoch added a comment - (Hm, how did this get set to "major" priority? I should pay more attention...)
            mhenoch mhenoch made changes -
            Field Original Value New Value
            Priority Major [ 3 ] Minor [ 4 ]
            Hide
            mhenoch mhenoch added a comment -

            Got it built and tested now. I confirm that the patch fixes the problem.

            Show
            mhenoch mhenoch added a comment - Got it built and tested now. I confirm that the patch fixes the problem.
            Hide
            zkrakko Zoltan Krakko added a comment -

            Hi, is there a plan when this bug will be corrected? Thanks.

            Show
            zkrakko Zoltan Krakko added a comment - Hi, is there a plan when this bug will be corrected? Thanks.
            Hide
            alex_barna Tzuchien added a comment -

            I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between:

            (1) Unstable - if any test fails.
            (2) Not executed/abort/skipped - if none of the test is actually executed.

            My GUI-automated tests require some complicated setup procedures,
            which are not very robust. Without proper setup, the "real/interesting
            part" of test cannot be executed, but I don't want them to be
            considered "failed/red" because the "test" is actually not be able to
            be performed.

            Show
            alex_barna Tzuchien added a comment - I propose to consider the build result as "aborted" - the gray icon, if all tests are skipped, in order to distinguish between: (1) Unstable - if any test fails. (2) Not executed/abort/skipped - if none of the test is actually executed. My GUI-automated tests require some complicated setup procedures, which are not very robust. Without proper setup, the "real/interesting part" of test cannot be executed, but I don't want them to be considered "failed/red" because the "test" is actually not be able to be performed.
            Hide
            alex_barna Tzuchien added a comment -

            Quotation from Testing in Python mailing list:

            On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote:
            > When we last discussed this kohusuke felt it was a good backstop
            > against broken configurations (no tests actually run is more likely to
            > indicate a broken setup than a successful test run on a brand new
            > project).

            Good point. However, if each test case is executed in a separated
            downstream job of a matrix job, the build result of the parent matrix
            job will be considered "failed" if any of the downstream job/test
            case is skipped. This is actually what is happening to me.

            Show
            alex_barna Tzuchien added a comment - Quotation from Testing in Python mailing list: On Mon, Mar 21, 2011 at 8:04 PM, Robert Collins <robertc@robertcollins.net> wrote: > When we last discussed this kohusuke felt it was a good backstop > against broken configurations (no tests actually run is more likely to > indicate a broken setup than a successful test run on a brand new > project). Good point. However, if each test case is executed in a separated downstream job of a matrix job, the build result of the parent matrix job will be considered "failed" if any of the downstream job/test case is skipped. This is actually what is happening to me.
            Hide
            rbondi Richard Bondi added a comment -

            Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped?

            Here's an example use case where the project should not fail if there are no tests.

            I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.

            Show
            rbondi Richard Bondi added a comment - Can't this just be made an option, a checkbox to optionally mark a build as not failed if all tests were skipped? Here's an example use case where the project should not fail if there are no tests. I have to run integration tests in project A against an unreliable customer's system; I use xslt in a separate project B to filter out only project A tests that failed due to that customer system, by marking all other tests as skipped. When there are no tests after filtering, project B contains no tests, indicating merely that the customer's system behaved for once. Currently Jenkins says such builds are failures; obviously in this use case the opposite is true.
            Hide
            davehunt Dave Hunt added a comment -

            Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.

            Show
            davehunt Dave Hunt added a comment - Any update on this issue? The patch looks simple enough, and seems like the right thing to do. I'm also open to the suggestion of a checkbox (preferably in the main configuration) to allow 100% skip to be considered a pass if there is still reluctance to merge the patch as it is.
            Hide
            triangletodd Todd Edwards added a comment -

            I would also like to see this patch merged.

            Show
            triangletodd Todd Edwards added a comment - I would also like to see this patch merged.
            kutzi kutzi made changes -
            Assignee kutzi [ kutzi ]
            Hide
            kutzi kutzi added a comment -

            Agree, it's absolutely reasonable to expect the build to only fail if there are really no test results - i.e. also no skipped tests.

            Show
            kutzi kutzi added a comment - Agree, it's absolutely reasonable to expect the build to only fail if there are really no test results - i.e. also no skipped tests.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Christoph Kutzinski
            Path:
            changelog.html
            core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
            core/src/main/java/hudson/tasks/junit/TestResult.java
            http://jenkins-ci.org/commit/jenkins/9bc7e21dc0bd679e7b6506f264079630493237d4
            Log:
            [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Christoph Kutzinski Path: changelog.html core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java core/src/main/java/hudson/tasks/junit/TestResult.java http://jenkins-ci.org/commit/jenkins/9bc7e21dc0bd679e7b6506f264079630493237d4 Log: [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests.
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2427
            [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests. (Revision 9bc7e21dc0bd679e7b6506f264079630493237d4)

            Result = SUCCESS
            Christoph Kutzinski : 9bc7e21dc0bd679e7b6506f264079630493237d4
            Files :

            • changelog.html
            • core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
            • core/src/main/java/hudson/tasks/junit/TestResult.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2427 [FIXED JENKINS-7970] JUnit result archiver should only fail builds if there are really no results - i.e. also no skipped tests. (Revision 9bc7e21dc0bd679e7b6506f264079630493237d4) Result = SUCCESS Christoph Kutzinski : 9bc7e21dc0bd679e7b6506f264079630493237d4 Files : changelog.html core/src/main/java/hudson/tasks/junit/JUnitResultArchiver.java core/src/main/java/hudson/tasks/junit/TestResult.java
            Hide
            triangletodd Todd Edwards added a comment -

            Thank you so much!

            Show
            triangletodd Todd Edwards added a comment - Thank you so much!
            Hide
            joelbdoin Joel Beaudoin added a comment -

            Any estimate as to when this will make it into a release package? I don't see it in the changelog yet.

            Show
            joelbdoin Joel Beaudoin added a comment - Any estimate as to when this will make it into a release package? I don't see it in the changelog yet.
            Hide
            kutzi kutzi added a comment - - edited

            Will be in 1.511. Look under 'upcoming changes': http://jenkins-ci.org/changelog

            Show
            kutzi kutzi added a comment - - edited Will be in 1.511. Look under 'upcoming changes': http://jenkins-ci.org/changelog
            Hide
            joelbdoin Joel Beaudoin added a comment -

            Thanks mate. I missed the 'upcoming changes' link.

            Show
            joelbdoin Joel Beaudoin added a comment - Thanks mate. I missed the 'upcoming changes' link.
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            olivergondza Oliver Gond┼ża made changes -
            Labels lts-candidate 1.509.4-rejected
            landoltjp Paul Landolt made changes -
            Link This issue is related to JENKINS-22747 [ JENKINS-22747 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 138073 ] JNJira + In-Review [ 187828 ]

              People

              Assignee:
              kutzi kutzi
              Reporter:
              mhenoch mhenoch
              Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: