Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36759

MSTest Plugin reports failed data driven tests as passed in certain situations.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • mstest-plugin
    • None
    • Windows 2012, Jenkins 1.653, mstest-plugin 0.19

    Description

      If a data driven test fails before ever running the test - for example, if the data file cannot be found - then the "failed" outcome in the .trx output file is not reported at all. It is not even included in the test counts. Also, the Jenkins Job passes as Stable, so it appears to everybody that everything is OK, whereas at best the test didn't run at all.
      The attached file UnitTest1.cs is a minimal test which reproduces the issue.
      The attached file TestResults.trx is the mstest output showing the test failure.
      The attached file Results.jpg is a screenshot of the Jenkins results as displayed by the mstest plugin.
      This is pretty serious, we had a product failure but when we looked at the Jenkins tests all appeared fine. The problem only showed up when we ran the tests manually and they failed.

      Attachments

        1. Results.jpg
          91 kB
          Jon Bowen
        2. TestResults.trx
          5 kB
          Jon Bowen
        3. UnitTest1.cs
          0.4 kB
          Jon Bowen

        Activity

          People

            nilleb Ivo Bellin Salarin
            jonbowen Jon Bowen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: