Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36759

MSTest Plugin reports failed data driven tests as passed in certain situations.


    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • mstest-plugin
    • None
    • Windows 2012, Jenkins 1.653, mstest-plugin 0.19

      If a data driven test fails before ever running the test - for example, if the data file cannot be found - then the "failed" outcome in the .trx output file is not reported at all. It is not even included in the test counts. Also, the Jenkins Job passes as Stable, so it appears to everybody that everything is OK, whereas at best the test didn't run at all.
      The attached file UnitTest1.cs is a minimal test which reproduces the issue.
      The attached file TestResults.trx is the mstest output showing the test failure.
      The attached file Results.jpg is a screenshot of the Jenkins results as displayed by the mstest plugin.
      This is pretty serious, we had a product failure but when we looked at the Jenkins tests all appeared fine. The problem only showed up when we ran the tests manually and they failed.

        1. Results.jpg
          91 kB
        2. TestResults.trx
          5 kB
        3. UnitTest1.cs
          0.4 kB

            nilleb Ivo Bellin Salarin
            jonbowen Jon Bowen
            0 Vote for this issue
            2 Start watching this issue