Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36759

MSTest Plugin reports failed data driven tests as passed in certain situations.

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • mstest-plugin
    • None
    • Windows 2012, Jenkins 1.653, mstest-plugin 0.19

    Description

      If a data driven test fails before ever running the test - for example, if the data file cannot be found - then the "failed" outcome in the .trx output file is not reported at all. It is not even included in the test counts. Also, the Jenkins Job passes as Stable, so it appears to everybody that everything is OK, whereas at best the test didn't run at all.
      The attached file UnitTest1.cs is a minimal test which reproduces the issue.
      The attached file TestResults.trx is the mstest output showing the test failure.
      The attached file Results.jpg is a screenshot of the Jenkins results as displayed by the mstest plugin.
      This is pretty serious, we had a product failure but when we looked at the Jenkins tests all appeared fine. The problem only showed up when we ran the tests manually and they failed.

      Attachments

        1. Results.jpg
          Results.jpg
          91 kB
        2. TestResults.trx
          5 kB
        3. UnitTest1.cs
          0.4 kB

        Activity

          jonbowen Jon Bowen created issue -
          jonbowen Jon Bowen made changes -
          Field Original Value New Value
          Attachment Pipeline hang after user abort.jpg [ 33343 ]
          jonbowen Jon Bowen made changes -
          Attachment Pipeline hang after user abort.jpg [ 33343 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 173492 ] JNJira + In-Review [ 185153 ]
          nilleb Ivo Bellin Salarin made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          nilleb Ivo Bellin Salarin added a comment - Fixed in  https://github.com/jenkinsci/mstest-plugin/commit/eb2cd6c07520ae2a5d4fd9e1f2d3d7569d445bb9
          nilleb Ivo Bellin Salarin made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          nilleb Ivo Bellin Salarin made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]

          People

            nilleb Ivo Bellin Salarin
            jonbowen Jon Bowen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: