(I posted this question on the user's mailinglist before)

I recently started to use Opencover with my MSTest unit tests to get code coverage results. Since then the xunit plugin does not recognize errors as a Failure anymore. The TRX file is parsed and you can see at the summary page that there are tests failed and since when, but the job is counted as success and no email is triggered.

Could it be that there is some delay now introduced inbetween writing the MSTest results file and processing by xunit that causes this behaviour? When I rerun the tests without coverage after processing the coverage results everything works like it should. What I can see from the last versions of xunit plugin and a short look into the source code I think there should be a mechanism that if the test result file is older than 3 seconds there should be an error which can be ignored if configured. But even when the error because of an outdated file is ignored (because I make sure that the workspace is always cleaned after the running the tests) the Jenkins test job should fail if one or more of the unit tests fail.

I paste a log and if more info is needed like job config xml etc I will be happy to provide it.

18:44:02 D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw>"C:\Program Files\OpenCover\OpenCover.Console.exe" -target:"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools\..\IDE\MSTEST.EXE" -output:coverage.xml -targetargs:"/testcontainer:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\Bin\Debug\CommonFunctionsTests.dll /testsettings:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\localtestrun.testsettings /resultsfile:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\TestResults\CommonFunctionsTests\CommonFunctionsTests.trx" -mergebyhash -targetdir:"Bin\Debug" -filter:+[CommonFunctions*]*
18:44:03 Executing: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTEST.EXE
18:44:06 Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1
18:44:06 Copyright (c) Microsoft Corporation. All rights reserved.
18:44:06
18:44:06 Loading D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\localtestrun.testsettings...
18:44:07 Loading D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\Bin\Debug\CommonFunctionsTests.dll...
18:44:08 Starting execution...
18:44:18
18:44:18 Results Top Level Tests
18:44:18 ------- ---------------
18:44:18 Passed CommonFunctionsTest.BufferedTraceListenerTest.BufferedTraceListenerConstructorTest
[... Passes removed ...]
18:44:31 Passed CommonFunctionsTest.CommonFunctionsTest.ResizeTwoDimensionalArrayTest2
18:44:31 Failed CommonFunctionsTest.ContestConfigurationParametersTest.ContestConfigurationParametersFileTest
18:44:31 Passed CommonFunctionsTest.ContestSecurityStreamTest.ContestSecurityStreamConstructorTest
[... Passes removed ...]
18:44:59 Passed TestcaseInformationTest.TestcaseXmlElementTest.TestcaseXmlElementConstructorTest2
18:45:00 216/217 test(s) Passed, 1 Failed
18:45:00
18:45:00 Summary
18:45:00 -------
18:45:00 Test Run Failed.
18:45:00 Passed 216
18:45:00 Failed 1
18:45:00 -----------
18:45:00 Total 217
18:45:00 Results file: D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\TestResults\CommonFunctionsTests\CommonFunctionsTests.trx
18:45:00 Test Settings: localtestrun
18:45:00 Committing...
18:45:00 Visited Classes 70 of 114 (61.4)
18:45:00 Visited Methods 588 of 1049 (56.05)
18:45:00 Visited Points 5763 of 12043 (47.85)
18:45:00 Visited Branches 1629 of 4073 (40.00)
18:45:00
18:45:00 ==== Alternative Results (includes all methods including those without corresponding source) ====
18:45:00 Alternative Visited Classes 70 of 127 (55.12)
18:45:00 Alternative Visited Methods 676 of 1266 (53.4)
18:45:00
18:45:00 D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw>"C:\Program Files\ReportGenerator_1.8.1.0.\bin\ReportGenerator.exe" -reports:coverage.xml -targetdir:coverage
18:45:01 Loading report 'D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\coverage.xml'
18:45:01 Preprocessing report
18:45:02 Indexing classes in directory 'c:\Snapshots\ContestContinuous\contest_sw\CommonFunctions\CommonFunctions\'
18:45:03 Added coverage information of 86/90 auto properties to module 'CommonFunctions'
18:45:03 Initiating parser for OpenCover
18:45:03 Current Assembly: CommonFunctions
18:45:03 Current Assembly: CommonFunctionsTests
18:45:03 Analyzing 89 classes
18:45:03 Creating report 1/89 (Assembly: CommonFunctionsTests, Class: CommonFunctionsTest.CommonFunctionsTest)
[....]
18:45:04 Creating report 84/89 (Assembly: CommonFunctions, Class: TSTraceOrigin)
18:45:04 Creating summary
18:45:04 Report generation took 3 seconds
18:45:04 Archiving artifacts
18:45:04 [htmlpublisher] Archiving HTML reports...
18:45:04 [htmlpublisher] Archiving at BUILD level D:\WS\CommonFunctionsTests\ContestContinuous\contest_sw\coverage to D:\Jenkins\jobs\CommonFunctionsTests\builds\2013-05-24_18-43-55\htmlreports\Coverage_Report
18:45:06 [xUnit] [INFO] - Starting to record.
18:45:06 [xUnit] [INFO] - Processing MSTest-Version N/A (default)
18:45:06 [xUnit] [INFO] - [MSTest-Version N/A (default)] - 1 test report file(s) were found with the pattern 'ContestContinuous\contest_sw\TestResults\CommonFunctionsTests\CommonFunctionsTests.trx' relative to 'D:\WS\CommonFunctionsTests' for the testing framework 'MSTest-Version N/A (default)'.
18:45:06 [xUnit] [INFO] - Check 'Failed Tests' threshold.
18:45:06 [xUnit] [INFO] - Check 'Skipped Tests' threshold.
18:45:06 [xUnit] [INFO] - Setting the build status to SUCCESS
18:45:06 [xUnit] [INFO] - Stopping recording.
18:45:06 Recording fingerprints
18:45:06 No emails were triggered.
18:45:06
18:45:06 Deleting project workspace...
18:45:18 done
18:45:18
18:45:18 Notifying upstream projects of job completion
18:45:18 Finished: SUCCESS