-
Bug
-
Resolution: Fixed
-
Major
-
None
we have a setup in which we run the same tests on windows and linux nodes. The test results are copied to the master into separate folders and a junit report is done for the base folder.
If, for example, linux fails and windows succeeds, I see 2 failed tests with the same name and in both tests I see the same log output, in my case the log output of the successful test.
This is quite annoying, since I see a test failure that absolutely does not match the log output, which looks completely successful.
I will setup a stripped down test scenario that demonstrates the problem.
- causes
-
-
- Closed
-
- depends on
-
JENKINS-27395 distinguish test reports filed from different parts of workflow
-
- Resolved
-
- duplicates
-
-
- Resolved
-
- relates to
-
-
- Closed
-
[JENKINS-41031] in Junit reports wrong output is shown for tests with same name
PS: you can also find this testsetup in my github repo: https://github.com/rbrindl/jenkins-41031-junit-reports. there is a branch for each scenario.
{
"_class" : "hudson.tasks.junit.TestResult",
"testActions" : [
],
"duration" : 30.0,
"empty" : false,
"failCount" : 1,
"passCount" : 1,
"skipCount" : 0,
"suites" : [
{
"cases" : [
{
"testActions" : [
],
"age" : 0,
"className" : "this.is.TheTest",
"duration" : 10.0,
"errorDetails" : null,
"errorStackTrace" : null,
"failedSince" : 0,
"name" : "myTest",
"skipped" : false,
"skippedMessage" : null,
"status" : "PASSED",
"stderr" : "\n\tNo error on Linux\n ",
"stdout" : "\n\tTest run on Linux\n "
},
{
"testActions" : [
],
"age" : 1,
"className" : "this.is.TheTest",
"duration" : 20.0,
"errorDetails" : "A Failure",
"errorStackTrace" : "The Stacktrace",
"failedSince" : 1,
"name" : "myTest",
"skipped" : false,
"skippedMessage" : null,
"status" : "FAILED",
"stderr" : "\n\tNo error on Linux\n ",
"stdout" : "\n\tTest run on Linux\n "
}
],
"duration" : 10.0,
"id" : null,
"name" : "this.is.TheTest",
"stderr" : "\n\tNo error on Linux\n ",
"stdout" : "\n\tTest run on Linux\n ",
"timestamp" : "2017-01-12T14:47:16"
}
]
}
{
"_class" : "hudson.tasks.junit.TestResult",
"testActions" : [
],
"duration" : 30.0,
"empty" : false,
"failCount" : 1,
"passCount" : 1,
"skipCount" : 0,
"suites" : [
{
"cases" : [
{
"testActions" : [
],
"age" : 1,
"className" : "this.is.TheTest",
"duration" : 20.0,
"errorDetails" : "A Failure",
"errorStackTrace" : "The Stacktrace",
"failedSince" : 1,
"name" : "myTest",
"skipped" : false,
"skippedMessage" : null,
"status" : "FAILED",
"stderr" : "\n\tError happened on windows\n ",
"stdout" : "\n\tTest run on windows\n "
},
{
"testActions" : [
],
"age" : 0,
"className" : "this.is.TheTest",
"duration" : 10.0,
"errorDetails" : null,
"errorStackTrace" : null,
"failedSince" : 0,
"name" : "myTest",
"skipped" : false,
"skippedMessage" : null,
"status" : "PASSED",
"stderr" : "\n\tError happened on windows\n ",
"stdout" : "\n\tTest run on windows\n "
}
],
"duration" : 20.0,
"id" : null,
"name" : "this.is.TheTest",
"stderr" : "\n\tError happened on windows\n ",
"stdout" : "\n\tTest run on windows\n ",
"timestamp" : "2017-01-12T14:48:16"
}
]
}
Closing as I think this was fixed in https://issues.jenkins-ci.org/browse/JENKINS-27395 (out in beta 1.4) - correct me if wrong.
I attached 2 files, which contain a folder junit with 2 subfolders. Those two subfolders contain a junit xml each, which has the same suitname, testclassname and testname, but they differ in the result, one is successful (lin), and one is failed (win).
And there is a Jenkinsfile in each of them which does a checkout and a junit report on those junit xml files.
I configured this to run in a multibranch pipeline build.
The 2 zip files are only different in the order of the junit folders, either 1-lin, 2-win or the other way round: 1-win, 2-lin.
The order is relevant for this problem, in one case you see the successful log output in both junit reports, in the other case you see the unsuccessful log in both reports.
I also added the junit json exported from the respective build, which clearly shows that both test results show the same log output.