we have a setup in which we run the same tests on windows and linux nodes. The test results are copied to the master into separate folders and a junit report is done for the base folder.
If, for example, linux fails and windows succeeds, I see 2 failed tests with the same name and in both tests I see the same log output, in my case the log output of the successful test.
This is quite annoying, since I see a test failure that absolutely does not match the log output, which looks completely successful.
I will setup a stripped down test scenario that demonstrates the problem.
I attached 2 files, which contain a folder junit with 2 subfolders. Those two subfolders contain a junit xml each, which has the same suitname, testclassname and testname, but they differ in the result, one is successful (lin), and one is failed (win).
And there is a Jenkinsfile in each of them which does a checkout and a junit report on those junit xml files.
I configured this to run in a multibranch pipeline build.
The 2 zip files are only different in the order of the junit folders, either 1-lin, 2-win or the other way round: 1-win, 2-lin.
The order is relevant for this problem, in one case you see the successful log output in both junit reports, in the other case you see the unsuccessful log in both reports.
I also added the junit json exported from the respective build, which clearly shows that both test results show the same log output.
Ronald Brindl
added a comment - I attached 2 files, which contain a folder junit with 2 subfolders. Those two subfolders contain a junit xml each, which has the same suitname, testclassname and testname, but they differ in the result, one is successful (lin), and one is failed (win).
And there is a Jenkinsfile in each of them which does a checkout and a junit report on those junit xml files.
I configured this to run in a multibranch pipeline build.
The 2 zip files are only different in the order of the junit folders, either 1-lin, 2-win or the other way round: 1-win, 2-lin.
The order is relevant for this problem, in one case you see the successful log output in both junit reports, in the other case you see the unsuccessful log in both reports.
I also added the junit json exported from the respective build, which clearly shows that both test results show the same log output.
Ronald Brindl
added a comment - PS: you can also find this testsetup in my github repo: https://github.com/rbrindl/jenkins-41031-junit-reports . there is a branch for each scenario.
I attached 2 files, which contain a folder junit with 2 subfolders. Those two subfolders contain a junit xml each, which has the same suitname, testclassname and testname, but they differ in the result, one is successful (lin), and one is failed (win).
And there is a Jenkinsfile in each of them which does a checkout and a junit report on those junit xml files.
I configured this to run in a multibranch pipeline build.
The 2 zip files are only different in the order of the junit folders, either 1-lin, 2-win or the other way round: 1-win, 2-lin.
The order is relevant for this problem, in one case you see the successful log output in both junit reports, in the other case you see the unsuccessful log in both reports.
I also added the junit json exported from the respective build, which clearly shows that both test results show the same log output.