-
Bug
-
Resolution: Not A Defect
-
Blocker
-
None
It seems like the "testsuite" element in my junit.xml file that Jenkins aggregated is not correct.
Somehow, the "Test Result" page I see in Jenkins shows me the correct numbers - good job with that! However, when I examine the junit.xml file, the errors, failures, skips, and tests elements do not match what I see in Jenkins.
I use Jenkins parallel jobs to run my tests because I have thousands of python unit tests that are run with py.test. I use py.test's junitxml output file which I later ingest into jenkins.
My projects has over 5,000 tests. In a recent run, the junit file suggests there were 0 failures and 1 skips which is wrong....
<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skips="1" tests="33"....
However, the Jenkins rendering engine was able to correctly deduce that there really are 11 failures at 525 skipped tests as you can see in my attached screenshot.
I think the bug here is that as each parallel shard is ingested by the jenkins master, that the errors, failures, skips, and tests count needs to be updated.
[JENKINS-53236] junit.xml testsuite element shows incorrect number of failures when you run tests with Jenkins parallel jobs
Jon B
added a comment - I did not understand how to properly do this when running things in parallel.
In my app, I run 35 parallel test runners and I import the junit.xml result file with both `junit <filename>` as well as archiveArtifacts.
Whereas junit.xml is additive - it builds up a nice looking test results UI summary for the end user, archiveArtifacts is not additive.. it will overwrite the junit.xml file it seems.
In my case, the corrective action was to have the junit.xml file filename vary to something like junit1.xml, junit2.xml etc such that all of the original files are stored normally.
This must be a mistake a lot of people make.
Closing this ticket since this was ultimately user error on my part.
Jon B
added a comment - I did not understand how to properly do this when running things in parallel.
In my app, I run 35 parallel test runners and I import the junit.xml result file with both `junit <filename>` as well as archiveArtifacts.
Whereas junit.xml is additive - it builds up a nice looking test results UI summary for the end user, archiveArtifacts is not additive.. it will overwrite the junit.xml file it seems.
In my case, the corrective action was to have the junit.xml file filename vary to something like junit1.xml, junit2.xml etc such that all of the original files are stored normally.
This must be a mistake a lot of people make.
Closing this ticket since this was ultimately user error on my part. 
junitresults.xml generated when running the parallel job from jenkins shows incorrect number of failures.It is showing the total number of failed assertions rather than the total number of failed tests.
In the below screenshot,Total test is showing as 1 but failures are shown as 3.Marked red color shows the number of failures shown for each testcases and that is added to total failure count.We are parsing the same xml for our email report which shows the wrong count.Please derive a logic to handle this when multijobs are run on different machines from jenkins.
davidvanlaatum We are using groovy script to parse this xml. sample code.Due to the wrong failure count in the junitxml,complete report projecting wrong counts.
resultXML.testsuite.each{resultXML.testsuite.each{ def currentSuiteName = "${it.@name}".toString() def currentSuiteTests = "${it.@tests}".toInteger() def currentSuiteFailed = "${it.@failures}".toInteger() def currentSuiteSkipped = "${it.@skipped}".toInteger() def currentSuitePassed = currentSuiteTests - (currentSuiteFailed+currentSuiteSkipped) def currentPassPercentage = (((currentSuitePassed/currentSuiteTests)*10000).toString().split('\\.')[0].toInteger())/100 awsMap = awsMap+["$currentSuiteName":['Total':currentSuiteTests,'Passed':currentSuitePassed,'Failed':currentSuiteFailed,'Skipped':currentSuiteSkipped,'Percentage':currentPassPercentage]]