-
Bug
-
Resolution: Not A Defect
-
Blocker
-
None
It seems like the "testsuite" element in my junit.xml file that Jenkins aggregated is not correct.
Somehow, the "Test Result" page I see in Jenkins shows me the correct numbers - good job with that! However, when I examine the junit.xml file, the errors, failures, skips, and tests elements do not match what I see in Jenkins.
I use Jenkins parallel jobs to run my tests because I have thousands of python unit tests that are run with py.test. I use py.test's junitxml output file which I later ingest into jenkins.
My projects has over 5,000 tests. In a recent run, the junit file suggests there were 0 failures and 1 skips which is wrong....
<?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skips="1" tests="33"....
However, the Jenkins rendering engine was able to correctly deduce that there really are 11 failures at 525 skipped tests as you can see in my attached screenshot.
I think the bug here is that as each parallel shard is ingested by the jenkins master, that the errors, failures, skips, and tests count needs to be updated.
[JENKINS-53236] junit.xml testsuite element shows incorrect number of failures when you run tests with Jenkins parallel jobs
Jon B
created issue -
Jon B
made changes -
| Description |
Original:
It seems like the "testsuite" element in my junit.xml file is not correct. Somehow, the "Test Result" page I see in Jenkins shows me the correct numbers - good job with that! However, when I examine the junit.xml file, the errors, failures, skips, and tests elements do not match what I see in Jenkins. I use Jenkins parallel jobs to run my tests because I have thousands of python unit tests that are run with py.test. I use py.test's junitxml output file which I later ingest into jenkins. My projects has over 5,000 tests. In a recent run, the junit file suggests there were 0 failures and 1 skips which is wrong.... {code:java} <?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skips="1" tests="33".... {code} However, the Jenkins rendering engine was able to correctly deduce that there really are 11 failures at 525 skipped tests as you can see in my attached screenshot. I think the bug here is that as each parallel shard is ingested by the jenkins master, that the errors, failures, skips, and tests count needs to be updated. !image-2018-08-25-09-13-05-479.png! |
New:
It seems like the "testsuite" element in my junit.xml file that Jenkins aggregated is not correct. Somehow, the "Test Result" page I see in Jenkins shows me the correct numbers - good job with that! However, when I examine the junit.xml file, the errors, failures, skips, and tests elements do not match what I see in Jenkins. I use Jenkins parallel jobs to run my tests because I have thousands of python unit tests that are run with py.test. I use py.test's junitxml output file which I later ingest into jenkins. My projects has over 5,000 tests. In a recent run, the junit file suggests there were 0 failures and 1 skips which is wrong.... {code:java} <?xml version="1.0" encoding="utf-8"?><testsuite errors="0" failures="0" name="pytest" skips="1" tests="33".... {code} However, the Jenkins rendering engine was able to correctly deduce that there really are 11 failures at 525 skipped tests as you can see in my attached screenshot. I think the bug here is that as each parallel shard is ingested by the jenkins master, that the errors, failures, skips, and tests count needs to be updated. !image-2018-08-25-09-13-05-479.png! |
Nikitha Susheem
made changes -
| Priority | Original: Minor [ 4 ] | New: Blocker [ 1 ] |
Nikitha Susheem
made changes -
| Attachment | New: Multijob run_Junitsxml.png [ 44624 ] |
Nikitha Susheem
made changes -
| Assignee | New: David van Laatum [ davidvanlaatum ] |
Nikitha Susheem
made changes -
| Rank | New: Ranked higher |
Nikitha Susheem
made changes -
| Rank | New: Ranked higher |
Jon B
made changes -
| Resolution | New: Not A Defect [ 7 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |