-
Bug
-
Resolution: Not A Defect
-
Major
-
None
-
Jenkins 2.127 - xUnit plugin 2.0.2
We're building a lot of Drupal projects and run Drupal's Simpletest. Simpletest generates jUnit resultfiles - with emphasis on pluralis. At the moment, I'm testing a setup which produces 11 jUnit files.
xUnit fails to parse all of the files. Most of the time, it parses just a single file out of the 11, but I eventually had a single run yesterday which parsed 10 of the 11 files in a single run. What's even more confusing is the fact that the single file xUnit normally parses seems to be selected at random - I cannot figure out a pattern, at least.
The relevant snippet of my Jenkinsfile looks like this:
sh """
rm -rf simpletest
docker exec ${PRODUCT}-www-develop /bin/sh -c 'rm -rf /tmp/simpletest && mkdir -p /tmp/simpletest && chown www-data:www-data /tmp/simpletest'
docker exec ${PRODUCT}-www-develop /bin/sh -c 'cd /var/www && sudo -u www-data php scripts/run-tests-jenkins.sh --php /usr/bin/php --xml /tmp/simpletest --url ${testURL} "Netpunkt / Bibliotek.dk"' || true
docker cp ${PRODUCT}-www-develop:/tmp/simpletest .
"""
step([$class: 'XUnitBuilder', testTimeMargin: '3000', thresholdMode: 1,
thresholds: [
[$class: 'FailedThreshold', failureNewThreshold: '', failureThreshold: '0', unstableNewThreshold: '', unstableThreshold: ''],
[$class: 'SkippedThreshold', failureNewThreshold: '', failureThreshold: '', unstableNewThreshold: '', unstableThreshold: '']],
tools: [
[$class: 'JUnitType', deleteOutputFiles: false, failIfNotNew: true, pattern: 'simpletest/**', skipNoTestFiles: false, stopProcessingIfError: false]]
])
I have attached a screenshot showing five consecutive runs for which I just pressed "Build now" with no code changes what so ever. If you parse all 11 results files, there is more than 100 tests being run; but as you can see from the Testresult-trend graphic, build 104 found 11 tests, build 105 and 106 found 7 tests, build 107 found 8 tests, and build 108 found 10 tests.
For each run I have confirmed that all 11 result files are present in the "simpletest" directory.
The jUnit result generator used is coded by myself and I have confirmed that the jUnit files validates against the jUnit schema used by the xUnit plugin. All files parse whenever xUnit attempts to.
The job log shows the following:
[Simpletest] INFO: Starting to record. [Simpletest] INFO: Processing JUnit [Simpletest] INFO: [JUnit] - 11 test report file(s) were found with the pattern 'simpletest/**' relative to '/var/lib/jenkins/workspace/netpunkt-test-setup' for the testing framework 'JUnit'. [Simpletest] INFO: Check 'Failed Tests' threshold. [Simpletest] INFO: Check 'Skipped Tests' threshold. [Simpletest] INFO: Setting the build status to SUCCESS [Simpletest] INFO: Stopping recording.
As you can see, xUnit claims it has found all 11 files. But it only parsed a single file for this build.
- is related to
-
-
- Resolved
-
[JENKINS-52057] xUnit test result trend has random statistics when parse multiple results files
Update on this issue:
Having read JENKINS-52056 I thought that my issue might be related as I'm running the xUnit publisher from a parallel run on a special node.
However, I tried again, this time stashing the jUnit resultfiles on the node where the test was run and unstashing them in a new stage on the master node, calling xUnit from this last stage on the master. It works, but it is as unstable as ealier, ie. only publishing some of the jUnit files. 8 of 11 in the last build.
BTW, all 11 files are found in the "generatedJUnitFiles" folder after the build.
jUnit files attached: simpletest-junit.tar.gz
BTW: The code I'm using is written by me - but it is identical to the code running on Drupal CI making dozens of Drupal builds every day. That code was written by me as well - and it's parsed by an old version of xUnit, I believe. 
I miss something...
You have a pipeline project.
Some stages run in parallel. Each of them produces junit reports and than for each one you run also the xunit publish.
Are parallel stages in the same node?
Hi Nikolas,
Yes and no: I do run tests in parallel, but for now, I'm only calling xunit in one of the stages, xunit calls are removed from the other stages - just for testing. And, as I wrote in an earlier comment, the problem persists even when I defer xunit to a final, non-parallel stage in the pipeline.
The parallel stages are run on different nodes. As a matter of fact, I'm running each node in a Docker container using the Jenkins Docker plugin.
The current setup with a final, non-parallel stage is running xUnit directly on the master, though - which is running "normally", ie. not dockerized.
The issue happens because reports you are generating with your custom logger are inconsistent.
For example:
- BibdkCustomSearchWebTestCase has a testsuite element that reports 25 tests and 2 failures but is has just 1 failures and only 6 testcase
- TingAgencyWebTest has a testsuite element that reports 38 tests and 2 failure but is has no failures and only 5 testcase
xUnit use under the hood junit-plugin to parse reports and it get confused also because all testsuite has same name and same timestamp that cause JUnit-plugin parser to override testsuites in its internal map (JENKINS-48583).
The random test result trend depends on which is the first report file parsed (depends on the name of the file). xUnit performs a flat copy from slave to master using a UUID name instead of the original to avoid duplicate).
If you correct the JUnit reports (like here correct-junit-report.zip ) the trend of test result will appear stable like in the picture from build #53
One of causes why since 2.x I have restrict JUnit support to most famous Ant and Maven schemas is avoid spend hours to debug similar cases or reject support for creative attributes like skip/s instead of skipped or disabled and so on
Could you post the report generated by your customisation when tests are less than expected?
By default all project parse multiple files and this issue never happens, maybe the produced junit reports report a wrong number of test.