We're building a lot of Drupal projects and run Drupal's Simpletest. Simpletest generates jUnit resultfiles - with emphasis on pluralis. At the moment, I'm testing a setup which produces 11 jUnit files.

xUnit fails to parse all of the files. Most of the time, it parses just a single file out of the 11, but I eventually had a single run yesterday which parsed 10 of the 11 files in a single run. What's even more confusing is the fact that the single file xUnit normally parses seems to be selected at random - I cannot figure out a pattern, at least.

The relevant snippet of my Jenkinsfile looks like this:

sh """
 rm -rf simpletest
 docker exec ${PRODUCT}-www-develop /bin/sh -c 'rm -rf /tmp/simpletest && mkdir -p /tmp/simpletest && chown www-data:www-data /tmp/simpletest'
 docker exec ${PRODUCT}-www-develop /bin/sh -c 'cd /var/www && sudo -u www-data php scripts/run-tests-jenkins.sh --php /usr/bin/php --xml /tmp/simpletest --url ${testURL} "Netpunkt / Bibliotek.dk"' || true
 docker cp ${PRODUCT}-www-develop:/tmp/simpletest .
 """
step([$class: 'XUnitBuilder', testTimeMargin: '3000', thresholdMode: 1,
 thresholds: [
 [$class: 'FailedThreshold', failureNewThreshold: '', failureThreshold: '0', unstableNewThreshold: '', unstableThreshold: ''],
 [$class: 'SkippedThreshold', failureNewThreshold: '', failureThreshold: '', unstableNewThreshold: '', unstableThreshold: '']],
 tools: [
 [$class: 'JUnitType', deleteOutputFiles: false, failIfNotNew: true, pattern: 'simpletest/**', skipNoTestFiles: false, stopProcessingIfError: false]]
 ])

I have attached a screenshot showing five consecutive runs for which I just pressed "Build now" with no code changes what so ever. If you parse all 11 results files, there is more than 100 tests being run; but as you can see from the Testresult-trend graphic, build 104 found 11 tests, build 105 and 106 found 7 tests, build 107 found 8 tests, and build 108 found 10 tests.

For each run I have confirmed that all 11 result files are present in the "simpletest" directory.

The jUnit result generator used is coded by myself and I have confirmed that the jUnit files validates against the jUnit schema used by the xUnit plugin. All files parse whenever xUnit attempts to.

The job log shows the following:

[Simpletest] INFO: Starting to record.
 [Simpletest] INFO: Processing JUnit
 [Simpletest] INFO: [JUnit] - 11 test report file(s) were found with the pattern 'simpletest/**' relative to '/var/lib/jenkins/workspace/netpunkt-test-setup' for the testing framework 'JUnit'.
 [Simpletest] INFO: Check 'Failed Tests' threshold.
 [Simpletest] INFO: Check 'Skipped Tests' threshold.
 [Simpletest] INFO: Setting the build status to SUCCESS
 [Simpletest] INFO: Stopping recording.

As you can see, xUnit claims it has found all 11 files. But it only parsed a single file for this build.