Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52202

xUnit reports "All test reports are empty" in case of parallel steps but reports are not empty

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • xunit-plugin
    •  xUnit 2.0.3
      Jenkins ver. 2.126

      I have created a simple pipeline with two parallel steps.

      In each step I create a valid JUnit xml file and use xUnit to analyse it.

      During the run xunit can analyse one of the JUnit xml file, but for the other one it only logs out the message: WARNING: All test reports are empty.

      Attached pipeline.log log.txt

       

      Example pipeline:

      timestamps {
          node {
              deleteDir()
              sleep 1
              parallel(
                          dateiEins: {
              stage('generate xml file 1'){
                  dir('file1') {
                  dir('test-output') {
                      dir('junit') {
                                  writeFile file: 'TEST-de.edict.services.portaltests.SlotSessionTest_ES.xml', text: '''<?xml version="1.0" encoding="UTF-8"?>
                  <!-- Generated by de.edict.listener.JUnitReportReporterBase -->
                  <testsuite hostname="hhbld03t" name="de.edict.services.portaltests.SlotSessionTest--ES" tests="11" failures="0" timestamp="26 Jun 2018 13:51:32 GMT" time="265.564" errors="1">
                    <testcase name="testExpireSlotSessionWhileGameStartLossCondition--ES" time="17.388" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testConsistencyOfSlotsessionAndAccountDataAfterHeartbeat--ES" time="59.843" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testSlotSessionWinning--ES" time="21.539" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testExpireSlotSessionWhileGameStartTimeCondition--ES" time="17.597" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testSlotSessionLimitAmountExceeded--ES" time="13.583" classname="de.edict.services.portaltests.SlotSessionTest">
                      <error type="java.lang.AssertionError" message="Expecting message type SLOT_LOSS_LIMIT_EXCEEDED in Response.
                  Expected: is &quot;SLOT_LOSS_LIMIT_EXCEEDED&quot;
                       but: was &quot;ACCOUNT_EMPTY&quot;">
                        <![CDATA[java.lang.AssertionError: Expecting message type SLOT_LOSS_LIMIT_EXCEEDED in Response.
                  Expected: is "SLOT_LOSS_LIMIT_EXCEEDED"
                       but: was "ACCOUNT_EMPTY"
                  	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
                  	at de.edict.services.portaltests.SlotSessionTest.testSlotSessionLimitAmountExceeded(SlotSessionTest.java:288)
                  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  	at java.lang.reflect.Method.invoke(Method.java:498)
                  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
                  	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
                  	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
                  	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
                  	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
                  	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
                  	at org.testng.TestRunner.privateRun(TestRunner.java:767)
                  	at org.testng.TestRunner.run(TestRunner.java:617)
                  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
                  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
                  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
                  	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
                  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
                  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
                  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
                  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
                  	at org.testng.TestNG.run(TestNG.java:1057)
                  	at org.testng.TestNG.privateMain(TestNG.java:1364)
                  	at org.testng.TestNG.main(TestNG.java:1333)
                  ]]>
                      </error>
                    </testcase> <!-- testSlotSessionLimitAmountExceeded-ES -->
                    <testcase name="testConsistencyOfSlotsessionAndAccountData--ES" time="18.456" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testCreateSlotSessionWithGameStart--ES" time="2.897" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testExpireSlotSessionWhileHeartbeat--ES" time="35.558" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testCheckPortalRestService--ES" time="28.069" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testCheckBORestService--ES" time="17.425" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testLossLimitWarning--ES" time="33.209" classname="de.edict.services.portaltests.SlotSessionTest"/>
                  </testsuite> <!-- de.edict.services.portaltests.SlotSessionTest-ES -->
                  '''
                  sh 'ls -l'
                  xunit testTimeMargin: '3000', thresholdMode: 1, thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0'), skipped()], tools: [JUnit(deleteOutputFiles: false, failIfNotNew: false, pattern: 'TEST-*.xml', skipNoTestFiles: false, stopProcessingIfError: true)]
                      }
                  }
              }
          }
      },
      dateiZwei: {
          stage('generate xml file 2'){
              dir('file2') {
                  dir('test-output') {
                      dir('junit') {                    writeFile file: 'TEST-de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default.xml', text: '''<?xml version="1.0" encoding="UTF-8"?>
                          <!-- Generated by de.edict.listener.JUnitReportReporterBase -->
                          <testsuite hostname="hhbld01t" name="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default" tests="15" failures="1" timestamp="26 Jun 2018 15:41:26 GMT" time="55.391" errors="0">
                            <testcase name="testCreateBonusOfferWithCountryRestrictions--IoM_default" time="12.167" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayInWith50EURWithVoucherCode--IoM_default" time="1.454" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferTrigger999--IoM_default" time="2.114" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testEveryPayInDuringThePeriodWithVisa--IoM_default" time="26.713" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests">
                              <failure type="com.googlecode.sardine.impl.SardineException" message="Unexpected response">
                                <![CDATA[com.googlecode.sardine.impl.SardineException: Unexpected response
                          	at com.googlecode.sardine.impl.handler.ValidatingResponseHandler.validateResponse(ValidatingResponseHandler.java:49)
                          	at com.googlecode.sardine.impl.handler.VoidResponseHandler.handleResponse(VoidResponseHandler.java:35)
                          	at com.googlecode.sardine.impl.SardineImpl.get(SardineImpl.java:478)
                          	at com.googlecode.sardine.impl.SardineImpl.get(SardineImpl.java:458)
                          	at de.edict.eoc.test.utils.TextHelper.connectWikiWebdav(TextHelper.java:75)
                          	at de.edict.eoc.test.bonus.BonusOfferTestHelper.checkDefaultStakerestrictionsAgainstWiki(BonusOfferTestHelper.java:339)
                          	at de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests.testEveryPayInDuringThePeriodWithVisa(BonusOfferDepositTests.java:178)
                          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                          	at java.lang.reflect.Method.invoke(Method.java:498)
                          	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
                          	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
                          	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
                          	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
                          	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
                          	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
                          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                          	at java.lang.Thread.run(Thread.java:748)
                          ]]>
                              </failure>
                            </testcase> <!-- testEveryPayInDuringThePeriodWithVisa-IoM_default -->
                            <testcase name="TryToCreateBonusWithoutCorrectPermission--IoM_default" time="0.610" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testBonusOfferWithSpecialCharacter--IoM_default" time="3.118" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayinWithOneHourDuration--IoM_default" time="0.693" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testBonusOfferWithSpecialCharacterMinus--IoM_default" time="0.501" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayinWithoutStartAndEnddate--IoM_default" time="0.542" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferWithCountryAndPaymentRestrictions--IoM_default" time="2.136" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testEveryfifthPayInWithVisa--IoM_default" time="0.675" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testEveryPayInDuringThePeriod--IoM_default" time="1.956" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayInWith50EUR--IoM_default" time="0.706" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferSartTodayTop5Restrictions--IoM_default" time="1.055" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferAllMax--IoM_default" time="0.951" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                          </testsuite> <!-- de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests-IoM_default -->
                          '''
                          
                                              sh 'ls -l'
                                              xunit testTimeMargin: '3000', thresholdMode: 1, thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0'), skipped()], tools: [JUnit(deleteOutputFiles: false, failIfNotNew: false, pattern: "TEST-*.xml", skipNoTestFiles: false, stopProcessingIfError: true)]
                                          }
                                      }
                                  }
                              }
                          }
                  )
                          
          
              }
          } 

          [JENKINS-52202] xUnit reports "All test reports are empty" in case of parallel steps but reports are not empty

          I would like to raise this problem to major - at least: This bug has been introduced with the upgrade from 2.0.2 to 2.0.3 and it's not only cosmetical.

          It also means that xUnit don't fail build jobs when it finds failures in the jUnit resultfiles - meaning that continuous deployments will deploy because they're not stopped by xUnit.

          I can add that even though xUnit warns that all test reports are empty, they're parsed correctly and can be seen in the usual Jenkins testresult view for the job.

          Lars Skjærlund added a comment - I would like to raise this problem to major - at least: This bug has been introduced with the upgrade from 2.0.2 to 2.0.3 and it's not only cosmetical. It also means that xUnit don't fail build jobs when it finds failures in the jUnit resultfiles - meaning that continuous deployments will deploy because they're not stopped by xUnit. I can add that even though xUnit warns that all test reports are empty, they're parsed correctly and can be seen in the usual Jenkins testresult view for the job.

          Nikolas Falco added a comment - - edited

          I agree, I will look this weekend.

          Nikolas Falco added a comment - - edited I agree, I will look this weekend.

          Nikolas Falco added a comment -

          With Jenkins 1.129, all plugins updated expect junit 1.20
          I'm not able to reproduce this issue. With your script I got this log:

          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] node
          Running on Jenkins in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline
          [Pipeline] {
          [Pipeline] deleteDir
          [Pipeline] sleep
          Sleeping for 1 s
          [Pipeline] parallel
          [Pipeline] [dateiEins] { (Branch: dateiEins)
          [Pipeline] [dateiZwei] { (Branch: dateiZwei)
          [Pipeline] [dateiEins] stage
          [Pipeline] [dateiEins] { (generate xml file 1)
          [Pipeline] [dateiZwei] stage
          [Pipeline] [dateiZwei] { (generate xml file 2)
          [Pipeline] [dateiEins] dir
          [dateiEins] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1
          [Pipeline] [dateiEins] {
          [Pipeline] [dateiZwei] dir
          [dateiZwei] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2
          [Pipeline] [dateiZwei] {
          [Pipeline] [dateiEins] writeFile
          [Pipeline] [dateiZwei] writeFile
          [Pipeline] [dateiEins] sh
          [dateiEins] [file1] Running shell script
          [Pipeline] [dateiZwei] sh
          [dateiEins] + ls -l
          [dateiEins] total 16
          [dateiEins] -rw-r--r--  1 nikolasfalco  staff  4511 Jun 30 12:24 TEST-de.edict.services.portaltests.SlotSessionTest_ES.xml
          [dateiZwei] [file2] Running shell script
          [dateiZwei] + ls -l
          [dateiZwei] total 16
          [dateiZwei] -rw-r--r--  1 nikolasfalco  staff  5665 Jun 30 12:24 TEST-de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default.xml
          [Pipeline] [dateiEins] xunit
          [dateiEins] INFO: Starting to record.
          [dateiEins] INFO: Processing JUnit
          [dateiEins] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1' for the testing framework 'JUnit'.
          [dateiEins] INFO: Check 'Failed Tests' threshold.
          [dateiEins] INFO: The total number of tests for this category exceeds the specified "failure" threshold value.
          [dateiEins] INFO: Setting the build status to FAILURE
          [dateiEins] INFO: Stopping recording.
          [Pipeline] [dateiZwei] xunit
          [dateiZwei] INFO: Starting to record.
          [dateiZwei] INFO: Processing JUnit
          [dateiZwei] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2' for the testing framework 'JUnit'.
          [Pipeline] [dateiEins] }
          [Pipeline] [dateiEins] // dir
          [Pipeline] [dateiEins] }
          [Pipeline] [dateiZwei] }
          [Pipeline] [dateiEins] // stage
          [Pipeline] [dateiZwei] // dir
          [Pipeline] [dateiEins] }
          [Pipeline] [dateiZwei] }
          [Pipeline] [dateiZwei] // stage
          [Pipeline] [dateiZwei] }
          [dateiZwei] Failed in branch dateiZwei
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          

          In case could you please attach a zip that contains the content of your workspace folder produced by the example you post in the issue?

          Nikolas Falco added a comment - With Jenkins 1.129, all plugins updated expect junit 1.20 I'm not able to reproduce this issue. With your script I got this log: Running in Durability level: MAX_SURVIVABILITY [Pipeline] node Running on Jenkins in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline [Pipeline] { [Pipeline] deleteDir [Pipeline] sleep Sleeping for 1 s [Pipeline] parallel [Pipeline] [dateiEins] { (Branch: dateiEins) [Pipeline] [dateiZwei] { (Branch: dateiZwei) [Pipeline] [dateiEins] stage [Pipeline] [dateiEins] { (generate xml file 1) [Pipeline] [dateiZwei] stage [Pipeline] [dateiZwei] { (generate xml file 2) [Pipeline] [dateiEins] dir [dateiEins] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1 [Pipeline] [dateiEins] { [Pipeline] [dateiZwei] dir [dateiZwei] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2 [Pipeline] [dateiZwei] { [Pipeline] [dateiEins] writeFile [Pipeline] [dateiZwei] writeFile [Pipeline] [dateiEins] sh [dateiEins] [file1] Running shell script [Pipeline] [dateiZwei] sh [dateiEins] + ls -l [dateiEins] total 16 [dateiEins] -rw-r--r-- 1 nikolasfalco staff 4511 Jun 30 12:24 TEST-de.edict.services.portaltests.SlotSessionTest_ES.xml [dateiZwei] [file2] Running shell script [dateiZwei] + ls -l [dateiZwei] total 16 [dateiZwei] -rw-r--r-- 1 nikolasfalco staff 5665 Jun 30 12:24 TEST-de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default.xml [Pipeline] [dateiEins] xunit [dateiEins] INFO: Starting to record. [dateiEins] INFO: Processing JUnit [dateiEins] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1' for the testing framework 'JUnit'. [dateiEins] INFO: Check 'Failed Tests' threshold. [dateiEins] INFO: The total number of tests for this category exceeds the specified "failure" threshold value. [dateiEins] INFO: Setting the build status to FAILURE [dateiEins] INFO: Stopping recording. [Pipeline] [dateiZwei] xunit [dateiZwei] INFO: Starting to record. [dateiZwei] INFO: Processing JUnit [dateiZwei] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2' for the testing framework 'JUnit'. [Pipeline] [dateiEins] } [Pipeline] [dateiEins] // dir [Pipeline] [dateiEins] } [Pipeline] [dateiZwei] } [Pipeline] [dateiEins] // stage [Pipeline] [dateiZwei] // dir [Pipeline] [dateiEins] } [Pipeline] [dateiZwei] } [Pipeline] [dateiZwei] // stage [Pipeline] [dateiZwei] } [dateiZwei] Failed in branch dateiZwei [Pipeline] // parallel [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline In case could you please attach a zip that contains the content of your workspace folder produced by the example you post in the issue?

          Nikolas Falco added a comment -

          I got it. It happens on next builds

          Nikolas Falco added a comment - I got it. It happens on next builds

          Slavko Fodor added a comment -

          How did you resolve this? I am seeing a similar problem when running Unity3d tests in play mode - it generates the xml files for test results with all the data, but reports "All test reports are empty."

          I am attaching the test results as well. https://pastebin.com/BfhaJj0M

          Slavko Fodor added a comment - How did you resolve this? I am seeing a similar problem when running Unity3d tests in play mode - it generates the xml files for test results with all the data, but reports "All test reports are empty." I am attaching the test results as well. https://pastebin.com/BfhaJj0M

            nfalco Nikolas Falco
            sfin Sven Finsterwalder
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: