Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18696

Not all test results are being reported

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Our build process produces multiple directories of JUnit result XML files. The workspace structure looks like this:

         build/
            integration-a/
               test-results/
            integration-b/
               test-results/
            integration-c/
               test-results/
            integration-d/
               test-results/
      

      The project is configured to collect the test results using the following file path:

      build/integration-*/test-results/*.xml

      I have verified looking in the workspace after the build that all the XMl files are there and that all the tests have run.

      The problem is that Jenkins is reporting slightly different counts of unit tests every build. I've looked back over older builds and this has been happening for months (could be even longer, but I just switched servers and lost my older history). For example, the last few build test counts have looked like this (the number of tests should not have changed at all):

      #23 18,875 tests (+210)
      #22 18,665 tests (±0)
      #21 18,665 tests (+204)
      #20 18,461 tests (-401)
      #19 18,862 tests (+186)
      #18 18,676 tests (-147)
      #17 18,823 tests (-39)
      

      When I review the test results in Jenkins, I find every test that it doesn't list. When I look at the workspace, I can find the XML file and there are also HTML files (in build/integration-*/reports/tests/) that are produced and all the tests are there as well. It just seems Jenkins is dropping various test XML files on each build. I just can't see a pattern. The count seems different every time.

        Attachments

          Issue Links

            Activity

            Hide
            mcantin Mathieu Cantin added a comment -

            We have the same issue on windows server 2012R2 with jenkins 1.596.2 and JUnit plugin 1.5.

            Show
            mcantin Mathieu Cantin added a comment - We have the same issue on windows server 2012R2 with jenkins 1.596.2 and JUnit plugin 1.5.
            Hide
            humer015 Aaron Humerickhouse added a comment - - edited

            Have the same issue. I was on 1.2-beta4 (default with Jenkins), just updated to 1.5 and am seeing the same issue.

            Files
            selenium-tests
            -results
            --results0.xml
            --results1.xml
            --results2.xml
            --results3.xml
            --results4.xml

            I've tried:
            **/results*.xml
            selenium-tests/results/results*.xml
            **/results/results.xml
            selenium-tests/results/results0.xml,selenium-tests/results/results1.xml,selenium-tests/results/results2.xml,selenium-tests/results/results3.xml,selenium-tests/results/results4.xml

            Four different patterns that should get me all of the files are failing to do so.

            Show
            humer015 Aaron Humerickhouse added a comment - - edited Have the same issue. I was on 1.2-beta4 (default with Jenkins), just updated to 1.5 and am seeing the same issue. Files selenium-tests -results --results0.xml --results1.xml --results2.xml --results3.xml --results4.xml I've tried: **/results*.xml selenium-tests/results/results*.xml **/results/results.xml selenium-tests/results/results0.xml,selenium-tests/results/results1.xml,selenium-tests/results/results2.xml,selenium-tests/results/results3.xml,selenium-tests/results/results4.xml Four different patterns that should get me all of the files are failing to do so.
            Hide
            khneal Kevin Neal added a comment -

            Usually the newer issue duplicates the older issue, but I found that one first and submitted a PR today. This issue has a PR open, but it's out of date.

            Show
            khneal Kevin Neal added a comment - Usually the newer issue duplicates the older issue, but I found that one first and submitted a PR today. This issue has a PR open, but it's out of date.

              People

              Assignee:
              khneal Kevin Neal
              Reporter:
              mayc Chuck May
              Votes:
              4 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: