Andrew Bayer added a comment - 2017-11-15 14:30 Any chance you could post your Jenkinsfile and your test reports xml?

Andrew Bayer added a comment - 2017-11-15 14:49 Ok, got one part - the failed tests lists don't include the stage name. If you drill down in the full test list to a particular test class, you'll see the test names including the stage name, but not in the list of failed tests, or the list of packages or classes. The latter two are sadly just what we're stuck with - it's entirely possible for a package or class result to contain tests from multiple stages, so there's no viable way I've found yet to connect package/class results with a specific stage. The failed tests, though, should have the stage included, but they pull their display name differently than in the case list, so I need to fix that. I can't reproduce the warnings, though - as mentioned, your Jenkinsfile, and particularly your test results XML file that the step is reading, would be hugely helpful for that.

Andrew Bayer added a comment - 2017-11-15 15:04 Check that, got the warning. It shows up when getting trends. Working on it.

Reinhold Füreder added a comment - 2017-11-15 15:07 Well, the Jenkinsfile actually depends heavily on a shared pipeline library, but the basic principle is that in two different stages ("docker-unit-tests" and "docker-integration-tests") PHPUnit (v6.4.3) is executed in a Docker Container via " phpunit --log-junit phpunit-result.xml ... " ... and the resulting XML file processed like this: def phpUnitXmlFilename = 'phpunit-result.xml' ... junit(keepLongStdio: true , testResults: phpUnitXmlFilename) (Maybe mind that it is "only" one single XML file in each run, and that the filename does not change.) Which of the test reports XML do you need: the one written by Jenkins in "/var/lib/jenkins/jobs/<job name>/builds/<build number>/junitResult.xml" or the PHPUnit output files? The former one looks like this: <?xml version='1.0' encoding='UTF-8'?> <result plugin="junit@1.22.1"> <suites> <suite> <file>/var/lib/jenkins/workspace/ACME-Pipeline-WS/phpunit-result.xml</file> <name>ACME_Accounting_Accountings_FileImporters_UBSTest</name> <duration>0.009598</duration> <time>0.009598</time> <nodeId>51</nodeId> <enclosingBlocks> <string>34</string> </enclosingBlocks> <enclosingBlockNames> <string>docker-unit-tests</string> </enclosingBlockNames> <cases> <case> <duration>0.00624</duration> <className>ACME_Accounting_Accountings_FileImporters_UBSTest</className> <testName>testCheckFile</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> <case> <duration>0.002944</duration> <className>ACME_Accounting_Accountings_FileImporters_UBSTest</className> <testName>testImport</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> <case> <duration>4.14E-4</duration> <className>ACME_Accounting_Accountings_FileImporters_UBSTest</className> <testName>testPossibleTenantsKeys</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> </cases> </suite> <suite> <file>/var/lib/jenkins/workspace/ACME-Pipeline-WS/phpunit-result.xml</file> <name>ACME_Accounting_Export_Abacus_GeneratorTest</name> ... <suite> <file>/var/lib/jenkins/workspace/ACME-Pipeline-WS/phpunit-result.xml</file> <name>TestFramework_DbFixtureGenerator_MeasuringModelBuilderTest</name> <duration>0.017027</duration> <time>0.017027</time> <nodeId>80</nodeId> <enclosingBlocks> <string>61</string> </enclosingBlocks> <enclosingBlockNames> <string>docker-integration-tests</string> </enclosingBlockNames> <cases> <case> <duration>0.004158</duration> <className>TestFramework.DbFixtureGenerator.MeasuringModelBuilderTest</className> <testName>testCreate</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> <case> <duration>0.00341</duration> <className>TestFramework.DbFixtureGenerator.MeasuringModelBuilderTest</className> <testName>testCreateNotCurrent</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> <case> <duration>0.004291</duration> <className>TestFramework.DbFixtureGenerator.MeasuringModelBuilderTest</className> <testName>testCreateWithOneDefaultPoint</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> <case> <duration>0.005168</duration> <className>TestFramework.DbFixtureGenerator.MeasuringModelBuilderTest</className> <testName>testCreateWithPoints</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> </cases> </suite> </suites> <duration>16.892485</duration> <keepLongStdio>true</keepLongStdio> </result> The latter ones I don't have now (they are thrown away), but I could of course try to get them if needed... Oh, I just found one from yesterday which should IMHO still be up-to-date: <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="./unittests/unit/" tests="482" assertions="925" errors="0" failures="0" skipped="0" time="0.780566"> <testsuite name="ACME_Accounting_Accountings_FileImporters_UBSTest" file="/var/lib/jenkins/workspace/ACME-Pipeline-WS@3/unittests/unit/app/admin/classes/accounting/accountings/fileimporters/ubsTest.php" tests="3" assertions="76" errors="0" failures="0" skipped="0" time="0.010088"> <testcase name="testCheckFile" class="ACME_Accounting_Accountings_FileImporters_UBSTest" classname="ACME_Accounting_Accountings_FileImporters_UBSTest" file="/var/lib/jenkins/workspace/ACME-Pipeline-WS@3/unittests/unit/app/admin/classes/accounting/accountings/fileimporters/ubsTest.php" line="11" assertions="2" time="0.006981"/> <testcase name="testImport" class="ACME_Accounting_Accountings_FileImporters_UBSTest" classname="ACME_Accounting_Accountings_FileImporters_UBSTest" file="/var/lib/jenkins/workspace/ACME-Pipeline-WS@3/unittests/unit/app/admin/classes/accounting/accountings/fileimporters/ubsTest.php" line="21" assertions="73" time="0.002706"/> <testcase name="testPossibleTenantsKeys" class="ACME_Accounting_Accountings_FileImporters_UBSTest" classname="ACME_Accounting_Accountings_FileImporters_UBSTest" file="/var/lib/jenkins/workspace/ACME-Pipeline-WS@3/unittests/unit/app/admin/classes/accounting/accountings/fileimporters/ubsTest.php" line="45" assertions="1" time="0.000401"/> </testsuite> <testsuite name="ACME\Accounting\Export\Abacus\GeneratorTest" file="/var/lib/jenkins/workspace/ACME-Pipeline-WS@3/unittests/unit/app/admin/classes/accounting/export/abacus/GeneratorTest.php" tests="1" assertions="1" errors="0" failures="0" skipped="0" time="0.013121"> ... </testsuite> </testsuites>

Andrew Bayer added a comment - 2017-11-15 15:12 Perfect - thanks!

Reinhold Füreder added a comment - 2017-11-15 15:15 - edited Frankly I am not sure that I understood your comment in https://issues.jenkins-ci.org/browse/JENKINS-48020?focusedCommentId=319785&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-319785 Note that in my example: there were no failing tests there are definitely different test cases in same test classes (= test classes with same name) executed in the two different PHPUnit runs (filtered executions based on " @group " test annotations) but I would strongly assume there are some "unique" test classes per PHPUnit run as well

Andrew Bayer added a comment - 2017-11-15 15:28 Basically, I was saying that the stage info isn't shown except for on individual tests - so, not in the list of packages on the initial test results screen, not on the list of classes when you click on one of the packages. You will see it in the list of cases within an individual test class, though. And with https://github.com/jenkinsci/junit-plugin/pull/89 , the lists of failed tests will have the stage info as well. (this is all in the classic UI - Blue Ocean isn't yet updated to take advantage of this information)

Andrew Bayer added a comment - 2017-11-15 15:29 Also, I slept badly last night so may not be fully coherent. My apologies. =)

Reinhold Füreder added a comment - 2017-11-15 15:42 OK, now I got it, thanks for your patience; and I can actually see/confirm that ("shown for individual tests") Just out of interest: are there any plans to pimp the visualization and not merge all test results together, but – frankly I don't know what the best UX would be – have one TestResult tab/entry per run or stage or so?  

Andrew Bayer added a comment - 2017-11-15 15:48 Maaaaybe? That's definitely on the roadmap for Blue Ocean, and I'll see if I can figure out something that makes sense for classic UI as well.

Andrew Bayer added a comment - 2017-11-15 18:13 Releasing 1.22.2 with this fix right now.

SCM/JIRA link daemon added a comment - 2017-11-15 18:13 Code changed in jenkins User: Andrew Bayer Path: src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/SuiteResult.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java http://jenkins-ci.org/commit/junit-plugin/9ec93955137b9eeba7715e6f118e558be7dbb712 Log: [FIXED JENKINS-48020] Show stage in failed tests, fix warnings First, this changes `CaseResult#getFullDisplayName()` to include stage/branch names like `CaseResult#getDisplayName()`, so that the full information will show up in lists of failed tests, for example. Second, stop using `CaseResult#getDisplayName()` for indexing `CaseResult`s in `SuiteResult`. That method will spam warnings if called too early in the run now due to `CaseResult#getRun()` being called before the `SuiteResult`'s parent is set. Third, fix `JUnitResultsStepTest#testTrends` to actually test what it's supposed to, to fail if it doesn't find the expected display names, to test full display names as well, and to use the same filename for the test results each time due to one of the stages' test file not containing an explicit suite name, leading to its suite's name being set to the filename, breaking `SuiteResult#getPreviousResult()`.

Reinhold Füreder added a comment - 2017-11-16 07:33 Thanks a lot for the quick fixes => looks good to me (at a first quick glance)