Hello Sebastian!

First of all, sorry for taking so long to review the issues in the TAP plug-in. I'll have a cycle to work on the plug-in soon and I'm filtering the issues.

Thanks for reporting this issue, and for attaching the test results. The public Jenkins with the plug-in installed will be very useful too.

> As you can see, for all failed (sub-)tests there are lines starting with "#" in the table instead of the number of the failing test and the other columns

Could you point a test with that is missing the failing test, please?

I'm looking at test-results/t9903-bash-prompt.sh, and it has 2 failing tests, and below the test result, there's the comment-diagnostic info. Probably I'm looking at the wrong part of the test results.

Thanks in advance!

Hi Bruno,

something (probably on my side) seems to have changed recently. The original issue I was talking about is as follows: If you go to [1] and search for "test-results/t0061-run-command.sh" on that web page you'll see that in the table for test "t0061-run-command" subtest number 2 is missing. Instead, you see a number of comments in the table. But if you look at the slightly more recent test run at [2] there suddenly is subtest number 2, called "run_command can run a command", in the table. I'm not sure what caused the change.

I think you shoud simply ignore all lines

Anyway, even when looking at the (currently) most recent build at [3] there are a number of issues / inconsistencies:

1) That page contains the string "TAP Test Result (130 failures null)", not sure why "null" is in there. However, the total number of 130 failed tests is correct, that's the same number I'm calculating in my Groovy post-build script. The Groovy script also generates the output below "The following test(s) failed" on that page, which is my reference for the list of failing tests.

2) Slightly below, there's the text "This build contains 628 TAP test sets. Click here for details" and "1 parse error(s)". Again I'm not sure what causes the parse error, whether it's a bug in the TAP plugin or a bug it Git's test suite output. In any case it would be nice to see the name of the (sub-)test where the error occurred here.

3) The "TAP Test Result" from 1) and the "Click here for details" from 2) link to [4] and [5] respectively. At first I was confused that the TAP plugin provides two detailed results pages, and I'm still not sure whether this makes sense.

4) On [4], clicking on any if the ">>>" or test links under "All Failed Tests" results in "Status Code: 404". Also, the list of "All Failed Tests" does not seem to be correct as it includes the tests marked as "TODO known breakage". Maybe you could add an option to ignore results of tests marked as TODO? I'm also a bit confused why our total failed test count of 130 matches mine, although you count "TODO known breakage" as failures and I don't.

5) On [5], test marked as "TODO known breakage" are also marked red, i.e. as failures, but again the total number of failed tests oddly is correct. But the number of skipped tetsts is not. The header says "628 files 9549 tests, 8967 ok, 130 not ok, 1 skipped, 0 Bail Out!", but as you can see from the yellow entries in the table there are far more than 1 skipped tests. Also, whenever there's a comment in the raw TAP file, you make it part of the table. I think the plugin should simply ignore lines starting with "#" in the input files and not include them into the table. A good example on that page is at "test-results/t5801-remote-helpers.sh" where you see a lot of comments which should not be visible at all, IMHO. The bottom of the page finally contains the details about parse errors, which is great. Maybe the "1 parse error(s)" from the overview page could link here? In any case I like this result page much better than [4], and I'd like to suggest to completely drop [4] and just fix the issues with [5] instead. This will also lead to less confusion for the user which TAP result page to look at.

I hope all of this does not overwhelm you Thank in advance for any fixes!

[1] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/96/tapResults/
[2] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/99/tapResults/
[3] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/
[4] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/tapTestReport/
[5] https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/tapResults/

Wow, great feedback, much appreciated Sebastian!

The plugin started after a discussion in the mailing list. At that time, I was working with Selenium + TestLink and needed TAP for extending my test results.

Thus, the plug-in was tailored more or less to my own needs, and with time and with Pull Requests and issues here in Jenkins JIRA, it became more clear how to make it more generic for other users.

One thing that some users were missing, was the ability to merge their test results (JUnit, TestNG, etc) with the TAP results. Then I added the "TAP Test Result" link to the left. That link uses Jenkins testing API (kind like an abstract for JUnit, the default test format in Jenkins).

> At first I was confused that the TAP plugin provides two detailed results pages, and I'm still not sure whether this makes sense.

And that's why we have two detailed results page :o) I'll comment on the other items later, time to get the train.

Thanks a lot!

Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/main/java/org/tap4j/plugin/TapTestResultAction.java
http://jenkins-ci.org/commit/tap-plugin/4b259118ffc024a99c4cedf1bc13bfdae993440f
Log:
JENKINS-17941 Fixing test diff string

Hello Sebastian! How's it going?

Fixed two other issues in tap-plugin, now working on this one again

> 1) That page contains the string "TAP Test Result (130 failures null)", not sure why "null" is in there. However, the total number of 130 failed tests is correct, that's the same number I'm calculating in my Groovy post-build script. The Groovy script also generates the output below "The following test(s) failed" on that page, which is my reference for the list of failing tests.

I didn't recall what that "null" was supposed to be, so I had to check Jenkins core source code. That's the diff string. It used the previous build (if it had TAP tests) to compute the diff. I didn't override the AbstractTestResultAction#getPreviousResult() method and it wasn't being able to find the previous builds. I believe it has been fixed in 4b259118ffc024a99c4cedf1bc13bfdae993440f.

Moving on to the next item...

Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/main/resources/org/tap4j/plugin/TapResult/index.jelly
http://jenkins-ci.org/commit/tap-plugin/22dff5c9899056a3df0ea69b2bc8ff17a1d0c4a8
Log:
JENKINS-17941 Adding link to view file that failed to be parsed

> 2) Slightly below, there's the text "This build contains 628 TAP test sets. Click here for details" and "1 parse error(s)". Again I'm not sure what causes the parse error, whether it's a bug in the TAP plugin or a bug it Git's test suite output. In any case it would be nice to see the name of the (sub-)test where the error occurred here.

I believe that information is available in the build summary (the "Click here" link points to that page). Not sure if we should add that information in the project page, since it can contain a long list when many TAP streams failed to be parsed. Perhaps it could be added as a feature... since other users could find useful to have a list of failed tests... or bail out'ed tests.

I believe the TAP Stream of that parser error is this one: https://qa.nest-initiative.org/view/msysGit/job/msysgit-mingwGitDevEnv-test/125/tapResults/contents?f=test-results/t9200-git-cvsexportcommit.sh

I think the plug-in was right to fail to parse it. However, I had to manually enter the file name in the browser URL to view the file that the plug-in failed to parse.

Commit 22dff5c9899056a3df0ea69b2bc8ff17a1d0c4a8 adds a link to that file.

Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/main/resources/org/tap4j/plugin/TapBuildAction/summary.jelly
src/main/resources/org/tap4j/plugin/TapResult/index.jelly
http://jenkins-ci.org/commit/tap-plugin/7cab81f801c34a106b12391e04615e0ad1df5347
Log:
JENKINS-17941 Modify tapResults text on the UI. Was Tap test results, now Extended TAP Test Results