-
Bug
-
Resolution: Fixed
-
Critical
-
Jenkins: 1.500
TAP Plugin: 1.10
Optional: perl: 5.8.9, TAP::Harness: 3.26, TAP::Harness::Archive: 0.14
Thanks for your work on the Jenkins TAP Plugin!
In our tests, we've found it ignores parse errors where # tests run does not match the plan. Given this perl test:
$ cat fail.t
use Test::More tests => 2;
ok('before die');
die "eek!";
ok('after die');
And a Jenkins Job with this shell build step:
cd $JENKINS_HOME
mkdir -p $WORKSPACE/tap-output
set +e
prove -m --archive $WORKSPACE/tap-output/ fail.t
echo "prove exited with: $?"
Plus a Publish TAP step...
Running the job you get this TAP output:
$ cat ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t
1..2
ok 1
eek! at fail.t line 4.
- Looks like you planned 2 tests but ran 1.
- Looks like your test exited with 255 just after 1.
This should result in a Parse Error (because we planned 2 tests, but ran only 1). TAP::Harness confirms:
$ cp ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t ~/fail.tap
$ prove -v ~/fail.tap
... snip - TAP ...
Failed 1/2 subtests
Test Summary Report
-------------------
fail.tap (Wstat: 0 Tests: 1 Failed: 0)
Parse errors: Bad plan. You planned 2 tests but ran 1.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU)
Result: FAIL
But the Jenkins TAP plugin reports a successful build:
1 tests, 1 ok, 0 not ok, 0 skipped, 0 Bail Out!.
...
No parse errors found.
See screenshot for more details.
I've marked this as critical: as it stands Jenkins TAP will report false positives in these cases. As a worst-case scenario this may result in some users deploying bugs to production.
[JENKINS-17781] TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Steve Purkis
created issue -
Steve Purkis
made changes -
| Description |
Original:
Thanks for your work on the Jenkins TAP Plugin! In our tests, we've found it ignores this parse error. Given this perl test: $ cat fail.t use Test::More tests => 2; ok('before die'); die "eek!"; ok('after die'); And a Jenkins Job with this shell build step: cd $JENKINS_HOME mkdir -p $WORKSPACE/tap-output set +e prove -m --archive $WORKSPACE/tap-output/ fail.t echo "prove exited with: $?" Plus a Publish TAP step... Running the job you get this TAP output: $ cat ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t 1..2 ok 1 eek! at fail.t line 4. # Looks like you planned 2 tests but ran 1. # Looks like your test exited with 255 just after 1. This should result in a Parse Error (because we planned 2 tests, but ran only 1). TAP::Harness confirms: $ cp ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t ~/fail.tap $ prove -v ~/fail.tap ... snip - TAP ... Failed 1/2 subtests Test Summary Report ------------------- fail.tap (Wstat: 0 Tests: 1 Failed: 0) Parse errors: Bad plan. You planned 2 tests but ran 1. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU) Result: FAIL But the Jenkins TAP plugin reports a successful build: 1 tests, 1 ok, 0 not ok, 0 skipped, 0 Bail Out!. ... No parse errors found. See screenshot for more details. I've marked this as critical: as it stands Jenkins TAP will report false positives in these cases. As a worst-case scenario this may result in some users deploying bugs to production. |
New:
Thanks for your work on the Jenkins TAP Plugin! In our tests, we've found it ignores parse errors where # tests run does not match the plan. Given this perl test: $ cat fail.t use Test::More tests => 2; ok('before die'); die "eek!"; ok('after die'); And a Jenkins Job with this shell build step: cd $JENKINS_HOME mkdir -p $WORKSPACE/tap-output set +e prove -m --archive $WORKSPACE/tap-output/ fail.t echo "prove exited with: $?" Plus a Publish TAP step... Running the job you get this TAP output: $ cat ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t 1..2 ok 1 eek! at fail.t line 4. # Looks like you planned 2 tests but ran 1. # Looks like your test exited with 255 just after 1. This should result in a Parse Error (because we planned 2 tests, but ran only 1). TAP::Harness confirms: $ cp ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t ~/fail.tap $ prove -v ~/fail.tap ... snip - TAP ... Failed 1/2 subtests Test Summary Report ------------------- fail.tap (Wstat: 0 Tests: 1 Failed: 0) Parse errors: Bad plan. You planned 2 tests but ran 1. Files=1, Tests=1, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU) Result: FAIL But the Jenkins TAP plugin reports a successful build: 1 tests, 1 ok, 0 not ok, 0 skipped, 0 Bail Out!. ... No parse errors found. See screenshot for more details. I've marked this as critical: as it stands Jenkins TAP will report false positives in these cases. As a worst-case scenario this may result in some users deploying bugs to production. |
| Summary | Original: TAP Plugin: fails to report errors for | New: TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan |
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Successfully reproduced the issue here. I'll write a quick fix for it later. Thanks Steve!
| Labels | Original: parse-error tap | New: 1.13 parse-error tap |
Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/main/java/org/tap4j/plugin/TapParser.java
src/main/java/org/tap4j/plugin/TapPublisher.java
src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly
http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31
Log:
[FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed
Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java
src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d
Log:
[JENKINS-17960 JENKINS-17781] Fixing tests
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: In Progress [ 3 ] | New: Resolved [ 5 ] |
| Status | Original: Resolved [ 5 ] | New: Closed [ 6 ] |
Hi Steve!
Thanks for reporting the issue, and for including details. At moment I'm working on other open source projects and private projects.
But I'll have a new cycle to work on Jenkins plug-ins within the next days. I'll take a look on this issue and will try to cut a release as soon as I've found a fix for it.
In the meantime, feel free to comment on this issue and/or, in case you have time, submit a pull request via GitHub
All the best,
Bruno