Thanks for your work on the Jenkins TAP Plugin!

In our tests, we've found it ignores parse errors where # tests run does not match the plan. Given this perl test:

$ cat fail.t
use Test::More tests => 2;
ok('before die');
die "eek!";
ok('after die');

And a Jenkins Job with this shell build step:

cd $JENKINS_HOME
mkdir -p $WORKSPACE/tap-output
set +e
prove -m --archive $WORKSPACE/tap-output/ fail.t
echo "prove exited with: $?"

Plus a Publish TAP step...

Running the job you get this TAP output:

$ cat ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t
1..2
ok 1
eek! at fail.t line 4.

1. Looks like you planned 2 tests but ran 1.
2. Looks like your test exited with 255 just after 1.

This should result in a Parse Error (because we planned 2 tests, but ran only 1). TAP::Harness confirms:

$ cp ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t ~/fail.tap
$ prove -v ~/fail.tap
... snip - TAP ...
Failed 1/2 subtests

Test Summary Report
-------------------
fail.tap (Wstat: 0 Tests: 1 Failed: 0)
Parse errors: Bad plan. You planned 2 tests but ran 1.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU)
Result: FAIL

But the Jenkins TAP plugin reports a successful build:

1 tests, 1 ok, 0 not ok, 0 skipped, 0 Bail Out!.
...
No parse errors found.

See screenshot for more details.

I've marked this as critical: as it stands Jenkins TAP will report false positives in these cases. As a worst-case scenario this may result in some users deploying bugs to production.