Hi James,

Good idea. We can include a new checkbox in the job configuration, so we'll keep the previous behaviour while we include this new feature.

Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/main/java/org/tap4j/plugin/TapParser.java
src/main/java/org/tap4j/plugin/TapPublisher.java
src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly
http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31
Log:
[FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed

Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java
src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d
Log:
[JENKINS-17960 JENKINS-17781] Fixing tests

Fixed in 1.13

Ah, you've made it less useful. Now I don't get any results.
The plan check should come last.

If I plan 1000 tests but only run 999, I should a) get all 999 results reported and b) be unstable even if they all passed.

Like how I originally suggested.

Sorry @James, let's reopen it and work on a fix again

My bad, Bruno

In fact, it now fails regardless of settings if there is no plan, where before it was successful.

Surely it's simple just to have it do this?

Tests planned: 100, Tests run: 90, Tests passed: 90, Status: Unstable
Tests planned: 100, Tests run: 110, Tests passed: 110, Status: Unstable
Tests planned: 100, Tests run: 100, Tests passed: 100, Status: Stable
etc.

Hi James, I believe I'll start a new cycle on tap-plugin today or very soon. Would you have time to test a beta version of tap-plugin before we release a new version and close this issue again?

Yes, I think so.

Starting a dev cycle on tap plugin looking forward to working on this issue again James. Hope you have some spare time and patience to go through this issue again.

> Tests planned: 100, Tests run: 90, Tests passed: 90, Status: Unstable
> Tests planned: 100, Tests run: 110, Tests passed: 110, Status: Unstable
> Tests planned: 100, Tests run: 100, Tests passed: 100, Status: Stable

Tis indeed not so hard to do. However, should you have any failure, and the
job is configured to fail the build in case of failures, your build will
be marked as failure, instead of unstable.

On the other hand, if you planned 100 test cases, but only 90 were execute,
and the others were skipped, then with this new change your build will
be marked as unstable because the number of test cases doesn't match
the plan.

WDYT? Will release an alpha version to the alpha update site.

Code changed in jenkins
User: Bruno P. Kinoshita
Path:
src/main/java/org/tap4j/plugin/TapParser.java
src/main/java/org/tap4j/plugin/TapPublisher.java
src/main/java/org/tap4j/plugin/TapResult.java
src/main/java/org/tap4j/plugin/model/TapStreamResult.java
src/main/java/org/tap4j/plugin/model/TapTestResultResult.java
src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
http://jenkins-ci.org/commit/tap-plugin/9e60ea571d359a0c445a33f791e62ca6c5f87479
Log:
JENKINS-17960 Indicate if tests don't go to plan

1.18-alpha released to http://updates.jenkins-ci.org/update-center.json

Give it a shot and let me know if that is in the right direction. You'll see a message in the console output, but nothing in the UI yet (will work on this if you confirm we are in the right direction).

Thanks!
Bruno

> On the other hand, if you planned 100 test cases, but only 90 were execute,
> and the others were skipped, then with this new change your build will
> be marked as unstable because the number of test cases doesn't match
> the plan.

That's not following the TAP standard. Skipped tests should count.
If Passed + Failed + Skipped = Planned then do nothing, otherwise make a Stable job Unstable.

I'm also seeing in version 1.17 that if the plan doesn't match then all the results are discarded.
It should report them just as for any other build.

The logic for this feature should really be very simple.

if (planExists && numberAtEndOfPlan != numberOfLastTest) {
log message that tests didn't go to plan;
if (status == SUCCESS) then status = UNSTABLE;
}

The checkbox should not cause anything else to happen.

I liked this simpler logic. Going to commit now but will be able to release another alpha version only tonight. Hope you can give it another shot this week. I'd like to cut a new release by weekend.

Thanks James!

Does that commit also address the problem of all the results being lost if the tests don't go to plan?

Not that one, I believe it was fixed in https://github.com/jenkinsci/tap-plugin/commit/9e60ea571d359a0c445a33f791e62ca6c5f87479#diff-7ebebde3074c0bb507b4983802c5803cL149, already released in the experimental update center.

This new commit only simplifies the logic as you suggested. Let me know if I missed anything. Probably I can push more commits today before releasing 1.18-2-alpha tonight.

By the way, I'm not exactly comparing the number of the last test, but comparing if the total of tests is equal to the expected number in the test plan.

Do you see any problem in doing like that?

That looks like it might work now
I'd need something in the next couple of hours in order to test it before Monday.

Due to firewall policies in the office where I am atm I can't release an alpha version to the experimental update center

I've attached a snapshot HPI with the latest code though, don't know if you can install it in some environment and try it. If so let me know if that works and feel free to update this issue.

JENKINS-17960 is the only pending issue I'm waiting to cut a new release. Next release will focus in JVM crashes and memory management, and the following will be code review/smaller improvements.

Thanks!
Bruno

TAP Plug-in snapshot 2014-03-12 HPI file

This looks like it now works as I would expect.

Yay, thanks for testing it so quickly James. Cutting a new release this night, 1.18 should be available in the update center within hours.

Thanks again
Bruno

Fixed in 1.18