Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17960

Indicate if tests don't go to plan

XMLWordPrintable

    • Icon: New Feature New Feature
    • Resolution: Fixed
    • Icon: Critical Critical
    • tap-plugin

      One of the major features of TAP is that you have a test plan.
      If my TAP starts 1..100, but only 5 results are recorded before something broke, then I want to know about it. Likewise, if 105 results are recorded, I want some kind of error indication.

      I suggest counting this as 95 skipped tests by default, possibly with an config option to treat them as failed instead.

        1. tap-snapshot-20140312.hpi
          340 kB

          [JENKINS-17960] Indicate if tests don't go to plan

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/main/java/org/tap4j/plugin/TapPublisher.java
          http://jenkins-ci.org/commit/tap-plugin/275c599945025e3467bb42e9d82d3089ee25dc2b
          Log:
          JENKINS-17960 Simplify logic

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapPublisher.java http://jenkins-ci.org/commit/tap-plugin/275c599945025e3467bb42e9d82d3089ee25dc2b Log: JENKINS-17960 Simplify logic

          James Howe added a comment -

          Does that commit also address the problem of all the results being lost if the tests don't go to plan?

          James Howe added a comment - Does that commit also address the problem of all the results being lost if the tests don't go to plan?

          Bruno P. Kinoshita added a comment -

          Not that one, I believe it was fixed in https://github.com/jenkinsci/tap-plugin/commit/9e60ea571d359a0c445a33f791e62ca6c5f87479#diff-7ebebde3074c0bb507b4983802c5803cL149, already released in the experimental update center.

          This new commit only simplifies the logic as you suggested. Let me know if I missed anything. Probably I can push more commits today before releasing 1.18-2-alpha tonight.

          Bruno P. Kinoshita added a comment - Not that one, I believe it was fixed in https://github.com/jenkinsci/tap-plugin/commit/9e60ea571d359a0c445a33f791e62ca6c5f87479#diff-7ebebde3074c0bb507b4983802c5803cL149 , already released in the experimental update center. This new commit only simplifies the logic as you suggested. Let me know if I missed anything. Probably I can push more commits today before releasing 1.18-2-alpha tonight.

          Bruno P. Kinoshita added a comment -

          By the way, I'm not exactly comparing the number of the last test, but comparing if the total of tests is equal to the expected number in the test plan.

          Do you see any problem in doing like that?

          Bruno P. Kinoshita added a comment - By the way, I'm not exactly comparing the number of the last test, but comparing if the total of tests is equal to the expected number in the test plan. Do you see any problem in doing like that?

          James Howe added a comment -

          That looks like it might work now
          I'd need something in the next couple of hours in order to test it before Monday.

          James Howe added a comment - That looks like it might work now I'd need something in the next couple of hours in order to test it before Monday.

          Bruno P. Kinoshita added a comment -

          Due to firewall policies in the office where I am atm I can't release an alpha version to the experimental update center

          I've attached a snapshot HPI with the latest code though, don't know if you can install it in some environment and try it. If so let me know if that works and feel free to update this issue.

          JENKINS-17960 is the only pending issue I'm waiting to cut a new release. Next release will focus in JVM crashes and memory management, and the following will be code review/smaller improvements.

          Thanks!
          Bruno

          Bruno P. Kinoshita added a comment - Due to firewall policies in the office where I am atm I can't release an alpha version to the experimental update center I've attached a snapshot HPI with the latest code though, don't know if you can install it in some environment and try it. If so let me know if that works and feel free to update this issue. JENKINS-17960 is the only pending issue I'm waiting to cut a new release. Next release will focus in JVM crashes and memory management, and the following will be code review/smaller improvements. Thanks! Bruno

          Bruno P. Kinoshita added a comment -

          TAP Plug-in snapshot 2014-03-12 HPI file

          Bruno P. Kinoshita added a comment - TAP Plug-in snapshot 2014-03-12 HPI file

          James Howe added a comment -

          This looks like it now works as I would expect.

          James Howe added a comment - This looks like it now works as I would expect.

          Bruno P. Kinoshita added a comment -

          Yay, thanks for testing it so quickly James. Cutting a new release this night, 1.18 should be available in the update center within hours.

          Thanks again
          Bruno

          Bruno P. Kinoshita added a comment - Yay, thanks for testing it so quickly James. Cutting a new release this night, 1.18 should be available in the update center within hours. Thanks again Bruno

          Bruno P. Kinoshita added a comment -

          Fixed in 1.18

          Bruno P. Kinoshita added a comment - Fixed in 1.18

            kinow Bruno P. Kinoshita
            jameshowe James Howe
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: