Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17960

Indicate if tests don't go to plan

XMLWordPrintable

    • Icon: New Feature New Feature
    • Resolution: Fixed
    • Icon: Critical Critical
    • tap-plugin

      One of the major features of TAP is that you have a test plan.
      If my TAP starts 1..100, but only 5 results are recorded before something broke, then I want to know about it. Likewise, if 105 results are recorded, I want some kind of error indication.

      I suggest counting this as 95 skipped tests by default, possibly with an config option to treat them as failed instead.

        1. tap-snapshot-20140312.hpi
          340 kB

          [JENKINS-17960] Indicate if tests don't go to plan

          James Howe created issue -
          Bruno P. Kinoshita made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          Bruno P. Kinoshita added a comment -

          Hi James,

          Good idea. We can include a new checkbox in the job configuration, so we'll keep the previous behaviour while we include this new feature.

          Bruno P. Kinoshita added a comment - Hi James, Good idea. We can include a new checkbox in the job configuration, so we'll keep the previous behaviour while we include this new feature.
          Bruno P. Kinoshita made changes -
          Labels New: 1.13

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/main/java/org/tap4j/plugin/TapParser.java
          src/main/java/org/tap4j/plugin/TapPublisher.java
          src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly
          http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31
          Log:
          [FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapParser.java src/main/java/org/tap4j/plugin/TapPublisher.java src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31 Log: [FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed

          SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java
          src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
          http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d
          Log:
          [JENKINS-17960 JENKINS-17781] Fixing tests

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d Log: [JENKINS-17960 JENKINS-17781] Fixing tests
          Bruno P. Kinoshita made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]

          Bruno P. Kinoshita added a comment -

          Fixed in 1.13

          Bruno P. Kinoshita added a comment - Fixed in 1.13
          Bruno P. Kinoshita made changes -
          Status Original: Resolved [ 5 ] New: Closed [ 6 ]

          James Howe added a comment -

          Ah, you've made it less useful. Now I don't get any results.
          The plan check should come last.

          If I plan 1000 tests but only run 999, I should a) get all 999 results reported and b) be unstable even if they all passed.

          James Howe added a comment - Ah, you've made it less useful. Now I don't get any results. The plan check should come last. If I plan 1000 tests but only run 999, I should a) get all 999 results reported and b) be unstable even if they all passed.

            kinow Bruno P. Kinoshita
            jameshowe James Howe
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: