Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27037

Identifying output files that don't match their plan

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • tap-plugin
    • None
    • Jenkins 1.552, tap-plugin 1.18, RHEL 6.6

      The output from the TAP Plugin does not identify which .tap files don't meet their plan, it just marks the whole job as unstable. I have jobs with several thousand tests which have no failures, but where one or more of the test programs have died before they finished running. There are test results missing from the output files from those tests which could easily be identified by the fact that they don't meet their plan, but all I can see from the TAP reports is that the job is unstable, not which of my many test files the problem came from.

      When a .tap file includes a plan (and the plugin has been configured to check the plans), any missing test lines should be treated as failures and counted as such in the test reports (e.g. if the plan says "1..7" but there are only 6 "ok" lines below it and no "not ok" lines at all, that should be counted as 6 passing and one failed test).

      This is really important IMHO, it's a fundamental issue with the meaning of the TAP output which the plugin is not following. The https://testanything.org/tap-specification.html gives an example with a missing test and shows how the harness should respond to this, see the section TESTS LINES AND THE PLAN/The test line/Test number where the harness reports the missing test #6 as having failed.

            kinow Bruno P. Kinoshita
            anjohnson Andrew Johnson
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: