Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27037

Identifying output files that don't match their plan

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: tap-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.552, tap-plugin 1.18, RHEL 6.6
    • Similar Issues:

      Description

      The output from the TAP Plugin does not identify which .tap files don't meet their plan, it just marks the whole job as unstable. I have jobs with several thousand tests which have no failures, but where one or more of the test programs have died before they finished running. There are test results missing from the output files from those tests which could easily be identified by the fact that they don't meet their plan, but all I can see from the TAP reports is that the job is unstable, not which of my many test files the problem came from.

      When a .tap file includes a plan (and the plugin has been configured to check the plans), any missing test lines should be treated as failures and counted as such in the test reports (e.g. if the plan says "1..7" but there are only 6 "ok" lines below it and no "not ok" lines at all, that should be counted as 6 passing and one failed test).

      This is really important IMHO, it's a fundamental issue with the meaning of the TAP output which the plugin is not following. The https://testanything.org/tap-specification.html gives an example with a missing test and shows how the harness should respond to this, see the section TESTS LINES AND THE PLAN/The test line/Test number where the harness reports the missing test #6 as having failed.

        Attachments

          Activity

          Hide
          kinow Bruno P. Kinoshita added a comment -

          Hi Andrew;

          Good points. Let me work on a fix for this. Will post a possible solution in the next days

          Show
          kinow Bruno P. Kinoshita added a comment - Hi Andrew; Good points. Let me work on a fix for this. Will post a possible solution in the next days
          Hide
          psrcode Jonathan Rajotte-Julien added a comment -

          Any news on this ?

          Show
          psrcode Jonathan Rajotte-Julien added a comment - Any news on this ?

            People

            Assignee:
            kinow Bruno P. Kinoshita
            Reporter:
            anjohnson Andrew Johnson
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: