Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17887

Jenkins TAP plugin should scale to millions of tests in thousands of test sets

      The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

      My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

      prove --archive $WORKSPACE/tap-output t

      And passing that on to Jenkins TAP plugin, we see:

      Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

      I'd suggest setting these goals:

      • minimise on-disk storage of TAP output & parsed TAP results
      • minimise load-time of TAP Results & TAP Test Report pages
      • enhance usability of these pages

      Here are a few ideas:

      • Don't pre-render TAP results
        • At a glance, I don't think the Jenkins TAP plugin is, but included just in case
        • Avoids: large files on disk of pre-rendered content
      • Render results client-side
        • Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
        • Render test-set summaries by default
        • Only render test-set details on request
          • Avoids: a high up-front load time for large result sets
        • Gracefully degrade to individual pages
      • Introduce a filter for failed tests
      • Stop storing parsed TAP results in build.xml
        • Move to an sqlite db?
        • At least: remove extra whitespace
      • Compress everything in tap-master-files
        • deliver it to client compressed to minimise over-the-wire transfer size?

      See also:

      • JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.

          [JENKINS-17887] Jenkins TAP plugin should scale to millions of tests in thousands of test sets

          +1, sounds doable and very reasonable.

          I think the last release is bugged, so I'll try to fix all the bugs, cut a bugfix release and include this issue in a major release. Hopefully I'll manage release both versions within three or four weeks.

          Thanks Steve.

          Bruno P. Kinoshita added a comment - +1, sounds doable and very reasonable. I think the last release is bugged, so I'll try to fix all the bugs, cut a bugfix release and include this issue in a major release. Hopefully I'll manage release both versions within three or four weeks. Thanks Steve.

          Todd Rinaldo added a comment - - edited

          Our test count is close to 100,000

          FWIW, I don't really care so much about the passing tests. Most of the time, I'd like to filter out either all the files that pass or sometimes, I just want ONLY the tests that didn't pass, not the whole files. Note: I don't say failed, rather didn't pass. Reason being that interesting things to look at are:
          (In order of importance)
          1. Failed
          2. TODO Passed
          3. TODO Failed
          4. Skipped

          Also turning captured debug comments on/off might be nice.

          Todd Rinaldo added a comment - - edited Our test count is close to 100,000 FWIW, I don't really care so much about the passing tests. Most of the time, I'd like to filter out either all the files that pass or sometimes, I just want ONLY the tests that didn't pass, not the whole files. Note: I don't say failed, rather didn't pass. Reason being that interesting things to look at are: (In order of importance) 1. Failed 2. TODO Passed 3. TODO Failed 4. Skipped Also turning captured debug comments on/off might be nice.

          Hi @Steve

          I'm preparing to cut a new release of tap-plugin and unfortunately I won't have enough time to fix your issue.

          But I've re-read it and now think I understand it better. Probably I will create sub issues for some of your suggestions, and will try to include them in a next release.

          @Todd
          Created https://issues.jenkins-ci.org/browse/JENKINS-19472 to work on filters. I'm linking that issue to this one too, but probably you'll want to watch that one too.

          > Also turning captured debug comments on/off might be nice.

          It's disabled by default now, but you can turn it on under the advanced panel of the TAP post build step.

          Thanks!
          Bruno

          Bruno P. Kinoshita added a comment - Hi @Steve I'm preparing to cut a new release of tap-plugin and unfortunately I won't have enough time to fix your issue. But I've re-read it and now think I understand it better. Probably I will create sub issues for some of your suggestions, and will try to include them in a next release. @Todd Created https://issues.jenkins-ci.org/browse/JENKINS-19472 to work on filters. I'm linking that issue to this one too, but probably you'll want to watch that one too. > Also turning captured debug comments on/off might be nice. It's disabled by default now, but you can turn it on under the advanced panel of the TAP post build step. Thanks! Bruno

          Actually, there's an existing issue for filters.

          https://issues.jenkins-ci.org/browse/JENKINS-17804

          Bruno P. Kinoshita added a comment - Actually, there's an existing issue for filters. https://issues.jenkins-ci.org/browse/JENKINS-17804

            kinow Bruno P. Kinoshita
            spurkis Steve Purkis
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: