Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17887

Jenkins TAP plugin should scale to millions of tests in thousands of test sets


      The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

      My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

      prove --archive $WORKSPACE/tap-output t

      And passing that on to Jenkins TAP plugin, we see:

      Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

      I'd suggest setting these goals:

      • minimise on-disk storage of TAP output & parsed TAP results
      • minimise load-time of TAP Results & TAP Test Report pages
      • enhance usability of these pages

      Here are a few ideas:

      • Don't pre-render TAP results
        • At a glance, I don't think the Jenkins TAP plugin is, but included just in case
        • Avoids: large files on disk of pre-rendered content
      • Render results client-side
        • Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
        • Render test-set summaries by default
        • Only render test-set details on request
          • Avoids: a high up-front load time for large result sets
        • Gracefully degrade to individual pages
      • Introduce a filter for failed tests
      • Stop storing parsed TAP results in build.xml
        • Move to an sqlite db?
        • At least: remove extra whitespace
      • Compress everything in tap-master-files
        • deliver it to client compressed to minimise over-the-wire transfer size?

      See also:

      • JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.

            kinow Bruno P. Kinoshita
            spurkis Steve Purkis
            3 Vote for this issue
            5 Start watching this issue