Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17887

Jenkins TAP plugin should scale to millions of tests in thousands of test sets

      The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

      My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

      prove --archive $WORKSPACE/tap-output t

      And passing that on to Jenkins TAP plugin, we see:

      Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

      I'd suggest setting these goals:

      • minimise on-disk storage of TAP output & parsed TAP results
      • minimise load-time of TAP Results & TAP Test Report pages
      • enhance usability of these pages

      Here are a few ideas:

      • Don't pre-render TAP results
        • At a glance, I don't think the Jenkins TAP plugin is, but included just in case
        • Avoids: large files on disk of pre-rendered content
      • Render results client-side
        • Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
        • Render test-set summaries by default
        • Only render test-set details on request
          • Avoids: a high up-front load time for large result sets
        • Gracefully degrade to individual pages
      • Introduce a filter for failed tests
      • Stop storing parsed TAP results in build.xml
        • Move to an sqlite db?
        • At least: remove extra whitespace
      • Compress everything in tap-master-files
        • deliver it to client compressed to minimise over-the-wire transfer size?

      See also:

      • JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.

          [JENKINS-17887] Jenkins TAP plugin should scale to millions of tests in thousands of test sets

          Steve Purkis created issue -
          Steve Purkis made changes -
          Description Original: The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

          My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

             prove --archive $WORKSPACE/tap-output t

          And passing that on to Jenkins TAP plugin, we see:

           * $WORKSPACE/tap-output is 172M
           * The last build dir is 789M. Breakdown:
          {code}
          452M build.xml
          4.0K changelog.xml
          167M log
          8.0K revision.txt
          171M tap-master-files
          {code}
           * The job's summary pages all load quickly, e.g.:
            * http://jenkins/job/ProjectName
            * http://jenkins/job/ProjectName/36/
           * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
            * load time: 46s
            * size: 16.2 MB
           * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
            * load time: 53s
            * size: 29.1 MB

          Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

          I'd suggest setting these goals:
           * minimise on-disk storage of TAP output & parsed TAP results
           * minimise load-time of TAP Results & TAP Test Report pages
           * enhance usability of these pages

          Here are a few ideas:
           * Don't pre-render TAP results
            * At a glance, I don't think the Jenkins TAP plugin is, but included just in case
            * Avoids: large files on disk of pre-rendered content
           * Render results client-side
            * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
            * Render test-set summaries by default
            * Only render test-set details on request
             * Avoids: a high up-front load time for large result sets
            * Gracefully degrade to individual pages
           * Introduce a filter for failed tests
           * Stop storing parsed TAP results in build.xml
            * Move to an sqlite db?
            * At least: remove extra whitespace
           * Compress everything in tap-master-files
            * deliver it to client compressed to minimise over-the-wire transfer size?

          See also:
           * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
          New: The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

          My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

             prove --archive $WORKSPACE/tap-output t

          And passing that on to Jenkins TAP plugin, we see:

           * $WORKSPACE/tap-output is 172M
           * The last build dir is 789M. Breakdown:
          {code}
          452M build.xml
          4.0K changelog.xml
          167M log
          8.0K revision.txt
          171M tap-master-files
          {code}
           * The job's summary pages all load quickly, e.g.:
           ** http://jenkins/job/ProjectName
           ** http://jenkins/job/ProjectName/36/
           * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
            * load time: 46s
            * size: 16.2 MB
           * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
            * load time: 53s
            * size: 29.1 MB

          Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

          I'd suggest setting these goals:
           * minimise on-disk storage of TAP output & parsed TAP results
           * minimise load-time of TAP Results & TAP Test Report pages
           * enhance usability of these pages

          Here are a few ideas:
           * Don't pre-render TAP results
            * At a glance, I don't think the Jenkins TAP plugin is, but included just in case
            * Avoids: large files on disk of pre-rendered content
           * Render results client-side
            * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
            * Render test-set summaries by default
            * Only render test-set details on request
             * Avoids: a high up-front load time for large result sets
            * Gracefully degrade to individual pages
           * Introduce a filter for failed tests
           * Stop storing parsed TAP results in build.xml
            * Move to an sqlite db?
            * At least: remove extra whitespace
           * Compress everything in tap-master-files
            * deliver it to client compressed to minimise over-the-wire transfer size?

          See also:
           * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
          Steve Purkis made changes -
          Description Original: The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

          My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

             prove --archive $WORKSPACE/tap-output t

          And passing that on to Jenkins TAP plugin, we see:

           * $WORKSPACE/tap-output is 172M
           * The last build dir is 789M. Breakdown:
          {code}
          452M build.xml
          4.0K changelog.xml
          167M log
          8.0K revision.txt
          171M tap-master-files
          {code}
           * The job's summary pages all load quickly, e.g.:
           ** http://jenkins/job/ProjectName
           ** http://jenkins/job/ProjectName/36/
           * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
            * load time: 46s
            * size: 16.2 MB
           * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
            * load time: 53s
            * size: 29.1 MB

          Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

          I'd suggest setting these goals:
           * minimise on-disk storage of TAP output & parsed TAP results
           * minimise load-time of TAP Results & TAP Test Report pages
           * enhance usability of these pages

          Here are a few ideas:
           * Don't pre-render TAP results
            * At a glance, I don't think the Jenkins TAP plugin is, but included just in case
            * Avoids: large files on disk of pre-rendered content
           * Render results client-side
            * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
            * Render test-set summaries by default
            * Only render test-set details on request
             * Avoids: a high up-front load time for large result sets
            * Gracefully degrade to individual pages
           * Introduce a filter for failed tests
           * Stop storing parsed TAP results in build.xml
            * Move to an sqlite db?
            * At least: remove extra whitespace
           * Compress everything in tap-master-files
            * deliver it to client compressed to minimise over-the-wire transfer size?

          See also:
           * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
          New: The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

          My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

             prove --archive $WORKSPACE/tap-output t

          And passing that on to Jenkins TAP plugin, we see:

           * $WORKSPACE/tap-output is 172M
           * The last build dir is 789M. Breakdown:
          {code}
          452M build.xml
          4.0K changelog.xml
          167M log
          8.0K revision.txt
          171M tap-master-files
          {code}
           * The job's summary pages all load quickly, e.g.:
           ** http://jenkins/job/ProjectName
           ** http://jenkins/job/ProjectName/36/
           * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
           ** load time: 46s
           ** size: 16.2 MB
           * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
           ** load time: 53s
           ** size: 29.1 MB

          Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

          I'd suggest setting these goals:
           * minimise on-disk storage of TAP output & parsed TAP results
           * minimise load-time of TAP Results & TAP Test Report pages
           * enhance usability of these pages

          Here are a few ideas:
           * Don't pre-render TAP results
           ** At a glance, I don't think the Jenkins TAP plugin is, but included just in case
           ** Avoids: large files on disk of pre-rendered content
           * Render results client-side
           ** Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
           ** Render test-set summaries by default
           ** Only render test-set details on request
           *** Avoids: a high up-front load time for large result sets
           ** Gracefully degrade to individual pages
           * Introduce a filter for failed tests
           * Stop storing parsed TAP results in build.xml
           ** Move to an sqlite db?
           ** At least: remove extra whitespace
           * Compress everything in tap-master-files
           ** deliver it to client compressed to minimise over-the-wire transfer size?

          See also:
           * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
          Bruno P. Kinoshita made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Bruno P. Kinoshita made changes -
          Link New: This issue is related to JENKINS-19472 [ JENKINS-19472 ]
          Bruno P. Kinoshita made changes -
          Status Original: In Progress [ 3 ] New: Open [ 1 ]
          Bruno P. Kinoshita made changes -
          Link New: This issue depends on JENKINS-17804 [ JENKINS-17804 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 149183 ] New: JNJira + In-Review [ 177275 ]
          Bruno P. Kinoshita made changes -
          Labels New: performance
          Bruno P. Kinoshita made changes -
          Link New: This issue is related to JENKINS-19647 [ JENKINS-19647 ]
          Bruno P. Kinoshita made changes -
          Link New: This issue is related to JENKINS-20212 [ JENKINS-20212 ]

            kinow Bruno P. Kinoshita
            spurkis Steve Purkis
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: