Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17887

Jenkins TAP plugin should scale to millions of tests in thousands of test sets

    XMLWordPrintable

Details

    Description

      The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

      My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

      prove --archive $WORKSPACE/tap-output t

      And passing that on to Jenkins TAP plugin, we see:

      Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

      I'd suggest setting these goals:

      • minimise on-disk storage of TAP output & parsed TAP results
      • minimise load-time of TAP Results & TAP Test Report pages
      • enhance usability of these pages

      Here are a few ideas:

      • Don't pre-render TAP results
        • At a glance, I don't think the Jenkins TAP plugin is, but included just in case
        • Avoids: large files on disk of pre-rendered content
      • Render results client-side
        • Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
        • Render test-set summaries by default
        • Only render test-set details on request
          • Avoids: a high up-front load time for large result sets
        • Gracefully degrade to individual pages
      • Introduce a filter for failed tests
      • Stop storing parsed TAP results in build.xml
        • Move to an sqlite db?
        • At least: remove extra whitespace
      • Compress everything in tap-master-files
        • deliver it to client compressed to minimise over-the-wire transfer size?

      See also:

      • JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.

      Attachments

        Issue Links

          Activity

            spurkis Steve Purkis created issue -
            spurkis Steve Purkis made changes -
            Field Original Value New Value
            Description The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

            My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

               prove --archive $WORKSPACE/tap-output t

            And passing that on to Jenkins TAP plugin, we see:

             * $WORKSPACE/tap-output is 172M
             * The last build dir is 789M. Breakdown:
            {code}
            452M build.xml
            4.0K changelog.xml
            167M log
            8.0K revision.txt
            171M tap-master-files
            {code}
             * The job's summary pages all load quickly, e.g.:
              * http://jenkins/job/ProjectName
              * http://jenkins/job/ProjectName/36/
             * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
              * load time: 46s
              * size: 16.2 MB
             * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
              * load time: 53s
              * size: 29.1 MB

            Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

            I'd suggest setting these goals:
             * minimise on-disk storage of TAP output & parsed TAP results
             * minimise load-time of TAP Results & TAP Test Report pages
             * enhance usability of these pages

            Here are a few ideas:
             * Don't pre-render TAP results
              * At a glance, I don't think the Jenkins TAP plugin is, but included just in case
              * Avoids: large files on disk of pre-rendered content
             * Render results client-side
              * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
              * Render test-set summaries by default
              * Only render test-set details on request
               * Avoids: a high up-front load time for large result sets
              * Gracefully degrade to individual pages
             * Introduce a filter for failed tests
             * Stop storing parsed TAP results in build.xml
              * Move to an sqlite db?
              * At least: remove extra whitespace
             * Compress everything in tap-master-files
              * deliver it to client compressed to minimise over-the-wire transfer size?

            See also:
             * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
            The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

            My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

               prove --archive $WORKSPACE/tap-output t

            And passing that on to Jenkins TAP plugin, we see:

             * $WORKSPACE/tap-output is 172M
             * The last build dir is 789M. Breakdown:
            {code}
            452M build.xml
            4.0K changelog.xml
            167M log
            8.0K revision.txt
            171M tap-master-files
            {code}
             * The job's summary pages all load quickly, e.g.:
             ** http://jenkins/job/ProjectName
             ** http://jenkins/job/ProjectName/36/
             * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
              * load time: 46s
              * size: 16.2 MB
             * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
              * load time: 53s
              * size: 29.1 MB

            Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

            I'd suggest setting these goals:
             * minimise on-disk storage of TAP output & parsed TAP results
             * minimise load-time of TAP Results & TAP Test Report pages
             * enhance usability of these pages

            Here are a few ideas:
             * Don't pre-render TAP results
              * At a glance, I don't think the Jenkins TAP plugin is, but included just in case
              * Avoids: large files on disk of pre-rendered content
             * Render results client-side
              * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
              * Render test-set summaries by default
              * Only render test-set details on request
               * Avoids: a high up-front load time for large result sets
              * Gracefully degrade to individual pages
             * Introduce a filter for failed tests
             * Stop storing parsed TAP results in build.xml
              * Move to an sqlite db?
              * At least: remove extra whitespace
             * Compress everything in tap-master-files
              * deliver it to client compressed to minimise over-the-wire transfer size?

            See also:
             * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
            spurkis Steve Purkis made changes -
            Description The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

            My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

               prove --archive $WORKSPACE/tap-output t

            And passing that on to Jenkins TAP plugin, we see:

             * $WORKSPACE/tap-output is 172M
             * The last build dir is 789M. Breakdown:
            {code}
            452M build.xml
            4.0K changelog.xml
            167M log
            8.0K revision.txt
            171M tap-master-files
            {code}
             * The job's summary pages all load quickly, e.g.:
             ** http://jenkins/job/ProjectName
             ** http://jenkins/job/ProjectName/36/
             * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
              * load time: 46s
              * size: 16.2 MB
             * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
              * load time: 53s
              * size: 29.1 MB

            Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

            I'd suggest setting these goals:
             * minimise on-disk storage of TAP output & parsed TAP results
             * minimise load-time of TAP Results & TAP Test Report pages
             * enhance usability of these pages

            Here are a few ideas:
             * Don't pre-render TAP results
              * At a glance, I don't think the Jenkins TAP plugin is, but included just in case
              * Avoids: large files on disk of pre-rendered content
             * Render results client-side
              * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
              * Render test-set summaries by default
              * Only render test-set details on request
               * Avoids: a high up-front load time for large result sets
              * Gracefully degrade to individual pages
             * Introduce a filter for failed tests
             * Stop storing parsed TAP results in build.xml
              * Move to an sqlite db?
              * At least: remove extra whitespace
             * Compress everything in tap-master-files
              * deliver it to client compressed to minimise over-the-wire transfer size?

            See also:
             * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
            The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:

            My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:

               prove --archive $WORKSPACE/tap-output t

            And passing that on to Jenkins TAP plugin, we see:

             * $WORKSPACE/tap-output is 172M
             * The last build dir is 789M. Breakdown:
            {code}
            452M build.xml
            4.0K changelog.xml
            167M log
            8.0K revision.txt
            171M tap-master-files
            {code}
             * The job's summary pages all load quickly, e.g.:
             ** http://jenkins/job/ProjectName
             ** http://jenkins/job/ProjectName/36/
             * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
             ** load time: 46s
             ** size: 16.2 MB
             * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
             ** load time: 53s
             ** size: 29.1 MB

            Its difficult to navigate both tapResults & tapTestReport pages because they are so big.

            I'd suggest setting these goals:
             * minimise on-disk storage of TAP output & parsed TAP results
             * minimise load-time of TAP Results & TAP Test Report pages
             * enhance usability of these pages

            Here are a few ideas:
             * Don't pre-render TAP results
             ** At a glance, I don't think the Jenkins TAP plugin is, but included just in case
             ** Avoids: large files on disk of pre-rendered content
             * Render results client-side
             ** Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
             ** Render test-set summaries by default
             ** Only render test-set details on request
             *** Avoids: a high up-front load time for large result sets
             ** Gracefully degrade to individual pages
             * Introduce a filter for failed tests
             * Stop storing parsed TAP results in build.xml
             ** Move to an sqlite db?
             ** At least: remove extra whitespace
             * Compress everything in tap-master-files
             ** deliver it to client compressed to minimise over-the-wire transfer size?

            See also:
             * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
            kinow Bruno P. Kinoshita made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            kinow Bruno P. Kinoshita made changes -
            Link This issue is related to JENKINS-19472 [ JENKINS-19472 ]
            kinow Bruno P. Kinoshita made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            kinow Bruno P. Kinoshita made changes -
            Link This issue depends on JENKINS-17804 [ JENKINS-17804 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 149183 ] JNJira + In-Review [ 177275 ]
            kinow Bruno P. Kinoshita made changes -
            Labels performance
            kinow Bruno P. Kinoshita made changes -
            Link This issue is related to JENKINS-19647 [ JENKINS-19647 ]
            kinow Bruno P. Kinoshita made changes -
            Link This issue is related to JENKINS-20212 [ JENKINS-20212 ]
            kinow Bruno P. Kinoshita made changes -
            Labels performance memory memory-leak performance

            People

              kinow Bruno P. Kinoshita
              spurkis Steve Purkis
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: