-
Improvement
-
Resolution: Unresolved
-
Major
-
Jenkins 1.500
TAP Plugin 1.10
The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example:
My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs:
prove --archive $WORKSPACE/tap-output t
And passing that on to Jenkins TAP plugin, we see:
- $WORKSPACE/tap-output is 172M
- The last build dir is 789M. Breakdown:
452M build.xml 4.0K changelog.xml 167M log 8.0K revision.txt 171M tap-master-files
- The job's summary pages all load quickly, e.g.:
- The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/
- load time: 46s
- size: 16.2 MB
- The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/
- load time: 53s
- size: 29.1 MB
Its difficult to navigate both tapResults & tapTestReport pages because they are so big.
I'd suggest setting these goals:
- minimise on-disk storage of TAP output & parsed TAP results
- minimise load-time of TAP Results & TAP Test Report pages
- enhance usability of these pages
Here are a few ideas:
- Don't pre-render TAP results
- At a glance, I don't think the Jenkins TAP plugin is, but included just in case
- Avoids: large files on disk of pre-rendered content
- Render results client-side
- Introduce server-side APIs for: test-set summaries, test-set details, original TAP output
- Render test-set summaries by default
- Only render test-set details on request
- Avoids: a high up-front load time for large result sets
- Gracefully degrade to individual pages
- Introduce a filter for failed tests
- Stop storing parsed TAP results in build.xml
- Move to an sqlite db?
- At least: remove extra whitespace
- Compress everything in tap-master-files
- deliver it to client compressed to minimise over-the-wire transfer size?
See also:
- JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here.
- depends on
-
JENKINS-17804 tapResults TAP Test Results filtering by test status
-
- Closed
-
- is related to
-
JENKINS-19647 memory issues with multiconfig jobs and TAP plugin
-
- Open
-
-
JENKINS-20212 TAP Plugin Uses High CPU/IO When Projects Have Lots of Tests
-
- Open
-
-
JENKINS-19472 Create filters
-
- Closed
-
[JENKINS-17887] Jenkins TAP plugin should scale to millions of tests in thousands of test sets
Description |
Original:
The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example: My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs: prove --archive $WORKSPACE/tap-output t And passing that on to Jenkins TAP plugin, we see: * $WORKSPACE/tap-output is 172M * The last build dir is 789M. Breakdown: {code} 452M build.xml 4.0K changelog.xml 167M log 8.0K revision.txt 171M tap-master-files {code} * The job's summary pages all load quickly, e.g.: * http://jenkins/job/ProjectName * http://jenkins/job/ProjectName/36/ * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/ * load time: 46s * size: 16.2 MB * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/ * load time: 53s * size: 29.1 MB Its difficult to navigate both tapResults & tapTestReport pages because they are so big. I'd suggest setting these goals: * minimise on-disk storage of TAP output & parsed TAP results * minimise load-time of TAP Results & TAP Test Report pages * enhance usability of these pages Here are a few ideas: * Don't pre-render TAP results * At a glance, I don't think the Jenkins TAP plugin is, but included just in case * Avoids: large files on disk of pre-rendered content * Render results client-side * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output * Render test-set summaries by default * Only render test-set details on request * Avoids: a high up-front load time for large result sets * Gracefully degrade to individual pages * Introduce a filter for failed tests * Stop storing parsed TAP results in build.xml * Move to an sqlite db? * At least: remove extra whitespace * Compress everything in tap-master-files * deliver it to client compressed to minimise over-the-wire transfer size? See also: * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here. |
New:
The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example: My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs: prove --archive $WORKSPACE/tap-output t And passing that on to Jenkins TAP plugin, we see: * $WORKSPACE/tap-output is 172M * The last build dir is 789M. Breakdown: {code} 452M build.xml 4.0K changelog.xml 167M log 8.0K revision.txt 171M tap-master-files {code} * The job's summary pages all load quickly, e.g.: ** http://jenkins/job/ProjectName ** http://jenkins/job/ProjectName/36/ * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/ * load time: 46s * size: 16.2 MB * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/ * load time: 53s * size: 29.1 MB Its difficult to navigate both tapResults & tapTestReport pages because they are so big. I'd suggest setting these goals: * minimise on-disk storage of TAP output & parsed TAP results * minimise load-time of TAP Results & TAP Test Report pages * enhance usability of these pages Here are a few ideas: * Don't pre-render TAP results * At a glance, I don't think the Jenkins TAP plugin is, but included just in case * Avoids: large files on disk of pre-rendered content * Render results client-side * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output * Render test-set summaries by default * Only render test-set details on request * Avoids: a high up-front load time for large result sets * Gracefully degrade to individual pages * Introduce a filter for failed tests * Stop storing parsed TAP results in build.xml * Move to an sqlite db? * At least: remove extra whitespace * Compress everything in tap-master-files * deliver it to client compressed to minimise over-the-wire transfer size? See also: * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here. |
Description |
Original:
The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example: My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs: prove --archive $WORKSPACE/tap-output t And passing that on to Jenkins TAP plugin, we see: * $WORKSPACE/tap-output is 172M * The last build dir is 789M. Breakdown: {code} 452M build.xml 4.0K changelog.xml 167M log 8.0K revision.txt 171M tap-master-files {code} * The job's summary pages all load quickly, e.g.: ** http://jenkins/job/ProjectName ** http://jenkins/job/ProjectName/36/ * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/ * load time: 46s * size: 16.2 MB * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/ * load time: 53s * size: 29.1 MB Its difficult to navigate both tapResults & tapTestReport pages because they are so big. I'd suggest setting these goals: * minimise on-disk storage of TAP output & parsed TAP results * minimise load-time of TAP Results & TAP Test Report pages * enhance usability of these pages Here are a few ideas: * Don't pre-render TAP results * At a glance, I don't think the Jenkins TAP plugin is, but included just in case * Avoids: large files on disk of pre-rendered content * Render results client-side * Introduce server-side APIs for: test-set summaries, test-set details, original TAP output * Render test-set summaries by default * Only render test-set details on request * Avoids: a high up-front load time for large result sets * Gracefully degrade to individual pages * Introduce a filter for failed tests * Stop storing parsed TAP results in build.xml * Move to an sqlite db? * At least: remove extra whitespace * Compress everything in tap-master-files * deliver it to client compressed to minimise over-the-wire transfer size? See also: * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here. |
New:
The Jenkins TAP plugin is awesome, but could be designed to scale a bit better. As an example: My client's test-suite has ~60,000 test cases in ~1,300 test scripts. After running the test suite with a Shell build step that ultimately runs: prove --archive $WORKSPACE/tap-output t And passing that on to Jenkins TAP plugin, we see: * $WORKSPACE/tap-output is 172M * The last build dir is 789M. Breakdown: {code} 452M build.xml 4.0K changelog.xml 167M log 8.0K revision.txt 171M tap-master-files {code} * The job's summary pages all load quickly, e.g.: ** http://jenkins/job/ProjectName ** http://jenkins/job/ProjectName/36/ * The last build's 'tapResults' page: http://jenkins/job/ProjectName/36/tapResults/ ** load time: 46s ** size: 16.2 MB * The last build's 'tapTestReport' page: http://jenkins/job/ProjectName/36/tapTestReport/ ** load time: 53s ** size: 29.1 MB Its difficult to navigate both tapResults & tapTestReport pages because they are so big. I'd suggest setting these goals: * minimise on-disk storage of TAP output & parsed TAP results * minimise load-time of TAP Results & TAP Test Report pages * enhance usability of these pages Here are a few ideas: * Don't pre-render TAP results ** At a glance, I don't think the Jenkins TAP plugin is, but included just in case ** Avoids: large files on disk of pre-rendered content * Render results client-side ** Introduce server-side APIs for: test-set summaries, test-set details, original TAP output ** Render test-set summaries by default ** Only render test-set details on request *** Avoids: a high up-front load time for large result sets ** Gracefully degrade to individual pages * Introduce a filter for failed tests * Stop storing parsed TAP results in build.xml ** Move to an sqlite db? ** At least: remove extra whitespace * Compress everything in tap-master-files ** deliver it to client compressed to minimise over-the-wire transfer size? See also: * JENKINS-13451 - I ran into a similar scaling issue with TAP::Formatter::HTML, though different because of the nature of the tool - static output was a requirement. Still, some ideas such as 'minimise size of html & CSS used in report', and 'remove unnecessary whitespace' may apply here. |
Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Link |
New:
This issue is related to |
Status | Original: In Progress [ 3 ] | New: Open [ 1 ] |
Link |
New:
This issue depends on |
Workflow | Original: JNJira [ 149183 ] | New: JNJira + In-Review [ 177275 ] |
Labels | New: performance |
Link | New: This issue is related to JENKINS-19647 [ JENKINS-19647 ] |
Link | New: This issue is related to JENKINS-20212 [ JENKINS-20212 ] |