-
Improvement
-
Resolution: Done
-
Major
-
-
Pipeline - October, Pipeline - December
As a pipeline user, I WANNA GO FAST. 
Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions. As-is, every action we attach triggers a persistence cycle and we attach several usually.
We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work.
Work items:
- Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
- Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
- Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
- Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions
This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected. An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.
- blocks
-
JENKINS-47173 Offer a high-performance storage engine for pipeline at some cost to resumability
-
- Closed
-
[JENKINS-47172] Provide more granular control of pipeline flownode persistence
Sam Van Oort
created issue -
Sam Van Oort
made changes -
| Epic Link | New: JENKINS-47170 [ 185575 ] |
Sam Van Oort
made changes -
| Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Sam Van Oort
made changes -
| Summary | Original: Pipeline Saves FlowNodes Multiple Times Redundantly | New: Provide more granular control of pipeline persistence |
Sam Van Oort
made changes -
| Summary | Original: Provide more granular control of pipeline persistence | New: Provide more granular control of pipeline flownode persistence |
Sam Van Oort
made changes -
| Link |
New:
This issue blocks |
| Sprint | New: Pipeline - October [ 406 ] |
Sam Van Oort
made changes -
| Description |
Original:
As a pipeline user, I WANNA GO FAST. :) Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions. As-is, every action we attach triggers a persistence cycle and we attach several usually. We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. Work items: * Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet * Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk * Provide FlowNodeStorage API to flush everything to disk (for restarts, etc) * Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions * Provide a way for Steps to mark that they are going to attach their own actions before running work, so we can wait on that to persist them. Useful for steps that add labels, etc. This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected. An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark. |
New:
As a pipeline user, I WANNA GO FAST. :) Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions. As-is, every action we attach triggers a persistence cycle and we attach several usually. We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. Work items: * Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet * Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk * Provide FlowNodeStorage API to flush everything to disk (for restarts, etc) * Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected. An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark. |
| Sprint | Original: Pipeline - October [ 406 ] | New: Pipeline - October, Pipeline - December [ 406, 446 ] |
Code changed in jenkins
User: Sam Van Oort
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java
src/main/java/org/jenkinsci/plugins/workflow/support/PipelineIOUtils.java
src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogActionImpl.java
src/main/java/org/jenkinsci/plugins/workflow/support/storage/BulkFlowNodeStorage.java
src/main/java/org/jenkinsci/plugins/workflow/support/storage/FlowNodeStorage.java
src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/AbstractStorageTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/BulkStorageTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/MockFlowExecution.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamStorageTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/StorageTestUtils.java
http://jenkins-ci.org/commit/workflow-support-plugin/5f8c0433d1b0d8d5f5062eedba0b791ffad449ec
Log:
Merge pull request #47 from svanoort/defer-action-write-mk2
Granular flownode persistence and new node storage engine
JENKINS-47172JENKINS-47173Compare: https://github.com/jenkinsci/workflow-support-plugin/compare/c0c4bbef3e36...5f8c0433d1b0