-
Improvement
-
Resolution: Done
-
Major
-
-
Pipeline - October, Pipeline - December
As a pipeline user, I WANNA GO FAST. 
Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions. As-is, every action we attach triggers a persistence cycle and we attach several usually.
We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work.
Work items:
- Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
- Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
- Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
- Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions
This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected. An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.
- blocks
-
JENKINS-47173 Offer a high-performance storage engine for pipeline at some cost to resumability
-
- Closed
-
[JENKINS-47172] Provide more granular control of pipeline flownode persistence
Sam Van Oort
added a comment - Feature released with workflow-support 2.17 – and consumed by downstream plugins
Sam Van Oort
added a comment - Feature released with workflow-support 2.17 – and consumed by downstream plugins 
Code changed in jenkins
User: Sam Van Oort
Path:
pom.xml
src/main/java/org/jenkinsci/plugins/workflow/support/DefaultStepContext.java
src/main/java/org/jenkinsci/plugins/workflow/support/PipelineIOUtils.java
src/main/java/org/jenkinsci/plugins/workflow/support/actions/LogActionImpl.java
src/main/java/org/jenkinsci/plugins/workflow/support/storage/BulkFlowNodeStorage.java
src/main/java/org/jenkinsci/plugins/workflow/support/storage/FlowNodeStorage.java
src/main/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamFlowNodeStorage.java
src/test/java/org/jenkinsci/plugins/workflow/ScalabilityTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/AbstractStorageTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/BulkStorageTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/MockFlowExecution.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/SimpleXStreamStorageTest.java
src/test/java/org/jenkinsci/plugins/workflow/support/storage/StorageTestUtils.java
http://jenkins-ci.org/commit/workflow-support-plugin/5f8c0433d1b0d8d5f5062eedba0b791ffad449ec
Log:
Merge pull request #47 from svanoort/defer-action-write-mk2
Granular flownode persistence and new node storage engine
JENKINS-47172JENKINS-47173Compare: https://github.com/jenkinsci/workflow-support-plugin/compare/c0c4bbef3e36...5f8c0433d1b0