Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47172

Provide more granular control of pipeline flownode persistence

XMLWordPrintable

    • Pipeline - October, Pipeline - December

      As a pipeline user, I WANNA GO FAST.  

      Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

      We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

      Work items:

      • Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
      • Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
      • Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
      • Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 

      This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.

          [JENKINS-47172] Provide more granular control of pipeline flownode persistence

          Sam Van Oort created issue -
          Sam Van Oort made changes -
          Epic Link New: JENKINS-47170 [ 185575 ]
          Sam Van Oort made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Sam Van Oort made changes -
          Summary Original: Pipeline Saves FlowNodes Multiple Times Redundantly New: Provide more granular control of pipeline persistence
          Sam Van Oort made changes -
          Summary Original: Provide more granular control of pipeline persistence New: Provide more granular control of pipeline flownode persistence
          Sam Van Oort made changes -
          Link New: This issue blocks JENKINS-47173 [ JENKINS-47173 ]
          James Dumay made changes -
          Sprint New: Pipeline - October [ 406 ]
          Sam Van Oort made changes -
          Description Original: As a pipeline user, I WANNA GO FAST.  :)

          Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

          We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

          Work items:
           * Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
           * Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
           * Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
           * Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 
           * Provide a way for Steps to mark that they are going to attach their own actions before running work, so we can wait on that to persist them. Useful for steps that add labels, etc.

          This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.           		New: As a pipeline user, I WANNA GO FAST.  :)

          Pipelines are highly IO-bound, and unfortunately are triggering multiple writes to a FlowNode (record of step execution) while it is being configured with the initial actions.  As-is, every action we attach triggers a persistence cycle and we attach several usually.

          We should defer persisting the FlowNode until the initial set-up is done and the step begins executing meaningful work. 

          Work items:
           * Provide FlowNodeStorage with ability to track FlowNodes that haven't been written to disk yet
           * Provide FlowNodeStorage with APIs to force a node to be flushed-to-disk
           * Provide FlowNodeStorage API to flush everything to disk (for restarts, etc)
           * Provide a way to enable auto-persistence of a FlowNode once we have attached initial actions 

          This will reduce IO use, reduce CPU use in persistence, and reduce memory garbage generated that has to be garbage-collected.  An early prototype version of this showed a 50% increase in build throughput (33% reduction in runtime) in one general case under a properly-constructed benchmark.
          James Dumay made changes -
          Sprint Original: Pipeline - October [ 406 ] New: Pipeline - October, Pipeline - December [ 406, 446 ]
          Sam Van Oort made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Closed [ 6 ]
          Sam Van Oort made changes -
          Resolution Original: Fixed [ 1 ]
          Status Original: Closed [ 6 ] New: Reopened [ 4 ]

            svanoort Sam Van Oort
            svanoort Sam Van Oort
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: