Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38381

[JEP-210] Optimize log handling in Pipeline and Durable Task

XMLWordPrintable

    • Pipeline - October, Pipeline - April 2018

      Extracted from JENKINS-30896:

      The current design of LogActionImpl, using $id.log, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, WorkflowRun.copyLogs must duplicate it all to log, doubling disk space requirements per build.

      It would be better to keep a single log file for the build. LogActionImpl should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When parallel blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by WorkflowRun will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)

          [JENKINS-38381] [JEP-210] Optimize log handling in Pipeline and Durable Task

          Jesse Glick created issue -
          Jesse Glick made changes -
          Epic Link New: JENKINS-38313 [ 174485 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-30896 [ JENKINS-30896 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-30777 [ JENKINS-30777 ]
          Jesse Glick made changes -
          Description Original: See discussion in JENKINS-30896 about log files. New: Extracted from JENKINS-30896:

          The current design of {{LogActionImpl}}, using {{$id.log}}, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, {{WorkflowRun.copyLogs}} must duplicate it all to {{log}}, doubling disk space requirements per build.

          It would be better to keep a single log file for the build. {{LogActionImpl}} should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When {{parallel}} blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by {{WorkflowRun}} will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)
          Jesse Glick made changes -
          Component/s New: durable-task-plugin [ 18622 ]
          Component/s New: workflow-durable-task-step-plugin [ 21715 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-31096 [ JENKINS-31096 ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-37575 [ JENKINS-37575 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "workflow-job PR 27 (Web Link)" [ 14876 ]
          Jesse Glick made changes -
          Remote Link New: This issue links to "jenkins-test-harness PR 37 (Web Link)" [ 14903 ]

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            35 Vote for this issue
            Watchers:
            83 Start watching this issue

              Created:
              Updated:
              Resolved: