Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38381

[JEP-210] Optimize log handling in Pipeline and Durable Task

XMLWordPrintable

    • Pipeline - October, Pipeline - April 2018

      Extracted from JENKINS-30896:

      The current design of LogActionImpl, using $id.log, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, WorkflowRun.copyLogs must duplicate it all to log, doubling disk space requirements per build.

      It would be better to keep a single log file for the build. LogActionImpl should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When parallel blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by WorkflowRun will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)

          [JENKINS-38381] [JEP-210] Optimize log handling in Pipeline and Durable Task

          Jesse Glick created issue -
          Jesse Glick made changes -
          Epic Link New: JENKINS-38313 [ 174485 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-30896 [ JENKINS-30896 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-30777 [ JENKINS-30777 ]
          Jesse Glick made changes -
          Description Original: See discussion in JENKINS-30896 about log files. New: Extracted from JENKINS-30896:

          The current design of {{LogActionImpl}}, using {{$id.log}}, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, {{WorkflowRun.copyLogs}} must duplicate it all to {{log}}, doubling disk space requirements per build.

          It would be better to keep a single log file for the build. {{LogActionImpl}} should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When {{parallel}} blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by {{WorkflowRun}} will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)
          Jesse Glick made changes -
          Component/s New: durable-task-plugin [ 18622 ]
          Component/s New: workflow-durable-task-step-plugin [ 21715 ]

          Jesse Glick added a comment -

          Expanding scope a bit since oleg_nenashev’s work requires org.jenkinsci.plugins.durabletask.Controller.writeLog to be reconsidered as well: we do not want to force step logs to be streamed back to the master. So need an extension point (which for ease of development could start off inside a plugin) which would allow a Run to obtain a BuildListener whose serial form would allow this alternate mode.

          Jesse Glick added a comment - Expanding scope a bit since oleg_nenashev ’s work requires org.jenkinsci.plugins.durabletask.Controller.writeLog to be reconsidered as well: we do not want to force step logs to be streamed back to the master. So need an extension point (which for ease of development could start off inside a plugin) which would allow a Run to obtain a BuildListener whose serial form would allow this alternate mode.

          Jesse Glick added a comment -

          Can roll in a fix of JENKINS-31096 by just using UTF-8 everywhere.

          Jesse Glick added a comment - Can roll in a fix of JENKINS-31096 by just using UTF-8 everywhere.
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-31096 [ JENKINS-31096 ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            35 Vote for this issue
            Watchers:
            83 Start watching this issue

              Created:
              Updated:
              Resolved: