Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38381

[JEP-210] Optimize log handling in Pipeline and Durable Task

    XMLWordPrintable

Details

    • Pipeline - October, Pipeline - April 2018

    Description

      Extracted from JENKINS-30896:

      The current design of LogActionImpl, using $id.log, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, WorkflowRun.copyLogs must duplicate it all to log, doubling disk space requirements per build.

      It would be better to keep a single log file for the build. LogActionImpl should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When parallel blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by WorkflowRun will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick added a comment -

            Yesterday I redesigned the API to more clearly separate layers of responsibility and allow cloud-based implementations to perform better. I have abandoned the logstash PoC implementation.

            jglick Jesse Glick added a comment - Yesterday I redesigned the API to more clearly separate layers of responsibility and allow cloud-based implementations to perform better. I have abandoned the logstash PoC implementation.

            any update on this? its currently marked as in progress.

            webrat Andreas Sieferlinger added a comment - any update on this? its currently marked as in progress.
            jglick Jesse Glick added a comment -

            Hoping for some review activity soon.

            jglick Jesse Glick added a comment - Hoping for some review activity soon.
            jglick Jesse Glick added a comment -

            I filed a JEP submission for this.

            jglick Jesse Glick added a comment - I filed a JEP submission for this.
            jglick Jesse Glick added a comment -

            Released in beta.

            jglick Jesse Glick added a comment - Released in beta.

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              35 Vote for this issue
              Watchers:
              82 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: