Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38381

[JEP-210] Optimize log handling in Pipeline and Durable Task

    • Pipeline - October, Pipeline - April 2018

      Extracted from JENKINS-30896:

      The current design of LogActionImpl, using $id.log, was considered the minimum necessary for a working 1.0 release, not a serious implementation. It has a major problem: when there is a large amount of output, WorkflowRun.copyLogs must duplicate it all to log, doubling disk space requirements per build.

      It would be better to keep a single log file for the build. LogActionImpl should deprecated in favor of an implementation that simply stores a rangeset of offsets into that file. When parallel blocks are producing concurrent output, the single log file will be a bit jumbled (probably still human-readable in most cases), but the rangesets will keep track of what output came from where. The final output produced by WorkflowRun will still be processed to split at line boundaries, add in thread labels, etc. (TBD how and whether JENKINS-30777 could be supported in this mode.)

          [JENKINS-38381] [JEP-210] Optimize log handling in Pipeline and Durable Task

          mishal shah added a comment -

          jglick Do you know when this will be released? We are also seeing infinite log issue.

          mishal shah added a comment - jglick Do you know when this will be released? We are also seeing infinite log issue.

          Any updates on this issue? Currently, this is slowing down builds and would be great to have fixed, or at least marked as "stalled" || "blocked" if that's the situation.

          Victor Bjelkholm added a comment - Any updates on this issue? Currently, this is slowing down builds and would be great to have fixed, or at least marked as "stalled" || "blocked" if that's the situation.

          Jesse Glick added a comment -

          PRs refreshed.

          Jesse Glick added a comment - PRs refreshed.

          Steven Foster added a comment -

          Nice, would this make any use of the work on artifact-manager-s3-plugin for the external storage part?

          Steven Foster added a comment - Nice, would this make any use of the work on artifact-manager-s3-plugin for the external storage part?

          Jesse Glick added a comment -

          No, external artifact management is unrelated.

          Jesse Glick added a comment - No, external artifact management is unrelated.

          Jesse Glick added a comment -

          Yesterday I redesigned the API to more clearly separate layers of responsibility and allow cloud-based implementations to perform better. I have abandoned the logstash PoC implementation.

          Jesse Glick added a comment - Yesterday I redesigned the API to more clearly separate layers of responsibility and allow cloud-based implementations to perform better. I have abandoned the logstash PoC implementation.

          any update on this? its currently marked as in progress.

          Andreas Sieferlinger added a comment - any update on this? its currently marked as in progress.

          Jesse Glick added a comment -

          Hoping for some review activity soon.

          Jesse Glick added a comment - Hoping for some review activity soon.

          Jesse Glick added a comment -

          I filed a JEP submission for this.

          Jesse Glick added a comment - I filed a JEP submission for this.

          Jesse Glick added a comment -

          Released in beta.

          Jesse Glick added a comment - Released in beta.

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            35 Vote for this issue
            Watchers:
            83 Start watching this issue

              Created:
              Updated:
              Resolved: