There are a number of places in Stapler, Jenkins core, and Blue Ocean code in which logs are displayed where it is implicitly assumed that

• obtaining the total content length (in bytes) is fast
• skipping to a certain offset (in bytes) is fast
• once you are reading from the log stream, it is cheap to read to the end

Clearly these assumptions do not match the reality of CloudWatch log storage well: it is network-based, so read operations are relatively laggy; it is line-oriented, so byte offsets are meaningless; output is paged (albeit at a large page size of 10k lines by default); content does not appear instantly after it was written (worked around in the plugin).

It is possible to make a CloudWatch-based log store behave semantically like a local log file, but it is relatively expensive; the current implementation will not scale well to very large logs.

The question is whether it is feasible to produce an efficient implementation without touching these other components. The current PipelineLargeText and AnnotatedLogAction.getLogText implementations are clearly not suited to large logs, but the Pipeline-based API could permit these to be overridden, and perhaps a more specialized implementation could perform better. For example, doProgressText could be overridden to dispense with the CharSpool buffer and stream content directly to the HTTP response. (The browser will only render the completed response, but at least the master then need not hold it all in memory.)

The key sticking point is the interpretation of the X-Text-Size header. Despite it being declared to be numeric (a byte offset), core JavaScript code seems to treat it as purely opaque, so it could be used as a CloudWatch FilterLogEventsResult.nextToken if doProgressText were overridden. The situation in Blue Ocean is more complicated (there are a number of different log-related JS classes centering around LogPager); KaraokeApi sometimes parses it as a number and it is unclear what the behavior would be if it were NaN. There is also a NodeLogResource which buffers the complete log on the server side, but it does not seem to be used, at least in the normal display; PipelineStepImpl seems to be serving content via LogResource. But LogResource also seems to be buffering output and bypassing doProgressText, instead using the lower-level writeLogTo and setting X-Text-Size based on byte offsets. It also defines a thresholdInKB query parameter which is evidently unused from the UI.

The length method is also a problem, though it seems to be used only from Run/console.jelly to decide when to truncate a long log, so it could be an estimate.

In sum, it seems like an efficient implementation could be produced for core (Classic UI) with some effort, but that it would not improve the performance of Blue Ocean at all. So if we want to provide high-performance log display from Jenkins itself (as opposed to an external log browser), we need to decide whether Classic or Blue Ocean is the primary consideration, and consider defining a fresh API that explicitly supports cloud-friendly line-oriented output and paging/cursors.

There are smaller optimizations that can be made in the meantime, and these can be evaluated: making PipelineLargeText.length() cheap; making it retrieve text only once.

See also https://github.com/jenkinsci/workflow-job-plugin/pull/27#discussion_r210328775.

Probably need to

• patch LargeText and everything above it to accept an opaque cursor rather than a byte offset
• fix some issues with log length estimation in Run + WorkflowRun