-
Improvement
-
Resolution: Unresolved
-
Major
-
None
-
Powered by SuggestiMate -
Blue Ocean - Candidates
Scope
- Switch from line numbers to time stamp
Original request
The console output in blue ocean seems to suppress the time stamps for each log line but this info can be useful. For example if we have a hanging test then in blue ocean it might say:
Test 123 started Sending interrupt signal
That could mean that the stage was aborted manually or because of a failure in another stage or because this stage timed out.
In regular log it might say:
3:23:00 Test 123 started 3:43:00 Sending interrupt signal
And this is where it's clear that the interrupt was due to a 20 minute timeout being reached. Makes the logs much more useful.
- blocks
-
JENKINS-48344 Log files generated by Jenkins pipeline scripts are bloated
-
- Resolved
-
-
JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task
-
- Resolved
-
- is duplicated by
-
JENKINS-47833 Logs should be timestamped
-
- Closed
-
-
JENKINS-45108 Integrate Timestamper plugin with Ocean Blue
-
- Resolved
-
- is related to
-
JENKINS-47833 Logs should be timestamped
-
- Closed
-
[JENKINS-44195] Add timestamps to Blue Ocean log
I'm happy to pick this ticket up, pending a few chats with other team members.
As per https://issues.jenkins-ci.org/browse/JENKINS-47833 - have we decided to use Timestamper for this solution? or a native solution without requiring an additional plugin.
dragoonis I think this doesn't need timestamper (looking at the duplicate ticket, the console notes seem to bloat things) and this could be done probably in a more baked in way (or at least thought about). But I don't have any specific ideas on implementation... (maybe time stamper could be adapted to work with blue ocean? I dunno)
Timestamer already works with pipeline, the problem is that blueocean is suppressing the output.
If you open the logs in the old console view it will show you the timestamps.
I'd vote against "baking this in", I think reusing a build wrapper is much better architecture when you consider plugins like https://wiki.jenkins.io/display/JENKINS/Logstash+Plugin I'd hate to have to have to remove those baked in timestamps with regex when sending log lines to an external system.
PS. We use timestamper on all our Jenkins jobs with 150 days retention. Some of the logs are quite big (~20 MB raw text). I haven't noticed any issues related to console notes like the ones trejkaz reported
I note that timestamper lets you fetch the logs as text (ie you can choose what you want to see, and then it adds it to text) which implies it might be able to work.
I think the idea of wrapping pipeline via a step to get this is very strange and a usability hassle though (wouldn't people want to see timestamps always as a base line?)
One of the hard things is that the whole console note thing kind of doesn't work if you are working with logs as text, outside of the classic web UI (as everything comes back as markup with assumptions as to how it works). It would be a bit like expecting the timestamper to work with LogStash logs.
jbochenski oh good to know re size.
Clarification: there is no supressing of the timestamps in the logs via blue ocean, the timestamps are not added to the log itself (directly), so it wouldn't be possible to supress it, they just aren't there.
The timestamper doesn't ad to the text you get via an api. It has another API to fetch the logs with the timestamps added as span/html fragments - so it must be possible. I can see the /wfapi used by the stage view shows it, so perhaps the best approach dragoonis is to find a way to support it in the log api that blue ocean uses - then it would work the same as people use now.
The data returned from it looks like the following, which would be messy/risky to inject into an alternate log viewer (see the html fragments - and explains why sometimes things look off in stage view).
The URI: http://localhost:8080/job/timestamper/3/execution/node/9/wfapi/log?_=1519944444778
Perhaps vivek may have some ideas on how the blue api server side could use the timestamper endpoint to fetch it with the stamps added?
Response:
{"nodeId": "18","nodeStatus": "SUCCESS","length": 8602,"hasMore": false,"text": "<span class=\"timestamp\"><b>00:00:14.594</b> </span>[timestamper] Running shell script\n<span class=\"timestamp\"><b>00:00:14.857</b> </span>+ ping -c 15 www.apple.com\n<span class=\"timestamp\"><b>00:00:14.857</b> </span>PING e6858.dsce9.akamaiedge.net (104.84.9.174): 56 data bytes\n<span class=\"timestamp\"><b>00:00:14.857</b> </span>64 bytes from 104.84.9.174: icmp_seq=0 ttl=52 time=46.146 ms\n<span class=\"timestamp\"><b>00:00:15.772</b> </span>64 bytes from 104.84.9.174: icmp_seq=1 ttl=52 time=24.984 ms\n<span class=\"timestamp\"><b>00:00:16.692</b> </span>64 bytes from 104.84.9.174: icmp_seq=2 ttl=52 time=40.099 ms\n<span class=\"timestamp\"><b>00:00:18.045</b> </span>64 bytes from 104.84.9.174: icmp_seq=3 ttl=52 time=39.247 ms\n<span class=\"timestamp\"><b>00:00:18.960</b> </span>64 bytes from 104.84.9.174: icmp_seq=4 ttl=52 time=38.471 ms\n<span class=\"timestamp\"><b>00:00:19.876</b> </span>64 bytes from 104.84.9.174: icmp_seq=5 ttl=52 time=37.845 ms\n<span class=\"timestamp\"><b>00:00:20.793</b> </span>64 bytes from 104.84.9.174: icmp_seq=6 ttl=52 time=37.656 ms\n<span class=\"timestamp\"><b>00:00:21.713</b> </span>64 bytes from 104.84.9.174: icmp_seq=7 ttl=52 time=35.950 ms\n<span class=\"timestamp\"><b>00:00:23.062</b> </span>64 bytes from 104.84.9.174: icmp_seq=8 ttl=52 time=37.010 ms\n<span class=\"timestamp\"><b>00:00:23.979</b> </span>64 bytes from 104.84.9.174: icmp_seq=9 ttl=52 time=34.426 ms\n<span class=\"timestamp\"><b>00:00:24.898</b> </span>64 bytes from 104.84.9.174: icmp_seq=10 ttl=52 time=33.600 ms\n<span class=\"timestamp\"><b>00:00:25.818</b> </span>64 bytes from 104.84.9.174: icmp_seq=11 ttl=52 time=21.033 ms\n<span class=\"timestamp\"><b>00:00:26.740</b> </span>64 bytes from 104.84.9.174: icmp_seq=12 ttl=52 time=19.620 ms\n<span class=\"timestamp\"><b>00:00:28.093</b> </span>64 bytes from 104.84.9.174: icmp_seq=13 ttl=52 time=30.910 ms\n<span class=\"timestamp\"><b>00:00:29.012</b> </span>64 bytes from 104.84.9.174: icmp_seq=14 ttl=52 time=57.460 ms\n<span class=\"timestamp\"><b>00:00:29.012</b> </span>\n<span class=\"timestamp\"><b>00:00:29.012</b> </span>--- e6858.dsce9.akamaiedge.net ping statistics ---\n<span class=\"timestamp\"><b>00:00:29.012</b> </span>15 packets transmitted, 15 packets received, 0.0% packet loss\n<span class=\"timestamp\"><b>00:00:29.012</b> </span>round-trip min/avg/max/stddev = 19.620/35.630/57.460/9.128 ms\n","consoleUrl": "/job/timestamper/3/execution/node/18/log"}
So some more thoughts:
- Perhaps there is a way for timestamper to add data to the log in a way that blue ocean or other systems can use (that isn't classic UI specific markup that won't work)
- Perhaps a way to show/toggle timestamps for all logs in blue ocean would be easier (would mean not using time stamper)
- The timestamper output has to be scraped for the actual time data out of the embedded markup returned in the json (ugh)
None are appealing really.
I'm fine with anything that gets the logs into the view in a more or less accurate way...
As long as it doesn't fill our disk with inefficient ConsoleNote serialisations like the timestamper plugin does. But maybe that just means fixing how its ConsoleNote is serialised? When I looked at how much was stored in it, I wasn't able to account for the enormous size it was writing into the file. (People who claim they aren't noticing it probably aren't using it with pipeline. It turns out that if you use it for non-pipeline jobs, it stores the timestamps in a much more sensible way.)
The comment about external logging systems seems a bit odd to me too actually, surely if you're going to store a timestamp, you'd want the time that it actually occurred, not the time at which the log event eventually entered something like logstash? Does logstash not support passing existing timestamps along?
Logstash definitely supports parsing existing timestamps and is generally the value you want to use vs just utilizing the received time like you mentioned.
This relates to another thing we noticed, which is that timestamps were being generated from Jenkins master, which meant they didn't really reflect the times the events were occurring on the slaves. Normally you'd think this wouldn't differ by much, but because of the already-mentioned ConsoleNote thing, our Jenkins was working so hard reading from sockets and writing to logs that it was being delayed in processing data coming from slaves.
We currently pipe all our commands through an executable to add the timestamps to the raw output so that we can see the true timestamps irrespective of how long the master is taking to read and write the data.
trejkaz interesting - yes they probably are large.
jakauppila log stash I don't think would see those timestamps from the timestamper plugin at all (it probably ads its own somehow...)
I think the idea of wrapping pipeline via a step to get this is very strange and a usability hassle though (wouldn't people want to see timestamps always as a base line?)
michaelneale one group of people who apparently has no use for timestamps are Jenkins plugin developers, c.f. https://ci.jenkins.io/job/Plugins/ YMMV
More seriously: there are more plugins one could consider obvious defaults, e.g: log rotate, colorize output or mask passwords.
I agree there is an usability problem, but it's a general pipeline problem. There is just no convenient way to apply common settings like the above.
I've already mentioned it before. You can define a common wrapper via shared libraries, but you need to remember to apply this (unless you want declarative pipeline, then your out of luck).
I'd love to see this general issue addressed instead of just baking in the timestamps
As long as it doesn't fill our disk with inefficient ConsoleNote serialisations like the timestamper plugin does. But maybe that just means fixing how its ConsoleNote is serialised? When I looked at how much was stored in it, I wasn't able to account for the enormous size it was writing into the file. (People who claim they aren't noticing it probably aren't using it with pipeline. It turns out that if you use it for non-pipeline jobs, it stores the timestamps in a much more sensible way.)
The comment about external logging systems seems a bit odd to me too actually, surely if you're going to store a timestamp, you'd want the time that it actually occurred, not the time at which the log event eventually entered something like logstash? Does logstash not support passing existing timestamps along?
trejkaz the logstash plugin supports two modes: sending each line individually or sending the whole log as a single document. In both modes the only time data that is being sent is hudson.model.Run.getTimestamp(). In each-line mode you have the receive time as a proxy for actual line time.
This might seem odd, but that's the way it works now. Being both an active user and maintainer of the logstash plugin I'd like to make sure it stays relevant. One thing that would break an existing use case is the baked-in timestamps interfering with text search of the build log.
I second the call to fix ConsoleNote serialization. It is the current API for annotating console lines so making it work properly with Pipeline seems like a better choice.
As a bonus: it would make implementing sendiing "real" timestamps to external systems much easier if somebody wants to add that feature one day (as opposed to having to hack into Pipeline internals, not to mention having to switch for old-style and pipeline jobs).
FTR the ConsoleNote and other reasons for pipeline logs bloat are discussed in JENKINS-38381 and JENKINS-48344
I agree timestamping ought to be always on, at the very least by default. One might not realize it is needed, until things start to go less well. Then, when trying to understand where time is spent, you're left with either enabling the option, or possibly trickier wild guesses.
would really like to see this feature, just stumpled upon it in Blue Ocean and thought there was a mistake with the missing timestamps
+1 - this would be awesome - we've recently migrated to Blue Ocean, and this is a regression for our debugging efforts
Is there any clue when this would be available, because otherwise we need to start manually logging to console instead?
The lack of this feature is also causing a step back in our ability to debug long running jobs. This is very needed. Thanks!
The release of JENKINS-48344 should open up a new implementation possibility for this in Blue Ocean.
Timestamper 1.9 provides support for timestamps in pipelines. It's even possible to enable them globally.
ethorsa it is not about timestamps in pipelines but the display of them in Blue Ocean
Proper timestamp visualization in the Blue Ocean view would be very much appreciated! Having to switch to the old UI for this is time-consuming and inconsistent.
In my case,
I am using timestamp plugin,
In console output, timezone is OK. but log is too long.
It shows
15:26:36 blah blah
So I want to see log in blueocean for each stage, step.
but int blue ocean, timezone is UTC maybe.
It shows
[2019-07-09T06:23:02.385Z] blah blah
even though I set timezone properly.
These two are related / duplicates, surely?