-
Story
-
Resolution: Unresolved
-
Major
-
Powered by SuggestiMate -
1.0-japan-m9
A common question asked when using CI is "did something change to break this build", not just who broke it (blame dark pattern) or where it was broken (which is already clear with stage graph).
1. Calling out that this specific run is where the build broke (ie previously it was ok) may invite users to look at the change (it is something I commonly do, after looking at the failure message)
2. For any failed view, it may be wise to have a quick link back to the last successful run.
[JENKINS-35806] Make it clear when current run broke the build
Nokia accomplished this ends by making the stage circles a little bigger and putting the last successful or first failing build number in the middle of the circle instead of an X or checkmark. The color was the only indicator of failure. In that way, the 2D pipeline view could also capture a sense of a dimension in time.
Here is the gist of the concept. Text spacing is just roughed in, but the build dots would now contain a little more information:
User could now click on a stage and be taken back to the build and stage that was last interesting.
greiber interesting, so that is showing a bit of history of things - ie when things last broken in a given stage?
The first question generally people want answered is "did the changes that triggered this break the build", or really "is the build now broken, but was ok before" - with a link to what was changed, or the last build (which was good). Really, its about working out what changed.
Having that clearly up the top makes sense, although the wording isn't quite that, not sure how to phrase it to make it clear. But I think this is good.
I like the other stuff though, which is really quite good for CD pipelines:
So, if I understand it correctly, when I look at the stage graph here, I am seeing the last attempted build numbers at each stage. Ie 423 is current failure. Previously 422 failed on some other browsers. 421 before that on Dev. 420 (SNOOOOOP) failed on staging (shame), and the last one to make it always the way to the end was 419.
Is that right?
Yes, that is right.
If on build 423 everything were to pass except "staging", staging would still say "420" and the final stage would still say "419", but everything else would be green checkmarks.
If on build 423 everything were to pass except "production", the final stage would still say "419" but would be the 2 tone red color indicating that it failed on this build (the rule here isn't quite consistent, but the last stage is special and signifies that all other stages have passed).
The rules for generating the display may seem complicated, but for the user the end result is simple. If it is red, it is broken and it broke for the first time on the run shown. If I click it, I go straight to that first failure. The first and last stages are a little special, because the first stage always runs and the last stage only runs if everything else passes. Thus they make handy places to show this run number compared to the last successful run number.
For the top bits, I am not sure if you really need that at all, but ultimately, I think you want to get as quickly as possible to the actual change list involved as well as the build log. You could tack in the change references with some sort of additional hover gesture or something if you wanted to squeeze it into the track diagram.
greiber just to make things more fun - what about unstable build states...
greiber yes, hover could provide a lot of info, eg if the comments are JIRA tickets, it could have info on them. Lots of possibilities.
I like this, but needs more thinking.
Unstable states are tougher, but could be handled the same way. Would just be orange or yellow. We would need to decide the replacement rules for unstable, as in, when does an unstable stage get repainted with a new run number. Would it be every time the stage is activated, every time the failing tests are different, or not until the stage is completely passed?
....for the more thinking, what does that mean? Do you want more drawings or just time to sit with it?
greiber I need some time for me to sit with it, but if you have any other options for visualising when things broke/passed, that would be good to see. Is this something one would only see when things break (I think so) - or perhaps something people turn on to diagnose?
For unstable - to me that is the same as failure, but it can be up to pipeline authors. By default unstable doesn't quit the pipeline the way a failure does, so it is possible that unstable could kind of be ignored and a deployment happen anyway (I think this is bad, but it is up to authors to opt out of that). Not sure if that makes things clearer...
Sorry I haven't gotten to this one just yet Gus - snowed under with a few things.
So my suggestions: use hover states where possible (means more room to "explain" as numbers alone take some training):
Also, showing any upcoming ones is very nice (who wants to debug something when there is a fix in the pipe). I do like the idea of showing the build numbers of last attempted ones on each stage (but not clear if it should be success or not).
greiber I like the idea of showing more info in the little ball things, possibly interesting when it is executing (not this ticket) so you can trace what it running where, but I think the hover things may give a lot more space to ease people into it.
mneale I think hovers is tough here. I would expect this to be a screen that folks would want to project on a big screen somewhere in their team area. "Broken since" which is what the red stage dots would show would be super helpful information.
Here I have added another state indication to show the stage didn't run this time (it is semi-opaque). To me, adding a gesture adds at least as much complexity and user burden as showing the combination of a number and a color and maintains the 'hands-free' use cases.
The Nokia guys also include time to complete stage within each stage circle, as one of their uses cases was pipeline CD optimization. They wanted to see at a glance how long the total run took and which stages were slow. They were expert users and designing specifically for themselves, but I found the few things that they showed immediately on the screen to be fairly compelling, generally. They wanted 'at a glance' diagnosis so they could project the results to a group and flip quickly between pipelines.
...anyway, that is my caution.
I think past stage failures is a really important bit of health information worth getting 'at a glance'. Stuffing important information into hovers, I doesn't always reduce user complexity, in fact, often it increases it. ...thus I would look for other places to trim clutter, if this feels too cluttered.
Another thing the Nokia guy talked about was wanting to see quickly when and which a CD run had been kicked off from some point other than the beginning. Other than resume, I am not sure if we allow that, but that also seems to me to be a common enough diagnostic issue.
The Nokia guy was describing that situation as an instance of team or developer bad behavior, where to speed things along they were skipping blocks of tests. Naughty or not, his teams were using that feature pretty often. My friend here in MS games as a matter of requirement needs to jump into midpoints of his CD do to the time consuming nature of his full pipeline. Most of the time, he cannot spare the 2 or 3 days that would be required to run a full regression.
....this isn't directly related to showing skipped build steps in a hover or not, but this was another piece of information the Nokia guy scanned for and their stage numbering system allowed him to see quickly without touching the screen. ...so I toss that in here as well.
greiber I think we could re-use this component for dashboard "large screen" like scenarious, but probably not as part of the default. As long as there is a visual indicator that things are "past" vs "now" that is interesting to me.
There are some other screens about diagnostics (trends? jdumay) that don't belong here as directly (they can show a lot more) - this is more a dev centric screen to start with, but the components can evolve to include these other methods.
As for skipping things... I am not sure how they would be doing that with pipeline.
greiber I really like the idea of showing past things, but I can't see how to do it without explaining it to people, which thus far hasn't been needed for this important screen. If we know everyone uses build numbers, I think that could be good (we can't show anything else though, as it would get too crowded).
Another option - below the stage graph there is the step listing, there is also space there to show more detailed information if we want.
The main thing is that it isn't visually obvious what is "now" vs past, if we can make that clear (I can't quite see it in the colours on my monitor) this could be really good.
I would also like (separate ticket) to look at this from the "hands off" perspective - in that case it won't be run specific - but that is not this feature (scope has increase enough for now).
I think there is also a ticket to show calculated average times per stage etc (there was going to be some backend maths to do it in a more robust way than it does now) - but also, not this ticket.
Explore two options: