JENKINS-26107 suggests a label for a block, but a (generally static) label set at the start, whereas this is mainly about adding summary information at the end. So that would probably have to be a different step.

JENKINS-26523 notes that the ability to set Run.result may be useful.

A related request was for a stage to be marked with a custom icon, to indicate what kind of stage this way, or to add some key/value pairs such as name of environment we deployed to.

Okay, to try & better explain what we're trying to achieve, here's our current implementation, which is split into three parts:

Custom logic in the job shell script to annotate results

roughly speaking, this looks like:

STEP="System.Security-4.5"
STARTTIME=`date +%s`
echo "*** start: ${STEP}"
${TIMEOUTCMD} --signal=ABRT --kill-after=60s 5m make -w -C mcs/class/System.Security run-test TEST_HARNESS_FLAGS=-labels PROFILE=net_4_x && echo -e "*** end($(echo $(date +%s) - ${STARTTIME} | bc)): ${STEP}: \\e[42mPassed\e[0m" || echo -e "*** end($(echo $(date +%s) - ${STARTTIME} | bc)): ${STEP}: \\e[43mUnstable\e[0m"

This assigns blocks of tests (roughly equivalent to Pipeline stages) to run together, with a common timeout, and either a "pass" status or an "unstable" status (unstable indicating this block had a non-zero exit status, but that execution of the script should continue). For steps where failure should be deemed critical, the colour code is red instead of yellow, status text says "failed", and there's some extra exit 1's thrown in.

Groovy script to parse the build log and generate a job status table

Pretty much just builds an HTML <table>, pairing "*** start" and "*** end" lines to build the table structure and write the execution time, and adding the most recent test runner output as an at-a-glance summary

Aggregate the data from multiple jobs in a giant table, with a custom plugin

this gives the most dense data view we can manage - 76 steps, 5 architectures, with an at-a-glance look for developers of "did I break tests grouped into X on architecture Y", or "is builder Z misbehaving and in need of attention"

I would love to move as much of this crap into a Jenkinsfile as possible. Certainly parts 1 and 2 of the above seem theoretically feasible to me, but there are gaps that means I'd just end up emulating bash really badly in Groovy, rather than taking advantage of Pipeline features (because the Pipeline versions are missing things)

I am already working on JENKINS-26107, which allows you to define a display label for a block (which could contain one step, a series of steps, some function calls, etc.). What I would propose is another step status that could take various parameters such as:

• result: 'ABORTED' (or any other hudson.model.Result)
• message: 'skipped 5 tests'
• tags: [env: 'production', version: 123]
• badge: 'url'
• error: x (after catch (x))

When run, the step would look up the enclosing BlockStartNode and attach some actions to it. (The BlockEndNode would be more logical, but that is not created until the block is actually closed. Alternately, the step could be placed outside and just after the block, but that makes it harder to access local variables and the like.) So your Jenkinsfile could look like

label('Security tests') {
  sh 'make test'
  def failureCount = Integer.parseInt(readFile 'failureCount')
  if (failureCount == 0) {
    status badge: 'http://mycdn.com/checkmark.png'
  } else {
    status badge: 'http://mycdn.com/redx.png', result: 'UNSTABLE', message: "${failureCount} test(s) failed"
  }
}

which I think is reasonably readable, and allows the job to record all relevant information in the flow graph in a way that is recoverable from the Java APIs (and, albeit awkwardly for now, from the REST API). The basic Pipeline Steps UI can certainly show all this information, and some kind of unobtrusive formatted text can be added to the textual build log too.

Then the question becomes how sophisticated you can expect the “stock” stage view visualization to be when it comes to representing this information. JENKINS-33185 proposes a significant overhaul to that view, in the context of which it would make sense to display this status information directly.

The UI you are showing requires more than that, however, since you are displaying multiple results per build according to matrix configurations. JENKINS-27395 requests a way for the Jenkinsfile to scope things like archived test results to a particular configuration of matrix axes, where this information would then become accessible to other plugins. In particular the stage view (or whatever the visualization of Pipeline builds is called by then) would need to render matrices of results when defined.

No matter what gets implemented in Pipeline, you are not going to have a UI as well tuned to your use case as what you currently have in your custom plugin, but perhaps the generic UI would be good enough, or at least your custom plugin could become a thinner veneer.

My immediate question is what features are safe to release incrementally. JENKINS-26107 itself is complete, but I want to spend some time studying or even prototyping this issue, JENKINS-28119, JENKINS-27395, JENKINS-26132, and JENKINS-33185 to make sure the design is not foreclosing or impeding future expansions.

Another example use case is that github-branch-source could look for the resulting status notes and publish them to the GH Status API, so you would be able to see progress of your build in different sections right in a pull request.

Per JENKINS-36087 this should probably get some sort of stage view UI representation.

To flesh out general implementation thoughts (in case I knock this out as a rainy day or weekend project).

Work breaks down roughly like this:

Pipeline core plugins work

• Action type to attach to the Blocks (probably on the BlockEndNode, so it can be accumulated during execution and then written out). Probably will take a

  Map<String, Object>

  for field, value pairs. Alternately, single Name,Value pairs for each action (with extension point for defined annotations that includes type restrictions).
• Built-in annotation: custom status (implementation of the action, or a default entry in map, if we go that way).
• Marker step object 'annotate' that attaches the action in question (so the flow graph shows when it is entered) – this gets propagated up to containing blocks, I think?
• Pipeline core UI (tree view): show annotations on steps

Stage View Work

• Pipeline rest API (stage view): add annotation info to the objects.
• Pipeline Stage view UI: add a new JS extension point impl (example exists) for the annotation displays, to display annotation notes and create a popup with them.
• Stage view REST/UI: override stage StatusExt status using status annotations on block OR add additional information to StageExt objects that the UI consumes if present (extending the existing status coloring) --> this is probably what is needed (for parallel use too).

Untagging stage view, since support in UI would probably be through Blue Ocean instead.

Proof-of-concept approach here: https://github.com/jenkinsci/workflow-api-plugin/pull/12

Code changed in jenkins
User: Sam Van Oort
Path:
src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
http://jenkins-ci.org/commit/workflow-api-plugin/f6de209ae70887dd47a28075f1972bf1642b68d7
Log:
Merge pull request #12 from jenkinsci/tagsaction-JENKINS-26522

Create a new TagsAction JENKINS-26522

Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/8ca82ae8180d...f6de209ae708

Code changed in jenkins
User: Sam Van Oort
Path:
src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
http://jenkins-ci.org/commit/workflow-api-plugin/428319414b7fa7288bec1652202188a1ea52daa5
Log:
Revert "Create a new TagsAction JENKINS-26522"

Code changed in jenkins
User: Sam Van Oort
Path:
src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
http://jenkins-ci.org/commit/workflow-api-plugin/43f7e86b0323d7031429b5e5552393eb9d027bfe
Log:
Merge pull request #14 from jenkinsci/revert-12-tagsaction-JENKINS-26522

Revert "Create a new TagsAction JENKINS-26522"

Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/f6de209ae708...43f7e86b0323

svanoort could you provide a brief example of how this will be used in a Jenkins file? From your description I'm imagining something like this, but I'm not sure if I've understood correctly or not.

node {
  stage('example') {
    if (condition) {
      note('condition was true')
    } else {
      note('condition was false')
    }
  }
}

and then the "condition was (true|false)" notation would be reported in the project's stage view?

alexdean There's a variety of different ways to use it, I think.

The simplest ones:

// Annotations are string-string key:value pairs now
stage('example') {
   annotate('WebLink','http://example.com');
   annotate('MySpecialCondition','true');
}

Possibly we might want to allow it to take blocks?

// Annotations are string-string key:value pairs now
stage('example') {
   annotate('WebLink','http://example.com') {
     sh './deploy.sh www.example.com'
   }
}

Also, we could use it for status-overrides, if simple support is added to the pipeline-graph-analysis plugin:

// Annotations are string-string key:value pairs now
stage('example') {
   try {
      sh 'mvn flakyTest'
   } catch (Exception ex) {
       stageStatus 'UNSTABLE'; // Sets *just this* stage status to unstable (once status computation supports it) without changing the overall build result. 
       annotate('currentStage.status', 'UNSTABLE'); // actually creates the same result?
    }
}

Or at least that's what would make sense to me, not sure what others think.

Code changed in jenkins
User: Sam Van Oort
Path:
src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
http://jenkins-ci.org/commit/workflow-api-plugin/a5a7e8cce052f430149575b02fb41d145b89147b
Log:
Revert "Revert "Create a new TagsAction JENKINS-26522""

This reverts commit 428319414b7fa7288bec1652202188a1ea52daa5.

jamesdumay Not quite in review – part of this is, but there's a bit more to be done before we can consider it complete.

Is it also possible to expose in the API, so subsequent stages in the pipeline can find the status of previous stages.

The stageStatus directive as shown above looks good for manual overrides.

We will also have to get existing plugins to start setting this as opposed to the build status. For example, junit plugin on test failure, findbugs on exceeding violation thresholds etc. etc.

How about making a step/stage programatically disabled/skipped (like in Gradle) so I can decide in runtime, based on previous steps/stages results if I want to execute next one(s) or not?

vanta that is something you can do already. Please reserve this issue for discussion of visualizations.

jglick I didn't know, can you redirect me to place when I can check it?

Just write whatever logic you need in your script. Please do not use JIRA for advice/support.

svanoort Did this PR land elsewhere? Would really like to have this in order to replace a 'promoted build' job with a pipeline.

m4dc4p I suppose you are rather looking for JENKINS-36089.

So we've got TagsAction for behind-the-scenes string-to-string metadata/annotations currently. There's no way to add tags from within a Pipeline directly, but I'm still not 100% convinced of that use case.

What I'm thinking about now is twofold - first, a more status-specific Action - perhaps called something as dumb as FlowNodeStatusAction, which would allow steps to set their status to a Result, either at the same time they set the entire build result or just setting the step status on its own. I'd like this to not solely be for Block - there's real value in being able to mark specific steps with their own statuses, and then aggregating the status of steps within a block to determine the block's status, etc. This would enable us to resolve issues like JENKINS-42863 and/or JENKINS-39203. Worth mentioning that this alone won't solve those issues, but it would provide the tools to solve those.

Second would be non-string metadata - most immediately and specifically, test results (hi there, JENKINS-26395). But I know there are some potential problems with adding too much data to FlowNode actions directly, so we'd probably go with the same approach the junit plugin (and its many cousins) are using with runs - saving the results themselves in a separate file and pointing to that file from the action on the FlowNode. 

svanoort - thoughts?

abayer I'd rather not add another Action  - we added the TagsAction specifically with an eye towards this, and Result values have straightforward canonical string representations.  

I think aggregating statuses within a block is not actually a trivial process because of the number of potential combinations of different statuses in different orders and across parallels.  I prefer to allow setting a single override for the overall block that determines its status (but can be modified later) which keeps the handling simple.  However either way we'd want a concrete and detailed proposal to work from – this is not something to do ad-hoc because we've been bitten by ignoring these layered complexities.

For test results with the TagsAction we would not be encapsulating a complete result, but an identifier for the test result (perhaps indicating the flownode it runs with) – similarly to how you describe. 

Aaaaah. TagsAction does make sense there. re: block status - would you put the status/test results pointer/whatever on the BlockStartStep or BlockEndStep? My instincts tend towards BlockStartStep, since we don't have the BlockEndStep yet at the time that something like junit could be adding test results/setting status, but I could be wrong. =)

abayer Convention (at least with ErrorActions, which we should follow the semantics of) is to put the status/results on the current node and the BlockEndNode.   This follows the convention that only the current head (tip) nodes are mutable, which permits caching and allows some simplifying assumptions to be made.  It also avoids the insanity of backtracking to modify every single BlockStartNode up to the beginning of the run, because we only need to track the current state. 

I'm sure we could find a straightforward way to keep information about currently executing blocks to simplify dealing with incomplete ones – I have several ideas for the future roadmap along those lines already.

svanoort Ok, so how would we propagate from the current node to the BlockEndNode? I may well be wrong, but I assumed that wasn't created until the end of the block.

svanoort Ok, dug into workflow-cps to figure that out. We aren't actually going from current node -> StepEndNode - the ErrorAction is ending up on StepEndNode via handling of an abnormal Outcome (with a Throwable) in CpsBodyExecution. So I my gut instinct at this point is that we'll need CpsBodyExecution.SuccessAdapter to traverse its nodes to find relevant TagsAction instances and add them on the StepEndNode. Which...I don't like. Maybe it makes more sense to do something like:

• When we start a body, add a TagsAction to its StepContext, somehow scoped to this specific body (so that nested bodies can each have their own TagsAction).
• When we hit a relevant status or test result, etc, add (if not existing) a TagsAction to the current node and add the relevant tags to it. This may not be needed - we may opt to bypass the current node completely, but I feel like we want to have that information on the specific node as well as on the whole block. Your call.
• At the same time, add the relevant tags to the body's TagsAction from the StepContext. Some logic will be needed for knowing, e.g., that status can't get better, just worse, tags that can have multiple values (i.e., while ideally we'd have only one junit invocation per block, we can't guarantee that, so we need to be able to record multiple test results per block - maybe TagsAction should support lists of Strings for values?).
• At completion of the body, in both FailureAdapter and SuccessAdapter, pull the body's TagsAction from the StepContext and add any values in it to the StepEndNode.
• And finally, propagate up the body's TagsAction contents to any possible parent body's TagsAction.

How's that sound?

Thinking further - some tag names will imply/require additional behavior, like status. So maybe we also need an extension point for "special tag names", that would provide a tag name and a method for actually setting the tag. Something like this:

public abstract class SpecialTag implements ExtensionPoint {
    @Nonnull
    public abstract String getTagName();

    public void setSpecialTag(@Nonnull TagsAction action, @Nonnull String value);

    public static ExtensionList<SpecialTag> all() {
        return ExtensionList.lookup(SpecialTag.class);
    }
}

Then, the theoretical StatusTag would be something like:

@Extension
public class StatusTag extends SpecialTag {
    public String getTagName() {
        return "BLOCK_STATUS";
    }

    public void setSpecialTag(@Nonnull TagsAction action, @Nonnull String value) {
        if (value != null) {
            String currentValue = action.getTagValue(getTagName());
            if (currentValue == null || Result.fromString(value).isWorseThan(Result.fromString(currentValue))) {
                action.addTag(this, value);
            }
        }
    }
}

And over in TagsAction itself, we'd change addTag to two versions...

public void addTag(@CheckForNull String tag, @CheckForNull String value) {
    if (tag != null && value != null && !tag.isEmpty() && !value.isEmpty()) {
        boolean foundSpecialTag = false;
        for (SpecialTag t : all()) {
            if (t.getTagName().equals(tagName)) {
                t.setSpecialTag(this, value);
                foundSpecialTag = true;
            }
        }
        if (!foundSpecialTag) {
            tags.put(tag, value);
        }
    }
}

public void addTag(@Nonnull SpecialTag specialTag, @Nonnull String value) {
    tags.put(specialTag.getTagName(), value);
}

Just a thought, but hey.

We aren't actually going from current node -> StepEndNode - the ErrorAction is ending up on StepEndNode via handling of an abnormal Outcome (with a Throwable) in CpsBodyExecution.

Right, this is not baked into the system at all, it is just an effect of the fact that an exception will be thrown up unless caught. workflow-cps attaches an ErrorAction to any step which ends in an exception, no more no less.

Would it not be simpler and more flexible to just attach any tag metadata to the current node, and have higher-level visualization APIs determine what scope(s) to associate that metadata with—later, on demand?

I probably missed a bunch of discussion so feel free to ignore if what I am saying does not make sense or has already been considered and rejected.

Right, this is not baked into the system at all, it is just an effect of the fact that an exception will be thrown up unless caught. workflow-cps attaches an ErrorAction to any step which ends in an exception, no more no less

As abayer observed correctly, it's just throwing it on there when it's closing out the blocks. 

I think we could bind this into the execution state (or context) associated with tracking open blocks during execution, to reduce complexity of attaching the statuses. 

Would it not be simpler and more flexible to just attach any tag metadata to the current node, and have higher-level visualization APIs determine what scope(s) to associate that metadata with—later, on demand?

NO. jglick this sort of approach has created a nightmare for Stage View, Blue Ocean before pipeline graph analysis libs ("Bismuth") and then in implementing the complex and nasty StatusAndTiming rules + UI representations.  We cannot make these rules even more complicated – it is at the tipping point where it is unsustainable to make them more complex, and I have to walk through long explanations for the results given because they are already hard to follow.  A new status coding must replace the previous in an either-or fashion and provide a simple, explicit result following from simple rules (probably the same rules of precedence we use for BuildResults being determined by the worse result).  It also needs to trickle up to the block level.

Feels like I could almost produce a PoC implementation in the time we're spending talking about this. 

I've got an initial design doc up at https://github.com/jenkinsci/workflow-api-plugin/pull/40 - I'm stuck on figuring out how to actually propagate up to the enclosing blocks, and have been for a few days, so I'm giving up on figuring it out solo and begging for help. =)

jamesdumay the tickets you linked seem to focus on the state of stages, but as I read the original ticket it was also about additional information by each stage (not only if a stage fails or is unstable, but also if everything went well).

This aspect seems to only vaguely covered by JENKINS-45579 with the message.
Should one open a new Request for this or will it be covered in one of the menitoned.

dageissl do you mean that you want to be able to attach a "note" to any stage or parallel? If so, please raise another ticket for that use cases and I can link it here.

jamesdumay I've opened JENKINS-46170 hope this coveres all the intentions.

Looks like the right initiative to close that one.

However I now wonder whether there's any ticket out there requesting to visualize stages inside parallel blocks using Blue Ocean. Can anyone tell? Didn't find anything so far but I doubt that I'm the only one who would make use of it.

Thanks.

macdrega you are looking for JENKINS-38442 but its unlikely to be implemented at this time (we can't find a sane way to visualise it). Would you mind drawing a picture of your ideal visualisation and a mock Jenkinsfile, to help us figure out what you want? (attach and comment to JENKINS-38442)

Thanks for the quick answer - found that one already.

I totally understand that nested stages are hard to visualize. I only need a small part of it: The only thing I need is just stages inside parallel blocks - without nesting one stage inside another stage, something like:

That should be easy to visualize.

Probably I'll create a new ticket hoping that it won't be closed in favor of  JENKINS-38442.

Mind posting this on the other ticket so that it's captured in the right place?

Sorry my fault . Post is already done now.

Thanks