Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-34038

New flow analyzer algorithm in pipeline stage view based on a visitor pattern

      Implementation of the fixes for https://issues.jenkins-ci.org/browse/JENKINS-33624 greatly increases scalability of stage view with large flow graphs, but it revealed some major underlying issues with the existing algorithms, in correctness, scalability, and extensibility. They're O(n log n) at least, and in practice something like O(m*n + n log n) with m being stage count.

      • This needs to track FlowSegments, which are linear sections of the flow graph with a defined start and end. This is needed because you need to handle and work with the structure of the flow (stages, labels, parallel blocks) without having to be aware of every node within, since there may be thousands.
      • This needs to support parallel flowsegments correctly, enabling JENKINS-33185
      • It needs to be aware of block scoping to correctly assign executor node labels to FlowNodes - Root cause for JENKINS-33290 and eventually for block labelling.
      • It needs to offer an (eventually pluggable) filter option to allow searching for specific nodes.
      • It needs to be largely O( n ), and where possible only do one pass through flownodes.
      • It should offer an API that is SAX-like, which tracks segment start/end, block start/end, and node visiting (in stage) with stages being mapped as a segment type.
      • It should be extensible for plugin authors to define their own analysis of flowgraphs (probably for visualization)

      If it's considered high enough quality, this can be added to the main pipeline plugin, as an available API.

      I have a PR (linked) that is addressing this need, which is currently a work in progress but close to covering this.

          [JENKINS-34038] New flow analyzer algorithm in pipeline stage view based on a visitor pattern

          Sam Van Oort added a comment -

          This is still in progress, and it will be consuming the ForkScanner workflow API work from https://github.com/jenkinsci/workflow-api-plugin/pull/2 because this is the only sane way to handle analysis of blocks and stages in the order encountered (otherwise you need to keep a complex record of all blocks).

          Sam Van Oort added a comment - This is still in progress, and it will be consuming the ForkScanner workflow API work from https://github.com/jenkinsci/workflow-api-plugin/pull/2 because this is the only sane way to handle analysis of blocks and stages in the order encountered (otherwise you need to keep a complex record of all blocks).

          Sam Van Oort added a comment -

          Yes, it's still in progress. It's mutated into a more general API set since we have a cluster of similar problems cropping up.

          Sam Van Oort added a comment - Yes, it's still in progress. It's mutated into a more general API set since we have a cluster of similar problems cropping up.

          Sam Van Oort added a comment -

          Out of review and merged to master, pending release as v2.0 after a bit of additional testing.

          Sam Van Oort added a comment - Out of review and merged to master, pending release as v2.0 after a bit of additional testing.

          Sam Van Oort added a comment -

          Released as version 2.0

          Sam Van Oort added a comment - Released as version 2.0

          think think added a comment -

          since the latest release I cannot color failing stages anymore. Before I could color failing stages dark red. Am I doing something wrong or is it because of this change?

          think think added a comment - since the latest release I cannot color failing stages anymore. Before I could color failing stages dark red. Am I doing something wrong or is it because of this change?

          Andrzej Chmielewski added a comment - - edited

          yeah, since the 2.0 release, the stages no longer indicate "failed" status, has anything changed on that matter svanoort ?

          Andrzej Chmielewski added a comment - - edited yeah, since the 2.0 release, the stages no longer indicate "failed" status, has anything changed on that matter svanoort ?

          Sam Van Oort added a comment -

          think Could you clarify what you mean in a separate ticket with instructions to reproduce please?

          I suspect you're referring to a fix done much earlier (several versions ago) that ensured caught errors would not result in a failed stage – unfortunately some people were taking advantage of that bug for tagging stages as failed... but can't tell without more info.

          Sam Van Oort added a comment - think Could you clarify what you mean in a separate ticket with instructions to reproduce please? I suspect you're referring to a fix done much earlier (several versions ago) that ensured caught errors would not result in a failed stage – unfortunately some people were taking advantage of that bug for tagging stages as failed... but can't tell without more info.

          Sam Van Oort added a comment -

          andrzej_zadorozny Same question as above: please open a separate ticket with instructions to reproduce. I suspect what you're wanting will be covered under JENKINS-26522 but cannot confirm off the top of my head.

          Sam Van Oort added a comment - andrzej_zadorozny Same question as above: please open a separate ticket with instructions to reproduce. I suspect what you're wanting will be covered under JENKINS-26522 but cannot confirm off the top of my head.

            svanoort Sam Van Oort
            svanoort Sam Van Oort
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: