Any resolution dates on this ? I have the same issue with the nested stages not being displayed at all.

kaulk as the issue description says we currently do not have a resolution for this just yet. Make sure you watch this ticket for updates.

got it, thanks.

Proposal

This proposal takes the position that the vast majority of uses cases can be adequately represented in the graph by only 2 levels, with deeper nesting merely indicated on the graph. Nesting beyond the 2nd level defaults the list view with indentation indicating nesting depth.

Representing more than 2 levels in the graph would lead to complex hierarchies that would detract from the role of the graph as an overview.

1. New 'parallel_stage' command (I'm not proposing this exact naming, just the concept)

• Acts like the existing 'stage' command, however can be executed at the same time as other 'parallel_stage' regardless of nesting level
• In this design, all 18 'parallel_stage' of the "Test" stage containing a 'node' are added to the build queue simultaneously. You can think of this a bit like a matrix, however without fixed axes.

2. 'stage' and 'parallel_stage' can be nested arbitrarily

• The graph only shows the 1st and 2nd levels
• If all stages at the 2nd level do not themselves contain further stages, they are presented as a vertical linear chain (e.g Build stage in design)
• If any of the stages at the 2nd level contain nested stages, then they are represented as siblings (e.g Test stage in design)
• Stages at the 3rd level are indicated in the graph by the number of immediate descendants on the parent node (e.g "Functional (iPhone 7 10.2)" in the Build stage).
• Stages at the 3rd level and beyond are presented in the list view, using indentation to indicate nesting level
• Neither can be called within a 'node' block
• Not sure what desired behavior should be if a 'stage' is nested within a 'parallel_stage', perhaps only 'parallel_stage' can be nested within another 'parallel_stage'?

3. The existing 'parallel' command is superseded by multiple `parallel_stage` and can be deprecated

stage("Lint") { node { sh "lint" }  }

stage("Build") {
  parallel_stage("Shared") { node { sh "build" } }
  parallel_stage("App 1") { node { sh "build" } }
  parallel_stage("App 2") { node { sh "build" } }
}

stage("Test") {
  parallel_stage("Shared") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
  }

  parallel_stage("App 1") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPhone 7 10.2)") {
      parallel_stage("Batch 1") { node { sh "test" } }
      parallel_stage("Batch 2" { node { sh "test" } }
      parallel_stage("Batch 3") { node { sh "test" } }
    }
  }

  parallel_stage("App 2") {
    parallel_stage("Unit (9.3)") { node { sh "test" } }
    parallel_stage("Unit (10.2)") { node { sh "test" } }
    parallel_stage("Snapshot (iPhone 7 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPhone 7 10.2)") {
      parallel_stage("Batch 1") { node { sh "test" } }
      parallel_stage("Batch 2" { node { sh "test" } }
      parallel_stage("Batch 3") { node { sh "test" } }
    }
    parallel_stage("Functional (iPad Air 2 10.2)") { node { sh "test" } }
    parallel_stage("Functional (iPad Pro 10.2)") { node { sh "test" } }
  }
}

stage("Collect") { node { sh "collect" }  }

ileitch Thanks a lot for the update, can you please comment on how the following case can be represent with your proposal?

stage('build and test') {
parallel('Slave1' : {
stage('configure') {
}
stage('build') {
}
stage('test') {
}
}
parallel('Slave2' : {
stage('configure') {
}
stage('build') {
}
stage('test') {
}
}
}

If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

If stages nested within a parallel stage is not supported, does that mean stages in slave1 and slave2 cannot be shown as desired?

alanwu yes, that is correct - we can't display those right now.

alanwu jamesdumay I'd still love for some kind of `named_step`, which has nothing to do without how the graph is shown, but for naming/grouping the individual steps taken within each stage.

ileitch That's great, it will be very useful to have named_step. Thank you.

FWIW here is an example of a build where information about different substages has been omitted. It looks worse during the build, when some of the nested stages are temporarily displayed, but in a confused order.

jglick yes it should be flattened out sensibly (there are a few things going on that should eventually fix that) so it is less confusing even without showing it as explicitly nested

There has been a fair bit of off-list discussion on this, the vast majority seem to be satisfied by: 

• One level of stages shown in parallel streams of execution
• some first class support for matrix of combinations (in paralllel)

I haven't heard a strong case for nested parallel "circles" inside a non parallel stage (in that case people want to label a step or a group of steps, which is different to having a different circle/node during a parallel stream of execution). 

Here's a real world example of why one might have parallel stages and with some of those stages needing parallel "circles". We have an application that has these basic stages:

1. Clone SCM
2. Build
3. Build documentation on Windows slave (slow)
4. Custom static analysis tool (slow)
5. Multiple Test suites
6. Archive build artifacts and test results

Steps 1-3 can all run in parallel and since 2 and 3 both take a long time, it's actually desirable for them to be running while we're doing all our various suites in parallel. Here's what the ideal Jenkinsfile would look like (IMHO):

stage('Clone') {
    checkout scm
}

testSuites = [ 'suite1', 'suite2', 'suite3' ]

parallel([
    buildAndTest: {
        stage('Build') {
            node('linux') {
                sh 'make'
                stash includes: 'pkg/**/*', name: 'build-artifacts'
            }
        }
        stage('Test') {
            testSteps = [:]
            for (suite in testSuites) {
                testSteps[suite] = {
                    node('linux') {
                        unstash 'build-artifacts'
                        sh "run_tests $suite"
                        stash includes: 'test_results/**/*.xml', name: "test-$suite"
                    }
                }
            }
            parallel(testSteps)
        }
    },
    docs: {
        stage('Documentation') {
            node('windows') {
                bat 'build_docs.exe'
            }
        }
    },
    staticAnalysis: {
        stage('Static Analysis') {
            node('linux') {
                sh 'run_static_analysis'
            }
        }
    }
])

stage('Archive') {
    unstash 'build-artifacts'
    for (suite in testSuites) {
        unstash "test-$suite"
    }
    archive 'pkg/**/*'
    junit 'test_results/**/*.xml'
}

I'm not exactly sure about where that checkout scm should go with all the parallel stuff, I haven't actually run this particular thing yet. Also while writing this I wasn't really sure what to name the parallel sections at the top level, since I really want them to get their names from the nested stages.

As for how this would look in the UI (only really thinking of Blue Ocean), I would want it to look how this graphviz DAG would render:

digraph {
  graph [rankdir=LR]

  Clone -> Build
  Clone -> Documentation
  Clone -> "Static Analysis"

  Documentation -> Archive
  "Static Analysis" -> Archive

  Build -> "Test suite1"
  Build -> "Test suite2"
  Build -> "Test suite3"

  "Test suite1" -> Archive
  "Test suite2" -> Archive
  "Test suite3" -> Archive
}

Here's what I was thinking it might look like in Blue Ocean

The workaround I have planned for this particular build is to have the docs and static analysis happen in other builds that I then have to wait for them to finish. Something like this:

stage('Clone') {
    checkout scm
}

stage('Build') {
    parallel([
        build: {
            node('linux') {
                sh 'make'
                stash includes: 'pkg/**/*', name: 'build'
            }
        },
        docs: {
            build job: 'docs', wait: false
        },
        staticAnalysis: {
            build job: 'staticAnalysis', wait: false
        }
    ])
}

testSuites = [ 'suite1', 'suite2', 'suite3' ]

stage('Test') {
    testSteps = [:]
    for (suite in testSuites) {
        testSteps[suite] = {
            node('linux') {
                unstash 'build'
                sh "run_tests $suite"
                stash includes: 'test_results/**/*.xml', name: "test-$suite"
            }
        }
    }
    parallel(testSteps)
}

// method to lookup the docs and staticAnalysis jobs we started and see if they're done
@NonCPS
def findSomeBuild(jobName)
    // do Jenkins groovy stuff
    return correctBuildNumber || null
end

stage('Archive') {
    unstash 'build'

    def docsBuildNumber = null    
    waitUntil {
        docsBuildNumber = findSomeBuild('docs')
    }
    
    def staticAnalysisBuildNumber = null
    waitUntil {
        staticAnalysisBuildNumber = findSomeBuild('staticAnalysis')
    }
    
    step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'docs', selector: [$class: 'SpecificBuildSelector', buildNumber: docsBuildNumber]])
    step([$class: 'CopyArtifact', filter: 'pkg/**/*', projectName: 'staticAnalysis', selector: [$class: 'SpecificBuildSelector', buildNumber: staticAnalysisBuildNumber]])

    unstash 'documentation'
    unstash 'static_analysis'
    for (suite in testSuites) {
        unstash "test-$suite"
    }
    archive 'pkg/**/*'
    junit 'test_results/**/*.xml'
}

@b_dean: I totally agree with your proposition!

We have a similar need. Currently, we are using FreestyleJob and DeliveryPipeline plugin to handle it. I'm looking forward to having such feature implemented to switch to JenkinsFile and BlueOcean.

May I suggest the following as a means to represent nested stages visually?

Display the top-most stages and consider the result of the children stages as their success/failure. Additionally, to view the nested stages, you could then click on a stage in the display, which would "drill-down" and show the sub-stages within that stage, like a zoom of sorts.

This seems it would allow for an infinite level of nesting without creating a drastic overhead in display logic.

I know it doesn't give a 100% overhead of all tasks simultaneously, but it would at least be some kind of a middle-road in my opinion.

ikari7789 could you draw a picture and upload it here please?

Sorry for my lack of artistic ability, but I've tried to portray what I'm thinking in my mind.

So at the top layer, you'd be presented with the normal staged output for the pipeline, if any stages have nested stages, it would display a zoom-in button or something (this is obviously not final).

When you click the zoom button, it will then display the stage(s) within that stage (in a single level increment):

This process could basically allow for an infinite expansion inwards.

The red/green status would be determined by the final result of all children stages.

Please look into this issue as this will give neater and intuitive UI for debugging purpose where we can see separate logs stage wise.

In my opinion, syntax given in Example2 should be the correct syntax to visualize stages in parallel and also the word "Parallel" can be removed from parallel step and may be we can show Labels there instead over the parallel steps with stage names for each steps.

Lable foo
---O--O--O-----------

---O--O--O-
s1 s2 s3

Above illustration is not looking good in visual mode, please ignore the table structure that got created. Stages for each label are in parallel with label name over stages of a particular line of execution.

Please refer to "Proposed Parallel Stages.JPG" attached here.

I like Andrew's idea, perhaps simply represent the sub-pipeline as a pipeline in itself? Continuing to re-use existing resources

As he mentioned, this should provide a UI framework to support additional nesting.

Here's a sketch that might help Andrew feel better about his artwork  Note the continuation from & to the rest of the build, perhaps add the previous & next step names and/or a better location for zoom out icons.

I agree, "zooming" into a nested stage seems like the most sensible and scalable solution, with the best possible user experience for all potential situations.

It also allows for re-usability of existing designs/assets, as this would be an addition on top of the existing way of working, instead of a re-thinking of the current way of working.

How about implementing a small improvement first, like only one level of stages inside parallel blocks, e.g.:

stage ('Build') {
    // Build something
}
parallel('Test1' : {
    stage('JUnit') {
    }
    stage('Firefox') {
    }
}, 'Test2' : {
    stage('DBUnit') {
    }
    stage('Edge') {
    }
}, 'Test3' : {
    stage('Jasmine') {
    }
    stage('Safari') {
    }    
}
stage ('Dev') {
}

Of course it'd be nice if the parallel block's names would be displayed somehow - but most important to me would be just the possibility to somehow split the different parallel blocks into smaller junks using something like stages.

macdrega that I think is what most people agree on as some kind of next step for this. 

Great than, was not sure about that.

macdrega +1, no more words on this needed from my side.

Hm, the one you removed, was even more my favorite. To visualize enclosing Steps, some kind of background bubble could be used and if a pipeline intermixes nested stages between parallel splits, then just draw the pipeline as far as it is done and identify the current steps by there name to display an estimate for each.

manschwetus

Sorry for that - just wanted to be crystal clear about what I think should be the next step.

Leaving out any possibility to have nested stages at all probably would make it easier to at least get a first shot for partitioning a parallel block into smaller parts.

But of course on the long run I'd be happy to see if we could also put the parallel block itself inside a stage, and possibly even have stages in stages in parallel blocks which can have parallel blocks

However for the time being I'd suggest to concentrate on a hopefully tiny additional feature for visualizing stages inside parallel blocks. Probably it'll even be easier to discuss about the remaining features once this is available so everybody is able to validate this against their use cases.

Coming from the embedded development world I'd like this in the form of an ability to run same stages for multiple platforms in parallel. I guess I'm after "parallel sets of stages". Like so:

These should be able to complete independently, but if one fails, the whole pipeline should be marked as unstable.

xtrom0rt do you mean more like a matrix of combinations? 

Maybe we should start thinking about a BO-style renderer for matrix jobs?

sophistifunk yes - and I know abayer wants it as an explicit thing in declarative (there are a few precursors though - like being able to have statuses/results per stage or step so it isn't all yellow or red or green, but nuanced) - once that is in place, yeah a matrix is absolutely needed. 

I'm not entirely sure, to be honest. Never fiddled much with the GUI jobs of old, as I went straight for the pipeline as code. It certainly sounds about right. Basically, what I'd need to accomplish is running the same set of steps for a number of different platforms. The steps are the same, but how they are accomplished for a particular platform might vary. That difference is handled in the shared library based on some parameterization.

For example, a build step for one device might just be a single compilation of latest commit on git master branch. Another device might need to compile one set of latest master source code and after that a set of latest master source code from another git repo using artifacts from the first one. Same goes for the other steps. The stage goal is the same (checkout, build, deploy for test, test, etc), but the way to accomplish it might vary.

After all the steps are through, the job result should reflect the status for that particular platform. Even though different platforms might utilize the same source code, they are separate products and I need to know everything for that particular product is working as intended

I am actually looking to do the same thing as xtromort

Went to stack overflow asking for suggestions for a problem that requires a nested parallel stage and was pointed to this item. Have there been any updates to this at all?

cryptomatt not yet no. One level of nesting has some low level support in the visualisation (ie a paralell branch with multiple stages) but is not glued to the api yet. 

This would be really nice to have!

+1 from me as well.

In our company, I also have a use case for nested parallel stages. We have a huge monorepo building multiple maven modules.  We are currently converting them to pipelines, and we parallelize a lot of steps:

1. Build Jars 
2. Package & Test (Parallel)
   1. Build RPM 
   2. Build Docker
   3. Run Tests (Parallel)
3. Promote build

We have about 20k unit tests which take a long while to run - to speed up the build, we paralleized the tests on multiple nodes, and during the test run, we also want to perform other steps.

Currently, when the Run Tests stage is ran, everything in blue ocean is broken - the view does not display any nodes even, only steps. 

I also think the ikari7789 suggestion seems like the best option.

I'm a software developer at SAP, and we use Jenkins including Blue Ocean to provide a build pipeline for our partners. We've also stumbled upon this issue, which holds us back from enabling parallel test execution by default. We introduced a feature toggle for this, which we would like to remove as soon as this issue is resolved.

I understand this issue is not easy to solve, but for our short term needs it would be good enough if nested parallel stages were not displayed at all in Blue Ocean. Is it possible to do this in the foreseeable future?

Best regards,
Florian

Something that I find interesting is that nested stage visualizations kind of work while your pipeline is running and when your nested stage is created within a script block. If I have some pipeline code such as the following:

pipeline {
    agent any
    stages {
        stage("Testing") {
            steps {
                script {
                    stage("A") {
                        parallel "1": {
                            println "1"
			    sleep 500
                        }, "2": {
                            println "2"
			    sleep 500
                        }, "3": {
                            println "3"
			    sleep 500
                        }
                    }
                }
            }
        }
    }
}

Then while this pipeline is running I see the following in Blue Ocean:

But after the pipeline finishes the nested stage and the parent stage get merged:

What would be really great is if the visualization that I see while the pipeline is running does not change when the pipeline finishes.

This is quite important for many CI scenarios. Quick example:

first stage: build - executed on a build node
parallel launch of 3 logical "blocks":
___ 1 block for 1 type of integration test - executed on a specialized integration node

___ 1 block for 1 other type of integration test - executed on another specialized integration node
___ 1 bigger block for deployment and functional tests:
       ___1 stage for deployment - executed on the target app environment
       ___1 stage for functional tests - executed on a specialized functional test runner node

Unfortunately... normal stages can only nest either "parallel" or "steps", not further stages. But here we need another stage cause the third block is actually split into two stages - steps don't work because they can't be executed on different nodes. And these stages are sequential, can't be put in a parallel block: running the tests before deployment is useless.

In my opinion perfect is the enemy of good.

Just define a maximum level of nesting and stick to it (call it a design decision, limitation, whatever). I know it's hard to say what that level should be, and we all know the apocryphal statement by Gates ("640k of RAM ought to be enough for anyone"), but I'd say 3 levels of nesting should be enough.

This should simplify the implementation of both the actual pipeline code and also the visualization.

+1 for ccaraivan

Have long running step and can't execute it in a parallel way because of this nested pipeline limitation.

3 level of nested pipeline should be enought

+1

I think this is really really needed for making flexible pipeline.

Because of feature toggling, 

we need

checkout - build - test

checkout - build - test

checkout - build - test 

... (so many)

(only different is source code checkouted and build command line)

If this feature can not be implemented, I will search another tool.

Intel is also very interested in this visualization enhancement.

We are working on the visualization now for sequential/nested stages. PR for the support in Declarative 1.3: https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/227. 

Great to hear 

Two questions:

1. Are you also planning to support the scripted pipeline?
2. As far as I see this the most important thing is to support stages within parallel blocks (in the first step no further nesting required). Will this be covered? According to the description I fear that this will not be included.

I see that the above PR is merged but issue is still in progress. Is this still being worked on?

When done will it support nested parallels (parallel in parallel)? As shown on pipeline_mockup.png ? Because with version 1.3.1 it still doesn't do that.

Thanx

AFAIK there's no plan to support nested parallels at the moment.

We really need that feature. Currently, we have two levels of parallelization and the visualization in BlueOcean is totally broken. Really, looking forward to supporting at least two level of parallelization (parallel within parallel).

I might be mistake as I don't have numbers, but the parallel pipeline described in https://jenkins.io/blog/2018/07/02/whats-new-declarative-piepline-13x-sequential-stages/ seems a pretty common use-case for pipelines. It might be the most natural and most common pipeline even. Therefore at least this kind of pipeline should get appropiate visualiation, IMHO.

First of all, I really like nested sequential stages, because I use a build tool already running things in maximum parallel (bazel), I don't want to run things with parallel stages which will be blocking on the Bazel server anyway.

However, I noticed a very annoying visualization issue when a sub-stage has FAILED (due to a failed step): the parent stage is marked SUCCESS. I would assume the failed state would propagate up to its parent.

It not only hides the error in the graph and makes it's hard to spot where in the pipeline the error occurred, it also hides the existence of the sub-stage. In the screenshot below, the "Code Style" stage has multiple sub-stages, that got somehow merged with its parent in this view, but the steps shown are part of one of the sub-stages.

Sub-stages are shown during the run, though.

As this Jira issue is still in-progress, I didn't want to open a new one. And I couldn't find any other issue reporting similar issues.
(Seeing this on 1.7.2 / Core 2.121.2)

Update: in 1.8.2 I now see all stages, but all flattened, as if there is no nesting in stages. It has improved the situation a lot to spot the errors, though, but it's far from ideal.

Blue Ocean 1.8.x adds support for visualizing sequential stages, so I am closing this as a dupe of JENKINS-49050 (although note that I believe that issue is specific to declarative pipeline).

macdrega I don't think there are any open feature requests specific to nested stages in scripted pipelines. We could reopen this issue and modify the title to be a little more specific, or we could open a new issue (probably better to have a more focused issue, assuming some of the people here are satisfied with the nested stage support for declarative in blue ocean 1.8), but I think either would be fine.

I think originally, this ticketed was also about having parallel blocks within parallel blocks. So at least two level of parallelization. 
Although the declarative syntax does not support it, you can have scripted parallel blocks within a parallel block in a declarative pipeline. The result is not visualized properly.

kurzy Yes, this issue has multiple feature requests in the comments, so it seems to me like it would make more sense to open individual issues so each request can be discussed separately, but we could also just reopen this issue.

dnusbaum: Hmm, looking at the examples above think this issue was originally targeted to scripted Pipeline. However as I like the idea of having smaller feature requests which are easier to implement on its own. Therefore I will create a new feature request hoping it'll not be merged with similar feature requests.

For those interested: I raised a new fr here: JENKINS-53162

sophistifunk is there still no plans to support nested parallels? we have built a nested parallel scripted pipeline, and the blue ocean view seems to change between various representations. More worryingly, if a task fails it shows as GREEN  . Something like pipeline_mockups.png from above would be amazing

dnusbaum As suggested, I have created an issue for two level of parallelization: https://issues.jenkins-ci.org/browse/JENKINS-54010

Is there any luck for implementing the pipeline mockup proposed by Ben Dean? https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=292647&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-292647

Should we create a separate issue for this?

This is a very long thread so forgive me if this is already covered...

Can anyone explain why a stage that contains sub-stages is displayed as a preceding node instead of a container?

Take this example:

The stage "Test" performs no actions of its own, but it contains the sub-stages that follow.

Wouldn't it make more sense to display it as a container, something like: