Thanks ileitch! For that second point would you be interested in something like JENKINS-38442 ?

jamesdumay It's not clear in JENKINS-38442 how the graph would be rendered. Nested stages would solve the issue for me provided they do alter the graph, however ideally both nested stages and parallel stages would provide the most flexibility.

ileitch we are not even clear how things should be rendered because users ideas of how that should look are so diverse

I think there are some things we can do here to make the existing graph more acceptable in your case. We will also be looking at how to solve the nesting problem this year but I can't give a good ETA on it yet.

The sketch you provided: https://issues.jenkins-ci.org/secure/attachment/34100/blueocean.sketch%202016-09-28%2015-03-57.png
would not be ideal as it only alters the steps table, not the graph. If we applied this approach to our pipeline, the graph would only display a single "Test" stage, with all of the meaningful stages being shown in the list below. This would make the graph linear, and somewhat redundant.

ileitch yeah I think that design is totally out for the reason you provided and because two steps can appear on the screen it makes the running behaviour really complex.

ileitch actually would you be up for a Google Hangout in the next few weeks? We can discuss this in greater detail. Would love to see some of the Pipelines you have built and how they show up in Blue Ocean.

jamesdumay Please see my proposal here: https://issues.jenkins-ci.org/browse/JENKINS-38442?focusedCommentId=284175&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-284175

jamesdumay Sorry just noticed the Google Hangout question. Yeah I'm happy to do so, though be warned that I'm very new to Jenkins (I was tasked with switching away from TeamCity). We only have a single Pipeline at the moment.

Oh no that's perfect that you are new to things! Can you email me on your work email? I'm jdumay@cloudbees.com

A multiaxis way of displaying steps would be nice too, consider the following:

def hostPlatforms = ['ppc64le', 'x86_64']
def targetPlatforms = ['x86_64', 'power32', 'ppc64le', 'ppc64be', 'arm']
def hostOSs = ['ubuntu-16.04']

def builds = [:]

for (hostPlatform in hostPlatforms) {
    for (targetPlatform in targetPlatforms) {
        if (hostPlatform.equals('ppc64le') && targetPlatform.equals('x86_64')) {
            continue;
        }

        for (hostOS in hostOSs) {
            def name = 'build-' + hostPlatform + '-' + targetPlatform
            def myHostPlatform = hostPlatform
            def myHostOS = hostOS
            def myTargetPlatform = targetPlatform

            builds[name] = {buildToolchain(myHostPlatform, myHostOS, myTargetPlatform)}
        }
    }
}

stage('build') {
    parallel builds
}

This currently ends up with a long list scrolling down the page, but would be better represented by a display similar to the Matrix Project plugin.

This pipeline of ours is unworkable in the current view (1.0.0rc3) because we cannot click through to any information on our later stages (we have approximately 250 stages total, and this view only shows the first 90). Even if the full graph is unrenderable, it would be nice to have some way of loading these stages. Just a dropdown list would be preferable to the current state.

scgruber great description of your use case here. I have a few questions:

• What actions are you performing with the parallels?
• Why 250 stages? What meaning do they have?

jamesdumay this pipeline is an integration test against our different hardware platforms. We have a sequence of tests (asdf3, asdf4, ...) which run in parallel over our hardware platforms (zxcv1, zxcv2, zxcv3, ...). So we have a stage for each combination of test and hardware, and that's how we get to 250.

scgruber very interesting! Thanks for the additional context. When we plan this feature, could we contact you for a Google Hangout to discuss the new design?

jdumay sure, as long as it can be scheduled during US Pacific (UTC-7) business hours.

Any insight on this issue ? I'm in the same situation with even more parallel steps (over 400 and growing). Basically we have so much browser tests that we need to split them a lot. Previously we were using a distributed system of our own using a queue system and consumers that upload the results back to the Jenkins master server but it kinda suck and we really want to move back to a basic master / slave management of your Jenkins.

How can I help ? From my point of view, if we could simply say, using parallel, to not create a visual step for each branch of the parallel would be enough as a quickfix. Do you have any slack or whatever tool to talk directly ? I'm not used to Java so I might need some help with the contribution process.

+1 on the limit of 100 being too low.  I have some pipelines that are slightly larger than 100, and this is a real pain.  My pipelines only have a few (non-nested) stages, but have lots of parallel steps per stage.  I'm fine with dealing with a slightly long UI graph if that means I get to see all the nodes.

Is there any chance that a short-term solution could be put in place to allow the limit to be configurable, with no changes to the current UI graph display?  I have a feeling that refactoring the UI using some of the solutions proposed here will take quite a while, especially since the roadmap indicates it is "Not planned.

I looked into it a bit.  

The current UI makes one call to:

http://jenkins/blue/rest/organizations/jenkins/pipelines/_job_/runs/_runId_/nodes/

Internally, the nodes are retrieved via the PipelineNodeContainerImpl, which is a BluePipelineNodeContainer, which is a Container, which is a Pageable.

Therefore, the response that is constructed is a PagedResponse, which has a default limit of 100.  Since no start/limit is passed in the request, the default start=0 and limit=100 are used.  So even though the container has all of the nodes in it, only the first 100 are returned in the HTTP response.

The response also includes a Link header for the next page:

Link: </blue/rest/organizations/jenkins/pipelines/_job_/runs/_runId_/nodes/?start=100&limit=100>; rel="next"

However, the UI does not follow the link to retrieve the next page.

While debugging, I was able to set a breakpoint on the server side, and hack around the arbitrary 100 limit to make the server return more than 100.  The UI worked just fine with the number of nodes in my pipelines.

There are a few hacky ways to make all the nodes show on the current UI:

1. Increase the default limit constant
2. Provide a way to configure the default limit (and don't rely on a constant), therefore allowing users to modify the limit as they desire
3. Have the call to .../nodes pass a higher limit by default... e.g. .../nodes/?limit=500
4. Have the UI follow links to next pages (could be done by default, OR done if the user clicks on the "Unable to display more" node.

In any case, a short term solution would be much appreciated.

My team has the necessary skills to contribute a fix to the UI to fetch all available pages. If we were to submit such a PR, is there a reasonable chance it would be merged for inclusion in the next BlueOcean release, or is there more roadmap politics to navigate than that?

For anyone who is curious, the deficient object is the PipelinePager which doesn't actually page in data: https://github.com/jenkinsci/blueocean-plugin/blob/master/blueocean-dashboard/src/main/js/components/karaoke/services/pagers/PipelinePager.js past the initial page.

This bug it greatly hindering my work. Could it be moved up the roadmap or could someone respond to the gentleman that offered to fix it in the above comment?

benlangfeld if you submit a PR that fixes the issue it would absolutely be considered for inclusion; we'd just have a look and make sure tests pass, etc.. Submissions are always welcome!

Seconded. benlangfeld I had looked at this problem in the past. Another option is to do successive fetches until all nodes / stages are loaded. If you look at the REST responses, you'll see there is pagination data written into a "Link" response header IIRC. That's a way to determine whether there is additional data to be fetched, and you could write some logic to grab say n=100 and just perform successive fetches until the Link header indicates there is no more data.

We may want to be careful about doing a massive fetch up front (say n=500) as for complex pipelines this might have a perf impact server wide. I recall discussing this with vivek a while back, can you refresh my memory on whether it might be preferable to do a single large fetch (say n=500) or several smaller fetches (n=100) until all data is loaded? Intuitively fewer large fetches seems more efficient from client's perspective, but I seem to recall a concern with loading a large number of nodes concurrently in the context of a single request?

Following Link is precisely what I had in mind cliffmeyers. We'll prep a patch. Thanks everyone.

benlangfeld just make sure you are nice and up to date with master as some recent changes were merged for how it follows along (may not affect you, just FYI). As for fetching the pages - absolutely why not, if you have a PR that would be wonderful. Go for it!

A patch to resolve this is proposed at https://github.com/jenkinsci/blueocean-plugin/pull/1517. I would appreciate a review, particularly from cliffmeyers.