Overview

Jenkins core never received the critical features or capabilities for plugin writers to do anything related to the job dependency graph (upstreams/downstreams) in the context of Declarative Pipelines.  Currently, the graph is near impossible to work with for plugin authors, and thus none of the plugins dealing with dependency graphs support declarative pipelines, even though they have now existed for several years.  So, a great deal of critical Jenkins functionality is effectively unavailable for users of declarative pipelines including: 

• Visualizing the Dependency Graph
• FanOut and FanIn of Jobs
• Blocking Execution based on Upstream/Downstream Status
• Evaluating any condition based on a dependency

Organizations which require these capabilities simply can't use declarative pipelines. 

Here are just a few other issues relating to dependency graph needs:

https://issues.jenkins-ci.org/browse/JENKINS-29913

https://issues.jenkins-ci.org/browse/JENKINS-19728

https://issues.jenkins-ci.org/browse/JENKINS-33577

https://issues.jenkins-ci.org/browse/JENKINS-19727

Background

Prior to pipelines, there were extensive, advanced, and widely used plugins related to the dependency graph. These types of plugins were vital to using Jenkins in many non-trivial codebases, and are no-less vital to using Declarative Pipelines. These were empowered by the extremely robust and extensive API surface surrounding the following class and it's constituents: 

hudson.model.DependencyGraph

When declarative pipelines were envisioned and implemented, all new classes and strategies surrounding job and builds were used.  The runtime construction of job instances, with their nodes and steps, became much more dynamic.  Thus, it's no surprise that the existing API's (and the entire ecosystem of plugins for visualization and job control) were completely unequipped to adapt to the declarative pipeline model.  They all require completely new logic for graph traversal if they are going to work, and that new logic requires a deep understanding of core to figure it out. 

Challenge

Simply put, nobody the core team never managed to retrofit Jenkins Core with a suitable replacement for the DependencyGraph API's for Declarative Pipelines, and it's created a huge gap for declarative pipeline users.  It seems very likely that the underlying reason this work was never done, is that it's really hard to do, AND that it hasn't reached the right level priority to justify it to the core team to work through it.  Also, it seems that some people might believe it's not possible to do in a perfectly and completely general way that addresses all use cases, and therefor not worth doing.  Nonetheless, there are numerous tickets asking how to solve dependency graph problems in declarative pipelines, most seem to be closed after suggestions on taking manual steps to work around the lack of a first-class DependencyGraph interface.  

It's worth noting that within the Jenkins plugin ecosystem, individual plugin maintainers are responsible for adapting these plugins to work with declarative pipelines (and NOT the core team).  This is true, however, in this case, it's unreasonable for each of these plugins to create their own implementations of the foundation logic for traversing the DependencyGraph.  This really is something that should be done in the core and exposed as an API.  Alternatively a new plugin which is at least backed by the core team which provides the necessary API's would suffice. Again, in either case, this is highly complicated logic. 

Prior Art

Nobody has tried to solve this problem, however some people have done some code "in the area".  

dnusbaum has created a plugin which contains logic which is likely to be very useful for this purpose.  This plugin provides a good API surface for walking the graph of a single pipeline, which should include the comprehension of upstream/downstream declarations, which will be the cornerstone for creating and traversing the larger DependencyGraph. 

https://github.com/jenkinsci/pipeline-graph-analysis-plugin/

lance513 created the following PR to try to add pipeline support to the job-fan-in plugin.  This PR contains one example of how one could potentially get handles to concrete jobs and emulate the old DependencyGraph structure within declarative pipelines. 

https://github.com/lance513/job-fan-in-plugin/commit/5c084ce5c21ebd039f5c175a5bf6b1b4cd4f058f

Summary

Please vote for this issue if you'd like to see these features developed.

This problem is certainly solvable, and this type of functionality can be added to Jenkins declarative pipelines.  The biggest barrier at the moment is that none of the people who have the required understanding of the existing dependency graph structure are working on it, nor motivated to do so.  The second biggest barrier would be the suggestions that being able to enumerate upstreams and downstreams of jobs is somehow impossible, unsupportable, or infeasible. 

In either case, feedback from the core team is requested as to whether or not they could ever be convinced to care about this problem and allocate resources to it.