Background:

I have started to use Jenkins to manage infrastructure as code a whole lot more as DevOps engineering has shifted to owning CloudOps as well. I use Jenkins because other "solutions" do not have any sort of flexibility and do not scale well for a single person. When I have found an issues is when I am trying to do things like manage a series of infrastructure which is all related, or maintain infrastructure code which involves a lot of pieces which depend on each other. Essentially the repository looks like this: you have multiple Jenkinsfiles in a single repository and then a single branch where you would modify a couple of them. Then when that work is done, you would use a PR to merge that branch into master which would contain your "normally configured infrastructure"

Issue:

Right now due to the way that Multibranch works, you would need each of those infrastructure as code pieces to either be in multiple branches on the same repository or in separate repositories altogether. For an example of the multiple branches: you have a single IAC repo with a branch named something like "Machine master", and "Route master" which would maintain your cloud machines and the route master would maintain the hostnames of all of those machines. When you build a new machine, you obviously need a new route. There is no easy way to update both of those Jenkinsfiles which means creating a new branch based on both and then doing 2 PRs and you can't end-to-end test at the same time. The same problem happens when you talk about separate repos.

The ideal scenario would be that you have separate folders within the same repository and create one branch called "new machine" which modifies the "Machine master" Jenkinsfile and "Route master" Jenkinsfile, then on that new branch you can run your tests of both Jenkinsfiles and determine if the PR will be valid when it is submitted to fix both.

As of right now there is no IAC tool (Jenkins included) to be able to auto scan for those Jenkinsfiles/"infrastructure details file" (name I just made up but I think it will catch on) automatically, then give you details about if the PR will work when submitted

Proposal:

What I would like to do is to extend the scm-api plugin with a new type of probe (assuming that is the way that makes the most sense after looking over the code) which will, instead of checking on a single file to see if it exists and is buildable, run a recursive search of all the paths looking for a file of a specific name (by default Jenkinsfile). Then it will create that structure with either just named files or actual folder structure similar to a github org. Finally because we also need to be looking at branches as well as PRs it will use the tabbing structure which is currently reserved for PRs and that will be the various branches.

I attached a bit of a visual aid to the issue because my goal is to not mess with the UI elements of a multibranch pipeline at all and leverage what is already there.

My thoughts for the configuration would be just a radial that says something like "Traditional search or Recursive search" so that it is obvious what the previous behavior is but I could also change Traditional to "Specific Path" and just leverage all the other configuration options.

Final Thoughts:

From my perspective, I think the need is very much there for this to be a feature even if it would be used for situations that are not "ideal". The situation where you have Jenkinsfiles which depend on other Jenkinsfiles but should not be within the same build is pretty common when looking at infrastructure as code. I think at the moment making Jenkins more flexible in this way is going to make significantly more people to use multibranch

There is the concern over performance issues on very large repositories and I am not sure if there is a fix for that. I am hoping that everyone is as good as Jgit when navigating over the repository and have methods like filtered Tree walking but I am not certain if they do. The concern would come with very large/wide repos where you have to find a ton of Jenkinsfiles. Additionally when you have that situation, you have everyone building all at once on a single branch which is not ideal. I am not certain if there is a fix for this and it just has to be a warning somewhere that I was probably going to add in the help button.