Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43749

Support multiple Jenkinsfiles from the same repository

      This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

      If I had multiple Jenkinsfiles... repository github.com/apple/swift

      /Package.jenkinsfile 
      /Incremental.jenkinsfile
      /Incremental-RA.jenkinsfile
      /Assert.jenkinsfile
      /src/…
      

      I would like to create multibranch Pipelines for each so I have the resulting structure:

      /Apple
      /Apple/Swift - Package
      /Apple/Swift - Incremental
      /Apple/Swift - Incremental-RA
      /Apple/Swfit - Assert
      

      Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

      I have written up examples and use cases in this doc

          [JENKINS-43749] Support multiple Jenkinsfiles from the same repository

          Francesco Pretto added a comment - - edited

          I'm relatively new to Jenkins but for me the reason this feature can't just be straightforwardly implemented in Jenkins is that it lacks a concept of a first-citizen SCM resource, meaning that SCM repositories can be configured in jobs but they are not treated as shareable resources themselves. If repositories could be configured as shareable resources (with no build steps configurable in them) and job could be configured to source on these resources (instead of configuring a separate git/svn connection for each job) it would be easier to implement a single queue of jobs to trigger for SCM changes on a single repository and it would be easier to filter which pipeline to trigger in case Job1 is configured with Jenkinsfile1 and Job2 is configured with JenkinsFile2. The single queue and the filter can probably be implemented with scripting but to have UI support and more declarative approach in the upstream pipelines (as opposed to imperatively select which job to trigger in the downstream SCM projects) something like what I am suggesting could be needed.

          Francesco Pretto added a comment - - edited I'm relatively new to Jenkins but for me the reason this feature can't just be straightforwardly implemented in Jenkins is that it lacks a concept of a first-citizen SCM resource, meaning that SCM repositories can be configured in jobs but they are not treated as shareable resources themselves. If repositories could be configured as shareable resources (with no build steps configurable in them) and job could be configured to source on these resources (instead of configuring a separate git/svn connection for each job) it would be easier to implement a single queue of jobs to trigger for SCM changes on a single repository and it would be easier to filter which pipeline to trigger in case Job1 is configured with Jenkinsfile1 and Job2 is configured with JenkinsFile2. The single queue and the filter can probably be implemented with  scripting  but to have UI support and more declarative approach in the upstream pipelines (as opposed to imperatively select which job to trigger in the downstream SCM projects) something like what I am suggesting could be needed.

          victor paul added a comment - - edited

          fajran Is it possible to share the code removing your firms references? Seems this has been the issue with Jenkins since years and there is not a stable solution.

          victor paul added a comment - - edited fajran Is it possible to share the code removing your firms references? Seems this has been the issue with Jenkins since years and there is not a stable solution.

          I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer. The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?

          Francesco Pretto added a comment - I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer . The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?

          Jeffrey Bennett added a comment - - edited

          I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

          In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating around|https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).

          That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

           

          Jeffrey Bennett added a comment - - edited I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline. In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating around| https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968 ] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa). That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.  

          Tim Black added a comment - - edited

          As noted above there are actually LOTS of reasons why someone would want multiple Jenkins sub-projects nested inside a repo. For example, this issue creates problems for folks that try to avoid submodules and tend to favor monorepos. In one such case, we have a "configuration-management" repo that we use to contain all of our configuration management tools and projects, e.g. packer, ansible, vagrant, etc.. Workflows are A LOT simpler having a monorepo for this, but when it comes to standing these processes up in Jenkins CI, we crash into this core problem.

          To support multiple Jenkinsfiles/multibranch projects in the same repo, we currently use jobdsl to explicitly and statically create a multibranch project for each known project location/subfolder. E.g. in this repo tree, we have 2 packer multibranch pipeline projects and 2 ansible multibranch pipeline projects:

          configuration-management
            - packer
              - subproject1
                - Jenkinsfile
                - src
              - subproject2
                - Jenkinsfile
                - src 
            - ansible
              - subproject1
                - Jenkinsfile
                - src
              - subproject2
                - Jenkinsfile
                - src 
          

          We use Folders to mirror the repo structure in Jenkins. The jobdsl to create the items in Jenkins looks like:

          folder("Packer") {
          // TODO: dynamically-create this list by searching repo on all branches
          //       for the superset of `packer/<packer_project_name>` subfolders that
          //       contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749
          packer_projects = ['subproject1', 'subproject2']
          for (packer_project in packer_projects)
          {
              multibranchPipelineJob("Packer/${packer_project}") {
                  .
                  .
                  factory {
                      workflowBranchProjectFactory {
                          scriptPath("packer/${packer_project}/Jenkinsfile")
                      }
                  }
              }
          }

          and

          folder("Ansible") {
          // TODO: dynamically-create this list by searching repo on all branches
          //       for the superset of `ansible/<ansible_project_name>` subfolders that
          //       contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749
          ansible_projects = ['subproject1', 'subproject2']
          for (ansible_project in ansible_projects)
          {
              multibranchPipelineJob("Ansible/${ansible_project}") {
                  .
                  .
                  factory {
                      workflowBranchProjectFactory {
                          scriptPath("ansible/${ansible_project}/Jenkinsfile")
                      }
                  }
               }
          }
          
          

          This approach works well, except that every time we want to add a new sub-project (ansible or packer subfolder with Jenkinsfile, in this case), we have to modify and push the jobdsl. It'd be great if workflow-multibranch could recursively scan for Jenkinsfiles and basically do what the jobdsl above is doing dynamically for me, as noted by my comments.

          I was planning on making this dynamic by modifying my jobdsl to execute a script to perform this Jenkinsfile scanning for me. I assume this is possible - but it's not a complete solution since jobdsl is  only updated during Jenkins infrastructure provisioning - infrequently - so this wouldn't detect most cases where someone adds, commits and pushes a subproject with a Jenkinsfile. We really need this scanning to be done by workflow-multibranch.

          Tim Black added a comment - - edited As noted above there are actually LOTS of reasons why someone would want multiple Jenkins sub-projects nested inside a repo. For example, this issue creates problems for folks that try to avoid submodules and tend to favor monorepos. In one such case, we have a "configuration-management" repo that we use to contain all of our configuration management tools and projects, e.g. packer, ansible, vagrant, etc.. Workflows are A LOT simpler having a monorepo for this, but when it comes to standing these processes up in Jenkins CI, we crash into this core problem. To support multiple Jenkinsfiles/multibranch projects in the same repo, we currently use jobdsl to explicitly and statically create a multibranch project for each known project location/subfolder. E.g. in this repo tree, we have 2 packer multibranch pipeline projects and 2 ansible multibranch pipeline projects: configuration-management - packer - subproject1 - Jenkinsfile - src - subproject2 - Jenkinsfile - src - ansible - subproject1 - Jenkinsfile - src - subproject2 - Jenkinsfile - src We use Folders to mirror the repo structure in Jenkins. The jobdsl to create the items in Jenkins looks like: folder( "Packer" ) { // TODO: dynamically-create this list by searching repo on all branches // for the superset of `packer/<packer_project_name>` subfolders that // contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749 packer_projects = [ 'subproject1' , 'subproject2' ] for (packer_project in packer_projects) { multibranchPipelineJob( "Packer/${packer_project}" ) { . . factory { workflowBranchProjectFactory { scriptPath( "packer/${packer_project}/Jenkinsfile" ) } } } } and folder( "Ansible" ) { // TODO: dynamically-create this list by searching repo on all branches // for the superset of `ansible/<ansible_project_name>` subfolders that // contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749 ansible_projects = [ 'subproject1' , 'subproject2' ] for (ansible_project in ansible_projects) { multibranchPipelineJob( "Ansible/${ansible_project}" ) { . . factory { workflowBranchProjectFactory { scriptPath( "ansible/${ansible_project}/Jenkinsfile" ) } } } } This approach works well, except that every time we want to add a new sub-project (ansible or packer subfolder with Jenkinsfile, in this case), we have to modify and push the jobdsl. It'd be great if workflow-multibranch could recursively scan for Jenkinsfiles and basically do what the jobdsl above is doing dynamically for me , as noted by my comments. I was planning on making this dynamic by modifying my jobdsl to execute a script to perform this Jenkinsfile scanning for me. I assume this is possible - but it's not a complete solution since jobdsl is  only updated during Jenkins infrastructure provisioning - infrequently - so this wouldn't detect most cases where someone adds, commits and pushes a subproject with a Jenkinsfile. We really need this scanning to be done by workflow-multibranch.

          If it may interest someone, I have an approach (https://stackoverflow.com/a/60316968/213871) that actually supports multiple pipelines in the same repository. It's not exactly a trivial solution (meaning that a first citizen support in Jenkins would just be better and it's good if this is considered for future releases) but it's very reliable and I have no compromises. I used it only in single branch scenarios, so far, so I don't know if it can be used also in multibranch. I also talk about it in my previous post (link).

          Francesco Pretto added a comment - If it may interest someone, I have an approach ( https://stackoverflow.com/a/60316968/213871 ) that actually supports multiple pipelines in the same repository. It's not exactly a trivial solution (meaning that a first citizen support in Jenkins would just be better and it's good if this is considered for future releases) but it's very reliable and I have no compromises. I used it only in single branch scenarios, so far, so I don't know if it can be used also in multibranch. I also talk about it in my previous post ( link ).

          Edgars Batna added a comment - - edited

          So far I had to work around this on every Jenkins installation that I've worked on, so +1. Usually the reasons being:

          • dependency build orchestration
          • complex trigger conditions
          • multiple build streams/environments (production/development)

          Edgars Batna added a comment - - edited So far I had to work around this on every Jenkins installation that I've worked on, so +1. Usually the reasons being: dependency build orchestration complex trigger conditions multiple build streams/environments (production/development)

          Philipp Hahn added a comment -

          My company also uses mono-repos and would like to have multiple `Jeninsfile`s discovered automatically. While researching this myself I found a Blog Post by Alexis Gauthiez where he describes their approach using JobDSL

          Philipp Hahn added a comment - My company also uses mono-repos and would like to have multiple `Jeninsfile`s discovered automatically. While researching this myself I found a Blog Post by Alexis Gauthiez  where he describes their approach using JobDSL

          Adam added a comment -

          When I ran into this same issue with my team I also came across the blog by Alexis Gauthiez, however it seemed a bit overkill for the size of our problem. I ended up going with a different approach. Multiple pipelines in parallel within one Jenkinsfile, I just wrote a blog about that approach too. Hopefully it might be useful to someone. 

          Adam added a comment - When I ran into this same issue with my team I also came across the blog by Alexis Gauthiez, however it seemed a bit overkill for the size of our problem. I ended up going with a different approach. Multiple pipelines in parallel within one Jenkinsfile, I just wrote a blog about that approach too . Hopefully it might be useful to someone. 

          I think the best solution is enhancing the pipeline declarative to allow a master jenkinsfiles to check the jobs condition (branch, tag, parameter, upstream …) and then execute any other declarative jenkinsfiles in project at any name base on the condition and the code in the project.

          NhatKhai Nguyen added a comment - I think the best solution is enhancing the pipeline declarative to allow a master jenkinsfiles to check the jobs condition (branch, tag, parameter, upstream …) and then execute any other declarative jenkinsfiles in project at any name base on the condition and the code in the project.

            Unassigned Unassigned
            jamesdumay James Dumay
            Votes:
            105 Vote for this issue
            Watchers:
            129 Start watching this issue

              Created:
              Updated: