Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43749

Support multiple Jenkinsfiles from the same repository

    XMLWordPrintable

Details

    Description

      This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

      If I had multiple Jenkinsfiles... repository github.com/apple/swift

      /Package.jenkinsfile 
      /Incremental.jenkinsfile
      /Incremental-RA.jenkinsfile
      /Assert.jenkinsfile
      /src/…
      

      I would like to create multibranch Pipelines for each so I have the resulting structure:

      /Apple
      /Apple/Swift - Package
      /Apple/Swift - Incremental
      /Apple/Swift - Incremental-RA
      /Apple/Swfit - Assert
      

      Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

      I have written up examples and use cases in this doc

      Attachments

        Issue Links

          Activity

            I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer. The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?

            ceztko Francesco Pretto added a comment - I don't know if this precisely answer the original issue here but I achieved to have multiple Jenkinsfile in the same repository/branch with the current feature set, I explained the procedure in this StackOverflow answer . The main idea is having separate Build and CSM projects configured in Jenkins, with the former that can be a simple Freestyle project with a polling on a repository and the latter a pipeline with a trigger on the CSM project. The trickier part was to add to add the skipDefaultCheckout directive in the Build pipeline, which is strange since I expected "Lightweight checkout" flag in the pipeline project to actually disable the checkout itself. Is this intended?
            jbennett20912 Jeffrey Bennett added a comment - - edited

            I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline.

            In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating around|https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa).

            That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.

             

            jbennett20912 Jeffrey Bennett added a comment - - edited I cannot speak to all the various use-cases that people have brought up, but it seems like a fair-number of people looking for this would be satisfied if multi-branch pipelines just supported an 'exclude regions' flag.  You get that behavior with regular-pipeline, but not multi-branch pipeline. In my situation (and seems like others have this as well), I have directoryA/Jenkinsfile and directoryB/Jenkinsfile, and I wish to trigger pipelineA when commits are made to DirectoryA, and pipelineB for DirectoryB.  There are [a few solutions floating around| https://stackoverflow.com/questions/49448029/multiple-jenkinsfile-in-one-repository/60316968 ] that effectively have both pipelineA and pipelineB starting, doing a 'validate' step, and then pipeline logic to short-circuit, causing completing when the commit is on the other side.  But I'd really prefer that the build NEVER start in the first place for the wrong pipeline.  Commit to DirectoryA -> only PipelineA is triggered.   That amounts to PipelineA configuring "DirectoryB" as an excludes region (and vice versa). That said, there are a bunch of use-cases all entwined herein, and this is only one of them.  But... it seems like a fairly well-defined one that might mitigate a lot of people's issues.  
            timblaktu Tim Black added a comment - - edited

            As noted above there are actually LOTS of reasons why someone would want multiple Jenkins sub-projects nested inside a repo. For example, this issue creates problems for folks that try to avoid submodules and tend to favor monorepos. In one such case, we have a "configuration-management" repo that we use to contain all of our configuration management tools and projects, e.g. packer, ansible, vagrant, etc.. Workflows are A LOT simpler having a monorepo for this, but when it comes to standing these processes up in Jenkins CI, we crash into this core problem.

            To support multiple Jenkinsfiles/multibranch projects in the same repo, we currently use jobdsl to explicitly and statically create a multibranch project for each known project location/subfolder. E.g. in this repo tree, we have 2 packer multibranch pipeline projects and 2 ansible multibranch pipeline projects:

            configuration-management
              - packer
                - subproject1
                  - Jenkinsfile
                  - src
                - subproject2
                  - Jenkinsfile
                  - src 
              - ansible
                - subproject1
                  - Jenkinsfile
                  - src
                - subproject2
                  - Jenkinsfile
                  - src 
            

            We use Folders to mirror the repo structure in Jenkins. The jobdsl to create the items in Jenkins looks like:

            folder("Packer") {
            // TODO: dynamically-create this list by searching repo on all branches
            //       for the superset of `packer/<packer_project_name>` subfolders that
            //       contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749
            packer_projects = ['subproject1', 'subproject2']
            for (packer_project in packer_projects)
            {
                multibranchPipelineJob("Packer/${packer_project}") {
                    .
                    .
                    factory {
                        workflowBranchProjectFactory {
                            scriptPath("packer/${packer_project}/Jenkinsfile")
                        }
                    }
                }
            }

            and

            folder("Ansible") {
            // TODO: dynamically-create this list by searching repo on all branches
            //       for the superset of `ansible/<ansible_project_name>` subfolders that
            //       contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749
            ansible_projects = ['subproject1', 'subproject2']
            for (ansible_project in ansible_projects)
            {
                multibranchPipelineJob("Ansible/${ansible_project}") {
                    .
                    .
                    factory {
                        workflowBranchProjectFactory {
                            scriptPath("ansible/${ansible_project}/Jenkinsfile")
                        }
                    }
                 }
            }
            
            

            This approach works well, except that every time we want to add a new sub-project (ansible or packer subfolder with Jenkinsfile, in this case), we have to modify and push the jobdsl. It'd be great if workflow-multibranch could recursively scan for Jenkinsfiles and basically do what the jobdsl above is doing dynamically for me, as noted by my comments.

            I was planning on making this dynamic by modifying my jobdsl to execute a script to perform this Jenkinsfile scanning for me. I assume this is possible - but it's not a complete solution since jobdsl is  only updated during Jenkins infrastructure provisioning - infrequently - so this wouldn't detect most cases where someone adds, commits and pushes a subproject with a Jenkinsfile. We really need this scanning to be done by workflow-multibranch.

            timblaktu Tim Black added a comment - - edited As noted above there are actually LOTS of reasons why someone would want multiple Jenkins sub-projects nested inside a repo. For example, this issue creates problems for folks that try to avoid submodules and tend to favor monorepos. In one such case, we have a "configuration-management" repo that we use to contain all of our configuration management tools and projects, e.g. packer, ansible, vagrant, etc.. Workflows are A LOT simpler having a monorepo for this, but when it comes to standing these processes up in Jenkins CI, we crash into this core problem. To support multiple Jenkinsfiles/multibranch projects in the same repo, we currently use jobdsl to explicitly and statically create a multibranch project for each known project location/subfolder. E.g. in this repo tree, we have 2 packer multibranch pipeline projects and 2 ansible multibranch pipeline projects: configuration-management - packer - subproject1 - Jenkinsfile - src - subproject2 - Jenkinsfile - src - ansible - subproject1 - Jenkinsfile - src - subproject2 - Jenkinsfile - src We use Folders to mirror the repo structure in Jenkins. The jobdsl to create the items in Jenkins looks like: folder( "Packer" ) { // TODO: dynamically-create this list by searching repo on all branches // for the superset of `packer/<packer_project_name>` subfolders that // contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749 packer_projects = [ 'subproject1' , 'subproject2' ] for (packer_project in packer_projects) { multibranchPipelineJob( "Packer/${packer_project}" ) { . . factory { workflowBranchProjectFactory { scriptPath( "packer/${packer_project}/Jenkinsfile" ) } } } } and folder( "Ansible" ) { // TODO: dynamically-create this list by searching repo on all branches // for the superset of `ansible/<ansible_project_name>` subfolders that // contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749 ansible_projects = [ 'subproject1' , 'subproject2' ] for (ansible_project in ansible_projects) { multibranchPipelineJob( "Ansible/${ansible_project}" ) { . . factory { workflowBranchProjectFactory { scriptPath( "ansible/${ansible_project}/Jenkinsfile" ) } } } } This approach works well, except that every time we want to add a new sub-project (ansible or packer subfolder with Jenkinsfile, in this case), we have to modify and push the jobdsl. It'd be great if workflow-multibranch could recursively scan for Jenkinsfiles and basically do what the jobdsl above is doing dynamically for me , as noted by my comments. I was planning on making this dynamic by modifying my jobdsl to execute a script to perform this Jenkinsfile scanning for me. I assume this is possible - but it's not a complete solution since jobdsl is  only updated during Jenkins infrastructure provisioning - infrequently - so this wouldn't detect most cases where someone adds, commits and pushes a subproject with a Jenkinsfile. We really need this scanning to be done by workflow-multibranch.

            If it may interest someone, I have an approach (https://stackoverflow.com/a/60316968/213871) that actually supports multiple pipelines in the same repository. It's not exactly a trivial solution (meaning that a first citizen support in Jenkins would just be better and it's good if this is considered for future releases) but it's very reliable and I have no compromises. I used it only in single branch scenarios, so far, so I don't know if it can be used also in multibranch. I also talk about it in my previous post (link).

            ceztko Francesco Pretto added a comment - If it may interest someone, I have an approach ( https://stackoverflow.com/a/60316968/213871 ) that actually supports multiple pipelines in the same repository. It's not exactly a trivial solution (meaning that a first citizen support in Jenkins would just be better and it's good if this is considered for future releases) but it's very reliable and I have no compromises. I used it only in single branch scenarios, so far, so I don't know if it can be used also in multibranch. I also talk about it in my previous post ( link ).
            gl1koz3 Edgars Batna added a comment - - edited

            So far I had to work around this on every Jenkins installation that I've worked on, so +1. Usually the reasons being:

            • dependency build orchestration
            • complex trigger conditions
            • multiple build streams/environments (production/development)
            gl1koz3 Edgars Batna added a comment - - edited So far I had to work around this on every Jenkins installation that I've worked on, so +1. Usually the reasons being: dependency build orchestration complex trigger conditions multiple build streams/environments (production/development)

            People

              Unassigned Unassigned
              jamesdumay James Dumay
              Votes:
              100 Vote for this issue
              Watchers:
              121 Start watching this issue

              Dates

                Created:
                Updated: