Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35415

Multiple branch projects per repository with different recognizers

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major Major
    • branch-api-plugin
    • None
    • Jenkins 2.3

      As a developer/QA... It could be necessary to have multiple build methods/options on a project. And it could be possible if we could have 2 or more multibranch project on the same git project, but with different jenkinsfile file.

          [JENKINS-35415] Multiple branch projects per repository with different recognizers

          Nick Walke added a comment - - edited

          It would also be nice if the specific script name were configurable at the Github org level, just so we could use something like `.jenkinsfile` rather than `Jenkinsfile`. Talking about this: https://github.com/jenkinsci/workflow-multibranch-plugin/blob/34c7574b0cdcaeb03228422760d82e9d363f83cf/src/main/java/org/jenkinsci/plugins/workflow/multibranch/WorkflowBranchProjectFactory.java#L40

          Nick Walke added a comment - - edited It would also be nice if the specific script name were configurable at the Github org level, just so we could use something like `.jenkinsfile` rather than `Jenkinsfile`. Talking about this: https://github.com/jenkinsci/workflow-multibranch-plugin/blob/34c7574b0cdcaeb03228422760d82e9d363f83cf/src/main/java/org/jenkinsci/plugins/workflow/multibranch/WorkflowBranchProjectFactory.java#L40

          Martin Sander added a comment -

          jglick isn't this a exact duplicate of JENKINS-34561?

          Martin Sander added a comment - jglick isn't this a exact duplicate of JENKINS-34561 ?

          Jesse Glick added a comment -

          Actually it is not.

          Jesse Glick added a comment - Actually it is not.

          Martin Sander added a comment -

          Ah, I see: this is about having two different recognizers on a single project, while the implementation of JENKINS-34561 only allows different recognizers on different projects in an organization folder.

          Martin Sander added a comment - Ah, I see: this is about having two different recognizers on a single project, while the implementation of JENKINS-34561 only allows different recognizers on different projects in an organization folder.

          Jesse Glick added a comment -

          Correct.

          With JENKINS-34561 you could create separate organization folders, one per filename. But there is not currently any mechanism in branch-api to create distinct branch projects for the same branch inside a single MultiBranchProject, nor to create distinct MultiBranchProject for the same repository inside a single OrganizationFolder. In principle it could do so, perhaps appending some sort of disambiguating name prefix to each, though the configuration screen would get pretty complicated I am afraid. At that point I am inclined to reject feature bloat like this in favor of a more generic system like that (sketchily) proposed in JENKINS-37220.

          Jesse Glick added a comment - Correct. With  JENKINS-34561  you could create separate organization folders, one per filename. But there is not currently any mechanism in branch-api to create distinct branch projects for the same branch inside a single MultiBranchProject , nor to create distinct MultiBranchProject for the same repository inside a single OrganizationFolder . In principle it could do so, perhaps appending some sort of disambiguating name prefix to each, though the configuration screen would get pretty complicated I am afraid. At that point I am inclined to reject feature bloat like this in favor of a more generic system like that (sketchily) proposed in  JENKINS-37220 .

          Gabriel Lopez added a comment - - edited

          Hi jglick

          I understand that cloudbees doesn't have a reason to do it, but are there any plans on releasing that functionality of CloudBees Pipeline: Template to non-enterprise users? 

          I'm referring to custom code as pipeline.

          Gabriel Lopez added a comment - - edited Hi jglick .  I understand that cloudbees doesn't have a reason to do it, but are there any plans on releasing that functionality of  CloudBees Pipeline: Template  to non-enterprise users?  I'm referring to custom code as pipeline .

          Jesse Glick added a comment -

          I cannot comment except to say that I am unaware of any public commitment to do so.

          Jesse Glick added a comment - I cannot comment except to say that I am unaware of any public commitment to do so.

          Patrick Wolf added a comment -

          gloparm We have no plans to open source this functionality at this time. This feature is typically needed by larger organizations or enterprises with strict governance controls to enforce a single Pipeline.

          Using Shared Libraries in Pipeline allows users to create a common pipeline that can be reused across all repositories. Depending on the type of Pipeline you want to create the Jenkinsfile can be a single line or a simple conditional:

           

          if (BRANCH_NAME == 'master') {
            runDeploy()
          } else {
            runSimpleBuild()
          }

          The entire Pipeline for a deployment or a simple build is then defined in the Shared Library and the end user doesn't have to write anything.

           

          cc: cleclerc

          Patrick Wolf added a comment - gloparm We have no plans to open source this functionality at this time. This feature is typically needed by larger organizations or enterprises with strict governance controls to enforce a single Pipeline. Using Shared Libraries in Pipeline allows users to create a common pipeline that can be reused across all repositories. Depending on the type of Pipeline you want to create the Jenkinsfile can be a single line or a simple conditional:   if (BRANCH_NAME == 'master' ) { runDeploy() } else { runSimpleBuild() } The entire Pipeline for a deployment or a simple build is then defined in the Shared Library and the end user doesn't have to write anything.   cc: cleclerc

          Gabriel Lopez added a comment -

          Thanks!. We are running into a similar use case and we are looking at possibilities to solve it. The aforementioned plugin was one, but maybe Shared libraries will do the trick.

          Thanks again for the info!.

          Gabriel Lopez added a comment - Thanks!. We are running into a similar use case and we are looking at possibilities to solve it. The aforementioned plugin was one, but maybe Shared libraries will do the trick. Thanks again for the info!.

          Tim Black added a comment - - edited

          Just noticed [the issue I commented on](https://issues.jenkins.io/browse/JENKINS-43749?focusedCommentId=405518&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-405518) is marked a duplicate of this one, so cc'ing here:

          As noted above there are actually LOTS of reasons why someone would want multiple Jenkins sub-projects nested inside a repo. For example, this issue creates problems for folks that try to avoid submodules and tend to favor monorepos. In one such case, we have a "configuration-management" repo that we use to contain all of our configuration management tools and projects, e.g. packer, ansible, vagrant, etc.. Workflows are A LOT simpler having a monorepo for this, but when it comes to standing these processes up in Jenkins CI, we crash into this core problem.

          To support multiple Jenkinsfiles/multibranch projects in the same repo, we currently use jobdsl to explicitly and statically create a multibranch project for each known project location/subfolder. E.g. in this repo tree, we have 2 packer multibranch pipeline projects and 2 ansible multibranch pipeline projects:

          configuration-management
            - packer
              - subproject1
                - Jenkinsfile
                - src
              - subproject2
                - Jenkinsfile
                - src 
            - ansible
              - subproject1
                - Jenkinsfile
                - src
              - subproject2
                - Jenkinsfile
                - src 
          

          We use Folders to mirror the repo structure in Jenkins. The jobdsl to create the items in Jenkins looks like:

          folder("Packer") {
          // TODO: dynamically-create this list by searching repo on all branches
          //       for the superset of `packer/<packer_project_name>` subfolders that
          //       contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749
          packer_projects = ['subproject1', 'subproject2']
          for (packer_project in packer_projects)
          {
              multibranchPipelineJob("Packer/${packer_project}") {
                  .
                  .
                  factory {
                      workflowBranchProjectFactory {
                          scriptPath("packer/${packer_project}/Jenkinsfile")
                      }
                  }
              }
          }

          and

          folder("Ansible") {
          // TODO: dynamically-create this list by searching repo on all branches
          //       for the superset of `ansible/<ansible_project_name>` subfolders that
          //       contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749
          ansible_projects = ['subproject1', 'subproject2']
          for (ansible_project in ansible_projects)
          {
              multibranchPipelineJob("Ansible/${ansible_project}") {
                  .
                  .
                  factory {
                      workflowBranchProjectFactory {
                          scriptPath("ansible/${ansible_project}/Jenkinsfile")
                      }
                  }
               }
          }
           

          This approach works well, except that every time we want to add a new sub-project (ansible or packer subfolder with Jenkinsfile, in this case), we have to modify and push the jobdsl. It'd be great if workflow-multibranch could recursively scan for Jenkinsfiles and basically do what the jobdsl above is doing dynamically for me, as noted by my comments.

          I was planning on making this dynamic by modifying my jobdsl to execute a script to perform this Jenkinsfile scanning for me. I assume this is possible - but it's not a complete solution since jobdsl is  only updated during Jenkins infrastructure provisioning - infrequently - so this wouldn't detect most cases where someone adds, commits and pushes a subproject with a Jenkinsfile. We really need this scanning to be done by workflow-multibranch.

          Tim Black added a comment - - edited Just noticed [the issue I commented on] ( https://issues.jenkins.io/browse/JENKINS-43749?focusedCommentId=405518&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-405518 ) is marked a duplicate of this one, so cc'ing here: As noted above there are actually LOTS of reasons why someone would want multiple Jenkins sub-projects nested inside a repo. For example, this issue creates problems for folks that try to avoid submodules and tend to favor monorepos. In one such case, we have a "configuration-management" repo that we use to contain all of our configuration management tools and projects, e.g. packer, ansible, vagrant, etc.. Workflows are A LOT simpler having a monorepo for this, but when it comes to standing these processes up in Jenkins CI, we crash into this core problem. To support multiple Jenkinsfiles/multibranch projects in the same repo, we currently use jobdsl to explicitly and  statically  create a multibranch project for each  known  project location/subfolder. E.g. in this repo tree, we have 2  packer  multibranch pipeline projects and 2  ansible  multibranch pipeline projects: configuration-management - packer - subproject1 - Jenkinsfile - src - subproject2 - Jenkinsfile - src - ansible - subproject1 - Jenkinsfile - src - subproject2 - Jenkinsfile - src We use Folders to mirror the repo structure in Jenkins. The jobdsl to create the items in Jenkins looks like: folder( "Packer" ) { // TODO: dynamically-create this list by searching repo on all branches // for the superset of `packer/<packer_project_name>` subfolders that // contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749 packer_projects = [ 'subproject1' , 'subproject2' ] for (packer_project in packer_projects) { multibranchPipelineJob( "Packer/${packer_project}" ) { . . factory { workflowBranchProjectFactory { scriptPath( "packer/${packer_project}/Jenkinsfile" ) } } } } and folder( "Ansible" ) { // TODO: dynamically-create this list by searching repo on all branches // for the superset of `ansible/<ansible_project_name>` subfolders that // contain a Jenkinsfile: https://issues.jenkins.io/browse/JENKINS-43749 ansible_projects = [ 'subproject1' , 'subproject2' ] for (ansible_project in ansible_projects) { multibranchPipelineJob( "Ansible/${ansible_project}" ) { . . factory { workflowBranchProjectFactory { scriptPath( "ansible/${ansible_project}/Jenkinsfile" ) } } } }   This approach works well, except that  every time we want to add a new sub-project (ansible or packer subfolder with Jenkinsfile, in this case), we have to modify and push the jobdsl.   It'd be great if workflow-multibranch could recursively scan for Jenkinsfiles and basically do what the jobdsl above is doing dynamically for me , as noted by my comments. I was planning on making this dynamic by modifying my jobdsl to execute a script to perform this Jenkinsfile scanning for me. I assume this is possible - but it's not a complete solution since jobdsl is  only updated during Jenkins infrastructure provisioning - infrequently - so this wouldn't detect most cases where someone adds, commits and pushes a subproject with a Jenkinsfile. We really need this scanning to be done by workflow-multibranch.

            Unassigned Unassigned
            elliotmoso Elliot Morales
            Votes:
            30 Vote for this issue
            Watchers:
            36 Start watching this issue

              Created:
              Updated: