Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47874

Support multiple repositories in multi-branch pipeline

    • Blue Ocean 1.5 - beta 3

      Within a single job, we need to source branches from multiple repositories that we do not have control over. Currently, if the same branch name occurs in different repositories, only the first is built and subsequent identical branches are dropped.

       

      I suggest that the remote name (if configured) be used to disambiguate the branches, so that they can all be built.

      eg.
      Remote name: linux
      Branch: master

      Remote Name: linux-next
      Branch: master11

      These would be resolved as linux/master and linux-next/master.

       

          [JENKINS-47874] Support multiple repositories in multi-branch pipeline

          Alastair D'Silva added a comment - - edited

          Interesting, that is the behavior I was expecting, but not what I am seeing.

          I'm using a Multibranch pipeline job, I've attached an excerpt from the scan log (branchindexing.log).

          I have configured multiple git sources, each one has a unique "Configure Remote Name" value. Is there some other configuration option required to make the origin appear in the branch name?

           Note the collision at the bottom of the log:

          Checking branches...
            Checking branch master
              Met criteria
          Ignoring Linux Kernel » master from source #5 as source #3 owns the branch name

           

          Alastair D'Silva added a comment - - edited Interesting, that is the behavior I was expecting, but not what I am seeing. I'm using a Multibranch pipeline job, I've attached an excerpt from the scan log (branchindexing.log). I have configured multiple git sources, each one has a unique "Configure Remote Name" value. Is there some other configuration option required to make the origin appear in the branch name?  Note the collision at the bottom of the log: Checking branches...   Checking branch master     Met criteria Ignoring Linux Kernel » master from source #5 as source #3 owns the branch name  

          Mark Waite added a comment -

          I don't understand how you've configured your multi-branch pipeline job, or more correctly, I don't understand what you're trying to achieve with your multi-branch pipeline job configuration.

          Based on your branch scanning log, I'd guess that you've configured multiple repositories in the multi-branch pipeline definition. I don't know what that should do, or how it should do it.

          As far as I understand a multi-branch pipeline, it is designed to create a single job for each branch in the repository which satisfies the branch name matching conditions. Why not just define a multi-branch pipeline job for each of your repositories, rather than combining them all into a single multi-branch pipeline job?

          I've confirmed in a test job that checkout of different repositories and/or different branches is well behaved from a pipeline job in a multi-branch pipeline. That test job runs as expected, reporting that each subdirectory where a checkout is performed contains the correct repository and branch.

          Can you confirm that you're placing multiple repositories in the multi-branch pipeline definition? If that's what you're doing, please explain more about the desired outcome of that multi-branch pipeline.

          Mark Waite added a comment - I don't understand how you've configured your multi-branch pipeline job, or more correctly, I don't understand what you're trying to achieve with your multi-branch pipeline job configuration. Based on your branch scanning log, I'd guess that you've configured multiple repositories in the multi-branch pipeline definition. I don't know what that should do, or how it should do it. As far as I understand a multi-branch pipeline, it is designed to create a single job for each branch in the repository which satisfies the branch name matching conditions. Why not just define a multi-branch pipeline job for each of your repositories, rather than combining them all into a single multi-branch pipeline job? I've confirmed in a test job that checkout of different repositories and/or different branches is well behaved from a pipeline job in a multi-branch pipeline. That test job runs as expected, reporting that each subdirectory where a checkout is performed contains the correct repository and branch. Can you confirm that you're placing multiple repositories in the multi-branch pipeline definition? If that's what you're doing, please explain more about the desired outcome of that multi-branch pipeline.

          Alastair D'Silva added a comment - - edited

          That is correct, what I am after is a single multibranch job that sources it's branches from separate repositories. Each job created by the multibranch job only ever refers to a single branch, which originates from one of the many repos specified by the multibranch pipeline.

          We need these to be managed as a single job to avoid a maintenance nightmare - we have ~20 multibranch jobs, each picking branches from 3-8 repos. Also, from a UX perspective, it's a lot easier for our maintainers to look at a single Blue Ocean results page with all the branches reported, than have to cycle through one for each repo.

          It sounds like the Git plugin is behaving as expected, and the issue lies with how the Multibranch Pipeline plugin handles the branches. Would you agree that this should be logged against that plugin instead?

           

          [later]Looks like the Branch API is responsible:

              public String getName() {
                  // TODO this could include a uniquifying prefix defined in BranchSource
                  return head.getName();
              }

          Alastair D'Silva added a comment - - edited That is correct, what I am after is a single multibranch job that sources it's branches from separate repositories. Each job created by the multibranch job only ever refers to a single branch, which originates from one of the many repos specified by the multibranch pipeline. We need these to be managed as a single job to avoid a maintenance nightmare - we have ~20 multibranch jobs, each picking branches from 3-8 repos. Also, from a UX perspective, it's a lot easier for our maintainers to look at a single Blue Ocean results page with all the branches reported, than have to cycle through one for each repo. It sounds like the Git plugin is behaving as expected, and the issue lies with how the Multibranch Pipeline plugin handles the branches. Would you agree that this should be logged against that plugin instead?   [later] Looks like the Branch API is responsible:     public String getName() {         // TODO this could include a uniquifying prefix defined in BranchSource         return head.getName();     }

          Mark Waite added a comment -

          I suspect that changes will be needed in multiple plugins to support your use case.

          I don't think changes are needed in the git plugin, but I would expect changes would be needed in the branch api, and possibly in the branch source plugins and the organization folder plugin.

          Mark Waite added a comment - I suspect that changes will be needed in multiple plugins to support your use case. I don't think changes are needed in the git plugin, but I would expect changes would be needed in the branch api, and possibly in the branch source plugins and the organization folder plugin.

          Rok Povse added a comment -

          We would really need this issue to be fixed. Could you please provide us with a solution?

          Rok Povse added a comment - We would really need this issue to be fixed. Could you please provide us with a solution?

          Jon Kelley added a comment -

          markewaite

          Here is a use-case for you. See the multi-lined comment areas below. Is there any ideas to achieve something similar??

          We need to run an external pipeline script which lives in a scripts repository, because we don't want to put our shared pipeline scripts in 900+ chef module repositories or use submodules.

          We've hit a brick wall for our use case since updating Job DSL 1.70 to 1.74 since pipelineJob is dead for supporting SCM.

          multibranchPipelineJob is lacking multi git functionality.

          // Generate PR push pipeline jobs for all of the repos passed in via REPO_LISTdef repoList = "${REPO_LIST}".trim().replaceAll('"', '').split(",")
          
          println "Generating PR push jobs for the following repos:: $repoList"
          repoList.each {
              def cookbook = it
              multibranchPipelineJob("chef-${cookbook}-PR-push") {
                  description("Chef push pipeline job")
                  branchSources {
                      github {
                          buildOriginPRHead() // Build fork PRs (unmerged head).
                          // buildOriginPRMerge() // Build fork PRs (merged with base branch).
                          buildOriginBranchWithPR() // Build origin branches also filed as PRs.
                          checkoutCredentialsId('fffff-fffff-fffff-fffff')
                          // scanCredentialsId('github-ci')
                          repoOwner('myOwner')
                          repository("${cookbook}")
                      }
                  }
                  configure {
                      // my Jenkinsfile doesn't exist in the chef coookbooks for 900 repositories...
                      // how can we set a seperate git repo for shared scripts?
                      // https://stackoverflow.com/questions/48284589/jenkins-jobdsl-multibranchpipelinejob-change-script-path
                      it / factory(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory') {
                          owner(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject', reference: '../..')
                          scriptPath("jenkins/[where ever you want]/Jenkinsfile")
                      }
                  }
              }
           }

           

          pipelineJob worked excellent for our needs, the old broken syntax is below:

          repoList.each {
            def cookbook = it
            pipelineJob("chef-${cookbook}-PR-push") {
              definition {
                cpsScm {
                  lightweight(true)
                  scriptPath('jenkins/[where ever you want]/Jenkinsfile')
                  scm {
                    git {
                      branch('master')
                      remote {
                        github("organization/scripts", 'ssh')
                        credentials('ffff-fff-fff-ffffff)
                      }
                    }
                  }
                }
              }
            }
          }
           

          Jon Kelley added a comment - markewaite Here is a use-case for you. See the multi-lined comment areas below. Is there any ideas to achieve something similar?? We need to run an external pipeline script which lives in a scripts repository, because we don't want to put our shared pipeline scripts in 900+ chef module repositories or use submodules. We've hit a brick wall for our use case since updating Job DSL 1.70 to 1.74 since  pipelineJob  is dead for supporting SCM. multibranchPipelineJob is lacking multi git functionality. // Generate PR push pipeline jobs for all of the repos passed in via REPO_LISTdef repoList = "${REPO_LIST}" .trim().replaceAll( ' "' , '').split(" ,") println "Generating PR push jobs for the following repos:: $repoList" repoList.each { def cookbook = it multibranchPipelineJob( "chef-${cookbook}-PR-push" ) { description( "Chef push pipeline job" ) branchSources { github { buildOriginPRHead() // Build fork PRs (unmerged head). // buildOriginPRMerge() // Build fork PRs (merged with base branch). buildOriginBranchWithPR() // Build origin branches also filed as PRs. checkoutCredentialsId( 'fffff-fffff-fffff-fffff' ) // scanCredentialsId( 'github-ci' ) repoOwner( 'myOwner' ) repository( "${cookbook}" ) } } configure { // my Jenkinsfile doesn't exist in the chef coookbooks for 900 repositories... // how can we set a seperate git repo for shared scripts? // https://stackoverflow.com/questions/48284589/jenkins-jobdsl-multibranchpipelinejob-change-script-path it / factory(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory' ) { owner(class: 'org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject' , reference: '../..' ) scriptPath( "jenkins/[where ever you want]/Jenkinsfile" ) } } } }   pipelineJob worked excellent for our needs, the old broken syntax is below: repoList.each { def cookbook = it pipelineJob( "chef-${cookbook}-PR-push" ) { definition { cpsScm { lightweight( true ) scriptPath( 'jenkins/[where ever you want]/Jenkinsfile' ) scm { git { branch( 'master' ) remote { github( "organization/scripts" , 'ssh' ) credentials('ffff-fff-fff-ffffff) } } } } } } }

          André Ilhicas Santos added a comment - - edited

          There is actually a TODO for exactly the same use case described in this issue in the responsible getName function for branch name in Branch class at branch-api-plugin

          https://github.com/jenkinsci/branch-api-plugin/blob/1039350970768b27c88bebb0dad6ff9c96e50e0e/src/main/java/jenkins/branch/Branch.java#L143

          What would be the impacts of adding another constructor to allow the remote name to be passed as a prefix for instance?

          André Ilhicas Santos added a comment - - edited There is actually a TODO for exactly the same use case described in this issue in the responsible getName function for branch name in Branch class at branch-api-plugin https://github.com/jenkinsci/branch-api-plugin/blob/1039350970768b27c88bebb0dad6ff9c96e50e0e/src/main/java/jenkins/branch/Branch.java#L143 What would be the impacts of adding another constructor to allow the remote name to be passed as a prefix for instance?

          Thomas Hirsch added a comment - - edited

          I am using a multibranch pipeline on several repositories for some time now, and generally it works.
          There is one issue that I have which is that the sidebar-links to the various (GitHub) repositories that are involved are only showing the text "GitHub", and not the specific repository name. This is on the overview page, as well as the "Pull-Requests" tab of the multibranch job.

          Another similar bug occurs on any specific Pull-Requests. The link that is attacked to the specific PR (in this case there is a little "branching" icon, instead of the GitHub icon in the sidebar) is always pointing to the first repository/branch source that is listed in the job configuration, instead of the one that actually triggered the multibranch job.
          This leads to broken links, with the first part of the URL, the repository, getting put together with the path part i.e. PR number from a different repository...

          This is easy to reproduce:
          1. Create a Multibranch PR job with two repositories (default settings should work, as long as PRs are discovered).
          2. Create a PR on the second repository in the list.
          3. Watch the pipeline trigger, and the GitHub link on the job that will be generated for the build will contain the base URL of the first repository, and the PR number of the second repository

          Thomas Hirsch added a comment - - edited I am using a multibranch pipeline on several repositories for some time now, and generally it works. There is one issue that I have which is that the sidebar-links to the various (GitHub) repositories that are involved are only showing the text "GitHub", and not the specific repository name. This is on the overview page, as well as the "Pull-Requests" tab of the multibranch job. Another similar bug occurs on any specific Pull-Requests. The link that is attacked to the specific PR (in this case there is a little "branching" icon, instead of the GitHub icon in the sidebar) is always pointing to the first repository/branch source that is listed in the job configuration, instead of the one that actually triggered the multibranch job. This leads to broken links, with the first part of the URL, the repository, getting put together with the path part i.e. PR number from a different repository... This is easy to reproduce: 1. Create a Multibranch PR job with two repositories (default settings should work, as long as PRs are discovered). 2. Create a PR on the second repository in the list. 3. Watch the pipeline trigger, and the GitHub link on the job that will be generated for the build will contain the base URL of the first repository, and the PR number of the second repository

          Hi, I was needed this feature and saw the comment in the getName method. I've developed this functionnality (by adding a new BranchProperty to allow the customization of the name).

          My specific use case was for Gitlab, but it should be working for any SCM

           

          I've created a pull request for this new feature : https://github.com/jenkinsci/branch-api-plugin/pull/262

           

          Frédéric Laugier added a comment - Hi, I was needed this feature and saw the comment in the getName method. I've developed this functionnality (by adding a new BranchProperty to allow the customization of the name). My specific use case was for Gitlab, but it should be working for any SCM   I've created a pull request for this new feature : https://github.com/jenkinsci/branch-api-plugin/pull/262  

          Jesse Glick added a comment -

          This feature does not seem like a good idea. In general, while MultiBranchProject supported a list of multiple SCM sources from the beginning, this never worked very well and should be discouraged. If you have multiple repositories to manage, it is best to set up an organization folder. The use cases described are unclear.

          Jesse Glick added a comment - This feature does not seem like a good idea. In general, while MultiBranchProject supported a list of multiple SCM sources from the beginning, this never worked very well and should be discouraged. If you have multiple repositories to manage, it is best to set up an organization folder. The use cases described are unclear.

            Unassigned Unassigned
            evildeece Alastair D'Silva
            Votes:
            18 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: