Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47874

Support multiple repositories in multi-branch pipeline

    • Blue Ocean 1.5 - beta 3

      Within a single job, we need to source branches from multiple repositories that we do not have control over. Currently, if the same branch name occurs in different repositories, only the first is built and subsequent identical branches are dropped.

       

      I suggest that the remote name (if configured) be used to disambiguate the branches, so that they can all be built.

      eg.
      Remote name: linux
      Branch: master

      Remote Name: linux-next
      Branch: master11

      These would be resolved as linux/master and linux-next/master.

       

          [JENKINS-47874] Support multiple repositories in multi-branch pipeline

          Alastair D'Silva created issue -
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]

          Mark Waite added a comment -

          Can you describe what you're envisioning in greater detail? The plugin already allows a separately named origin for each repository it clones, and allows the branch name to include that origin for disambiguation.

          Are you in a freestyle job context, or a PIpeline context when reading from multiple repositories?

          Mark Waite added a comment - Can you describe what you're envisioning in greater detail? The plugin already allows a separately named origin for each repository it clones, and allows the branch name to include that origin for disambiguation. Are you in a freestyle job context, or a PIpeline context when reading from multiple repositories?
          Alastair D'Silva made changes -
          Attachment New: branchindexing.log [ 40255 ]

          Alastair D'Silva added a comment - - edited

          Interesting, that is the behavior I was expecting, but not what I am seeing.

          I'm using a Multibranch pipeline job, I've attached an excerpt from the scan log (branchindexing.log).

          I have configured multiple git sources, each one has a unique "Configure Remote Name" value. Is there some other configuration option required to make the origin appear in the branch name?

           Note the collision at the bottom of the log:

          Checking branches...
            Checking branch master
              Met criteria
          Ignoring Linux Kernel » master from source #5 as source #3 owns the branch name

           

          Alastair D'Silva added a comment - - edited Interesting, that is the behavior I was expecting, but not what I am seeing. I'm using a Multibranch pipeline job, I've attached an excerpt from the scan log (branchindexing.log). I have configured multiple git sources, each one has a unique "Configure Remote Name" value. Is there some other configuration option required to make the origin appear in the branch name?  Note the collision at the bottom of the log: Checking branches...   Checking branch master     Met criteria Ignoring Linux Kernel » master from source #5 as source #3 owns the branch name  

          Mark Waite added a comment -

          I don't understand how you've configured your multi-branch pipeline job, or more correctly, I don't understand what you're trying to achieve with your multi-branch pipeline job configuration.

          Based on your branch scanning log, I'd guess that you've configured multiple repositories in the multi-branch pipeline definition. I don't know what that should do, or how it should do it.

          As far as I understand a multi-branch pipeline, it is designed to create a single job for each branch in the repository which satisfies the branch name matching conditions. Why not just define a multi-branch pipeline job for each of your repositories, rather than combining them all into a single multi-branch pipeline job?

          I've confirmed in a test job that checkout of different repositories and/or different branches is well behaved from a pipeline job in a multi-branch pipeline. That test job runs as expected, reporting that each subdirectory where a checkout is performed contains the correct repository and branch.

          Can you confirm that you're placing multiple repositories in the multi-branch pipeline definition? If that's what you're doing, please explain more about the desired outcome of that multi-branch pipeline.

          Mark Waite added a comment - I don't understand how you've configured your multi-branch pipeline job, or more correctly, I don't understand what you're trying to achieve with your multi-branch pipeline job configuration. Based on your branch scanning log, I'd guess that you've configured multiple repositories in the multi-branch pipeline definition. I don't know what that should do, or how it should do it. As far as I understand a multi-branch pipeline, it is designed to create a single job for each branch in the repository which satisfies the branch name matching conditions. Why not just define a multi-branch pipeline job for each of your repositories, rather than combining them all into a single multi-branch pipeline job? I've confirmed in a test job that checkout of different repositories and/or different branches is well behaved from a pipeline job in a multi-branch pipeline. That test job runs as expected, reporting that each subdirectory where a checkout is performed contains the correct repository and branch. Can you confirm that you're placing multiple repositories in the multi-branch pipeline definition? If that's what you're doing, please explain more about the desired outcome of that multi-branch pipeline.

          Alastair D'Silva added a comment - - edited

          That is correct, what I am after is a single multibranch job that sources it's branches from separate repositories. Each job created by the multibranch job only ever refers to a single branch, which originates from one of the many repos specified by the multibranch pipeline.

          We need these to be managed as a single job to avoid a maintenance nightmare - we have ~20 multibranch jobs, each picking branches from 3-8 repos. Also, from a UX perspective, it's a lot easier for our maintainers to look at a single Blue Ocean results page with all the branches reported, than have to cycle through one for each repo.

          It sounds like the Git plugin is behaving as expected, and the issue lies with how the Multibranch Pipeline plugin handles the branches. Would you agree that this should be logged against that plugin instead?

           

          [later]Looks like the Branch API is responsible:

              public String getName() {
                  // TODO this could include a uniquifying prefix defined in BranchSource
                  return head.getName();
              }

          Alastair D'Silva added a comment - - edited That is correct, what I am after is a single multibranch job that sources it's branches from separate repositories. Each job created by the multibranch job only ever refers to a single branch, which originates from one of the many repos specified by the multibranch pipeline. We need these to be managed as a single job to avoid a maintenance nightmare - we have ~20 multibranch jobs, each picking branches from 3-8 repos. Also, from a UX perspective, it's a lot easier for our maintainers to look at a single Blue Ocean results page with all the branches reported, than have to cycle through one for each repo. It sounds like the Git plugin is behaving as expected, and the issue lies with how the Multibranch Pipeline plugin handles the branches. Would you agree that this should be logged against that plugin instead?   [later] Looks like the Branch API is responsible:     public String getName() {         // TODO this could include a uniquifying prefix defined in BranchSource         return head.getName();     }

          Mark Waite added a comment -

          I suspect that changes will be needed in multiple plugins to support your use case.

          I don't think changes are needed in the git plugin, but I would expect changes would be needed in the branch api, and possibly in the branch source plugins and the organization folder plugin.

          Mark Waite added a comment - I suspect that changes will be needed in multiple plugins to support your use case. I don't think changes are needed in the git plugin, but I would expect changes would be needed in the branch api, and possibly in the branch source plugins and the organization folder plugin.
          Mark Waite made changes -
          Summary Original: Resolve branch names collisions New: Support multiple repositories in multi-branch pipeline
          Alastair D'Silva made changes -
          Component/s New: branch-api-plugin [ 18621 ]
          Component/s New: workflow-multibranch-plugin [ 21465 ]
          Component/s Original: git-plugin [ 15543 ]
          Environment Original: Jenkins 2.88
          Git Plugin 3.6.4
          New: Jenkins 2.88
          Git Plugin 3.6.4
          Branch API Plugin 2.0.15
          Pipeline Multibranch 2.16

            Unassigned Unassigned
            evildeece Alastair D'Silva
            Votes:
            18 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: