Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32018

Multibranch workflow project tracking multiple repositories

      In some cases it is inappropriate to keep Jenkinsfile in the same repository as project sources, because it has to refer to a particular CI environment that not all users care about. There are other cases where multiple repositories must be consulted during the course of the build. In all these cases currently you can use a standalone Workflow project to run builds, but you lose the advantages of multibranch:

      • ability to see the effect of changes made in unmerged branches
      • ability to customize Jenkinsfile atomically to match other changes, where relevant

      One possibility is another mode for workflow-multibranch, where the script comes not from a coversioned Jenkinsfile in the same repo but from some independent source. So sort of like CpsScmScriptDefinition in terms of picking up the script definition, but like current multibranch in terms of automatic generation of branch projects. You would not be able to make changes to the project sources (e.g., pom.xml) in tandem with changes to the Workflow script—you are back to having a single job definition which must be valid for all active branches—but this might be an acceptable price to pay for the ability to keep specifics of a particular CI environment out of the general project sources.

      Another option would be to use for example Git submodules (or Repo), whereby the project sources remain pristine in one repo, and Jenkinsfile for a particular CI environment lives in a super repository with a submodule for the main sources. The issue there is that you need to choose between strict versioning of modules, meaning you need a dummy commit to the super repo to test any source changes; vs. lax versioning (referring to a submodule by a ref rather than hash, if Git even allows that), meaning you lose determinacy in the revision. This also forces use of more exotic SCM features that are not universally available or desirable, though it does allow precise control of the linkage between repository versions.

      Perhaps we need an SCMSource which delegates to two or more regular sources and allows you to create a workspace out of multiple repositories, keeping deterministic revisions and allowing configurable options for matching up branches (so you could have a separate repository which could copy branch names from the main repository when necessary to customize Jenkinsfile on a per-branch basis). A potential problem is that some current SCMSource implementations attempt to checkcast the owner's sources list to their own type for various purposes, such as webhooks or status notifications. There needs to be an API in scm-api allowing an SCMSource to indicate that it is a composition.

          [JENKINS-32018] Multibranch workflow project tracking multiple repositories

          Andrew Bayer added a comment -

          Yeah, I'm used to living in a world of meta-repos that have scripts, repo and branch information, etc for pulling other repositories into the build process along the way, which right now doesn't quite organically fit into Multibranch. Mind you, it's a bit of a challenge to rationalize the meta-repo with the child repos in re: versioning without something like you said, submodules, Repo, etc.

          I'd be a strong +1 for such an SCMSource - coupling would be handy.

          Andrew Bayer added a comment - Yeah, I'm used to living in a world of meta-repos that have scripts, repo and branch information, etc for pulling other repositories into the build process along the way, which right now doesn't quite organically fit into Multibranch. Mind you, it's a bit of a challenge to rationalize the meta-repo with the child repos in re: versioning without something like you said, submodules, Repo, etc. I'd be a strong +1 for such an SCMSource - coupling would be handy.

          because it has to refer to a particular CI environment

          Why would you need to refer anything of the CI environment? URLs? I don't follow.

          Antonio Muñiz added a comment - because it has to refer to a particular CI environment Why would you need to refer anything of the CI environment? URLs? I don't follow.

          +1 for the SCMSource approach!

          Flávio Augusto Valones added a comment - +1 for the SCMSource approach!

          James Nord added a comment -

          Why would you need to refer anything of the CI environment? URLs? I don't follow.

          Tools (you need a specific one to build!) : (maven / JDK etc) may have different names
          Credentials (maybe for deploying you deploy as you - I deploy as me): IDs may not match
          Labels (where to build it): you may say linux - I may say RedHat

          James Nord added a comment - Why would you need to refer anything of the CI environment? URLs? I don't follow. Tools (you need a specific one to build!) : (maven / JDK etc) may have different names Credentials (maybe for deploying you deploy as you - I deploy as me): IDs may not match Labels (where to build it): you may say linux - I may say RedHat

          It seems possible to specify multiple GitHub and Git repositories with version 2.8 of the Pipelines Multibranch plugin.

          jglick, can you confirm this is fixed?

          Giovanni Tirloni added a comment - It seems possible to specify multiple GitHub and Git repositories with version 2.8 of the Pipelines Multibranch plugin. jglick , can you confirm this is fixed?

          Jesse Glick added a comment -

          It seems possible to specify multiple GitHub and Git repositories

          No. You can specify multiple SCM sources. Any given branch project will use one of them, according to a first-match semantics. An example use case would be if you had one publicly visible repository containing release branches, and a private clone containing developer branches: you might want an internal CI multibranch job to build each branch from whichever location most authoritatively hosted it.

          Jesse Glick added a comment - It seems possible to specify multiple GitHub and Git repositories No. You can specify multiple SCM sources . Any given branch project will use one of them, according to a first-match semantics. An example use case would be if you had one publicly visible repository containing release branches, and a private clone containing developer branches: you might want an internal CI multibranch job to build each branch from whichever location most authoritatively hosted it.

          Jesse Glick added a comment -

          One possibility is another mode for workflow-multibranch, where the script comes not from a coversioned Jenkinsfile in the same repo but from some independent source.

          Note that this is already implemented in CloudBees Jenkins Platform: documentation.

          Jesse Glick added a comment - One possibility is another mode for workflow-multibranch , where the script comes not from a coversioned Jenkinsfile in the same repo but from some independent source. Note that this is already implemented in CloudBees Jenkins Platform: documentation .

          This is a blocker for being able to evaluate the Blue Oceans plugin against various open source projects that we contribute to.

          I have a pull request here for evaluation, which allows for a user provided script via the config UI:
          https://github.com/jenkinsci/workflow-multibranch-plugin/pull/51

          Alastair D'Silva added a comment - This is a blocker for being able to evaluate the Blue Oceans plugin against various open source projects that we contribute to. I have a pull request here for evaluation, which allows for a user provided script via the config UI: https://github.com/jenkinsci/workflow-multibranch-plugin/pull/51

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            17 Vote for this issue
            Watchers:
            23 Start watching this issue

              Created:
              Updated: