Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27668

Add support for multiple git repositories in a single workspace

    XMLWordPrintable

Details

    Description

      I have two repos namely - commonlib.git, project1.git, and both of the have the root directories Source and CM. My Jenkins job is MultiGit_Job. I tried to configure a Jenkins job with two repos. During build, the source code for the first project is extracted (found when monitoring during the build execution) and the job fails during the second repo cloning. When I have analyzed the job's workspace, the commonlib.git source is extracted directly under workspace. There is identifier for commonlib repo. Because project1 repo also same root structure, the cloning is failing. It is a critical item for us as we are migrating our projects to GIT now. Thanks

      Attachments

        Activity

          markewaite Mark Waite added a comment -

          The git plugin does not directly support checkout of distinct repositories under a single workspace.

          If you want to support distinct repositories under a single workspace, you'll need to use the multi-SCM's plugin to configure the separate repositories.

          markewaite Mark Waite added a comment - The git plugin does not directly support checkout of distinct repositories under a single workspace. If you want to support distinct repositories under a single workspace, you'll need to use the multi-SCM's plugin to configure the separate repositories.

          Ok - But I wonder if GIT plugin does not support checkout of distinct repositories under single workspace, why would it have the option to add multiple repositories in the configuration?

          I even tried Multi-SCM plugin - which is also not differentiating the different repo code in workspace. Everything appears directly cluttering under the JOB's workspace

          sagayd Sagayaraj David added a comment - Ok - But I wonder if GIT plugin does not support checkout of distinct repositories under single workspace, why would it have the option to add multiple repositories in the configuration? I even tried Multi-SCM plugin - which is also not differentiating the different repo code in workspace. Everything appears directly cluttering under the JOB's workspace

          Ok - But I wonder if GIT plugin does not support checkout of distinct repositories under single workspace, why would it have the option to add multiple repositories in the configuration?

          I even tried Multi-SCM plugin - which is also not differentiating the different repo code in workspace. Everything appears directly cluttering under the JOB's workspace

          sagayd Sagayaraj David added a comment - Ok - But I wonder if GIT plugin does not support checkout of distinct repositories under single workspace, why would it have the option to add multiple repositories in the configuration? I even tried Multi-SCM plugin - which is also not differentiating the different repo code in workspace. Everything appears directly cluttering under the JOB's workspace
          markewaite Mark Waite added a comment -

          The git plugin allows multiple copies of the same repository and will poll for changes within those repositories. It assumes that the repositories are the same basic repository, potentially with newer commits in some of the repositories rather than others.

          For example, I have a job which includes a reference to the jenkinsci/git-plugin.git repository and also includes a reference to MarkEWaite/git-plugin.git. That allows me to detect changes pushed by others, and changes which I push to my own repository.

          markewaite Mark Waite added a comment - The git plugin allows multiple copies of the same repository and will poll for changes within those repositories. It assumes that the repositories are the same basic repository, potentially with newer commits in some of the repositories rather than others. For example, I have a job which includes a reference to the jenkinsci/git-plugin.git repository and also includes a reference to MarkEWaite/git-plugin.git. That allows me to detect changes pushed by others, and changes which I push to my own repository.

          Not sure if I understood correctly. I understand it as - Clone of same repository could be available to multiple remote users and the plugin can poll for changes of any number of such repositories and try to populate the local repo with pull from all of the configured remote repos. Is that correct?
          If the answer is yes, there could be merge conflicts of common files from different remote repos - how does the plugin handles it?

          Thanks for your support in tracking this.

          sagayd Sagayaraj David added a comment - Not sure if I understood correctly. I understand it as - Clone of same repository could be available to multiple remote users and the plugin can poll for changes of any number of such repositories and try to populate the local repo with pull from all of the configured remote repos. Is that correct? If the answer is yes, there could be merge conflicts of common files from different remote repos - how does the plugin handles it? Thanks for your support in tracking this.
          sagayd Sagayaraj David added a comment - - edited

          I also think that identifying the repo name on local workspace is needed for every SCM plugin. For example, if we take Multi-SCM plugin which configures a TFS project ($/Project1/Source) and a Git repo http://gitstore-host/project2.git (sub directory starts with Source directory) - then the directories from both SCM tool will be merged and create junk workspace useful for nothing.

          When configuring the distinct directory of SCM repo, have option for the user to choose the local directory name. For example, the local config will look something like: http://host1/project1.git (WORKSPACE\Source\h1_project1), http://host1/project2.git (WORKSPACE\Source\h1_project2) and http://host2/project1.git (WORKSPACE\Source\h2_project1)

          sagayd Sagayaraj David added a comment - - edited I also think that identifying the repo name on local workspace is needed for every SCM plugin. For example, if we take Multi-SCM plugin which configures a TFS project ($/Project1/Source) and a Git repo http://gitstore-host/project2.git (sub directory starts with Source directory) - then the directories from both SCM tool will be merged and create junk workspace useful for nothing. When configuring the distinct directory of SCM repo, have option for the user to choose the local directory name. For example, the local config will look something like: http://host1/project1.git (WORKSPACE\Source\h1_project1), http://host1/project2.git (WORKSPACE\Source\h1_project2) and http://host2/project1.git (WORKSPACE\Source\h2_project1)
          markewaite Mark Waite added a comment - - edited

          I also think that identifying the repo name on local workspace is needed for every SCM plugin. For example, if we take Multi-SCM plugin which configures a TFS project ($/Project1/Source) and a Git repo http://gitstore-host/project2.git (sub directory starts with Source directory) - then the directories from both SCM tool will be merged and create junk workspace useful for nothing.

          I don't know if the TFS plugin works with the multi-SCM plugin, but the git plugin does work with it, and allows the user to specify a subdirectory of the workspace for each of the repositories used in that job. With the git plugin, after I've added the git repository to the Multi-SCM configuration of the job, then I need to add an "Additional behaviour" to that git configuration to checkout to a specific subdirectory.

          markewaite Mark Waite added a comment - - edited I also think that identifying the repo name on local workspace is needed for every SCM plugin. For example, if we take Multi-SCM plugin which configures a TFS project ($/Project1/Source) and a Git repo http://gitstore-host/project2.git (sub directory starts with Source directory) - then the directories from both SCM tool will be merged and create junk workspace useful for nothing. I don't know if the TFS plugin works with the multi-SCM plugin, but the git plugin does work with it, and allows the user to specify a subdirectory of the workspace for each of the repositories used in that job. With the git plugin, after I've added the git repository to the Multi-SCM configuration of the job, then I need to add an "Additional behaviour" to that git configuration to checkout to a specific subdirectory.
          benza benza added a comment -

          multi-scm plugin is nice but it's not good enough, it does not keep the git env vars for all repos but overrides, so they are available only for the last one cloned.
          git publisher can not be used in the post build actions since it doesn't recognize any git settings
          "Project not currently configured to use Git; cannot check remote repository"

          benza benza added a comment - multi-scm plugin is nice but it's not good enough, it does not keep the git env vars for all repos but overrides, so they are available only for the last one cloned. git publisher can not be used in the post build actions since it doesn't recognize any git settings "Project not currently configured to use Git; cannot check remote repository"
          greenscar James Sandlin added a comment - - edited

          This should definitely be part of Jenkin's GIT plugin itself. With CI gaining popularity, more and more companies use multiple repos. Unfortunately, with Jenkins not supporting it natively, the multi-scm plugin breaks other plugins' CI functionality.
          We've seen problems with:

          • github pull request builder
          • github plugin's push functionality

          This also caused us to checkout with the multi-scm plugin in a specific order to ensure Jenkins has the correct version in its variables.

          When using Build Forge 10 years ago, a single repo was normal. Not so much any more.

          greenscar James Sandlin added a comment - - edited This should definitely be part of Jenkin's GIT plugin itself. With CI gaining popularity, more and more companies use multiple repos. Unfortunately, with Jenkins not supporting it natively, the multi-scm plugin breaks other plugins' CI functionality. We've seen problems with: github pull request builder github plugin's push functionality This also caused us to checkout with the multi-scm plugin in a specific order to ensure Jenkins has the correct version in its variables. When using Build Forge 10 years ago, a single repo was normal. Not so much any more.

          I have the same problem. I need to use 3 Git repos and I don't want to configure git modules.

          The multiSCM plugin as it says in it's own wiki page is more a proof-of-concept currently then a fully implemented plugin. It has it's shortcommings (although I think the idea behind it should be a core feature of Jenkins) which we did hit in the past even when using SVN.
          Voted up.

          costescuandrei Andrei Costescu added a comment - I have the same problem. I need to use 3 Git repos and I don't want to configure git modules. The multiSCM plugin as it says in it's own wiki page is more a proof-of-concept currently then a fully implemented plugin. It has it's shortcommings (although I think the idea behind it should be a core feature of Jenkins) which we did hit in the past even when using SVN. Voted up.
          dennisbell Dennis Bell added a comment -

          Really annoying that Jenkins doesnt support multiple GIT repos
          Voted up.

          +1

          dennisbell Dennis Bell added a comment - Really annoying that Jenkins doesnt support multiple GIT repos Voted up. +1
          tewing Terry Ewing added a comment -

          Found this thread after experiencing the same issue.

          My job is configured with two git repos. When the job runs it appears to check out both repos but the workspace only shows contents from the first.

          I believe this is a bug because the configuration in the UI isn't being produced in the workspace. I'm sure I can work around the issue and am hoping maintainers can find a solution soon.

          tewing Terry Ewing added a comment - Found this thread after experiencing the same issue. My job is configured with two git repos. When the job runs it appears to check out both repos but the workspace only shows contents from the first. I believe this is a bug because the configuration in the UI isn't being produced in the workspace. I'm sure I can work around the issue and am hoping maintainers can find a solution soon.
          markewaite Mark Waite added a comment -

          Adding multiple independent repository support to the existing git plugin looks extremely difficult to me.

          You might investigate using a Jenkinsfile in the root directory of your starting repository. I believe that Jenkinsfile can then be used to define the set of repositories you would like to checkout. There is a stackoverflow article which provides an example.

          markewaite Mark Waite added a comment - Adding multiple independent repository support to the existing git plugin looks extremely difficult to me. You might investigate using a Jenkinsfile in the root directory of your starting repository. I believe that Jenkinsfile can then be used to define the set of repositories you would like to checkout. There is a stackoverflow article which provides an example.
          jekeller Jacob Keller added a comment -

          You can use Multiple SCM plugin for this. Just be sure that you order the repositories so that the "most root" subdirectory is cloned first, otherwise Git plugin will erase the files due to how git-clone requires an empty directory.

          jekeller Jacob Keller added a comment - You can use Multiple SCM plugin for this. Just be sure that you order the repositories so that the "most root" subdirectory is cloned first, otherwise Git plugin will erase the files due to how git-clone requires an empty directory.
          greenscar James Sandlin added a comment -

          jekeller: Read comments above and read the documentation of the multi-scm plugin. It was written as a POC; it was not written for everyone to use and accept as "the best Jenkins has to offer". It is not a production ready plugin.

          We currently do use the muti-scm plugin and we have problems weekly due to the fact that "This plugin is more of a proof-of-concept than a robust and fully functional component. It does work in my particular build environment, and is meant to serve as a demonstration of what might be possible with more work."

          This is a limitation in Jenkins and anyone who accepts a POC plugin as a real solution needs to back up and think a bit.

          greenscar James Sandlin added a comment - jekeller : Read comments above and read the documentation of the multi-scm plugin. It was written as a POC; it was not written for everyone to use and accept as "the best Jenkins has to offer". It is not a production ready plugin. We currently do use the muti-scm plugin and we have problems weekly due to the fact that "This plugin is more of a proof-of-concept than a robust and fully functional component. It does work in my particular build environment, and is meant to serve as a demonstration of what might be possible with more work." This is a limitation in Jenkins and anyone who accepts a POC plugin as a real solution needs to back up and think a bit.
          jekeller Jacob Keller added a comment -

          I'm not saying it's the best or that it couldn't be improved, but it's what we have today that doesn't require more work.

          jekeller Jacob Keller added a comment - I'm not saying it's the best or that it couldn't be improved, but it's what we have today that doesn't require more work.

          Yes, but Multiple SCM plugin does need more work to not cause trouble...
          In some cases it doesn't work at all (generates exceptions in the defined child SCM entry plugins), in some cases it works incorrectly, in some scenarios it doesn't report changes correctly; see cases related to that plugin. I like that it exists though and I did use it for a while when I needed to combine multiple types of repos into one job (but I also did bump into these problems from time to time when using it).

          I still think that what MultipleSCM plugin does should be a core Jenkins feature (complete/fully operational of course) - is there a place where I can suggest this in a more official way? And then this case is no longer relevant as you could do it that way. And any other SCM plugin would only need to be implemented to handle one repo.

          costescuandrei Andrei Costescu added a comment - Yes, but Multiple SCM plugin does need more work to not cause trouble... In some cases it doesn't work at all (generates exceptions in the defined child SCM entry plugins), in some cases it works incorrectly, in some scenarios it doesn't report changes correctly; see cases related to that plugin. I like that it exists though and I did use it for a while when I needed to combine multiple types of repos into one job (but I also did bump into these problems from time to time when using it). I still think that what MultipleSCM plugin does should be a core Jenkins feature (complete/fully operational of course) - is there a place where I can suggest this in a more official way? And then this case is no longer relevant as you could do it that way. And any other SCM plugin would only need to be implemented to handle one repo.
          markewaite Mark Waite added a comment -

          This is the right place for feature requests. The Jenkins pipeline allows multiple repositories while defining a pipeline. You might try that for your use case

          markewaite Mark Waite added a comment - This is the right place for feature requests. The Jenkins pipeline allows multiple repositories while defining a pipeline. You might try that for your use case

          Thanks for the hint.

          So the pipeline basically skips all the UI for building jobs and configuring them and replaces that by using groovy scripts right?
          We will not move to that right now I but we might in the future.

          Is this ability to support multiple repos. a pipeline plugin ability or is it just based on some underlying API that the UI for configuring jobs uses as well? Cause in that case it might not be hard to add that ability there as well.

          costescuandrei Andrei Costescu added a comment - Thanks for the hint. So the pipeline basically skips all the UI for building jobs and configuring them and replaces that by using groovy scripts right? We will not move to that right now I but we might in the future. Is this ability to support multiple repos. a pipeline plugin ability or is it just based on some underlying API that the UI for configuring jobs uses as well? Cause in that case it might not be hard to add that ability there as well.
          markewaite Mark Waite added a comment -

          Pipeline scripts are groovy. A "snippet generator" is included in the pipeline job type assists by generating snippets of groovy code that can be inserted into a pipeline script. There is a "checkout" snippet, and there is a "git" snippet. One of those two should be able to help with the transition from pure UI driven job definition to pipeline based job definition.

          markewaite Mark Waite added a comment - Pipeline scripts are groovy. A "snippet generator" is included in the pipeline job type assists by generating snippets of groovy code that can be inserted into a pipeline script. There is a "checkout" snippet, and there is a "git" snippet. One of those two should be able to help with the transition from pure UI driven job definition to pipeline based job definition.
          jekeller Jacob Keller added a comment - - edited

          I suspect job definition would require re-work to support multiple SCMs. It's not un-doable but no one has put forth effort to work on it. I think it's best to be a core Jenkins feature, and get the review necessary to support the workflow. I don't think it's too far out there, and I think even changes could be handled properly that way.

          I think that this extension of features is worth doing even though pipelines support multiple repositories, as pipelines are more complex and not for everyone.

          jekeller Jacob Keller added a comment - - edited I suspect job definition would require re-work to support multiple SCMs. It's not un-doable but no one has put forth effort to work on it. I think it's best to be a core Jenkins feature, and get the review necessary to support the workflow. I don't think it's too far out there, and I think even changes could be handled properly that way. I think that this extension of features is worth doing even though pipelines support multiple repositories, as pipelines are more complex and not for everyone.
          greenscar James Sandlin added a comment -

          Pipeline is Groovy, which is easier than many scripting languages. If it's too complex for users, they probably need to find another industry.

          As Pipeline is the way of the future with Jenkins, it is an acceptable solution to this problem to recommend Pipeline. With 2.x, Pipeline becomes part of the product (a core plugin). It's available as a plugin now. My team accepted there is no reason to put an enormous effort into 1.x functionality and has moved on to using Pipeline.

          greenscar James Sandlin added a comment - Pipeline is Groovy, which is easier than many scripting languages. If it's too complex for users, they probably need to find another industry. As Pipeline is the way of the future with Jenkins, it is an acceptable solution to this problem to recommend Pipeline. With 2.x, Pipeline becomes part of the product (a core plugin). It's available as a plugin now. My team accepted there is no reason to put an enormous effort into 1.x functionality and has moved on to using Pipeline.
          markewaite Mark Waite added a comment -

          Since pipeline supports multiple git repositories, that is the preferred solution for this.

          Supporting multiple repositories from inside the git plugin will introduce too many compatibility problems for the git plugin.

          markewaite Mark Waite added a comment - Since pipeline supports multiple git repositories, that is the preferred solution for this. Supporting multiple repositories from inside the git plugin will introduce too many compatibility problems for the git plugin.

          Yes, the pipeline script supports multiple git repositories to be checked out (cloned) into the corresponding the sub-directories.
          But how should one write this script so that the polling of these repositories would trigger the job in case of updates? Please, see my question in details here: https://stackoverflow.com/questions/47439718/trigger-a-job-by-polling-multiple-git-repos-inside-jenkinsfile.

          alexander_samoylov Alexander Samoylov added a comment - Yes, the pipeline script supports multiple git repositories to be checked out (cloned) into the corresponding the sub-directories. But how should one write this script so that the polling of these repositories would trigger the job in case of updates? Please, see my question in details here: https://stackoverflow.com/questions/47439718/trigger-a-job-by-polling-multiple-git-repos-inside-jenkinsfile .
          ip1981 Igor Pashev added a comment - - edited

          Pipelines are not usable as for now, because any triggers and parameters defined in them are ignored until you manually start the job.

          ip1981 Igor Pashev added a comment - - edited Pipelines are not usable as for now, because any triggers and parameters defined in them are ignored until you manually start the job.

          People

            ndeloof Nicolas De Loof
            sagayd Sagayaraj David
            Votes:
            11 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: