Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44729

Optimize git clones via shared reference repo for multi branch builds of same repository

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We love the new pipeline multi branch (and github org) plugin models, but have a general issue of bloated disk usage and slow clones for some of our more active and larger repositories. Specifically we have an active repo that has a lot of open PRs at any given time and a few long-running mainline/maintenance branches.

      Every time a new branch or PR is opened, we generate a new clone on at least one build node for the new build (and typically as the PR gets more commits and the build nodes get assigned for subsequent builds differently it multiples to more clones on other machines).

      Given that these branches and PRs are all for the same repository, it'd be great to automatically have it silently share a common reference repo on each build node. Something like automatically incorporating https://github.com/orrc/git-webhook-proxy into the workflow.

      Otherwise with a single repo having ~70 open PRs at any given time and 4 main branches, we end up with ~74 clones of the *same* repo on each build node. That's a significant unnecessary disk bloat and network bandwidth hog. As a first level, just having a full clone to use as a reference per-repo per-agent would be a big help.

        Attachments

          Issue Links

            Activity

            seadub Chris Williams created issue -
            seadub Chris Williams made changes -
            Field Original Value New Value
            Labels git multibranch pipeline
            seadub Chris Williams made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            seadub Chris Williams made changes -
            Link This issue relates to JENKINS-43194 [ JENKINS-43194 ]
            seadub Chris Williams made changes -
            Link This issue relates to JENKINS-33273 [ JENKINS-33273 ]
            jglick Jesse Glick made changes -
            Component/s git-plugin [ 15543 ]
            Component/s workflow-multibranch-plugin [ 21465 ]
            Labels git multibranch pipeline multibranch pipeline
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-43894 [ JENKINS-43894 ]
            stephenconnolly Stephen Connolly made changes -
            Link This issue blocks JENKINS-44295 [ JENKINS-44295 ]
            stephenconnolly Stephen Connolly made changes -
            Link This issue is duplicated by JENKINS-44295 [ JENKINS-44295 ]
            stephenconnolly Stephen Connolly made changes -
            Link This issue blocks JENKINS-44295 [ JENKINS-44295 ]
            bitwiseman Liam Newman made changes -
            Link This issue is related to JENKINS-57315 [ JENKINS-57315 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              seadub Chris Williams
              Votes:
              9 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated: