Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64031

Jenkins multi branch pipeline scans failing due to timeouts because cache repo needs gc

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.235.5
      git-plugin 4.3.0
    • Similar Issues:

      Description

      We've been having issues with timeouts during the following command in branch scans and at the start of builds:

      git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10

       

      I assumed it was the our repo size because we have a fairly large mono repo (~7.5GB). However, after experimenting with copying the repo without modification I found that an unmodified copy wouldn't hang on the command. Then I found where the clone was stored on disk (ours was in /var/jenkins_home/caches/) and found that the clone was double the size at ~15GB. Running the same command from the command line stalled there as well. However, once I let the command run long enough it output the following:

      Auto packing the repository in background for optimum performance.
      See "git help gc" for manual housekeeping.
      error: The last gc run reported the following. Please correct the root cause
      and remove .git/gc.log.
      Automatic cleanup will not be performed until the file is removed.
      
      warning: There are too many unreachable loose objects; run 'git prune' to remove them.
      

      I ran git gc and git prune manually and they both completed without error. The clone size went back down to 7.5GB. I then removed .git/gc.log as suggested, ran the git fetch command again and it completes instantly without stalling like before.

       

      I'm not sure if any of that can be automated to prevent it from happening again in the future, however at the very least it would be nice if the plugin could output some of those messages so it would be more visible what the issue was. Perhaps it's in a log somewhere that I didn't see.

       

        Attachments

          Activity

          swbrowne Steve Browne created issue -
          markewaite Mark Waite made changes -
          Field Original Value New Value
          Assignee Mark Waite [ markewaite ]
          Hide
          markewaite Mark Waite added a comment -

          I agree that the git plugin needs some way to notify administrators when it detects surprises and unexpected results during background operations. We've seen a similar case on ci.jenkins.io where a cache repository was twice the expected size because it had never been garbage collected. The problem is more visible on large repositories because they can be much slower if they need garbage collection.

          Once the administrator has been notified, it then needs a way to resolve the notice that has been shown to the administrator. In this case, that would likely be offering to run git gc on the cached repository or possibly deleting the cached repository and recreating it.

          Show
          markewaite Mark Waite added a comment - I agree that the git plugin needs some way to notify administrators when it detects surprises and unexpected results during background operations. We've seen a similar case on ci.jenkins.io where a cache repository was twice the expected size because it had never been garbage collected. The problem is more visible on large repositories because they can be much slower if they need garbage collection. Once the administrator has been notified, it then needs a way to resolve the notice that has been shown to the administrator. In this case, that would likely be offering to run git gc on the cached repository or possibly deleting the cached repository and recreating it.
          markewaite Mark Waite made changes -
          Summary Jenkins multi branch pipeline scans failing due to timeouts Jenkins multi branch pipeline scans failing due to timeouts because cache repo needs gc

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            swbrowne Steve Browne
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: