Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64031

Jenkins multi branch pipeline scans failing due to timeouts because cache repo needs gc

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Jenkins 2.235.5
      git-plugin 4.3.0

      We've been having issues with timeouts during the following command in branch scans and at the start of builds:

      git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10

       

      I assumed it was the our repo size because we have a fairly large mono repo (~7.5GB). However, after experimenting with copying the repo without modification I found that an unmodified copy wouldn't hang on the command. Then I found where the clone was stored on disk (ours was in /var/jenkins_home/caches/) and found that the clone was double the size at ~15GB. Running the same command from the command line stalled there as well. However, once I let the command run long enough it output the following:

      Auto packing the repository in background for optimum performance.
      See "git help gc" for manual housekeeping.
      error: The last gc run reported the following. Please correct the root cause
      and remove .git/gc.log.
      Automatic cleanup will not be performed until the file is removed.
      
      warning: There are too many unreachable loose objects; run 'git prune' to remove them.
      

      I ran git gc and git prune manually and they both completed without error. The clone size went back down to 7.5GB. I then removed .git/gc.log as suggested, ran the git fetch command again and it completes instantly without stalling like before.

       

      I'm not sure if any of that can be automated to prevent it from happening again in the future, however at the very least it would be nice if the plugin could output some of those messages so it would be more visible what the issue was. Perhaps it's in a log somewhere that I didn't see.

       

            Unassigned Unassigned
            swbrowne Steve Browne
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: