-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins 2.235.5
git-plugin 4.3.0
We've been having issues with timeouts during the following command in branch scans and at the start of builds:
git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
I assumed it was the our repo size because we have a fairly large mono repo (~7.5GB). However, after experimenting with copying the repo without modification I found that an unmodified copy wouldn't hang on the command. Then I found where the clone was stored on disk (ours was in /var/jenkins_home/caches/) and found that the clone was double the size at ~15GB. Running the same command from the command line stalled there as well. However, once I let the command run long enough it output the following:
Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. error: The last gc run reported the following. Please correct the root cause and remove .git/gc.log. Automatic cleanup will not be performed until the file is removed. warning: There are too many unreachable loose objects; run 'git prune' to remove them.
I ran git gc and git prune manually and they both completed without error. The clone size went back down to 7.5GB. I then removed .git/gc.log as suggested, ran the git fetch command again and it completes instantly without stalling like before.
I'm not sure if any of that can be automated to prevent it from happening again in the future, however at the very least it would be nice if the plugin could output some of those messages so it would be more visible what the issue was. Perhaps it's in a log somewhere that I didn't see.
I agree that the git plugin needs some way to notify administrators when it detects surprises and unexpected results during background operations. We've seen a similar case on ci.jenkins.io where a cache repository was twice the expected size because it had never been garbage collected. The problem is more visible on large repositories because they can be much slower if they need garbage collection.
Once the administrator has been notified, it then needs a way to resolve the notice that has been shown to the administrator. In this case, that would likely be offering to run git gc on the cached repository or possibly deleting the cached repository and recreating it.