Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67874

JENKINS_HOME/caches/git-... is prone to conflicts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-client-plugin
    • None
    • Jenkins-2.303
      git-client-plugin 3.10 (approx, was updated to 3.11 after the issue before the report)

      For context, we use an NFS-shared git reference repository to speed up checkouts across our CI farm (connection from subnet with the farm to github is weak). The NFS server had a hardware issue and got disconnected, so existing git workspaces that referenced parts of the shared index got invalidated.

      Most of the workers were just recycled wholesale and started life with empty workspaces; however the cache tracked by Jenkins controller internally (for Jenkinsfile and shared library checkouts I suppose) was just cleaned by moving /var/lib/jenkins/cache/git* directories away.

      Sizing for dozens of jobs to be handled remained, however - and many were queued (including needed copies of same infra jobs with different arguments). And many of those failed quickly.

      Looking at the log, I suppose that numerous builds began making new git-caches, with names somehow related to the checked out repo/commits, without proper sequential locking - so several git processes were init'ing the same directory and everyone failed.

      At least, forcing the "built-in" worker to one executor temporarily seems to have resolved the issue, as it now evaluated a lightweight checkout of only one job at a time. (Normally we allow the "built-in" node half a dozen executors for such processings, with zero jobs matching its label for "actual" workload).

      Typical build console of these failed builds:

      > git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb/.git # timeout=10
ERROR: Workspace has a .git repository, but it appears to be corrupt.
hudson.plugins.git.GitException: Error performing git command: git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb/.git
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2800)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2719)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2715)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:2080)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:2092)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.hasGitRepo(CliGitAPIImpl.java:403)
	at hudson.plugins.git.GitAPI.hasGitRepo(GitAPI.java:246)
	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:351)
	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
	at hudson.model.ResourceController.execute(ResourceController.java:100)
	at hudson.model.Executor.run(Executor.java:433)
Caused by: java.io.IOException: Cannot run program "git" (in directory "/var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb"): error=11, Resource temporarily unavailable
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.<init>(Proc.java:253)
	at hudson.Proc$LocalProc.<init>(Proc.java:222)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
	at hudson.Launcher$ProcStarter.start(Launcher.java:508)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2785)
	... 14 more
Caused by: java.io.IOException: error=11, Resource temporarily unavailable
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 19 more
Creating git repository in /var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb
 > git init /var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb # timeout=10
java.io.IOException: error=11, Resource temporarily unavailable
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
Caused: java.io.IOException: Cannot run program "git" (in directory "/var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb"): error=11, Resource temporarily unavailable
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at hudson.Proc$LocalProc.<init>(Proc.java:253)
	at hudson.Proc$LocalProc.<init>(Proc.java:222)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
	at hudson.Launcher$ProcStarter.start(Launcher.java:508)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2785)
Caused: hudson.plugins.git.GitException: Error performing git command: git init /var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2800)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2719)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2715)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:2080)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1071)
Caused: hudson.plugins.git.GitException: Could not init /var/lib/jenkins/caches/git-d267a14f82f160e10c6f84ee0ace3cdb
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1073)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.init(CliGitAPIImpl.java:358)
	at hudson.plugins.git.GitAPI.init(GitAPI.java:223)
	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:353)
	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
	at hudson.model.ResourceController.execute(ResourceController.java:100)
	at hudson.model.Executor.run(Executor.java:433)
Finished: FAILURE

            Unassigned Unassigned
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: