Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65277

git plugin left lock in caches repo, blocked all jobs using that cache

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • Jenkins 2.277.1 with git plugin 4.7.0 and other plugins as described in my docker-lfs repo dated 31 Mar 2021

      The git plugin left a config.lock file in one of the caches folders The folder was two weeks old and was showing indications that it might have benefited from a git gc.

      Because the config.lock file was left in that folder, many jobs failed (hundreds). They could not acquire a configuration lock on that repository. The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.

      Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.

      The directory contents looked like this when the lock was held for 7 or more hours:

      -rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 FETCH_HEAD
      -rw-r--r--   1 jenkins jenkins    164 Mar 22 11:24 config
      -rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 config.lock
      drwxr-xr-x   8 jenkins jenkins   4096 Mar 31 19:11 .
      jenkins@bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date
      Wed 31 Mar 2021 07:12:05 PM MDT
      

      The build log using command line git reported:

      15:04:49   > git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git # timeout=11
      15:04:49  hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git" returned status code 255:
      15:04:49  stdout: 
      15:04:49  stderr: error: could not lock config file .git/config: File exists
      15:04:49  
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2608)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2538)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2534)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1920)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1932)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1546)
      15:04:49  	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
      15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
      15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
      15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
      15:04:49  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      15:04:49  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
      15:04:49  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      15:04:49  	at hudson.model.ResourceController.execute(ResourceController.java:97)
      15:04:49  	at hudson.model.Executor.run(Executor.java:429)
      15:04:49  Finished: FAILURE
      

      The build log using JGit reported:

      5:03:01  Started by remote host Started from Groovy Console by Mark Waite with note: Groovy console started
      15:03:01  Setting origin to https://github.com/MarkEWaite/jenkins-bugs.git
      15:03:06  org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config. Ensure that no other process has an open file handle on the lock file /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config.lock, then you may delete the lock file and retry.
      15:03:06  	at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:220)
      15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:945)
      15:03:06  Caused: hudson.plugins.git.GitException
      15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:947)
      15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
      15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
      15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
      15:03:06  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      15:03:06  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
      15:03:06  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      15:03:06  	at hudson.model.ResourceController.execute(ResourceController.java:97)
      15:03:06  	at hudson.model.Executor.run(Executor.java:429)
      15:03:06  Finished: FAILURE
      

          [JENKINS-65277] git plugin left lock in caches repo, blocked all jobs using that cache

          Mark Waite created issue -
          Mark Waite made changes -
          Description Original: The git plugin left a {{config.lock}} file in one of the caches folders The folder was two weeks old and was showing indications that it might have benefited from a {{git gc}}.

          Because the {{config.lock}} file was left in that folder, many jobs failed (hundreds). They could not acquire a configuration lock on that repository. The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.

          Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.

          The directory contents looked like this when the lock was held for 7 or more hours:

          {noformat}
          -rw-r--r-- 1 jenkins jenkins 0 Mar 22 11:24 FETCH_HEAD
          -rw-r--r-- 1 jenkins jenkins 164 Mar 22 11:24 config
          -rw-r--r-- 1 jenkins jenkins 0 Mar 22 11:24 config.lock
          drwxr-xr-x 8 jenkins jenkins 4096 Mar 31 19:11 .
          jenkins@bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date
          Wed 31 Mar 2021 07:12:05 PM MDT
          {noformat}
          New: The git plugin left a {{config.lock}} file in one of the caches folders The folder was two weeks old and was showing indications that it might have benefited from a {{git gc}}.

          Because the {{config.lock}} file was left in that folder, many jobs failed (hundreds). They could not acquire a configuration lock on that repository. The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.

          Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.

          The directory contents looked like this when the lock was held for 7 or more hours:

          {noformat}
          -rw-r--r-- 1 jenkins jenkins 0 Mar 22 11:24 FETCH_HEAD
          -rw-r--r-- 1 jenkins jenkins 164 Mar 22 11:24 config
          -rw-r--r-- 1 jenkins jenkins 0 Mar 22 11:24 config.lock
          drwxr-xr-x 8 jenkins jenkins 4096 Mar 31 19:11 .
          jenkins@bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date
          Wed 31 Mar 2021 07:12:05 PM MDT
          {noformat}

          The build log using command line git reported:

          {noformat}
          15:04:49 > git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git # timeout=11
          15:04:49 hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git" returned status code 255:
          15:04:49 stdout:
          15:04:49 stderr: error: could not lock config file .git/config: File exists
          15:04:49
          15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2608)
          15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2538)
          15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2534)
          15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1920)
          15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1932)
          15:04:49 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1546)
          15:04:49 at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
          15:04:49 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
          15:04:49 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
          15:04:49 at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
          15:04:49 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
          15:04:49 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
          15:04:49 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
          15:04:49 at hudson.model.ResourceController.execute(ResourceController.java:97)
          15:04:49 at hudson.model.Executor.run(Executor.java:429)
          15:04:49 Finished: FAILURE
          {noformat}

          The build log using JGit reported:

          {noformat}
          5:03:01 Started by remote host Started from Groovy Console by Mark Waite with note: Groovy console started
          15:03:01 Setting origin to https://github.com/MarkEWaite/jenkins-bugs.git
          15:03:06 org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config. Ensure that no other process has an open file handle on the lock file /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config.lock, then you may delete the lock file and retry.
          15:03:06 at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:220)
          15:03:06 at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:945)
          15:03:06 Caused: hudson.plugins.git.GitException
          15:03:06 at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:947)
          15:03:06 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
          15:03:06 at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
          15:03:06 at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
          15:03:06 at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
          15:03:06 at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
          15:03:06 at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
          15:03:06 at hudson.model.ResourceController.execute(ResourceController.java:97)
          15:03:06 at hudson.model.Executor.run(Executor.java:429)
          15:03:06 Finished: FAILURE
          {noformat}
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]
          Mark Waite made changes -
          Summary Original: git plugin left dead lock in a caches repo, blocked all jobs using that cache New: git plugin left lock in caches repo, blocked all jobs using that cache
          Hung Doan made changes -
          Assignee New: Mark Waite [ markewaite ]
          Mark Waite made changes -
          Assignee Original: Mark Waite [ markewaite ]

            Unassigned Unassigned
            markewaite Mark Waite
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: