Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65277

git plugin left lock in caches repo, blocked all jobs using that cache

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • Jenkins 2.277.1 with git plugin 4.7.0 and other plugins as described in my docker-lfs repo dated 31 Mar 2021

      The git plugin left a config.lock file in one of the caches folders The folder was two weeks old and was showing indications that it might have benefited from a git gc.

      Because the config.lock file was left in that folder, many jobs failed (hundreds). They could not acquire a configuration lock on that repository. The plugin needs to either be less concerned about acquiring locks when reading the repository configuration or it needs to be more aggressive in cleaning locks and possibly detecting repositories that need to be garbage collected.

      Might consider an Admin Monitor to report cached repositories that would benefit from garbage collection along with an operation that would perform garbage collection on repositories that were detected as needing cleanup.

      The directory contents looked like this when the lock was held for 7 or more hours:

      -rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 FETCH_HEAD
      -rw-r--r--   1 jenkins jenkins    164 Mar 22 11:24 config
      -rw-r--r--   1 jenkins jenkins      0 Mar 22 11:24 config.lock
      drwxr-xr-x   8 jenkins jenkins   4096 Mar 31 19:11 .
      jenkins@bfe5372c3e46:~/caches/git-c3acd33583448548d14667df83b93ee1$ date
      Wed 31 Mar 2021 07:12:05 PM MDT
      

      The build log using command line git reported:

      15:04:49   > git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git # timeout=11
      15:04:49  hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/MarkEWaite/jenkins-bugs.git" returned status code 255:
      15:04:49  stdout: 
      15:04:49  stderr: error: could not lock config file .git/config: File exists
      15:04:49  
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2608)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2538)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2534)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1920)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1932)
      15:04:49  	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1546)
      15:04:49  	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
      15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
      15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
      15:04:49  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
      15:04:49  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      15:04:49  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
      15:04:49  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      15:04:49  	at hudson.model.ResourceController.execute(ResourceController.java:97)
      15:04:49  	at hudson.model.Executor.run(Executor.java:429)
      15:04:49  Finished: FAILURE
      

      The build log using JGit reported:

      5:03:01  Started by remote host Started from Groovy Console by Mark Waite with note: Groovy console started
      15:03:01  Setting origin to https://github.com/MarkEWaite/jenkins-bugs.git
      15:03:06  org.eclipse.jgit.errors.LockFailedException: Cannot lock /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config. Ensure that no other process has an open file handle on the lock file /var/jenkins_home/caches/git-c3acd33583448548d14667df83b93ee1/.git/config.lock, then you may delete the lock file and retry.
      15:03:06  	at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:220)
      15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:945)
      15:03:06  Caused: hudson.plugins.git.GitException
      15:03:06  	at org.jenkinsci.plugins.gitclient.JGitAPIImpl.setRemoteUrl(JGitAPIImpl.java:947)
      15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
      15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:330)
      15:03:06  	at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:396)
      15:03:06  	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:582)
      15:03:06  	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:100)
      15:03:06  	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309)
      15:03:06  	at hudson.model.ResourceController.execute(ResourceController.java:97)
      15:03:06  	at hudson.model.Executor.run(Executor.java:429)
      15:03:06  Finished: FAILURE
      

          [JENKINS-65277] git plugin left lock in caches repo, blocked all jobs using that cache

          Hung Doan added a comment -

          Hi,

          I'm having the same issue. I already deleted the caches folder in our node machine but didn't help. I don't have Admin Monitor right for my Jenkins. Do you have any recommend on this?

          Started
          [Fri Feb 25 14:21:46 ICT 2022] Starting branch indexing...
          > git.exe --version # timeout=10
          > git --version # 'git version 2.31.1.windows.1'
          using GIT_SSH to set credentials functional
          > git.exe ls-remote --symref --  # timeout=10
          > git.exe rev-parse --resolve-git-dir  # timeout=10
          Setting origin to 
          > git.exe config remote.origin.url  # timeout=10
          ERROR: [Fri Feb 25 14:21:47 ICT 2022] Could not fetch branches from source
          [Fri Feb 25 14:21:47 ICT 2022] Finished branch indexing. Indexing took 1.3 sec
          FATAL: Failed to recompute children of aNFCNCIStack » Continuous-Integration
          hudson.plugins.git.GitException: Command "git.exe config remote.origin.url " returned status code 255:
          stdout:
          stderr: error: could not lock config file .git/config: File exists

          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2639)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2569)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2565)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1951)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1963)
          at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1577)
          at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161)
          at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357)
          at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:566)
          at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
          at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
          at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641)
          at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
          at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166)
          at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
          Finished: FAILURE

          Hung Doan added a comment - Hi, I'm having the same issue. I already deleted the caches folder in our node machine but didn't help. I don't have Admin Monitor right for my Jenkins. Do you have any recommend on this? Started [Fri Feb 25 14:21:46 ICT 2022] Starting branch indexing... > git.exe --version # timeout=10 > git --version # 'git version 2.31.1.windows.1' using GIT_SSH to set credentials functional > git.exe ls-remote --symref --  # timeout=10 > git.exe rev-parse --resolve-git-dir  # timeout=10 Setting origin to  > git.exe config remote.origin.url  # timeout=10 ERROR: [Fri Feb 25 14:21:47 ICT 2022] Could not fetch branches from source [Fri Feb 25 14:21:47 ICT 2022] Finished branch indexing. Indexing took 1.3 sec FATAL: Failed to recompute children of aNFCNCIStack » Continuous-Integration hudson.plugins.git.GitException: Command "git.exe config remote.origin.url " returned status code 255: stdout: stderr: error: could not lock config file .git/config: File exists at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2639) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2569) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2565) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1951) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1963) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1577) at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:161) at jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:357) at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:566) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:166) at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Finished: FAILURE

          Mark Waite added a comment -

          minhhungft9 the locked configuration file is on the Jenkins controller, not on the Jenkins agent. Since it appears your controller is running on Windows, you'll need to check that there are no git.exe processes that are still running and holding locks. You may also need to stop and restart the Jenkins service for the controller to assure that open files are all closed.

          Mark Waite added a comment - minhhungft9 the locked configuration file is on the Jenkins controller, not on the Jenkins agent. Since it appears your controller is running on Windows, you'll need to check that there are no git.exe processes that are still running and holding locks. You may also need to stop and restart the Jenkins service for the controller to assure that open files are all closed.

          Hung Doan added a comment -

          Hi markewaite,

          Thank you for the information. After I deleted the gitconfig lock file on my Jenkins controller. The jobs can run again.

          Can you give any advices so that I won't get this kind of issues again?

          Hung Doan added a comment - Hi markewaite , Thank you for the information. After I deleted the gitconfig lock file on my Jenkins controller. The jobs can run again. Can you give any advices so that I won't get this kind of issues again?

          Mark Waite added a comment -

          minhhungft9 I have no advice to offer. I have only seen this failure once in all the time that I've used the git plugin. I don't know how to duplicate the problem or what caused the problem initially.

          Mark Waite added a comment - minhhungft9 I have no advice to offer. I have only seen this failure once in all the time that I've used the git plugin. I don't know how to duplicate the problem or what caused the problem initially.

          Hung Doan added a comment -

          markewaite for me, I made a mistake in my Discover Branches regular expression and there were lots of jobs running on different branches. I didn't want to stop them one by one so I deleted the whole multibranch folder on Jenkins, then this issue happened.

           

          Hung Doan added a comment - markewaite  for me, I made a mistake in my Discover Branches regular expression and there were lots of jobs running on different branches. I didn't want to stop them one by one so I deleted the whole multibranch folder on Jenkins, then this issue happened.  

            Unassigned Unassigned
            markewaite Mark Waite
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: