Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25353

Git operations fail due to "dead", but lingering lock file

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • git-client-plugin
    • None

      Time to time I see the following error. I suppose, it can be caused by job abort in git checkout stage.

      java.io.IOException: Could not checkout 154d73b8218b3a4c0db7808853565ca5ed0b8999
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:966)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1253)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      	at hudson.model.Run.execute(Run.java:1745)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:89)
      	at hudson.model.Executor.run(Executor.java:240)
      Caused by: hudson.plugins.git.GitLockFailedException: Could not lock repository. Please try again
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1619)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:328)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at hudson.remoting.Engine$1$1.run(Engine.java:63)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: hudson.plugins.git.GitException: Command "C:\Program Files (x86)\Git\bin\git.exe checkout -f 154d73b8218b3a4c0db7808853565ca5ed0b8999" returned status code 128:
      stdout: 
      stderr: fatal: Unable to create 'e:/jenkins/workspace/xxx/.git/index.lock': File exists.
      
      If no other git process is currently running, this probably means a
      git process crashed in this repository earlier. Make sure no other git
      process is running and remove the file manually to continue.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1437)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1616)
      	... 12 more
      

          [JENKINS-25353] Git operations fail due to "dead", but lingering lock file

          Pavel Baranchikov created issue -

          Daniel Beck added a comment -

          What exactly is the bug here? I mean, it even says

          If no other git process is currently running, this probably means a
          git process crashed in this repository earlier. Make sure no other git
          process is running and remove the file manually to continue.

          This does not look like something that could be resolved automatically.

          Daniel Beck added a comment - What exactly is the bug here? I mean, it even says If no other git process is currently running, this probably means a git process crashed in this repository earlier. Make sure no other git process is running and remove the file manually to continue. This does not look like something that could be resolved automatically.

          My understanding, is that Jenkins prevents multiple builds in the same workspace. So, we can be sure, that if new build is triggered, no other git processes should access the workspace, and .git repository as well. And I believe, that no other users are interacting git at Jenkins slave.

          My suggestion is to unlock the git repository (removing .git/git.lock) file before checkout.

          Pavel Baranchikov added a comment - My understanding, is that Jenkins prevents multiple builds in the same workspace. So, we can be sure, that if new build is triggered, no other git processes should access the workspace, and .git repository as well. And I believe, that no other users are interacting git at Jenkins slave. My suggestion is to unlock the git repository (removing .git/git.lock) file before checkout.

          Daniel Beck added a comment -

          My understanding, is that Jenkins prevents multiple builds in the same workspace

          Only if the workspace is not a custom workspace (and details like this should be transparent to the SCM).

          And I believe, that no other users are interacting git at Jenkins slave.

          Given how many weird setups I've seen, I don't think this is necessarily universal.


          That said, it may be possible to make sure the file is deleted when aborting a build during a Git operation (if it's ensured Git gets properly killed).

          Daniel Beck added a comment - My understanding, is that Jenkins prevents multiple builds in the same workspace Only if the workspace is not a custom workspace (and details like this should be transparent to the SCM). And I believe, that no other users are interacting git at Jenkins slave. Given how many weird setups I've seen, I don't think this is necessarily universal. That said, it may be possible to make sure the file is deleted when aborting a build during a Git operation (if it's ensured Git gets properly killed).

          A agree with the approach to force unlocking git repository on job aborting.
          For today, I've seen this error 3 times on my build server.

          Pavel Baranchikov added a comment - A agree with the approach to force unlocking git repository on job aborting. For today, I've seen this error 3 times on my build server.

          Mark Waite added a comment - - edited

          Reliably removing the lock file and not harming some other use model seems complicated and risky. If a custom workspace is involved, there may be multiple git processes operating in the directory. If a typical workspace is involved, there may be build steps running which start a git process and expect the build step to complete before the git process has completed.

          What if we removed lock files as part of the optional "clean" step? Would that have resolved the case you detected? Would you have been willing to clean the workspace to remove the lock file?

          Wiping the workspace should be one way to work around this problem, since wiping the workspace will reconstruct the git repository from a freshly cloned copy.

          Mark Waite added a comment - - edited Reliably removing the lock file and not harming some other use model seems complicated and risky. If a custom workspace is involved, there may be multiple git processes operating in the directory. If a typical workspace is involved, there may be build steps running which start a git process and expect the build step to complete before the git process has completed. What if we removed lock files as part of the optional "clean" step? Would that have resolved the case you detected? Would you have been willing to clean the workspace to remove the lock file? Wiping the workspace should be one way to work around this problem, since wiping the workspace will reconstruct the git repository from a freshly cloned copy.

          I have no objections on separate "clean" step. For now, I've set "wipe out repository and force clean", but it take too much time to recustruct

          Pavel Baranchikov added a comment - I have no objections on separate "clean" step. For now, I've set "wipe out repository and force clean", but it take too much time to recustruct
          Mark Waite made changes -
          Summary Original: Git returned status code 128 New: Git aborts due to "dead", but lingering lock file
          Mark Waite made changes -
          Summary Original: Git aborts due to "dead", but lingering lock file New: Git operations fail due to "dead", but lingering lock file

          I tried on my local PC (openSuSE) and found that index.lock only left in the .git directory, when git is killed with SIGKILL. But there is not index.lock file when killing with SIGTERM.

          The operation, causing the index.lock file creation is

          git checkout
          

          Pavel Baranchikov added a comment - I tried on my local PC (openSuSE) and found that index.lock only left in the .git directory, when git is killed with SIGKILL. But there is not index.lock file when killing with SIGTERM. The operation, causing the index.lock file creation is git checkout

            ndeloof Nicolas De Loof
            pbaranchikov Pavel Baranchikov
            Votes:
            8 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: