Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25353

Git operations fail due to "dead", but lingering lock file

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • git-client-plugin
    • None

      Time to time I see the following error. I suppose, it can be caused by job abort in git checkout stage.

      java.io.IOException: Could not checkout 154d73b8218b3a4c0db7808853565ca5ed0b8999
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:966)
      	at hudson.model.AbstractProject.checkout(AbstractProject.java:1253)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:622)
      	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:528)
      	at hudson.model.Run.execute(Run.java:1745)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:89)
      	at hudson.model.Executor.run(Executor.java:240)
      Caused by: hudson.plugins.git.GitLockFailedException: Could not lock repository. Please try again
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1619)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:328)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at hudson.remoting.Engine$1$1.run(Engine.java:63)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: hudson.plugins.git.GitException: Command "C:\Program Files (x86)\Git\bin\git.exe checkout -f 154d73b8218b3a4c0db7808853565ca5ed0b8999" returned status code 128:
      stdout: 
      stderr: fatal: Unable to create 'e:/jenkins/workspace/xxx/.git/index.lock': File exists.
      
      If no other git process is currently running, this probably means a
      git process crashed in this repository earlier. Make sure no other git
      process is running and remove the file manually to continue.
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1437)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:87)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$8.execute(CliGitAPIImpl.java:1616)
      	... 12 more
      

          [JENKINS-25353] Git operations fail due to "dead", but lingering lock file

          Mark Waite added a comment -

          Fixed in git-client-plugin 1.13.0 released 18 Dec 2014

          Mark Waite added a comment - Fixed in git-client-plugin 1.13.0 released 18 Dec 2014

          Jan Hudec added a comment -

          It still happens with 1.16.1 when the job is cancelled during Git operation.

          Jan Hudec added a comment - It still happens with 1.16.1 when the job is cancelled during Git operation.

          Ron MacNeil added a comment - - edited

          Could we just have an optional "clean lock file(s) before checking out" additional behaviour for the Git SCM plugin please? I think that's exactly what folks are looking for.

          Presently, this can be emulated with a pre-checkout "del /f /s index.lock" (or similar) via the pre-checkout-buildstep plugin.

          I appreciate the cleverness of the clean-up-on-exception idea but, as Pavel points out, it doesn't work in all cases and I, for one, would like something that works in all cases.

          Ron MacNeil added a comment - - edited Could we just have an optional "clean lock file(s) before checking out" additional behaviour for the Git SCM plugin please? I think that's exactly what folks are looking for. Presently, this can be emulated with a pre-checkout "del /f /s index.lock" (or similar) via the pre-checkout-buildstep plugin. I appreciate the cleverness of the clean-up-on-exception idea but, as Pavel points out , it doesn't work in all cases and I, for one, would like something that works in all cases.

          Mark Waite added a comment -

          I don't think an optional "clean lock file(s) before checkout out" will fix the problem completely for Windows. If the git process which creates the lock file (on windows) is still running, then the file cannot be deleted due to the Windows file system rules which will not allow open files to be deleted.

          Mark Waite added a comment - I don't think an optional "clean lock file(s) before checkout out" will fix the problem completely for Windows. If the git process which creates the lock file (on windows) is still running, then the file cannot be deleted due to the Windows file system rules which will not allow open files to be deleted.

          Ron MacNeil added a comment -

          True, but if we assume the common case where jobs have their own isolated workspaces, then the only place a conflicting Git process could come from would be a previous run of the same job. For that to happen, the process would have to have somehow escaped Jenkins' process tracking/killing behaviour, something we've never seen in our particular environment and, I submit, is unlikely in general.

          On the other hand, we have evidence that folks are still coming up against abandoned lock files, whether due to processes being KILL'ed or otherwise. So the optional "clean locks" behaviour would seem to at least put those folks in a better place than they're in now.

          Either way, thank you for your work on Jenkins, much appreciated.

          Ron MacNeil added a comment - True, but if we assume the common case where jobs have their own isolated workspaces, then the only place a conflicting Git process could come from would be a previous run of the same job. For that to happen, the process would have to have somehow escaped Jenkins' process tracking/killing behaviour, something we've never seen in our particular environment and, I submit, is unlikely in general. On the other hand, we have evidence that folks are still coming up against abandoned lock files, whether due to processes being KILL'ed or otherwise. So the optional "clean locks" behaviour would seem to at least put those folks in a better place than they're in now. Either way, thank you for your work on Jenkins, much appreciated.

          Jan Hudec added a comment -

          On Windows the lock file can't be deleted while still held, which is what makes unconditional removal safe. Unless somebody is messing up with the Jenkins workspaces by hand, there is no process running in them when build starts, so it still does fix the issue.

          Jan Hudec added a comment - On Windows the lock file can't be deleted while still held, which is what makes unconditional removal safe . Unless somebody is messing up with the Jenkins workspaces by hand, there is no process running in them when build starts, so it still does fix the issue.

          Hi guys, so I manage to reproduce a posible cause for this bug (there could be other cases also):
          I am certain the build triggers are one of the problems: "pull scm" and "build periodically" collide. So the pull scm will work ok but when build periodically comes, the build is going to fail with the lock issue.
          Replicate: have pull scm at 5 minute and build periodically at 5 minute and look what happens.
          Note: I am using the latest plugins (as of 22 mar 2016).

          Nicolae Dragos Sava added a comment - Hi guys, so I manage to reproduce a posible cause for this bug (there could be other cases also): I am certain the build triggers are one of the problems: "pull scm" and "build periodically" collide. So the pull scm will work ok but when build periodically comes, the build is going to fail with the lock issue. Replicate: have pull scm at 5 minute and build periodically at 5 minute and look what happens. Note: I am using the latest plugins (as of 22 mar 2016).

          Is there any progress on this?

          matthew giardina added a comment - Is there any progress on this?

          Mark Waite added a comment - - edited

          No, and not likely to be any progress in the near term.  

          You can install the "pre-scm build" plugin and and a pre-scm step to unconditionally remove lock files in the .git directory if you're seeing this frequently enough to justify changing your job definition.

          Mark Waite added a comment - - edited No, and not likely to be any progress in the near term.   You can install the "pre-scm build" plugin and and a pre-scm step to unconditionally remove lock files in the .git directory if you're seeing this frequently enough to justify changing your job definition.

          trejkaz added a comment -

          Copying from JENKINS-47652, commands leading up to the error were:

          [macosx]  > git rev-parse --is-inside-work-tree # timeout=10
          [macosx]  > git config remote.origin.url git@_git-server_:_app_/core.git # timeout=10
          [macosx]  > git --version # timeout=10
          [macosx]  > git fetch --tags --progress git@_git-server_:_app_/core.git +refs/pull/50/head:refs/remotes/origin/PR-50 +refs/heads/master:refs/remotes/origin/master --prune # timeout=40
          [macosx]  > git config core.sparsecheckout # timeout=10
          [macosx]  > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30
          [macosx]  > git merge 0abbc94cdff38bfe76dad45562011f6f703143ce # timeout=10
          [macosx]  > git config core.sparsecheckout # timeout=10
          [macosx]  > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30
          

          The sequence of commands seems odd to me, but surely it wasn't locked before SCM commands started, or the first `git checkout` would have failed, so I'm not sure a pre-SCM step would work either, unless there is a way to do it half way through the SCM step? Or are we saying that this log represents multiple SCM steps, so the pre would run before each? (Jenkins' logs make what's being done very unclear.)

          trejkaz added a comment - Copying from JENKINS-47652 , commands leading up to the error were: [macosx] > git rev-parse --is-inside-work-tree # timeout=10 [macosx] > git config remote.origin.url git@_git-server_:_app_/core.git # timeout=10 [macosx] > git --version # timeout=10 [macosx] > git fetch --tags --progress git@_git-server_:_app_/core.git +refs/pull/50/head:refs/remotes/origin/PR-50 +refs/heads/master:refs/remotes/origin/master --prune # timeout=40 [macosx] > git config core.sparsecheckout # timeout=10 [macosx] > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30 [macosx] > git merge 0abbc94cdff38bfe76dad45562011f6f703143ce # timeout=10 [macosx] > git config core.sparsecheckout # timeout=10 [macosx] > git checkout -f 72d57d4d7854d9b2b9eb78e0fe158869b3f04f80 # timeout=30 The sequence of commands seems odd to me, but surely it wasn't locked before SCM commands started, or the first `git checkout` would have failed, so I'm not sure a pre-SCM step would work either, unless there is a way to do it half way through the SCM step? Or are we saying that this log represents multiple SCM steps, so the pre would run before each? (Jenkins' logs make what's being done very unclear.)

            ndeloof Nicolas De Loof
            pbaranchikov Pavel Baranchikov
            Votes:
            8 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: