-
Bug
-
Resolution: Duplicate
-
Minor
-
Jenkins 2.14
Windows Server 2012 R2
workflow-multibranch-plugin 2.8
Description
When git checkout crash (for example, due to a timeout) before the Jenkinsfile execution, the internal multibranch workspace is broken. Indeed, git add .lock files in its repository when it runs. If git crash, these files are not removed. When you try to restart the build, it will fails because git finds a .lock file and think another git operation is running.
Workaround
You have to manually remove the workspace of your branch in the Jenkins Home (or at least the .lock files)
Logs
Here is the git crash caused by a timeout (probably a problem with my internet connection).
Started by user anonymous Setting origin to https://my.server.com/my-repo.git Fetching origin... > git.exe rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git.exe config remote.origin.url https://my.server.com/my-repo.git # timeout=10 Cleaning workspace > git.exe rev-parse --verify HEAD # timeout=10 Resetting working tree > git.exe reset --hard # timeout=10 > git.exe clean -fdx # timeout=10 Fetching upstream changes from https://my.server.com/my-repo.git > git.exe --version # timeout=10 using GIT_ASKPASS to set credentials vsts > git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=120 ERROR: Timeout after 120 minutes ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://my.server.com/my-repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:799) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1055) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108) at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:85) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:404) Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1" returned status code -1: stdout: stderr: Fatal: InvalidOperationException encountered. remote: Microsoft (R) Visual Studio (R) Team Services remote: remote: Found 459 objects to send. remote: Found 5449 objects to send. (14062 ms) Receiving objects: 0% (1/5449) Receiving objects: 1% (55/5449) Receiving objects: 2% (109/5449) Receiving objects: 3% (164/5449) Receiving objects: 4% (218/5449) Receiving objects: 5% (273/5449) Receiving objects: 6% (327/5449) Receiving objects: 7% (382/5449) Receiving objects: 8% (436/5449) Receiving objects: 9% (491/5449) Receiving objects: 10% (545/5449) Receiving objects: 11% (600/5449) Receiving objects: 12% (654/5449) Receiving objects: 13% (709/5449) Receiving objects: 14% (763/5449) Receiving objects: 15% (818/5449) Receiving objects: 16% (872/5449) Receiving objects: 17% (927/5449) Receiving objects: 18% (981/5449) Receiving objects: 19% (1036/5449) Receiving objects: 20% (1090/5449) Receiving objects: 21% (1145/5449) Receiving objects: 22% (1199/5449) Receiving objects: 23% (1254/5449) Receiving objects: 24% (1308/5449) Receiving objects: 25% (1363/5449) Receiving objects: 26% (1417/5449) Receiving objects: 27% (1472/5449) Receiving objects: 28% (1526/5449) Receiving objects: 29% (1581/5449) Receiving objects: 30% (1635/5449) Receiving objects: 30% (1687/5449), 1.65 MiB | 1.93 MiB/s Receiving objects: 31% (1690/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 32% (1744/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 33% (1799/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 34% (1853/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 35% (1908/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 36% (1962/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 37% (2017/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 38% (2071/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 39% (2126/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 40% (2180/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 41% (2235/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 42% (2289/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 43% (2344/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 44% (2398/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 45% (2453/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 46% (2507/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 47% (2562/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 48% (2616/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 49% (2671/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 50% (2725/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 51% (2779/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 52% (2834/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 53% (2888/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 54% (2943/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 55% (2997/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 56% (3052/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 57% (3106/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 58% (3161/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 59% (3215/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 60% (3270/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 61% (3324/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 62% (3379/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 63% (3433/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 64% (3488/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 65% (3542/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 66% (3597/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 67% (3651/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 68% (3706/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 69% (3760/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 70% (3815/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 71% (3869/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 72% (3924/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 72% (3962/5449), 2.66 MiB | 1.81 MiB/s Receiving objects: 73% (3978/5449), 4.53 MiB | 1.99 MiB/s Receiving objects: 74% (4033/5449), 6.83 MiB | 2.46 MiB/s Receiving objects: 74% (4051/5449), 6.83 MiB | 2.46 MiB/s Receiving objects: 74% (4074/5449), 8.88 MiB | 2.05 MiB/s Receiving objects: 74% (4074/5449), 10.19 MiB | 1.90 MiB/s Receiving objects: 74% (4074/5449), 11.63 MiB | 1.81 MiB/s Receiving objects: 74% (4074/5449), 13.25 MiB | 1.37 MiB/s Receiving objects: 74% (4075/5449), 14.44 MiB | 1.40 MiB/s Receiving objects: 74% (4082/5449), 18.06 MiB | 1.57 MiB/s Receiving objects: 74% (4082/5449), 19.06 MiB | 1.51 MiB/s Receiving objects: 74% (4083/5449), 19.81 MiB | 1.50 MiB/s Receiving objects: 75% (4087/5449), 19.96 MiB | 1.31 MiB/s Receiving objects: 75% (4112/5449), 19.96 MiB | 1.31 MiB/s Receiving objects: 76% (4142/5449), 21.62 MiB | 1.46 MiB/s Receiving objects: 76% (4159/5449), 22.50 MiB | 1.46 MiB/s Receiving objects: 77% (4196/5449), 22.95 MiB | 1.32 MiB/s Receiving objects: 78% (4251/5449), 22.95 MiB | 1.32 MiB/s Receiving objects: 79% (4305/5449), 22.95 MiB | 1.32 MiB/s Receiving objects: 80% (4360/5449), 22.95 MiB | 1.32 MiB/s Receiving objects: 81% (4414/5449), 22.95 MiB | 1.32 MiB/s Receiving objects: 82% (4469/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 83% (4523/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 84% (4578/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 85% (4632/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 86% (4687/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 87% (4741/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 88% (4796/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 88% (4807/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 89% (4850/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 90% (4905/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 91% (4959/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 92% (5014/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 93% (5068/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 94% (5123/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 95% (5177/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 96% (5232/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 97% (5286/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 98% (5341/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 99% (5395/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 100% (5449/5449), 24.34 MiB | 1.69 MiB/s Receiving objects: 100% (5449/5449), 24.68 MiB | 1.32 MiB/s, done. Resolving deltas: 0% (0/4170) Resolving deltas: 1% (44/4170) Resolving deltas: 2% (84/4170) Resolving deltas: 3% (141/4170) Resolving deltas: 4% (167/4170) Resolving deltas: 5% (210/4170) Resolving deltas: 6% (254/4170) Resolving deltas: 7% (309/4170) Resolving deltas: 9% (397/4170) Resolving deltas: 10% (427/4170) Resolving deltas: 11% (463/4170) Resolving deltas: 12% (511/4170) Resolving deltas: 13% (543/4170) Resolving deltas: 14% (590/4170) Resolving deltas: 15% (631/4170) Resolving deltas: 16% (669/4170) Resolving deltas: 17% (709/4170) Resolving deltas: 18% (752/4170) Resolving deltas: 19% (797/4170) Resolving deltas: 20% (835/4170) Resolving deltas: 21% (883/4170) Resolving deltas: 22% (927/4170) Resolving deltas: 23% (962/4170) Resolving deltas: 24% (1001/4170) Resolving deltas: 25% (1043/4170) Resolving deltas: 26% (1095/4170) Resolving deltas: 27% (1153/4170) Resolving deltas: 28% (1168/4170) Resolving deltas: 29% (1225/4170) Resolving deltas: 30% (1251/4170) Resolving deltas: 31% (1295/4170) Resolving deltas: 32% (1346/4170) Resolving deltas: 33% (1397/4170) Resolving deltas: 34% (1422/4170) Resolving deltas: 35% (1479/4170) Resolving deltas: 36% (1502/4170) Resolving deltas: 37% (1544/4170) Resolving deltas: 38% (1587/4170) Resolving deltas: 39% (1631/4170) Resolving deltas: 40% (1681/4170) Resolving deltas: 41% (1710/4170) Resolving deltas: 41% (1712/4170) Resolving deltas: 41% (1714/4170) Resolving deltas: 42% (1752/4170) Resolving deltas: 43% (1795/4170) Resolving deltas: 44% (1835/4170) Resolving deltas: 45% (1885/4170) Resolving deltas: 46% (1921/4170) Resolving deltas: 47% (1967/4170) Resolving deltas: 48% (2003/4170) Resolving deltas: 49% (2053/4170) Resolving deltas: 50% (2087/4170) Resolving deltas: 51% (2137/4170) Resolving deltas: 52% (2169/4170) Resolving deltas: 53% (2211/4170) Resolving deltas: 54% (2252/4170) Resolving deltas: 55% (2298/4170) Resolving deltas: 56% (2345/4170) Resolving deltas: 57% (2396/4170) Resolving deltas: 58% (2430/4170) Resolving deltas: 59% (2462/4170) Resolving deltas: 60% (2502/4170) Resolving deltas: 61% (2548/4170) Resolving deltas: 62% (2594/4170) Resolving deltas: 63% (2635/4170) Resolving deltas: 64% (2677/4170) Resolving deltas: 65% (2711/4170) Resolving deltas: 66% (2753/4170) Resolving deltas: 67% (2802/4170) Resolving deltas: 68% (2836/4170) Resolving deltas: 69% (2878/4170) Resolving deltas: 69% (2911/4170) Resolving deltas: 70% (2919/4170) Resolving deltas: 71% (2963/4170) Resolving deltas: 72% (3003/4170) Resolving deltas: 73% (3049/4170) Resolving deltas: 74% (3086/4170) Resolving deltas: 75% (3130/4170) Resolving deltas: 76% (3170/4170) Resolving deltas: 77% (3214/4170) Resolving deltas: 78% (3260/4170) Resolving deltas: 79% (3296/4170) Resolving deltas: 80% (3337/4170) Resolving deltas: 81% (3382/4170) Resolving deltas: 82% (3422/4170) Resolving deltas: 83% (3464/4170) Resolving deltas: 84% (3504/4170) Resolving deltas: 85% (3545/4170) Resolving deltas: 86% (3588/4170) Resolving deltas: 87% (3629/4170) Resolving deltas: 88% (3670/4170) Resolving deltas: 89% (3712/4170) Resolving deltas: 90% (3762/4170) Resolving deltas: 91% (3795/4170) Resolving deltas: 92% (3849/4170) Resolving deltas: 93% (3881/4170) Resolving deltas: 94% (3928/4170) Resolving deltas: 95% (3962/4170) Resolving deltas: 96% (4005/4170) Resolving deltas: 97% (4045/4170) Resolving deltas: 98% (4087/4170) Resolving deltas: 99% (4129/4170) Resolving deltas: 100% (4170/4170) Resolving deltas: 100% (4170/4170), completed with 951 local objects. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1739) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1482) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797) ... 8 more ERROR: null Finished: FAILURE
And after that crash, I've the following error each time I want to start a new build:
Started by user anonymous Setting origin to https://my.server.com/my-repo.git Fetching origin... > git.exe rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git.exe config remote.origin.url https://my.server.com/my-repo.git # timeout=10 Cleaning workspace > git.exe rev-parse --verify HEAD # timeout=10 Resetting working tree > git.exe reset --hard # timeout=10 > git.exe clean -fdx # timeout=10 Fetching upstream changes from https://my.server.com/my-repo.git > git.exe --version # timeout=10 using GIT_ASKPASS to set credentials vsts qdu pat > git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=120 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://my.server.com/my-repo.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:799) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1055) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108) at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:85) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:206) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:404) Caused by: hudson.plugins.git.GitException: Command "git.exe fetch --tags --progress https://my.server.com/my-repo.git +refs/heads/*:refs/remotes/origin/* --depth=1" returned status code 128: stdout: stderr: Fatal: InvalidOperationException encountered. fatal: Unable to create 'M:/workspace/b/f2a6f488@script/.git/shallow.lock': File exists. Another git process seems to be running in this repository, e.g. an editor opened by 'git commit'. Please make sure all processes are terminated then try again. If it still fails, a git process may have crashed in this repository earlier: remove the file manually to continue. at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1739) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1482) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:64) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:315) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797) ... 8 more ERROR: null Finished: FAILURE
How to reproduce
- Create a git repository with a huge file and a Jenkinsfile
- Create a multibranch pipeline linked to the previous repository
- Set the git checkout and clone timeout to the smallest value you can to cause a git timeout
- Try to restart the build, you'll have the error with the .lock file
Ideas to fix it
I don't know if it's hard to fix. Some idea would be to scan the git repository to search for .lock files and remove them ? Or implement the equivalent of this Jenkinsfile groovy script:
try { checkout scm } catch(caughtError) { deleteDir(); checkout scm }
Note 1: Even if it's a problem linked to git, I thought that you'll want to handle this case in the workflow multibranch pipeline. I may be wrong.
- duplicates
-
JENKINS-12513 git plugin should automatically remove .git/index.lock if the job is killed
-
- Open
-
- relates to
-
-
- Closed
-
