carsten_hilber, please provide more details to duplicate the problem. A detailed set of steps which others can use to see the problem is preferred. If that is not possible, then please answer the following questions:

• What is WS-CLEANUP and how is it used in that job?
• Is the Jenkins job a Pipeline job or a Freestyle job?
• If a Freestyle job, does the same problem exist if you modify the job definition to add the "Wipe out repository and force clone" additional behavior to the git configuration of the job?
• If a Pipeline job, does the same problem exist if you use the deleteDir step?
• Is the Jenkins job configured to use command line git or JGit to clone and checkout the workspace?
• What git related options are included in the job definition?
• Is there an anti-virus program running on the Windows computer? If so, is the problem still visible if virus scanning is disabled for the workspace directory?

I can't duplicate the problem in my experiment. I don't recognize WS-CLEANUP, so I approximated it with "Wipe out repository and force clone".

The Job is a freestyle job.
The option I set is called 'Delete workspace before build starts', within Build Environment.
It is configured to use the command line git. There are no more options given to the git task, only auth and branch is set.

I additionally set the "Wipe out repository and force clone" option in git and i saw the same behavior as before.
Then I set the wipe out exclusive, without the cleanup task and I got a different error message from git:

Started by user Carsten Hilber
Building remotely on webserv_runner_01 (webserv_runner) in workspace C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master
Wiping out workspace first.
java.nio.file.FileSystemException: C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
	at java.nio.file.Files.deleteIfExists(Unknown Source)
	at hudson.Util.tryOnceDeleteFile(Util.java:316)
	at hudson.Util.deleteFile(Util.java:272)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from wxp-turm-p13.ads.de/10.xx.xx.161:1697
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:389)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1072)
		at hudson.FilePath.act(FilePath.java:1061)
		at hudson.FilePath.deleteContents(FilePath.java:1283)
		at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:30)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1183)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1810)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
Caused: java.io.IOException: Unable to delete 'C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts.
	at hudson.Util.deleteFile(Util.java:277)
	at hudson.FilePath.deleteRecursive(FilePath.java:1305)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.deleteRecursive(FilePath.java:1296)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.deleteRecursive(FilePath.java:1296)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.deleteRecursive(FilePath.java:1296)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.access$1800(FilePath.java:213)
	at hudson.FilePath$DeleteContents.invoke(FilePath.java:1289)
	at hudson.FilePath$DeleteContents.invoke(FilePath.java:1285)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Unknown Source)
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\report.xml is 3 Minuten 13 Sekunden old

Starting import task...
Import Cucumber features Task started...

Same issue here. We run our Jenkins master on a Windows Server 2008 R2 server and anytime a freestyle job has the "Build Environment" check box checked for "Delete workspace before build starts", there is a lock that Jenkins has on a .pack file in the git directory.

Our work-around is to add an advanced configuration to the "Delete workspace before build starts" which specifically excludes 

*/.git/**

We are having this issue as well since we upgraded from 3.8.0 to 3.9.1.  At random times we have workspaces that cannot be cleaned anymore and the only solution seems to be to restart the Jenkins agent service.  This only occurs on our windows slaves, Linux slaves are not impacted.

The fact that cleanup workspace before build is mentioned here is not really a cause.  It's a symptom of a problem that occurs earlier.  If we do a build and then manually try to wipe the workspace we also get the same error.

Also it seems like if you wait a while (somewhere between half an hour and 2 hours the lock is removed and the cleanup works again.

I can confirm with Bart, after a couple of hours the lock is gone and the workspace cleanup/deleting is acting as expected.

We too noticed the same issue with Windows slaves under Jenkins version 2.150.3. Workaround mentioned by Aaron Nelson wasn't working because the next git clone identifies ".git" directory as suspicious.

I am able to replicate this issue with "Freestyle" job with below configurations.

1. SCM - Git checkout
2. Build Environment - Delete workspace before build starts
3. Build Trigger - For every 15 mins ( H/15 * * * * (
4. Build - This is not important for this testing.

As Bart explained, we are not able to delete the workspace/directory manually as well. Only way is to restart the Jenkins slave process.

Workaround:

Use "Delete workspace when build is done" in post build section apart of Build Environment. But this doesn't suit some use cases though.

Why do Jenkins process use the workspace  even after the build completes ? I see especially all these cases are throwing error with .git/objects/pack/<sha file> is in use

I can actually reproduce the bug on Git 3.8.0 and couldn't reproduce it on 3.0.5. I hope to bisect it to later versions to narrow the changes caused it.

I've isolated the bug to a change that happened between the 3.0.5(no bug) and 3.1.0 release

It may be a case of a conflicting change with another plugin change. If anyone else can try testing 3.0.5 fixes the issue it will be helpful

I looked at https://github.com/jenkinsci/git-plugin/compare/git-3.0.5...git-3.1.0 which led me to https://github.com/jenkinsci/git-client-plugin/compare/git-client-2.1.0...git-client-2.3.0 

markewaite do you think it could be related to think it could be caused by https://github.com/jenkinsci/git-client-plugin/compare/git-client-2.1.0...git-client-2.3.0#diff-4a13310f7c84a9d6e133def48c0aed5a  thanks in advance!

iplaman I don't see anything in the differences you listed that would leave a file open. The git client sequence of changes added support for git large files, but I assume you're not using git large files and thus are not executing the new code paths.

Per markewaite guidance and suggestions - big thanks! I've used Kosuke's tool https://file-leak-detector.kohsuke.org/ which helped me identify the open file handlers of the Jenkins node.

This the stack trace the tool provided, notice com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor 

I disabled this plugin and the issue is resolved now.

 
#17 PATH_TOWORKSPACE\GitWinBug\.git\objects\pack\pack-1db6aa468a129acaa6b8055b79680fec66793541.pack by thread:pool-1-thread-7 for JNLP4-connect connection to JENKINS_URL/IP_ADDR:60048 id=1013 on Sat Mar 16 21:44:38 EDT 2019#17 PATH_TOWORKSPACE\GitWinBug\.git\objects\pack\pack-1db6aa468a129acaa6b8055b79680fec66793541.pack by thread:pool-1-thread-7 for JNLP4-connect connection to JENKINS_URL/IP_ADDR:60048 id=1013 on Sat Mar 16 21:44:38 EDT 2019 at java.io.RandomAccessFile.<init>(Unknown Source) at org.eclipse.jgit.internal.storage.file.PackFile.doOpen(PackFile.java:643) at org.eclipse.jgit.internal.storage.file.PackFile.beginWindowCache(PackFile.java:625) at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:295) at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:379) at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:184) at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:360) at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:259) at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:601) at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:773) at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:285) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:486) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:444) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:435) at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165) at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236) at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:890) at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:800) at com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:128) at com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:105) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3087) at hudson.remoting.UserRequest.perform(UserRequest.java:207) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:358) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:98) at java.lang.Thread.run(Unknown Source)

Assigned to hp-application-automation-tools-plugin.
Workaround: Since I didn't need the HP plugin anymore I simply uninstalled it but for those which may still use it It could be worth trying to revert to 5.0 version(last known working version I used)
For hp-application-automation-tools-plugin maintainers, it's recommended you either to place the repository manipulating calls inside try with resources, or switch (where feasible) to use calls to the git client plugin rather than calling directly to JGit.

We also have a Windows node and we ran the fileleak and it is pointing to the same issue/plugin -

com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:128)

We will report back if reverting helps.

Okay, the revert could only go to 5.6.2 of  the Microfocus plugin and this did not help.  The issue is when you have the Delete Workspace selected (see pic below for where this option is set).  If I unselect it, stop the slave service, manually clean the workspace, start the service and run the job, then I am able to run the job successfully thereafter.

**It turned out that we had used a different user than the user account the Jenkins slave service was using when troubleshooting/working with the Jenkins workspace directory. Despite both accounts being members of the Administrators group it seems everything needed to be owned/created/modified using the same account**

Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....?

Note: The below should says 

*/.git/** 

 but the second asterisk isn't coming through.

csmith_sgas you don't mention in your description if you are using the hp-automation-tools-plugin. If you are not using the hp-automation-tools-plugin, then you've found a different problem and it most likely has a different root cause. If you are not using the hp-automation-tools-plugin, please submit a separate bug report with the details of your environment and your best guess at the steps that will allow others to recreate the problem.

It happens to us as well.

• We are using Workspace Cleanup  plugin 0.37 
• Jenkins 3.183.3 Freestyle 

It keeps running the same error
```
Triggered by Benson GitLab Merge Request #181: omg/feature/gateway/IotRequestServer_v2 => master
Building in workspace /var/lib/jenkins/workspace/omg_smoke_test
[WS-CLEANUP] Deleting project workspace...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete '/var/lib/jenkins/workspace/omg_smoke_test'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
ERROR: Cannot delete workspace: Unable to delete '/var/lib/jenkins/workspace/omg_smoke_test'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
Finished: FAILURE
```

Util the other user kick a new job from different repo.

radislavb, was it fixed in the latest version of the plugin?

radislavb do we have any updates on this? I can confirm that workspace is not getting deleted because of locked .pack file.

radislavb,

Thank you so much!!!

I can confirm that 5.8.5-beta have fixed workspace delete issue.

We are on latest 5.8 version. Any plan on releasing stable version with the fix?

Hey manishchristian, our release process takes some time in order to ensure that everything works correctly and no bugs are found. We are hoping to release a stable version in October.

I am moving this issue to fixed.

Hello, Can please someone tell the ETA when the fix will be released? Thanks in advance.

Is this definitely solved in the 5.9 version?

When will this fix actually be Released ?
As far as I can see it was not in 5.9.

In versions 5.9 we did several related fixes in the area.

Version 5.9 was released in september.

Can you confirm this beta fix was included in version 5.9 ?
Status of this issue suggests otherwise ...

included,

I also will change status of issue.

OK, thx for the confirmation.