Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55836

Unable to delete workspace due to locked git .pack file

XMLWordPrintable

      In our Jenkins site, all builds which are run on a windows Jenkins client and which do not delete their working direktory prior to run the job will fail with a message like:

      [WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\.git\objects\pack\pack-5e75a4a1a37fced205964f0aeaadd0b304e779f8.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts.
ERROR: Cannot delete workspace: Unable to delete 'C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\.git\objects\pack\pack-5e75a4a1a37fced205964f0aeaadd0b304e779f8.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts.
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\Results29012019125534536.xml is 3 Minuten 34 Sekunden old

Recording test results
ERROR: Step ‘Publish Micro Focus tests result’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\Results29012019125534536.xml is 3 Minuten 34 Sekunden old
...
Sucessfully imported JUnit XML results from Results29012019125534536.xml
Finished: FAILURE

        1. image-2019-03-21-15-40-39-729.png
          image-2019-03-21-15-40-39-729.png
          14 kB

          [JENKINS-55836] Unable to delete workspace due to locked git .pack file

          Carsten Hilber created issue -
          Carsten Hilber made changes -
          Component/s New: git-plugin [ 15543 ]
          Component/s Original: github [ 18925 ]
          Key Original: INFRA-1988 New: JENKINS-55836
          Workflow Original: classic default workflow [ 234632 ] New: JNJira + In-Review [ 234633 ]
          Project Original: Infrastructure [ 10301 ] New: Jenkins [ 10172 ]
          Daniel Beck made changes -
          Assignee Original: Daniel Beck [ danielbeck ]
          Carsten Hilber made changes -
          Priority Original: Minor [ 4 ] New: Critical [ 2 ]

          Mark Waite added a comment - - edited

          carsten_hilber, please provide more details to duplicate the problem. A detailed set of steps which others can use to see the problem is preferred. If that is not possible, then please answer the following questions:

          • What is WS-CLEANUP and how is it used in that job?
          • Is the Jenkins job a Pipeline job or a Freestyle job?
          • If a Freestyle job, does the same problem exist if you modify the job definition to add the "Wipe out repository and force clone" additional behavior to the git configuration of the job?
          • If a Pipeline job, does the same problem exist if you use the deleteDir step?
          • Is the Jenkins job configured to use command line git or JGit to clone and checkout the workspace?
          • What git related options are included in the job definition?
          • Is there an anti-virus program running on the Windows computer? If so, is the problem still visible if virus scanning is disabled for the workspace directory?

          I can't duplicate the problem in my experiment. I don't recognize WS-CLEANUP, so I approximated it with "Wipe out repository and force clone".

          Mark Waite added a comment - - edited carsten_hilber , please provide more details to duplicate the problem. A detailed set of steps which others can use to see the problem is preferred. If that is not possible, then please answer the following questions: What is WS-CLEANUP and how is it used in that job? Is the Jenkins job a Pipeline job or a Freestyle job? If a Freestyle job, does the same problem exist if you modify the job definition to add the "Wipe out repository and force clone" additional behavior to the git configuration of the job? If a Pipeline job, does the same problem exist if you use the deleteDir step? Is the Jenkins job configured to use command line git or JGit to clone and checkout the workspace? What git related options are included in the job definition? Is there an anti-virus program running on the Windows computer? If so, is the problem still visible if virus scanning is disabled for the workspace directory? I can't duplicate the problem in my experiment . I don't recognize WS-CLEANUP, so I approximated it with "Wipe out repository and force clone".

          Carsten Hilber added a comment - - edited

          The Job is a freestyle job.
          The option I set is called 'Delete workspace before build starts', within Build Environment.
          It is configured to use the command line git. There are no more options given to the git task, only auth and branch is set.

          I additionally set the "Wipe out repository and force clone" option in git and i saw the same behavior as before.
          Then I set the wipe out exclusive, without the cleanup task and I got a different error message from git:

          Started by user Carsten Hilber
Building remotely on webserv_runner_01 (webserv_runner) in workspace C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master
Wiping out workspace first.
java.nio.file.FileSystemException: C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
	at java.nio.file.Files.deleteIfExists(Unknown Source)
	at hudson.Util.tryOnceDeleteFile(Util.java:316)
	at hudson.Util.deleteFile(Util.java:272)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from wxp-turm-p13.ads.de/10.xx.xx.161:1697
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserResponse.retrieve(UserRequest.java:389)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1072)
		at hudson.FilePath.act(FilePath.java:1061)
		at hudson.FilePath.deleteContents(FilePath.java:1283)
		at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:30)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1183)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1810)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
Caused: java.io.IOException: Unable to delete 'C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts.
	at hudson.Util.deleteFile(Util.java:277)
	at hudson.FilePath.deleteRecursive(FilePath.java:1305)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.deleteRecursive(FilePath.java:1296)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.deleteRecursive(FilePath.java:1296)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.deleteRecursive(FilePath.java:1296)
	at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314)
	at hudson.FilePath.access$1800(FilePath.java:213)
	at hudson.FilePath$DeleteContents.invoke(FilePath.java:1289)
	at hudson.FilePath$DeleteContents.invoke(FilePath.java:1285)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Unknown Source)
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\report.xml is 3 Minuten 13 Sekunden old

Starting import task...
Import Cucumber features Task started...

          Carsten Hilber added a comment - - edited The Job is a freestyle job. The option I set is called 'Delete workspace before build starts', within Build Environment. It is configured to use the command line git. There are no more options given to the git task, only auth and branch is set. I additionally set the "Wipe out repository and force clone" option in git and i saw the same behavior as before. Then I set the wipe out exclusive, without the cleanup task and I got a different error message from git: Started by user Carsten Hilber Building remotely on webserv_runner_01 (webserv_runner) in workspace C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master Wiping out workspace first. java.nio.file.FileSystemException: C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird. at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) at java.nio.file.Files.deleteIfExists(Unknown Source) at hudson.Util.tryOnceDeleteFile(Util.java:316) at hudson.Util.deleteFile(Util.java:272) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from wxp-turm-p13.ads.de/10.xx.xx.161:1697 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.UserResponse.retrieve(UserRequest.java:389) at hudson.remoting.Channel.call(Channel.java:955) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at hudson.FilePath.deleteContents(FilePath.java:1283) at hudson.plugins.git.extensions.impl.WipeWorkspace.beforeCheckout(WipeWorkspace.java:30) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1183) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1810) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused: java.io.IOException: Unable to delete 'C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\.git\objects\pack\pack-5d0b10aae545c18f1c4b0ec7d499f31aa93e7b5a.pack' . Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts. at hudson.Util.deleteFile(Util.java:277) at hudson.FilePath.deleteRecursive(FilePath.java:1305) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.deleteRecursive(FilePath.java:1296) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.deleteRecursive(FilePath.java:1296) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.deleteRecursive(FilePath.java:1296) at hudson.FilePath.deleteContentsRecursive(FilePath.java:1314) at hudson.FilePath.access$1800(FilePath.java:213) at hudson.FilePath$DeleteContents.invoke(FilePath.java:1289) at hudson.FilePath$DeleteContents.invoke(FilePath.java:1285) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086) at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:94) at java.lang. Thread .run(Unknown Source) Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new . Did leafNodes run? For example, C:\Jenkins\workspace\Testautomatisierung\06_Testfälle_ReadyAPI\Guidewire\CC\RA_A_Smoketest_Master\report.xml is 3 Minuten 13 Sekunden old Starting import task... Import Cucumber features Task started...

          Aaron Nelson added a comment - - edited

          Same issue here. We run our Jenkins master on a Windows Server 2008 R2 server and anytime a freestyle job has the "Build Environment" check box checked for "Delete workspace before build starts", there is a lock that Jenkins has on a .pack file in the git directory.

           

          Our work-around is to add an advanced configuration to the "Delete workspace before build starts" which specifically excludes 

          */.git/**

          Aaron Nelson added a comment - - edited Same issue here. We run our Jenkins master on a Windows Server 2008 R2 server and anytime a freestyle job has the "Build Environment" check box checked for "Delete workspace before build starts", there is a lock that Jenkins has on a .pack file in the git directory.   Our work-around is to add an advanced configuration to the "Delete workspace before build starts" which specifically excludes  */.git/**

          Bart Devriendt added a comment -

          We are having this issue as well since we upgraded from 3.8.0 to 3.9.1.  At random times we have workspaces that cannot be cleaned anymore and the only solution seems to be to restart the Jenkins agent service.  This only occurs on our windows slaves, Linux slaves are not impacted.

          Bart Devriendt added a comment - We are having this issue as well since we upgraded from 3.8.0 to 3.9.1.  At random times we have workspaces that cannot be cleaned anymore and the only solution seems to be to restart the Jenkins agent service.  This only occurs on our windows slaves, Linux slaves are not impacted.

          Bart Devriendt added a comment -

          The fact that cleanup workspace before build is mentioned here is not really a cause.  It's a symptom of a problem that occurs earlier.  If we do a build and then manually try to wipe the workspace we also get the same error.

          Also it seems like if you wait a while (somewhere between half an hour and 2 hours the lock is removed and the cleanup works again.

          Bart Devriendt added a comment - The fact that cleanup workspace before build is mentioned here is not really a cause.  It's a symptom of a problem that occurs earlier.  If we do a build and then manually try to wipe the workspace we also get the same error. Also it seems like if you wait a while (somewhere between half an hour and 2 hours the lock is removed and the cleanup works again.

          Aaron Nelson added a comment -

          I can confirm with Bart, after a couple of hours the lock is gone and the workspace cleanup/deleting is acting as expected.

          Aaron Nelson added a comment - I can confirm with Bart, after a couple of hours the lock is gone and the workspace cleanup/deleting is acting as expected.

            radislavb Radi Berkovich
            carsten_hilber Carsten Hilber
            Votes:
            24 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: