Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17995

Jenkins produces DirectoryNotEmptyException when attempting to delete large directories

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • core, subversion-plugin
    • None
    • Jenkins version: 1.514
      Master: Debian Wheezy, standard openjdk (6b27-1.12.5-1, java version 1.6.0_27).
      Slave: Windows 7, Java 1.7.0_04

      We're getting DirectoryNotEmptyException's in 2 scenario's when dealing with workspaces that have directories that contain a lot of files and are heavily nested (specifically: maven local repositories). Both when applying the "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'" SVN checkout strategy, and when deleting a workspace (both on a Windows slave), Jenkins seems to sometimes have trouble deleting the local maven repository (which is contained in the workspace, and yes that's stupid and we changed it, but I thought I log this bug anyway). See the logs below.

      Here's the error I got in the build log:

      Bouwen op afstand op fitnesse-host in workspace C:\jenkins\workspace\Test_Job
      Checking out a fresh workspace because the workspace is not http://svn.local/fitnesse/tags/LAST_STABLE
      Cleaning local Directory .
      hudson.util.IOException2: remote file operation failed: C:\jenkins\workspace\Test_Job at hudson.remoting.Channel@5defe20d:fitnesse-host
        at hudson.FilePath.act(FilePath.java:900)
        at hudson.FilePath.act(FilePath.java:877)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:843)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:781)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
        at hudson.model.Run.execute(Run.java:1575)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
        at hudson.model.ResourceController.execute(ResourceController.java:88)
        at hudson.model.Executor.run(Executor.java:241)
      Caused by: java.io.IOException: Remote call on fitnesse-host failed
        at hudson.remoting.Channel.call(Channel.java:677)
        at hudson.FilePath.act(FilePath.java:893)
        ... 11 more
      Caused by: java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:266)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622)
        at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:116)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1609)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1768)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
        at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:211)
        at hudson.remoting.Channel.call(Channel.java:673)
        ... 12 more
      Recording test results
      [htmlpublisher] Archiving HTML reports...
      [htmlpublisher] Archiving at BUILD level 
      ......
      Notifying upstream projects of job completion
      Finished: FAILURE
      

      And here's the error I got from the "Wipe out current workspace" button:

      Status Code: 500
      
      Exception: remote file operation failed: C:\jenkins\workspace\Test_Job at hudson.remoting.Channel@536e8e87:fitnesse-host
      Stacktrace:
      hudson.util.IOException2: remote file operation failed: C:\jenkins\workspace\Test_Job at hudson.remoting.Channel@536e8e87:fitnesse-host
      	at hudson.FilePath.act(FilePath.java:900)
      	at hudson.FilePath.act(FilePath.java:877)
      	at hudson.FilePath.deleteRecursive(FilePath.java:1036)
      	at hudson.model.AbstractProject.doDoWipeOutWorkspace(AbstractProject.java:2023)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:616)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
      	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
      	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      	at java.lang.Thread.run(Thread.java:679)
      Caused by: java.io.IOException: Remote call on fitnesse-host failed
      	at hudson.remoting.Channel.call(Channel.java:677)
      	at hudson.FilePath.act(FilePath.java:893)
      	... 73 more
      Caused by: java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException
      	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
      	at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:266)
      	at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:622)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:116)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1609)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1768)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:211)
      	at hudson.remoting.Channel.call(Channel.java:673)
      	... 74 more
      

          [JENKINS-17995] Jenkins produces DirectoryNotEmptyException when attempting to delete large directories

          Brian Brooks added a comment - - edited

          This issue seems to be a duplicate of
          JENKINS-15331 Workaround Windows unpredictable
          The heart of this issue is JVM file locking, particularly with Windows 7 / Windows 2008.

          Brian Brooks added a comment - - edited This issue seems to be a duplicate of JENKINS-15331 Workaround Windows unpredictable The heart of this issue is JVM file locking, particularly with Windows 7 / Windows 2008.

          Actually it's not DirectoryNotEmptyException itself that causes problems, it's java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException, meaning the exception class could not be found. Any clues?

          Krzysztof Malinowski added a comment - Actually it's not DirectoryNotEmptyException itself that causes problems, it's java.lang.ClassNotFoundException: java.nio.file.DirectoryNotEmptyException, meaning the exception class could not be found. Any clues?

          OK, reading further that java.nio.file.DirectoryNotEmptyException was introduced in Java 7, I guess the issue is caused by your server running Java 6, while slaves run Java 7. This way server cannot deserialize exception coming from your slave and this issue occurs.

          I think I encountered the same issue, yet I am unable to switch to 1.7 on the server right now. Once I'll check I will update this issue. If it's possible for you to update java to 1.7 on the server, you could check it for yourself.

          Krzysztof Malinowski added a comment - OK, reading further that java.nio.file.DirectoryNotEmptyException was introduced in Java 7, I guess the issue is caused by your server running Java 6, while slaves run Java 7. This way server cannot deserialize exception coming from your slave and this issue occurs. I think I encountered the same issue, yet I am unable to switch to 1.7 on the server right now. Once I'll check I will update this issue. If it's possible for you to update java to 1.7 on the server, you could check it for yourself.

          Krzysztof: our issue was due to a master/server JDK version mismatch as well, sorry for not reporting back.

          Maarten Dirkse added a comment - Krzysztof: our issue was due to a master/server JDK version mismatch as well, sorry for not reporting back.

          As said in earlier comments, this issue was due to a mismatch in JDK version between the master and the slave machines. The exception thrown is one that doesn't exists in JDK 6, so deserialization fails. The issue disappeared when we downgraded the slaves to JDK 6.

          Maarten Dirkse added a comment - As said in earlier comments, this issue was due to a mismatch in JDK version between the master and the slave machines. The exception thrown is one that doesn't exists in JDK 6, so deserialization fails. The issue disappeared when we downgraded the slaves to JDK 6.

            Unassigned Unassigned
            mdirkse Maarten Dirkse
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: