Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35160

Unable to delete project using NFS. File descriptor leak in build folder

      I tried to delete a project, but getting following error:

      http://cheesecake.factory.corp.yahoo.com:9999/jenkins/user/jennyhung/my-views/view/All/job/GeminiE2E_SearchSharedLibrarySiteLinkTestSingle-verifyAdServingNoUse/doDelete

      A problem occurred while processing the request. Please check our bug tracker to see if a similar problem has already been reported. If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem. If you think this is a new issue, please file a new issue. When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins. The users list might be also useful in understanding what has happened.
      Stack trace

      java.nio.file.FileSystemException: /home/y/var/jenkins/jobs/GeminiE2E_SearchSharedLibrarySiteLinkTestSingle-verifyAdServingNoUse/builds/2016-05-18_03-38-23/.nfs000000000fbbd28800000082: Device or resource busy
      	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
      	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
      	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
      	at java.nio.file.Files.delete(Files.java:1126)
      	at sun.reflect.GeneratedMethodAccessor3466.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at hudson.Util.deleteFile(Util.java:247)
      	at hudson.Util.deleteRecursive(Util.java:310)
      	at hudson.Util.deleteContentsRecursive(Util.java:212)
      	at hudson.Util.deleteRecursive(Util.java:301)
      	at hudson.Util.deleteContentsRecursive(Util.java:212)
      	at hudson.Util.deleteRecursive(Util.java:301)
      	at hudson.Util.deleteContentsRecursive(Util.java:212)
      	at hudson.Util.deleteRecursive(Util.java:301)
      	at hudson.model.AbstractItem.performDelete(AbstractItem.java:596)
      	at hudson.model.Job.performDelete(Job.java:259)
      	at hudson.model.AbstractProject.performDelete(AbstractProject.java:372)
      	at hudson.model.AbstractItem.delete(AbstractItem.java:585)
      	at hudson.model.Job.delete(Job.java:648)
      	at hudson.model.AbstractItem.doDoDelete(AbstractItem.java:546)
      	at sun.reflect.GeneratedMethodAccessor3013.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
      	at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:391)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:102)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.servlet.filter.BouncerFilter.doFilter(BouncerFilter.java:217)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.servlet.filter.SSLCrimeFilter.doFilter(SSLCrimeFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:73)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:152)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:114)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:126)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:228)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at yjava.servlet.filter.StatsFilter.doFilter(StatsFilter.java:73)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
      	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.webapp.logging.ContextLogHandler.handle(ContextLogHandler.java:62)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:497)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      	at java.lang.Thread.run(Thread.java:745)
      

          [JENKINS-35160] Unable to delete project using NFS. File descriptor leak in build folder

          Jenny Hung created issue -
          Oleg Nenashev made changes -
          Summary Original: Unable to delete project New: Unable to delete project. File descriptor leak in build folder

          Oleg Nenashev added a comment -

          It's a file descriptor leak somewhere in the build folder.
          Could you please create a dump of open file descriptors?

          E.g. it can be done via https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin

          Oleg Nenashev added a comment - It's a file descriptor leak somewhere in the build folder. Could you please create a dump of open file descriptors? E.g. it can be done via https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin
          Oleg Nenashev made changes -
          Assignee New: Oleg Nenashev [ oleg_nenashev ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 171380 ] New: JNJira + In-Review [ 184285 ]
          Carlos Sanchez made changes -
          Description Original: I tried to delete a project, but getting following error:

          http://cheesecake.factory.corp.yahoo.com:9999/jenkins/user/jennyhung/my-views/view/All/job/GeminiE2E_SearchSharedLibrarySiteLinkTestSingle-verifyAdServingNoUse/doDelete

           

          A problem occurred while processing the request. Please check our bug tracker to see if a similar problem has already been reported. If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem. If you think this is a new issue, please file a new issue. When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins. The users list might be also useful in understanding what has happened.
          Stack trace

          java.nio.file.FileSystemException: /home/y/var/jenkins/jobs/GeminiE2E_SearchSharedLibrarySiteLinkTestSingle-verifyAdServingNoUse/builds/2016-05-18_03-38-23/.nfs000000000fbbd28800000082: Device or resource busy
          at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
          at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
          at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
          at java.nio.file.Files.delete(Files.java:1126)
          at sun.reflect.GeneratedMethodAccessor3466.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at hudson.Util.deleteFile(Util.java:247)
          at hudson.Util.deleteRecursive(Util.java:310)
          at hudson.Util.deleteContentsRecursive(Util.java:212)
          at hudson.Util.deleteRecursive(Util.java:301)
          at hudson.Util.deleteContentsRecursive(Util.java:212)
          at hudson.Util.deleteRecursive(Util.java:301)
          at hudson.Util.deleteContentsRecursive(Util.java:212)
          at hudson.Util.deleteRecursive(Util.java:301)
          at hudson.model.AbstractItem.performDelete(AbstractItem.java:596)
          at hudson.model.Job.performDelete(Job.java:259)
          at hudson.model.AbstractProject.performDelete(AbstractProject.java:372)
          at hudson.model.AbstractItem.delete(AbstractItem.java:585)
          at hudson.model.Job.delete(Job.java:648)
          at hudson.model.AbstractItem.doDoDelete(AbstractItem.java:546)
          at sun.reflect.GeneratedMethodAccessor3013.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
          at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:391)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
          at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:102)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.BouncerFilter.doFilter(BouncerFilter.java:217)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.SSLCrimeFilter.doFilter(SSLCrimeFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:73)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:152)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:114)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:126)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:228)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.StatsFilter.doFilter(StatsFilter.java:73)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
          at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.webapp.logging.ContextLogHandler.handle(ContextLogHandler.java:62)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.Server.handle(Server.java:497)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
          at java.lang.Thread.run(Thread.java:745)
          New: I tried to delete a project, but getting following error:

          http://cheesecake.factory.corp.yahoo.com:9999/jenkins/user/jennyhung/my-views/view/All/job/GeminiE2E_SearchSharedLibrarySiteLinkTestSingle-verifyAdServingNoUse/doDelete

           

          A problem occurred while processing the request. Please check our bug tracker to see if a similar problem has already been reported. If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem. If you think this is a new issue, please file a new issue. When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins. The users list might be also useful in understanding what has happened.
          Stack trace

          {code}
          java.nio.file.FileSystemException: /home/y/var/jenkins/jobs/GeminiE2E_SearchSharedLibrarySiteLinkTestSingle-verifyAdServingNoUse/builds/2016-05-18_03-38-23/.nfs000000000fbbd28800000082: Device or resource busy
          at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
          at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
          at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
          at java.nio.file.Files.delete(Files.java:1126)
          at sun.reflect.GeneratedMethodAccessor3466.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at hudson.Util.deleteFile(Util.java:247)
          at hudson.Util.deleteRecursive(Util.java:310)
          at hudson.Util.deleteContentsRecursive(Util.java:212)
          at hudson.Util.deleteRecursive(Util.java:301)
          at hudson.Util.deleteContentsRecursive(Util.java:212)
          at hudson.Util.deleteRecursive(Util.java:301)
          at hudson.Util.deleteContentsRecursive(Util.java:212)
          at hudson.Util.deleteRecursive(Util.java:301)
          at hudson.model.AbstractItem.performDelete(AbstractItem.java:596)
          at hudson.model.Job.performDelete(Job.java:259)
          at hudson.model.AbstractProject.performDelete(AbstractProject.java:372)
          at hudson.model.AbstractItem.delete(AbstractItem.java:585)
          at hudson.model.Job.delete(Job.java:648)
          at hudson.model.AbstractItem.doDoDelete(AbstractItem.java:546)
          at sun.reflect.GeneratedMethodAccessor3013.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
          at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:391)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
          at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:117)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:102)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.BouncerFilter.doFilter(BouncerFilter.java:217)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.SSLCrimeFilter.doFilter(SSLCrimeFilter.java:30)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.YHdrsFilter.doFilter(YHdrsFilter.java:73)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.cookie.CookieDataFilter.doFilter(CookieDataFilter.java:152)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.DoNotTrackFilter.doFilter(DoNotTrackFilter.java:114)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.remote.ip.RemoteIPFilter.doFilter(RemoteIPFilter.java:126)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.security.yiv.servlet.InputValidationFilter.doFilter(InputValidationFilter.java:228)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at yjava.servlet.filter.StatsFilter.doFilter(StatsFilter.java:73)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
          at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
          at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.webapp.logging.ContextLogHandler.handle(ContextLogHandler.java:62)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:309)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          at org.eclipse.jetty.server.Server.handle(Server.java:497)
          at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
          at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
          at java.lang.Thread.run(Thread.java:745)
          {code}

          This is easily reproducible when deleting a job that is running when backend is NFS

          The problem is that it leaves the dir in a half-deleted state.
          IMO Jenkins should not attempt to delete .nfs* files, or ignore any failed file deletion and log it as WARNING, not dump a stacktrace

          java.io.IOException: Unable to delete '/var/jenkins_home/jobs/generated-jobs-test-2/jobs/job1'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
          	at hudson.Util.deleteRecursive(Util.java:346)
          	at hudson.model.AbstractItem.performDelete(AbstractItem.java:581)
          	at hudson.model.Job.performDelete(Job.java:276)
          	at hudson.model.AbstractProject.performDelete(AbstractProject.java:379)
          	at hudson.model.AbstractItem.delete(AbstractItem.java:570)
          	at hudson.model.Job.delete(Job.java:683)
          	at hudson.model.AbstractItem.doDoDelete(AbstractItem.java:531)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:335)
          	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
          	at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
          	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:175)
          	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:108)
          	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236)
          	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
          	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
          	at com.cloudbees.jenkins.ha.HAHealthCheckFilter.doFilter(HAHealthCheckFilter.java:35)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
          	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          	at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:90)
          	at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:67)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          	at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:69)
          	at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:44)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
          	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:80)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
          	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
          	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
          	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
          	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
          	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
          	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
          	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
          	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
          	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
          	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
          	at org.eclipse.jetty.server.Server.handle(Server.java:499)
          	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
          	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
          	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
          	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          Caused by: java.nio.file.FileSystemException: /var/jenkins_home/jobs/generated-jobs-test-2/jobs/job1/builds/1/.nfs4ad125975b07daa2000003e3: Device or resource busy
          	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
          	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
          	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
          	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
          	at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
          	at java.nio.file.Files.deleteIfExists(Files.java:1165)
          	at hudson.Util.tryOnceDeleteFile(Util.java:287)
          	at hudson.Util.tryOnceDeleteRecursive(Util.java:364)
          	at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383)
          	at hudson.Util.tryOnceDeleteRecursive(Util.java:363)
          	at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383)
          	at hudson.Util.tryOnceDeleteRecursive(Util.java:363)
          	at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383)
          	at hudson.Util.tryOnceDeleteRecursive(Util.java:363)
          	at hudson.Util.deleteRecursive(Util.java:341)
          	... 92 more
          

          Carlos Sanchez added a comment - This is easily reproducible when deleting a job that is running when backend is NFS The problem is that it leaves the dir in a half-deleted state. IMO Jenkins should not attempt to delete .nfs* files, or ignore any failed file deletion and log it as WARNING, not dump a stacktrace java.io.IOException: Unable to delete '/ var /jenkins_home/jobs/generated-jobs-test-2/jobs/job1' . Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at hudson.Util.deleteRecursive(Util.java:346) at hudson.model.AbstractItem.performDelete(AbstractItem.java:581) at hudson.model.Job.performDelete(Job.java:276) at hudson.model.AbstractProject.performDelete(AbstractProject.java:379) at hudson.model.AbstractItem.delete(AbstractItem.java:570) at hudson.model.Job.delete(Job.java:683) at hudson.model.AbstractItem.doDoDelete(AbstractItem.java:531) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:335) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:175) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:108) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:236) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at com.cloudbees.jenkins.ha.HAHealthCheckFilter.doFilter(HAHealthCheckFilter.java:35) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter._doFilter(OfflineSecurityRealmFilter.java:90) at com.cloudbees.opscenter.client.plugin.OfflineSecurityRealmFilter.doFilter(OfflineSecurityRealmFilter.java:67) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at com.cloudbees.opscenter.security.ClusterSessionFilter._doFilter(ClusterSessionFilter.java:69) at com.cloudbees.opscenter.security.ClusterSessionFilter.doFilter(ClusterSessionFilter.java:44) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:80) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Caused by: java.nio.file.FileSystemException: / var /jenkins_home/jobs/generated-jobs-test-2/jobs/job1/builds/1/.nfs4ad125975b07daa2000003e3: Device or resource busy at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108) at java.nio.file.Files.deleteIfExists(Files.java:1165) at hudson.Util.tryOnceDeleteFile(Util.java:287) at hudson.Util.tryOnceDeleteRecursive(Util.java:364) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383) at hudson.Util.tryOnceDeleteRecursive(Util.java:363) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383) at hudson.Util.tryOnceDeleteRecursive(Util.java:363) at hudson.Util.tryOnceDeleteContentsRecursive(Util.java:383) at hudson.Util.tryOnceDeleteRecursive(Util.java:363) at hudson.Util.deleteRecursive(Util.java:341) ... 92 more

          You can try by deleting the log for the job from the shell

          I have no name!@ip-172-18-54-215:/var/jenkins_home/jobs$ ls -laFR test1/builds/1
          test1/builds/1:
          total 16
          drwxr-xr-x. 2 2000 root 4096 Feb 21 17:31 ./
          drwxr-xr-x. 3 2000 root 4096 Feb 21 17:31 ../
          -rw-r--r--. 1 2000 root    6 Feb 21 17:31 changelog.xml
          -rw-r--r--. 1 2000 root  715 Feb 21 17:31 log
          I have no name!@ip-172-18-54-215:/var/jenkins_home/jobs$ rm  test1/builds/1/log
          

          Then nfs will create the .nfs* file, which means the file is in use. The .nfs* file can't (and shouldn't) be deleted

          I have no name!@ip-172-18-54-215:/var/jenkins_home/jobs$ ls -laFR test1/builds/1
          test1/builds/1:
          total 16
          drwxr-xr-x. 2 2000 root 4096 Feb 21 17:35 ./
          drwxr-xr-x. 3 2000 root 4096 Feb 21 17:31 ../
          -rw-r--r--. 1 2000 root  715 Feb 21 17:31 .nfsb1ebe46654ce5b0b00000011
          -rw-r--r--. 1 2000 root    6 Feb 21 17:31 changelog.xml
          

          Carlos Sanchez added a comment - You can try by deleting the log for the job from the shell I have no name!@ip-172-18-54-215:/ var /jenkins_home/jobs$ ls -laFR test1/builds/1 test1/builds/1: total 16 drwxr-xr-x. 2 2000 root 4096 Feb 21 17:31 ./ drwxr-xr-x. 3 2000 root 4096 Feb 21 17:31 ../ -rw-r--r--. 1 2000 root 6 Feb 21 17:31 changelog.xml -rw-r--r--. 1 2000 root 715 Feb 21 17:31 log I have no name!@ip-172-18-54-215:/ var /jenkins_home/jobs$ rm test1/builds/1/log Then nfs will create the .nfs* file, which means the file is in use. The .nfs* file can't (and shouldn't) be deleted I have no name!@ip-172-18-54-215:/ var /jenkins_home/jobs$ ls -laFR test1/builds/1 test1/builds/1: total 16 drwxr-xr-x. 2 2000 root 4096 Feb 21 17:35 ./ drwxr-xr-x. 3 2000 root 4096 Feb 21 17:31 ../ -rw-r--r--. 1 2000 root 715 Feb 21 17:31 .nfsb1ebe46654ce5b0b00000011 -rw-r--r--. 1 2000 root 6 Feb 21 17:31 changelog.xml

          Oleg Nenashev added a comment -

          > This is easily reproducible when deleting a job that is running when backend is NFS

          I Suppose we should prohibit this use-case at all (return error) or "Disable & schedule for deletion".

          Oleg Nenashev added a comment - > This is easily reproducible when deleting a job that is running when backend is NFS I Suppose we should prohibit this use-case at all (return error) or "Disable & schedule for deletion".

          I think it is fine to delete all files but the locked ones, they will go away. You would get all the empty dirs though, maybe setting them to delete on restart could work just fine

          Carlos Sanchez added a comment - I think it is fine to delete all files but the locked ones, they will go away. You would get all the empty dirs though, maybe setting them to delete on restart could work just fine

            stephenconnolly Stephen Connolly
            jennyhung Jenny Hung
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: