Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25742

Concurrent "job rename" and "edit view" results in deadlock that puts Jenkins in a non responsive state

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      When there are multiple users using Jenkins:

      One user has clicked on "Yes" confirmation to rename a job.
      Another user has clicked on "OK" on the "Edit View" screen at the same time.

      Results in a race condition that leads to a deadlock. Immediately after this deadlock occurs, Jenkins "create new job" and other operations are unresponsive. And after some more time Jenkins goes down.

      Name: Handling GET /jenkins/view/ABCDEF/api/json : qtp817406040-243035
      State: BLOCKED on hudson.model.ListView@57affa54 owned by: Handling POST /jenkins/view/ABCDEF/configSubmit : qtp817406040-42
      Total blocked: 9  Total waited: 189
      
      Stack trace: 
      hudson.model.ListView.getItems(ListView.java:164)
      hudson.model.ListView.getItems(ListView.java:66)
      sun.reflect.GeneratedMethodAccessor1929.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:483)
      org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
      org.kohsuke.stapler.export.Property.writeTo(Property.java:116)
      org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:190)
      org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185)
      org.kohsuke.stapler.export.Model.writeTo(Model.java:157)
      org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267)
      hudson.model.Api.doJson(Api.java:204)
      sun.reflect.GeneratedMethodAccessor771.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:483)
      org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
      org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      
      
      
      Name: Handling POST /jenkins/view/ABCDEF/configSubmit : qtp817406040-42
      State: BLOCKED on hudson.model.Hudson@715617bb owned by: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
      Total blocked: 2,007  Total waited: 12,613
      
      Stack trace: 
      jenkins.model.Jenkins.save(Jenkins.java:2675)
      hudson.model.View.save(View.java:341)
      hudson.util.PersistedList.onModified(PersistedList.java:173)
      hudson.util.PersistedList.replaceBy(PersistedList.java:85)
      hudson.util.DescribableList.rebuildHetero(DescribableList.java:203)
      hudson.model.ListView.submit(ListView.java:348)
      hudson.model.View.doConfigSubmit(View.java:956)
         - locked hudson.model.ListView@57affa54
      sun.reflect.GeneratedMethodAccessor1966.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:483)
      org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
      org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
      org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
      org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      
      
      
      Name: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
      State: BLOCKED on hudson.model.ListView@7e01a6a2 owned by: Handling POST /jenkins/view/XYZXYZ/configSubmit : qtp817406040-117
      Total blocked: 54,155  Total waited: 11,725
      
      Stack trace: 
      hudson.model.ListView$Listener.renameViewItem(ListView.java:421)
      hudson.model.ListView$Listener.onLocationChanged(ListView.java:404)
      hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:199)
      hudson.model.AbstractItem.renameTo(AbstractItem.java:317)
         - locked hudson.model.FreeStyleProject@5dafef50
         - locked hudson.model.Hudson@715617bb
      hudson.model.Job.renameTo(Job.java:617)
      hudson.model.Job.doDoRename(Job.java:1403)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:483)
      org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
      org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
      org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
      org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      
      
      
      Name: Handling POST /jenkins/view/XYZXYZ/configSubmit : qtp817406040-117
      State: BLOCKED on hudson.model.Hudson@715617bb owned by: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
      Total blocked: 2,045  Total waited: 12,350
      
      Stack trace: 
      jenkins.model.Jenkins.save(Jenkins.java:2675)
      hudson.model.View.save(View.java:341)
      hudson.model.View.doConfigSubmit(View.java:967)
         - locked hudson.model.ListView@7e01a6a2
      sun.reflect.GeneratedMethodAccessor1966.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:483)
      org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
      org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
      org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
      org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
      org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
      org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
      org.kohsuke.stapler.Stapler.service(Stapler.java:225)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
      hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
      
      

        Attachments

          Activity

          Hide
          dspartz Daryl Spartz added a comment -

          I've submitted pull request with fix for this issue: https://github.com/jenkinsci/jenkins/pull/1488
          See comments there on how to reproduce. I've reproduced on 1.593 and LTS we are using 1.565.3

          Show
          dspartz Daryl Spartz added a comment - I've submitted pull request with fix for this issue: https://github.com/jenkinsci/jenkins/pull/1488 See comments there on how to reproduce. I've reproduced on 1.593 and LTS we are using 1.565.3
          Hide
          tonyyang Tony Yang added a comment - - edited

          Hi Daniel,

          We reproduced this in the latest weekly release Jenkins ver. 1.590

          Here is the thread deadlock stack trace we captured through jConsole for the pair of threads that got into deadlock

          Name: Handling POST /jenkins/view/deadlock1/configSubmit from 10.72.120.119 : qtp1955915048-52
          State: BLOCKED on hudson.model.Hudson@7e3b42ad owned by: Handling POST /jenkins/job/aaa_test2/doRename from 10.73.211.174 : qtp1955915048-45
          Total blocked: 1  Total waited: 14
          
          Stack trace: 
          jenkins.model.Jenkins.save(Jenkins.java:2756)
          hudson.model.View.save(View.java:333)
          hudson.model.View.doConfigSubmit(View.java:959)
             - locked hudson.model.ListView@1a6186c3
          sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          java.lang.reflect.Method.invoke(Method.java:483)
          org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
          org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
          org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
          hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
          net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
          net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
          org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          
          
          
          ============================================================================================
          
          Name: Handling POST /jenkins/job/aaa_test2/doRename from 10.73.211.174 : qtp1955915048-45
          State: BLOCKED on hudson.model.ListView@1a6186c3 owned by: Handling POST /jenkins/view/deadlock1/configSubmit from 10.72.120.119 : qtp1955915048-52
          Total blocked: 9  Total waited: 14
          
          Stack trace: 
          hudson.model.ListView$Listener.renameViewItem(ListView.java:464)
          hudson.model.ListView$Listener.locationChanged(ListView.java:446)
          hudson.model.ListView$Listener.access$000(ListView.java:434)
          hudson.model.ListView$Listener$1.run(ListView.java:438)
          hudson.security.ACL.impersonate(ACL.java:145)
          hudson.model.ListView$Listener.onLocationChanged(ListView.java:436)
          hudson.model.listeners.ItemListener$6.apply(ItemListener.java:237)
          hudson.model.listeners.ItemListener$6.apply(ItemListener.java:235)
          hudson.model.listeners.ItemListener.forAll(ItemListener.java:166)
          hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:235)
          hudson.model.AbstractItem.renameTo(AbstractItem.java:327)
             - locked hudson.model.FreeStyleProject@1ca6e63a
             - locked hudson.model.Hudson@7e3b42ad
          hudson.model.Job.renameTo(Job.java:623)
          hudson.model.Job.doDoRename(Job.java:1434)
          sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          java.lang.reflect.Method.invoke(Method.java:483)
          org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
          org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
          org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
          org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
          org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
          org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
          org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
          org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
          hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
          net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
          net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
          org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
          hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
          hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
          org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          
          Show
          tonyyang Tony Yang added a comment - - edited Hi Daniel, We reproduced this in the latest weekly release Jenkins ver. 1.590 Here is the thread deadlock stack trace we captured through jConsole for the pair of threads that got into deadlock Name: Handling POST /jenkins/view/deadlock1/configSubmit from 10.72.120.119 : qtp1955915048-52 State: BLOCKED on hudson.model.Hudson@7e3b42ad owned by: Handling POST /jenkins/job/aaa_test2/doRename from 10.73.211.174 : qtp1955915048-45 Total blocked: 1 Total waited: 14 Stack trace: jenkins.model.Jenkins.save(Jenkins.java:2756) hudson.model.View.save(View.java:333) hudson.model.View.doConfigSubmit(View.java:959) - locked hudson.model.ListView@1a6186c3 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:483) org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46) org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399) org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) org.kohsuke.stapler.Stapler.service(Stapler.java:238) javax.servlet.http.HttpServlet.service(HttpServlet.java:790) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) ============================================================================================ Name: Handling POST /jenkins/job/aaa_test2/doRename from 10.73.211.174 : qtp1955915048-45 State: BLOCKED on hudson.model.ListView@1a6186c3 owned by: Handling POST /jenkins/view/deadlock1/configSubmit from 10.72.120.119 : qtp1955915048-52 Total blocked: 9 Total waited: 14 Stack trace: hudson.model.ListView$Listener.renameViewItem(ListView.java:464) hudson.model.ListView$Listener.locationChanged(ListView.java:446) hudson.model.ListView$Listener.access$000(ListView.java:434) hudson.model.ListView$Listener$1.run(ListView.java:438) hudson.security.ACL.impersonate(ACL.java:145) hudson.model.ListView$Listener.onLocationChanged(ListView.java:436) hudson.model.listeners.ItemListener$6.apply(ItemListener.java:237) hudson.model.listeners.ItemListener$6.apply(ItemListener.java:235) hudson.model.listeners.ItemListener.forAll(ItemListener.java:166) hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:235) hudson.model.AbstractItem.renameTo(AbstractItem.java:327) - locked hudson.model.FreeStyleProject@1ca6e63a - locked hudson.model.Hudson@7e3b42ad hudson.model.Job.renameTo(Job.java:623) hudson.model.Job.doDoRename(Job.java:1434) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:483) org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46) org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399) org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) org.kohsuke.stapler.Stapler.service(Stapler.java:238) javax.servlet.http.HttpServlet.service(HttpServlet.java:790) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114) hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
          Hide
          danielbeck Daniel Beck added a comment -

          What are the stack traces in the newer versions? Do they also involve the API access ("Api.doJson")?

          Show
          danielbeck Daniel Beck added a comment - What are the stack traces in the newer versions? Do they also involve the API access ("Api.doJson")?
          Hide
          mujib Mujib Wahab added a comment -

          we are able to reproduce this with latest LTS 1.580.1 as well. Will test with latest release 1.590 and update the result.

          Show
          mujib Mujib Wahab added a comment - we are able to reproduce this with latest LTS 1.580.1 as well. Will test with latest release 1.590 and update the result.
          Hide
          danielbeck Daniel Beck added a comment -

          This deadlock involves the JSON API as well.

          Could you please try reproduce this issue on 1.580.x or the current weekly release?

          Show
          danielbeck Daniel Beck added a comment - This deadlock involves the JSON API as well. Could you please try reproduce this issue on 1.580.x or the current weekly release?

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            bitingsparrow Manav Chimakurthi
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: