• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • RedHat 5.6 java 1.6. jenkins 1.522

      While parallel deletion/renaming of jobs we have following deadlock:

      Found one Java-level deadlock:
      =============================
      "Handling POST /jenkins/view/QA/view/Verification/createItem : http-bio-8080-exec-189":
        waiting to lock monitor 0x000000004eba7ca8 (object 0x000000070032c4c8, a hudson.model.Hudson),
        which is held by "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140"
      "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140":
        waiting to lock monitor 0x000000004f70e1e0 (object 0x0000000752e08ea0, a hudson.maven.MavenModuleSet),
        which is held by "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166"
      "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166":
        waiting to lock monitor 0x000000004eba7ca8 (object 0x000000070032c4c8, a hudson.model.Hudson),
        which is held by "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140"
      

      Dump of threads:

      "Handling POST /jenkins/view/QA/view/Verification/createItem : http-bio-8080-exec-189":
              at jenkins.model.Jenkins.doCreateItem(Jenkins.java:2889)
              - waiting to lock <0x000000070032c4c8> (a hudson.model.Hudson)
              at jenkins.model.Jenkins.doCreateItem(Jenkins.java:312)
              at hudson.model.ListView.doCreateItem(ListView.java:262)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:217)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
              at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
              at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
      
      "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140":
              at hudson.model.AbstractProject.save(AbstractProject.java:278)
              - waiting to lock <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.plugins.copyartifact.CopyArtifact$ListenerImpl.onRenamed(CopyArtifact.java:331)
              at hudson.model.AbstractItem.renameTo(AbstractItem.java:294)
              - locked <0x0000000709ca3ef8> (a hudson.model.FreeStyleProject)
              - locked <0x000000070032c4c8> (a hudson.model.Hudson)
              at hudson.model.Job.renameTo(Job.java:575)
              at hudson.model.Job.doDoRename(Job.java:1294)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:29)
              at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:398)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:217)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
      
      "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166":
              at jenkins.model.Jenkins.save(Jenkins.java:2635)
              - waiting to lock <0x000000070032c4c8> (a hudson.model.Hudson)
              at jenkins.model.Jenkins.onDeleted(Jenkins.java:2435)
              at jenkins.model.Jenkins.onDeleted(Jenkins.java:312)
              at hudson.model.AbstractItem.invokeOnDeleted(AbstractItem.java:524)
              at hudson.model.AbstractItem.delete(AbstractItem.java:511)
              - locked <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.model.Job.delete(Job.java:585)
              - locked <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.model.AbstractProject.doDoDelete(AbstractProject.java:1915)
              at sun.reflect.GeneratedMethodAccessor1652.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:29)
              at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:398)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
      

          [JENKINS-19446] Deadlock while parallel deletion/rename of jobs

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/src/main/java/hudson/model/AbstractItem.java
          http://jenkins-ci.org/commit/jenkins/a5755cb3e1f901c98a2263a3ae1851489cb8e47b
          Log:
          [FIXED JENKINS-19446]

          Defer the notification to avoid deadlock.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/model/AbstractItem.java http://jenkins-ci.org/commit/jenkins/a5755cb3e1f901c98a2263a3ae1851489cb8e47b Log: [FIXED JENKINS-19446] Defer the notification to avoid deadlock.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3085
          [FIXED JENKINS-19446] (Revision a5755cb3e1f901c98a2263a3ae1851489cb8e47b)

          Result = SUCCESS
          kohsuke : a5755cb3e1f901c98a2263a3ae1851489cb8e47b
          Files :

          • core/src/main/java/hudson/model/AbstractItem.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3085 [FIXED JENKINS-19446] (Revision a5755cb3e1f901c98a2263a3ae1851489cb8e47b) Result = SUCCESS kohsuke : a5755cb3e1f901c98a2263a3ae1851489cb8e47b Files : core/src/main/java/hudson/model/AbstractItem.java changelog.html

          Jesse Glick added a comment -

          The current fix is just no good. It delivers events asynchronously, which makes the system unpredictable, and ruins functional tests.

          Better IMHO would be to call just save() asynchronously.

          Jesse Glick added a comment - The current fix is just no good. It delivers events asynchronously, which makes the system unpredictable, and ruins functional tests. Better IMHO would be to call just save() asynchronously.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2
          Log:
          JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html http://jenkins-ci.org/commit/jenkins/f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2 Log: JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3199
          JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543. (Revision f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2)

          Result = UNSTABLE
          Jesse Glick : f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3199 JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543. (Revision f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2) Result = UNSTABLE Jesse Glick : f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2 Files : changelog.html

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/model/AbstractItem.java
          http://jenkins-ci.org/commit/jenkins/41baac80fbf559aa3eb30c635abe0386815fd67f
          Log:
          [FIXED JENKINS-19446]

          Defer the notification to avoid deadlock.

          (cherry picked from commit a5755cb3e1f901c98a2263a3ae1851489cb8e47b)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractItem.java http://jenkins-ci.org/commit/jenkins/41baac80fbf559aa3eb30c635abe0386815fd67f Log: [FIXED JENKINS-19446] Defer the notification to avoid deadlock. (cherry picked from commit a5755cb3e1f901c98a2263a3ae1851489cb8e47b) Conflicts: changelog.html

          Jesse Glick added a comment -

          Filed JENKINS-22001 for the problematic fix.

          Jesse Glick added a comment - Filed JENKINS-22001 for the problematic fix.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/model/ListView.java
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/3aa0dd5368444728b608bce9790e59983dabbc8a
          Log:
          JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill.
          This is already handled by ListView.Listener (other kinds of views are now responsible for their own listeners).
          And handled better—if there are no actual changes
          (because you have no such views, or they do not mention this job), there is no need to save.
          Also moving the actual saving in those remaining cases out of the lock on the ListView itself, just in case.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/ListView.java core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/3aa0dd5368444728b608bce9790e59983dabbc8a Log: JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. This is already handled by ListView.Listener (other kinds of views are now responsible for their own listeners). And handled better—if there are no actual changes (because you have no such views, or they do not mention this job), there is no need to save. Also moving the actual saving in those remaining cases out of the lock on the ListView itself, just in case.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/model/AbstractItem.java
          core/src/main/java/hudson/model/Job.java
          test/src/test/java/hudson/model/ListViewTest.java
          http://jenkins-ci.org/commit/jenkins/00d27176f3a452e38bd0caafebe78c7bc2fa822f
          Log:
          [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior.

          Compare: https://github.com/jenkinsci/jenkins/compare/334ce1be691c...00d27176f3a4

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/Job.java test/src/test/java/hudson/model/ListViewTest.java http://jenkins-ci.org/commit/jenkins/00d27176f3a452e38bd0caafebe78c7bc2fa822f Log: [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. Compare: https://github.com/jenkinsci/jenkins/compare/334ce1be691c...00d27176f3a4

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3204
          JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. (Revision 3aa0dd5368444728b608bce9790e59983dabbc8a)
          [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. (Revision 00d27176f3a452e38bd0caafebe78c7bc2fa822f)

          Result = SUCCESS
          Jesse Glick : 3aa0dd5368444728b608bce9790e59983dabbc8a
          Files :

          • core/src/main/java/jenkins/model/Jenkins.java
          • core/src/main/java/hudson/model/ListView.java

          Jesse Glick : 00d27176f3a452e38bd0caafebe78c7bc2fa822f
          Files :

          • test/src/test/java/hudson/model/ListViewTest.java
          • core/src/main/java/hudson/model/AbstractItem.java
          • core/src/main/java/hudson/model/Job.java

          dogfood added a comment - Integrated in jenkins_main_trunk #3204 JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. (Revision 3aa0dd5368444728b608bce9790e59983dabbc8a) [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. (Revision 00d27176f3a452e38bd0caafebe78c7bc2fa822f) Result = SUCCESS Jesse Glick : 3aa0dd5368444728b608bce9790e59983dabbc8a Files : core/src/main/java/jenkins/model/Jenkins.java core/src/main/java/hudson/model/ListView.java Jesse Glick : 00d27176f3a452e38bd0caafebe78c7bc2fa822f Files : test/src/test/java/hudson/model/ListViewTest.java core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/Job.java

            Unassigned Unassigned
            joesephz Joe Zeldin
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: