Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19446

Deadlock while parallel deletion/rename of jobs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      RedHat 5.6 java 1.6. jenkins 1.522
    • Similar Issues:

      Description

      While parallel deletion/renaming of jobs we have following deadlock:

      Found one Java-level deadlock:
      =============================
      "Handling POST /jenkins/view/QA/view/Verification/createItem : http-bio-8080-exec-189":
        waiting to lock monitor 0x000000004eba7ca8 (object 0x000000070032c4c8, a hudson.model.Hudson),
        which is held by "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140"
      "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140":
        waiting to lock monitor 0x000000004f70e1e0 (object 0x0000000752e08ea0, a hudson.maven.MavenModuleSet),
        which is held by "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166"
      "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166":
        waiting to lock monitor 0x000000004eba7ca8 (object 0x000000070032c4c8, a hudson.model.Hudson),
        which is held by "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140"
      

      Dump of threads:

      "Handling POST /jenkins/view/QA/view/Verification/createItem : http-bio-8080-exec-189":
              at jenkins.model.Jenkins.doCreateItem(Jenkins.java:2889)
              - waiting to lock <0x000000070032c4c8> (a hudson.model.Hudson)
              at jenkins.model.Jenkins.doCreateItem(Jenkins.java:312)
              at hudson.model.ListView.doCreateItem(ListView.java:262)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:217)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
              at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:208)
              at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:181)
              at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86)
              at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84)
      
      "Handling POST /jenkins/view/RE/view/Onsite/view/Dev/job/RE-Onsite-Build-ATG-Stg/doRename : http-bio-8080-exec-140":
              at hudson.model.AbstractProject.save(AbstractProject.java:278)
              - waiting to lock <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.plugins.copyartifact.CopyArtifact$ListenerImpl.onRenamed(CopyArtifact.java:331)
              at hudson.model.AbstractItem.renameTo(AbstractItem.java:294)
              - locked <0x0000000709ca3ef8> (a hudson.model.FreeStyleProject)
              - locked <0x000000070032c4c8> (a hudson.model.Hudson)
              at hudson.model.Job.renameTo(Job.java:575)
              at hudson.model.Job.doDoRename(Job.java:1294)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:29)
              at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:398)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
              at org.kohsuke.stapler.Stapler.service(Stapler.java:217)
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
      
      "Handling POST /jenkins/view/Maintenance/job/__tst1__/doDelete : http-bio-8080-exec-166":
              at jenkins.model.Jenkins.save(Jenkins.java:2635)
              - waiting to lock <0x000000070032c4c8> (a hudson.model.Hudson)
              at jenkins.model.Jenkins.onDeleted(Jenkins.java:2435)
              at jenkins.model.Jenkins.onDeleted(Jenkins.java:312)
              at hudson.model.AbstractItem.invokeOnDeleted(AbstractItem.java:524)
              at hudson.model.AbstractItem.delete(AbstractItem.java:511)
              - locked <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.model.Job.delete(Job.java:585)
              - locked <0x0000000752e08ea0> (a hudson.maven.MavenModuleSet)
              at hudson.model.AbstractProject.doDoDelete(AbstractProject.java:1915)
              at sun.reflect.GeneratedMethodAccessor1652.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:297)
              at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:29)
              at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:398)
              at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:160)
              at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:95)
              at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239)
              at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
              at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:684)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:777)
              at org.kohsuke.stapler.Stapler.invoke(Stapler.java:586)
      

        Attachments

          Issue Links

            Activity

            joesephz Joe Zeldin created issue -
            Hide
            akirillov Artem Kirillov added a comment -

            Pull request containing fix: https://github.com/jenkinsci/jenkins/pull/932

            Show
            akirillov Artem Kirillov added a comment - Pull request containing fix: https://github.com/jenkinsci/jenkins/pull/932
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/hudson/model/AbstractItem.java
            http://jenkins-ci.org/commit/jenkins/a5755cb3e1f901c98a2263a3ae1851489cb8e47b
            Log:
            [FIXED JENKINS-19446]

            Defer the notification to avoid deadlock.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/model/AbstractItem.java http://jenkins-ci.org/commit/jenkins/a5755cb3e1f901c98a2263a3ae1851489cb8e47b Log: [FIXED JENKINS-19446] Defer the notification to avoid deadlock.
            scm_issue_link SCM/JIRA link daemon made changes -
            Field Original Value New Value
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3085
            [FIXED JENKINS-19446] (Revision a5755cb3e1f901c98a2263a3ae1851489cb8e47b)

            Result = SUCCESS
            kohsuke : a5755cb3e1f901c98a2263a3ae1851489cb8e47b
            Files :

            • core/src/main/java/hudson/model/AbstractItem.java
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3085 [FIXED JENKINS-19446] (Revision a5755cb3e1f901c98a2263a3ae1851489cb8e47b) Result = SUCCESS kohsuke : a5755cb3e1f901c98a2263a3ae1851489cb8e47b Files : core/src/main/java/hudson/model/AbstractItem.java changelog.html
            ndeloof Nicolas De Loof made changes -
            Labels lts-candidate
            Hide
            jglick Jesse Glick added a comment -

            The current fix is just no good. It delivers events asynchronously, which makes the system unpredictable, and ruins functional tests.

            Better IMHO would be to call just save() asynchronously.

            Show
            jglick Jesse Glick added a comment - The current fix is just no good. It delivers events asynchronously, which makes the system unpredictable, and ruins functional tests. Better IMHO would be to call just save() asynchronously.
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2
            Log:
            JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html http://jenkins-ci.org/commit/jenkins/f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2 Log: JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3199
            JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543. (Revision f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2)

            Result = UNSTABLE
            Jesse Glick : f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2
            Files :

            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3199 JENKINS-19446 Amending changelog; this actually went into 1.544, not 1.543. (Revision f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2) Result = UNSTABLE Jesse Glick : f22cfbf31b733d8eb64ecdfc7444f7e61ad155b2 Files : changelog.html
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/src/main/java/hudson/model/AbstractItem.java
            http://jenkins-ci.org/commit/jenkins/41baac80fbf559aa3eb30c635abe0386815fd67f
            Log:
            [FIXED JENKINS-19446]

            Defer the notification to avoid deadlock.

            (cherry picked from commit a5755cb3e1f901c98a2263a3ae1851489cb8e47b)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/model/AbstractItem.java http://jenkins-ci.org/commit/jenkins/41baac80fbf559aa3eb30c635abe0386815fd67f Log: [FIXED JENKINS-19446] Defer the notification to avoid deadlock. (cherry picked from commit a5755cb3e1f901c98a2263a3ae1851489cb8e47b) Conflicts: changelog.html
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-22001 [ JENKINS-22001 ]
            Hide
            jglick Jesse Glick added a comment -

            Filed JENKINS-22001 for the problematic fix.

            Show
            jglick Jesse Glick added a comment - Filed JENKINS-22001 for the problematic fix.
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate 1.532.3-fixed
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/ListView.java
            core/src/main/java/jenkins/model/Jenkins.java
            http://jenkins-ci.org/commit/jenkins/3aa0dd5368444728b608bce9790e59983dabbc8a
            Log:
            JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill.
            This is already handled by ListView.Listener (other kinds of views are now responsible for their own listeners).
            And handled better—if there are no actual changes
            (because you have no such views, or they do not mention this job), there is no need to save.
            Also moving the actual saving in those remaining cases out of the lock on the ListView itself, just in case.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/ListView.java core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/3aa0dd5368444728b608bce9790e59983dabbc8a Log: JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. This is already handled by ListView.Listener (other kinds of views are now responsible for their own listeners). And handled better—if there are no actual changes (because you have no such views, or they do not mention this job), there is no need to save. Also moving the actual saving in those remaining cases out of the lock on the ListView itself, just in case.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/AbstractItem.java
            core/src/main/java/hudson/model/Job.java
            test/src/test/java/hudson/model/ListViewTest.java
            http://jenkins-ci.org/commit/jenkins/00d27176f3a452e38bd0caafebe78c7bc2fa822f
            Log:
            [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior.

            Compare: https://github.com/jenkinsci/jenkins/compare/334ce1be691c...00d27176f3a4

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/Job.java test/src/test/java/hudson/model/ListViewTest.java http://jenkins-ci.org/commit/jenkins/00d27176f3a452e38bd0caafebe78c7bc2fa822f Log: [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. Compare: https://github.com/jenkinsci/jenkins/compare/334ce1be691c...00d27176f3a4
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3204
            JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. (Revision 3aa0dd5368444728b608bce9790e59983dabbc8a)
            [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. (Revision 00d27176f3a452e38bd0caafebe78c7bc2fa822f)

            Result = SUCCESS
            Jesse Glick : 3aa0dd5368444728b608bce9790e59983dabbc8a
            Files :

            • core/src/main/java/jenkins/model/Jenkins.java
            • core/src/main/java/hudson/model/ListView.java

            Jesse Glick : 00d27176f3a452e38bd0caafebe78c7bc2fa822f
            Files :

            • test/src/test/java/hudson/model/ListViewTest.java
            • core/src/main/java/hudson/model/AbstractItem.java
            • core/src/main/java/hudson/model/Job.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3204 JENKINS-19446 Saving Jenkins configuration in onRenamed/onDeleted is often overkill. (Revision 3aa0dd5368444728b608bce9790e59983dabbc8a) [FIXED JENKINS-22001] Simpler fix of JENKINS-19446 that does not introduce asynchronous behavior. (Revision 00d27176f3a452e38bd0caafebe78c7bc2fa822f) Result = SUCCESS Jesse Glick : 3aa0dd5368444728b608bce9790e59983dabbc8a Files : core/src/main/java/jenkins/model/Jenkins.java core/src/main/java/hudson/model/ListView.java Jesse Glick : 00d27176f3a452e38bd0caafebe78c7bc2fa822f Files : test/src/test/java/hudson/model/ListViewTest.java core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/Job.java
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-18071 [ JENKINS-18071 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 150895 ] JNJira + In-Review [ 193725 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              joesephz Joe Zeldin
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: