Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29936

deadlock between OldDataMonitor and AuthorizationStrategy.

    XMLWordPrintable

Details

    Description

      A deadlock has been observed between The OldDataMonitor and an AuthorizationStrategy.

      When a job has a new build it will call save which will trigger the OldDataMonitor via the SaveableListener.
      This will call referTo which in the case of a Run will lookup the parent Job via it's full name. This will perform a permission check .

      Now if the authorization strategy is also being saved at the same time then it is highly likely that a deadlock will occur.

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick added a comment -

            No thread dump for reference?

            jglick Jesse Glick added a comment - No thread dump for reference?
            teilo James Nord added a comment -

            jglick unfortunately I only have a trimmed stack trace that doesn't fully show this. The rest has been identified with knowing what's been occuring at this time and inspecting the code.

            "Handling GET /job/myJob/groups/ from 192.168.13.182 : http-bio-8080-exec-81 GroupContainerMixIn/index.jelly" Id=149 BLOCKED on hudson.diagnosis.OldDataMonitor@145b2dc1 owned by "Computer.threadPoolForRemoting [#1622] for Channel to Maven [/usr/java/jdk1.7.0_05/bin/java, -Xmx1024m, -XX:MaxPermSize=256m, -cp, /data/jenkins_slave/epjenkins81/maven3-agent.jar:/opt/apache-maven/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar, org.jvnet.hudson.maven3.agent.Maven3Main, /opt/apache-maven/apache-maven-3.0.4, /data/jenkins_slave/epjenkins81/slave.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor-commons.jar, 37246]" Id=279517
                at hudson.diagnosis.OldDataMonitor.remove(OldDataMonitor.java:106)
                -  blocked on hudson.diagnosis.OldDataMonitor@145b2dc1
                at hudson.diagnosis.OldDataMonitor.access$000(OldDataMonitor.java:67)
                at hudson.diagnosis.OldDataMonitor$1.onChange(OldDataMonitor.java:120)
                at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:80)
                at hudson.model.User.save(User.java:708)
                at hudson.model.User.addProperty(User.java:273)
                at hudson.plugins.active_directory.ActiveDirectoryUserDetail.updateUserInfo(ActiveDirectoryUserDetail.java:163)
                at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:321)
                ...
            
            
            "Computer.threadPoolForRemoting [#1622] for Channel to Maven [/usr/java/jdk1.7.0_05/bin/java, -Xmx1024m, -XX:MaxPermSize=256m, -cp, /data/jenkins_slave/epjenkins81/maven3-agent.jar:/opt/apache-maven/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar, org.jvnet.hudson.maven3.agent.Maven3Main, /opt/apache-maven/apache-maven-3.0.4, /data/jenkins_slave/epjenkins81/slave.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor-commons.jar, 37246]" Id=279517 BLOCKED on nectar.plugins.rbac.groups.Group@2f247cf9 owned by "Handling GET /job/myJob/groups/ from 192.168.13.182 : http-bio-8080-exec-81 GroupContainerMixIn/index.jelly" Id=149
                at nectar.plugins.rbac.groups.Group.isMatch(Group.java:376)
                -  blocked on nectar.plugins.rbac.groups.Group@2f247cf9
                at nectar.plugins.rbac.groups.Group.isMatch(Group.java:349)
                at nectar.plugins.rbac.groups.Group.isMatch(Group.java:309)
                at nectar.plugins.rbac.groups.GroupContainerACL.hasPermission(GroupContainerACL.java:127)
                at nectar.plugins.rbac.groups.GroupContainerACL._hasPermission(GroupContainerACL.java:94)
                at nectar.plugins.rbac.groups.GroupContainerACL.hasPermission(GroupContainerACL.java:65)
                at hudson.security.ACL.hasPermission(ACL.java:73)
                at hudson.model.AbstractItem.hasPermission(AbstractItem.java:505)
                ...
            
            teilo James Nord added a comment - jglick unfortunately I only have a trimmed stack trace that doesn't fully show this. The rest has been identified with knowing what's been occuring at this time and inspecting the code. "Handling GET /job/myJob/groups/ from 192.168.13.182 : http-bio-8080-exec-81 GroupContainerMixIn/index.jelly" Id=149 BLOCKED on hudson.diagnosis.OldDataMonitor@145b2dc1 owned by "Computer.threadPoolForRemoting [#1622] for Channel to Maven [/usr/java/jdk1.7.0_05/bin/java, -Xmx1024m, -XX:MaxPermSize=256m, -cp, /data/jenkins_slave/epjenkins81/maven3-agent.jar:/opt/apache-maven/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar, org.jvnet.hudson.maven3.agent.Maven3Main, /opt/apache-maven/apache-maven-3.0.4, /data/jenkins_slave/epjenkins81/slave.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor-commons.jar, 37246]" Id=279517 at hudson.diagnosis.OldDataMonitor.remove(OldDataMonitor.java:106) - blocked on hudson.diagnosis.OldDataMonitor@145b2dc1 at hudson.diagnosis.OldDataMonitor.access$000(OldDataMonitor.java:67) at hudson.diagnosis.OldDataMonitor$1.onChange(OldDataMonitor.java:120) at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:80) at hudson.model.User.save(User.java:708) at hudson.model.User.addProperty(User.java:273) at hudson.plugins.active_directory.ActiveDirectoryUserDetail.updateUserInfo(ActiveDirectoryUserDetail.java:163) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:321) ... "Computer.threadPoolForRemoting [#1622] for Channel to Maven [/usr/java/jdk1.7.0_05/bin/java, -Xmx1024m, -XX:MaxPermSize=256m, -cp, /data/jenkins_slave/epjenkins81/maven3-agent.jar:/opt/apache-maven/apache-maven-3.0.4/boot/plexus-classworlds-2.4.jar, org.jvnet.hudson.maven3.agent.Maven3Main, /opt/apache-maven/apache-maven-3.0.4, /data/jenkins_slave/epjenkins81/slave.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor.jar, /data/jenkins_slave/epjenkins81/maven3-interceptor-commons.jar, 37246]" Id=279517 BLOCKED on nectar.plugins.rbac.groups.Group@2f247cf9 owned by "Handling GET /job/myJob/groups/ from 192.168.13.182 : http-bio-8080-exec-81 GroupContainerMixIn/index.jelly" Id=149 at nectar.plugins.rbac.groups.Group.isMatch(Group.java:376) - blocked on nectar.plugins.rbac.groups.Group@2f247cf9 at nectar.plugins.rbac.groups.Group.isMatch(Group.java:349) at nectar.plugins.rbac.groups.Group.isMatch(Group.java:309) at nectar.plugins.rbac.groups.GroupContainerACL.hasPermission(GroupContainerACL.java:127) at nectar.plugins.rbac.groups.GroupContainerACL._hasPermission(GroupContainerACL.java:94) at nectar.plugins.rbac.groups.GroupContainerACL.hasPermission(GroupContainerACL.java:65) at hudson.security.ACL.hasPermission(ACL.java:73) at hudson.model.AbstractItem.hasPermission(AbstractItem.java:505) ...

            Code changed in jenkins
            User: James Nord
            Path:
            core/src/main/java/hudson/diagnosis/OldDataMonitor.java
            http://jenkins-ci.org/commit/jenkins/8a077a801960aa74da455441cfb12d300c6d6e3a
            Log:
            JENKINS-29936 when removing an item use ACL.SYTEM.

            The OldDataMonitor should be using ACL.system not the ACL of the calling
            thread - this also avoids the deadlock when an authorization strategy is
            being saved (locking the auth strategy) which will call into the ODM at
            the same point the ODM is being called an a Run has been saved (which will
            cause a lookup of the job which will do a permissions check).

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: core/src/main/java/hudson/diagnosis/OldDataMonitor.java http://jenkins-ci.org/commit/jenkins/8a077a801960aa74da455441cfb12d300c6d6e3a Log: JENKINS-29936 when removing an item use ACL.SYTEM. The OldDataMonitor should be using ACL.system not the ACL of the calling thread - this also avoids the deadlock when an authorization strategy is being saved (locking the auth strategy) which will call into the ODM at the same point the ODM is being called an a Run has been saved (which will cause a lookup of the job which will do a permissions check).

            Code changed in jenkins
            User: Daniel Beck
            Path:
            core/src/main/java/hudson/diagnosis/OldDataMonitor.java
            http://jenkins-ci.org/commit/jenkins/c1b60f18b548852428b835ecba72a3467d726ba0
            Log:
            Merge pull request #1796 from jtnord/jenkins-29936

            JENKINS-29936 when removing an item use ACL.SYSTEM.

            Compare: https://github.com/jenkinsci/jenkins/compare/6bfbe0c79d9f...c1b60f18b548

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: core/src/main/java/hudson/diagnosis/OldDataMonitor.java http://jenkins-ci.org/commit/jenkins/c1b60f18b548852428b835ecba72a3467d726ba0 Log: Merge pull request #1796 from jtnord/jenkins-29936 JENKINS-29936 when removing an item use ACL.SYSTEM. Compare: https://github.com/jenkinsci/jenkins/compare/6bfbe0c79d9f...c1b60f18b548
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #4249
            JENKINS-29936 when removing an item use ACL.SYTEM. (Revision 8a077a801960aa74da455441cfb12d300c6d6e3a)

            Result = SUCCESS
            james nord : 8a077a801960aa74da455441cfb12d300c6d6e3a
            Files :

            • core/src/main/java/hudson/diagnosis/OldDataMonitor.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #4249 JENKINS-29936 when removing an item use ACL.SYTEM. (Revision 8a077a801960aa74da455441cfb12d300c6d6e3a) Result = SUCCESS james nord : 8a077a801960aa74da455441cfb12d300c6d6e3a Files : core/src/main/java/hudson/diagnosis/OldDataMonitor.java

            Code changed in jenkins
            User: James Nord
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/e0849bdaf84457246708c10e6cd8a5a10fa35725
            Log:
            Noting JENKINS-29936

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: changelog.html http://jenkins-ci.org/commit/jenkins/e0849bdaf84457246708c10e6cd8a5a10fa35725 Log: Noting JENKINS-29936

            Code changed in jenkins
            User: James Nord
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/68c1e351f8dc7f63973ff1fa0291b2907cd93f3f
            Log:
            Merge pull request #1799 from jtnord/changelog-JENKINS-29936

            Noting JENKINS-29936

            Compare: https://github.com/jenkinsci/jenkins/compare/7add0d1646b2...68c1e351f8dc

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: changelog.html http://jenkins-ci.org/commit/jenkins/68c1e351f8dc7f63973ff1fa0291b2907cd93f3f Log: Merge pull request #1799 from jtnord/changelog- JENKINS-29936 Noting JENKINS-29936 Compare: https://github.com/jenkinsci/jenkins/compare/7add0d1646b2...68c1e351f8dc
            teilo James Nord added a comment -

            Fixed with PR 1796

            teilo James Nord added a comment - Fixed with PR 1796
            danielbeck Daniel Beck added a comment -

            Nominating for next LTS (not 1.609.3).

            danielbeck Daniel Beck added a comment - Nominating for next LTS (not 1.609.3).
            jglick Jesse Glick added a comment -

            IMO this is a serious enough bug to be considered for 1.609.3 despite the youth of the fix (CC olivergondza).

            jglick Jesse Glick added a comment - IMO this is a serious enough bug to be considered for 1.609.3 despite the youth of the fix (CC olivergondza ).
            danielbeck Daniel Beck added a comment -

            As there's still no RC, we could even still make it part of that. Discussion probably later today in the project meeting.

            danielbeck Daniel Beck added a comment - As there's still no RC, we could even still make it part of that. Discussion probably later today in the project meeting.
            danielbeck Daniel Beck added a comment -

            May still not be in the baseline for next LTS.

            danielbeck Daniel Beck added a comment - May still not be in the baseline for next LTS.

            Code changed in jenkins
            User: James Nord
            Path:
            core/src/main/java/hudson/diagnosis/OldDataMonitor.java
            http://jenkins-ci.org/commit/jenkins/9a63d6f8cb734d99597c12263f232fc49604eeb0
            Log:
            JENKINS-29936 when removing an item use ACL.SYTEM.

            The OldDataMonitor should be using ACL.system not the ACL of the calling
            thread - this also avoids the deadlock when an authorization strategy is
            being saved (locking the auth strategy) which will call into the ODM at
            the same point the ODM is being called an a Run has been saved (which will
            cause a lookup of the job which will do a permissions check).

            (cherry picked from commit 8a077a801960aa74da455441cfb12d300c6d6e3a)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: James Nord Path: core/src/main/java/hudson/diagnosis/OldDataMonitor.java http://jenkins-ci.org/commit/jenkins/9a63d6f8cb734d99597c12263f232fc49604eeb0 Log: JENKINS-29936 when removing an item use ACL.SYTEM. The OldDataMonitor should be using ACL.system not the ACL of the calling thread - this also avoids the deadlock when an authorization strategy is being saved (locking the auth strategy) which will call into the ODM at the same point the ODM is being called an a Run has been saved (which will cause a lookup of the job which will do a permissions check). (cherry picked from commit 8a077a801960aa74da455441cfb12d300c6d6e3a)
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #4292
            JENKINS-29936 when removing an item use ACL.SYTEM. (Revision 9a63d6f8cb734d99597c12263f232fc49604eeb0)

            Result = UNSTABLE
            ogondza : 9a63d6f8cb734d99597c12263f232fc49604eeb0
            Files :

            • core/src/main/java/hudson/diagnosis/OldDataMonitor.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #4292 JENKINS-29936 when removing an item use ACL.SYTEM. (Revision 9a63d6f8cb734d99597c12263f232fc49604eeb0) Result = UNSTABLE ogondza : 9a63d6f8cb734d99597c12263f232fc49604eeb0 Files : core/src/main/java/hudson/diagnosis/OldDataMonitor.java

            People

              Unassigned Unassigned
              teilo James Nord
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: