Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22830

Slow login w/ Active directory plugin

    XMLWordPrintable

Details

    Description

      Logging in on version 1.37 of the Active Directory plugin takes too long and I think it is causing occasional proxy timeouts. At first I had blamed it on having too many groups in Active Directory. Then I took a look at the release notes and saw that version 1.37 add some extra group specific logic. I decided to try downgrading the plugin and sure enough I was able to log in almost immediately. Which made me very very happy!

      Version 1.37 (I've seen this take up to a minute)

      Apr 30, 2014 7:38:17 PM FINE hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider
      Stage 2: looking up via memberOf
      Apr 30, 2014 7:38:44 PM FINE hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider

      Version 1.36

      Apr 30, 2014 7:53:34 PM FINE hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider
      Stage 2: looking up via memberOf
      Apr 30, 2014 7:53:34 PM FINER hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider

      Attachments

        Issue Links

          Activity

            mrose Michael Rose created issue -
            sijoey si Joey added a comment -

            I have the same issue. I am not even able to authenticate using the 1.37 of the plugin. I see timeout errors in the jenkins log, with LTS release 1.532.2. I am also seeing the same problem with LTS 1.554.1 of Jenkins. I had to use version 1.35 of the plugin to resolve the login issues.

            sijoey si Joey added a comment - I have the same issue. I am not even able to authenticate using the 1.37 of the plugin. I see timeout errors in the jenkins log, with LTS release 1.532.2. I am also seeing the same problem with LTS 1.554.1 of Jenkins. I had to use version 1.35 of the plugin to resolve the login issues.
            mrose Michael Rose added a comment -

            I didn't try version 1.35. We are using version 1.36 right now. Did you try this version? Does 1.35 perform better?

            mrose Michael Rose added a comment - I didn't try version 1.35. We are using version 1.36 right now. Did you try this version? Does 1.35 perform better?
            sijoey si Joey added a comment -

            I just tried 1.36 and it works fine. I also tried upgrading from 1.36 to 1.37 with no change in the error message and login problems. I get the following message: Caused by: javax.naming.TimeLimitExceededException: [LDAP: error code 3 - Timelimit Exceeded]; remaining name 'DC=xyx,DC=acme,DC=org'
            at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3143)
            at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033)
            at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840)
            at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:147)
            at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:216)
            at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:189)
            at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.parseMembers(ActiveDirectoryUnixAuthenticationProvider.java:456)
            at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.resolveGroups(ActiveDirectoryUnixAuthenticationProvider.java:417)
            at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:293)

            sijoey si Joey added a comment - I just tried 1.36 and it works fine. I also tried upgrading from 1.36 to 1.37 with no change in the error message and login problems. I get the following message: Caused by: javax.naming.TimeLimitExceededException: [LDAP: error code 3 - Timelimit Exceeded] ; remaining name 'DC=xyx,DC=acme,DC=org' at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3143) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840) at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:147) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:216) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:189) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.parseMembers(ActiveDirectoryUnixAuthenticationProvider.java:456) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.resolveGroups(ActiveDirectoryUnixAuthenticationProvider.java:417) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:293)

            I see the same issue here. Downgrading from 1.37 to 1.36 fixed my issue as well...
            (download it from here https://updates.jenkins-ci.org/download/plugins/active-directory/1.36/active-directory.hpi and upload it manually to your server)

            I turned on logging for the plugin and it seemed to be spending a lot of time getting the groups from active directory which led me to this issue.

            I was getting frustrated comments from several users of the system...so could we bump up the priority? Problems like this (20 seconds to login or load the web page the first time in awhile) tend to deter people from using otherwise-awesome tools.

            gavinagoodtime Gavin Goodrich added a comment - I see the same issue here. Downgrading from 1.37 to 1.36 fixed my issue as well... (download it from here https://updates.jenkins-ci.org/download/plugins/active-directory/1.36/active-directory.hpi and upload it manually to your server) I turned on logging for the plugin and it seemed to be spending a lot of time getting the groups from active directory which led me to this issue. I was getting frustrated comments from several users of the system...so could we bump up the priority? Problems like this (20 seconds to login or load the web page the first time in awhile) tend to deter people from using otherwise-awesome tools.
            george_li George Li added a comment -

            I would second bumping up the priority as I am experiencing the same errors mentioned above. I am using the same workaround by downgrading to 1.36.

            george_li George Li added a comment - I would second bumping up the priority as I am experiencing the same errors mentioned above. I am using the same workaround by downgrading to 1.36.
            lawrenceong Lawrence Ong added a comment -

            This bug basically killed our LDAP server by making a really bad query. I would bump this as well.

            lawrenceong Lawrence Ong added a comment - This bug basically killed our LDAP server by making a really bad query. I would bump this as well.
            lawrenceong Lawrence Ong made changes -
            Field Original Value New Value
            Priority Minor [ 4 ] Major [ 3 ]

            Can you also remove 1.37? It's already caused a big problem in our organization, and may well do so in others.

            jono_g Jonathan Gowland added a comment - Can you also remove 1.37? It's already caused a big problem in our organization, and may well do so in others.
            jglick Jesse Glick made changes -
            Labels performance regression

            Oh boy.

            The original change was introduced in response to a bug someone filed indicating that 1.36 didn't list all the groups they should be seeing.

            I've just spent good time googling the performance of this operator. There are some that reports this is faster than recursive lookup, and others that report this is slower, as in the case of this bug report.

            Overall, people do seem to favor this recursive search operator over other options. Does anyone know any AD person who can shed some lights on this?

            Fow now I'm going to introduce a switch that can be used to force one way or the other.

            kohsuke Kohsuke Kawaguchi added a comment - Oh boy. The original change was introduced in response to a bug someone filed indicating that 1.36 didn't list all the groups they should be seeing. I've just spent good time googling the performance of this operator. There are some that reports this is faster than recursive lookup , and others that report this is slower, as in the case of this bug report. Overall, people do seem to favor this recursive search operator over other options . Does anyone know any AD person who can shed some lights on this? Fow now I'm going to introduce a switch that can be used to force one way or the other.

            For my internal book-keeping, the original request for this change came from ZD-16865.

            kohsuke Kohsuke Kawaguchi added a comment - For my internal book-keeping, the original request for this change came from ZD-16865.
            danielbeck Daniel Beck added a comment -

            Kohsuke: Also see ZD-16562. It seems this just won't work in large directories.

            danielbeck Daniel Beck added a comment - Kohsuke: Also see ZD-16562. It seems this just won't work in large directories.

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java
            src/main/java/hudson/plugins/active_directory/ActiveDirectoryUnixAuthenticationProvider.java
            src/main/java/hudson/plugins/active_directory/GroupLookupStrategy.java
            src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/config.jelly
            src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/help-groupLookupStrategy.html
            src/main/resources/hudson/plugins/active_directory/Messages.properties
            http://jenkins-ci.org/commit/active-directory-plugin/ba12cf0afd57cc643d5237c8be04d5a7e44296ee
            Log:
            [FIXED JENKINS-22830]

            Providing the option for the user to pick the lesser of two evils. Performance vs Correctness.

            Looking forward to collect more information in the future to improve this situation, but in the mean tim we need to make this plugin usable.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java src/main/java/hudson/plugins/active_directory/ActiveDirectoryUnixAuthenticationProvider.java src/main/java/hudson/plugins/active_directory/GroupLookupStrategy.java src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/config.jelly src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/help-groupLookupStrategy.html src/main/resources/hudson/plugins/active_directory/Messages.properties http://jenkins-ci.org/commit/active-directory-plugin/ba12cf0afd57cc643d5237c8be04d5a7e44296ee Log: [FIXED JENKINS-22830] Providing the option for the user to pick the lesser of two evils. Performance vs Correctness. Looking forward to collect more information in the future to improve this situation, but in the mean tim we need to make this plugin usable.
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            I've released 1.38 that contains the fix to this problem.

            You can go to "Configure global security" and under the "advanced" tab, you can choose "Group Membership Lookup Strategy".

            You basically have two choices.

            • "Recursive queries", which is the pre-1.37 behavior. This involves issuing a number of simple LDAP queries to recursively find all the groups that you belong to. Reported by some people not to find all groups, and also reported by others to be slow (presumably due to the number of queries it has to issue)
            • "LDAP_MATCHING_RULE_IN_CHAIN", which is what 1.37 added. People who are hit by JENKINS-22830 is likely not want to use this, but otherwise the consensus in the developer community appears to be that this is the way to go. After all, why would Microsoft go out of its way to add a custom extension to LDAP if it's not supposed to be used!?

            The default is "automatic" and it tries to do the right thing the best it can.

            The main challenge for me to work on a problem like this is that I do not have direct access to any large AD deployment. Looking forward to your feedback on relative merits/problems so that we can improve the situation over time.

            kohsuke Kohsuke Kawaguchi added a comment - I've released 1.38 that contains the fix to this problem. You can go to "Configure global security" and under the "advanced" tab, you can choose "Group Membership Lookup Strategy". You basically have two choices. "Recursive queries", which is the pre-1.37 behavior. This involves issuing a number of simple LDAP queries to recursively find all the groups that you belong to. Reported by some people not to find all groups, and also reported by others to be slow (presumably due to the number of queries it has to issue) "LDAP_MATCHING_RULE_IN_CHAIN", which is what 1.37 added. People who are hit by JENKINS-22830 is likely not want to use this, but otherwise the consensus in the developer community appears to be that this is the way to go. After all, why would Microsoft go out of its way to add a custom extension to LDAP if it's not supposed to be used!? The default is "automatic" and it tries to do the right thing the best it can. The main challenge for me to work on a problem like this is that I do not have direct access to any large AD deployment. Looking forward to your feedback on relative merits/problems so that we can improve the situation over time.
            lawrenceong Lawrence Ong added a comment -

            The LDAP_MATCHING_RULE_IN_CHAIN mechanism can be very intensive on large servers. The reference to this on Microsoft's site is:

            http://msdn.microsoft.com/en-us/library/aa746475%28v=vs.85%29.aspx

            It mentioned this specifically:

            "Note that when using LDAP_MATCHING_RULE_IN_CHAIN, scope is not limited—it can be base, one-level, or subtree. Some such queries on subtrees may be more processor intensive, such as chasing links with a high fan-out; that is, listing all the groups that a user is a member of. Inefficient searches will log appropriate event log messages, as with any other type of query."

            From what I can see, the search via the chain lookup in the plugin is only using the subtree scope, so it is unlikely that any developer with access to a large AD deployment would want this at all as it can easily kill the AD if there are sufficient queries to accounts with large number of group memberships.

            lawrenceong Lawrence Ong added a comment - The LDAP_MATCHING_RULE_IN_CHAIN mechanism can be very intensive on large servers. The reference to this on Microsoft's site is: http://msdn.microsoft.com/en-us/library/aa746475%28v=vs.85%29.aspx It mentioned this specifically: "Note that when using LDAP_MATCHING_RULE_IN_CHAIN, scope is not limited—it can be base, one-level, or subtree. Some such queries on subtrees may be more processor intensive, such as chasing links with a high fan-out; that is, listing all the groups that a user is a member of. Inefficient searches will log appropriate event log messages, as with any other type of query." From what I can see, the search via the chain lookup in the plugin is only using the subtree scope, so it is unlikely that any developer with access to a large AD deployment would want this at all as it can easily kill the AD if there are sufficient queries to accounts with large number of group memberships.
            danielbeck Daniel Beck added a comment -

            queries to accounts with large number of group memberships.

            I tried this a while ago in an LDAP tool against AD global catalog. The request timed out for me with an account with only (recursively) two group memberships. One of them was "All Employees Worldwide" though, with 10k+ accounts in it. So it seems the support article doesn't have the full story.

            danielbeck Daniel Beck added a comment - queries to accounts with large number of group memberships. I tried this a while ago in an LDAP tool against AD global catalog. The request timed out for me with an account with only (recursively) two group memberships. One of them was "All Employees Worldwide" though, with 10k+ accounts in it. So it seems the support article doesn't have the full story.
            lawrenceong Lawrence Ong added a comment - - edited

            This is the timed out query for us:

            member<==>CN=myid,OU=People,OU=eProfile,DC=core,DC=dir,DC=domain,DC=com

            lawrenceong Lawrence Ong added a comment - - edited This is the timed out query for us: member<==>CN=myid,OU=People,OU=eProfile,DC=core,DC=dir,DC=domain,DC=com
            russ519 Russ Frisch added a comment -

            I have just installed version 1.38 of this extension on a new Jenkins instance and I'm running into the issue of AD authentication taking a very long time. I have tried to follow the instructions KohSuke provided here a few comments up (https://issues.jenkins-ci.org/browse/JENKINS-22830?focusedCommentId=203052&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-203052) but I'm only seeing the Domain Name and Domain Controller text fields under advanced. I'm not seeing the Group Membership Lookup Strategy. Can anyone else see this option in their install?

            russ519 Russ Frisch added a comment - I have just installed version 1.38 of this extension on a new Jenkins instance and I'm running into the issue of AD authentication taking a very long time. I have tried to follow the instructions KohSuke provided here a few comments up ( https://issues.jenkins-ci.org/browse/JENKINS-22830?focusedCommentId=203052&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-203052 ) but I'm only seeing the Domain Name and Domain Controller text fields under advanced. I'm not seeing the Group Membership Lookup Strategy. Can anyone else see this option in their install?
            danielbeck Daniel Beck added a comment -

            Russ: This option is only available if your Jenkins master doesn't run on Windows, or com4j isn't working.

            danielbeck Daniel Beck added a comment - Russ: This option is only available if your Jenkins master doesn't run on Windows, or com4j isn't working.
            russ519 Russ Frisch added a comment -

            Daniel,

            Thanks for the response. I looked at the commit and was able to determine that the value is stored in the config file as <groupLookupStrategy> and updated my config.xml to include <groupLookupStrategy>Recursive</groupLookupStrategy> which fixes my slow login issue.

            Is there a technical reason why displaying this option in the Jenkins Global Security configuration will not work if Jenkins is running on Windows? Curious why the other advance settings for the AD plugin work but not this one?

            russ519 Russ Frisch added a comment - Daniel, Thanks for the response. I looked at the commit and was able to determine that the value is stored in the config file as <groupLookupStrategy> and updated my config.xml to include <groupLookupStrategy>Recursive</groupLookupStrategy> which fixes my slow login issue. Is there a technical reason why displaying this option in the Jenkins Global Security configuration will not work if Jenkins is running on Windows? Curious why the other advance settings for the AD plugin work but not this one?
            dennys Dennys Hsieh added a comment - - edited

            I use 1.38 and I got this error "AD chain lookup is taking too long (41ms). Falling back to recursive lookup". But I input the domain name in the configuration, I'm not sure will it impact the performance. But in 1.36 it only takes < 1 second.

            dennys Dennys Hsieh added a comment - - edited I use 1.38 and I got this error "AD chain lookup is taking too long (41ms). Falling back to recursive lookup". But I input the domain name in the configuration, I'm not sure will it impact the performance. But in 1.36 it only takes < 1 second.
            nkjensen Niels Kristian Jensen made changes -
            Attachment errorlog.zip [ 26476 ]
            danielbeck Daniel Beck made changes -
            Link This issue is related to JENKINS-24960 [ JENKINS-24960 ]
            podskalsky podskalsky added a comment -

            the workaround "<groupLookupStrategy>Recursive</groupLookupStrategy>" is ok for me

            podskalsky podskalsky added a comment - the workaround "<groupLookupStrategy>Recursive</groupLookupStrategy>" is ok for me
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 154958 ] JNJira + In-Review [ 195092 ]

            The problem has been fixed at least since 2.4 of the plugin.

            nkjensen Niels Kristian Jensen added a comment - The problem has been fixed at least since 2.4 of the plugin.
            nkjensen Niels Kristian Jensen made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            I'm experiencing slow lookups with version 2.6:

            Aug 09, 2017 1:48:23 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider resolveGroups
            WARNING: Group lookup via Active Directorys LDAP_MATCHING_RULE_IN_CHAIN extension matched users groups but took 24 seconds to run. Switching to recursive lookup for future group lookup queries

            I was configuring Active Directory as follows:

            import jenkins.model.*
            import hudson.security.*
            import hudson.plugins.active_directory.*
               
            String domain = 'ad.example.com'
            String site = 'FOO'
            String server = 'bar.ad.example.com'
            String bindName = 'service-jenkins@ad.example.com'
            String bindPassword = 'password'
            
            // Set the security realm to AD, so users can log in with their windows credentials.
            realm = new ActiveDirectorySecurityRealm(domain, site, bindName, bindPassword, server)
            def instance = Jenkins.getInstance()
            instance.setSecurityRealm(realm)
            

            I worked around this by setting Token Groups as my lookup strategy:

            import jenkins.model.*
            import hudson.security.*
            import hudson.plugins.active_directory.*
               
            String domain = 'ad.example.com'
            String site = 'FOO'
            String server = 'bar.ad.example.com'
            String bindName = 'service-jenkins@ad.example.com'
            String bindPassword = 'password'
            
            // AD lookups can be little slow due to looking up groups. Use Token
            // Groups, which is the fastest way of looking groups, although it
            // returns fewer groups.
            def lookupStrategy = GroupLookupStrategy.TOKENGROUPS
            
            // Set the security realm to AD, so users can log in with their windows credentials.
            realm = new ActiveDirectorySecurityRealm(domain, site, bindName, bindPassword, server, lookupStrategy)
            def instance = Jenkins.getInstance()
            instance.setSecurityRealm(realm)
            
            wilfredh Wilfred Hughes added a comment - I'm experiencing slow lookups with version 2.6: Aug 09, 2017 1:48:23 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider resolveGroups WARNING: Group lookup via Active Directorys LDAP_MATCHING_RULE_IN_CHAIN extension matched users groups but took 24 seconds to run. Switching to recursive lookup for future group lookup queries I was configuring Active Directory as follows: import jenkins.model.* import hudson.security.* import hudson.plugins.active_directory.*     String domain = 'ad.example.com' String site = 'FOO' String server = 'bar.ad.example.com' String bindName = 'service-jenkins@ad.example.com' String bindPassword = 'password' // Set the security realm to AD, so users can log in with their windows credentials. realm = new ActiveDirectorySecurityRealm(domain, site, bindName, bindPassword, server) def instance = Jenkins.getInstance() instance.setSecurityRealm(realm) I worked around this by setting Token Groups as my lookup strategy: import jenkins.model.* import hudson.security.* import hudson.plugins.active_directory.*     String domain = 'ad.example.com' String site = 'FOO' String server = 'bar.ad.example.com' String bindName = 'service-jenkins@ad.example.com' String bindPassword = 'password' // AD lookups can be little slow due to looking up groups. Use Token // Groups, which is the fastest way of looking groups, although it // returns fewer groups. def lookupStrategy = GroupLookupStrategy.TOKENGROUPS // Set the security realm to AD, so users can log in with their windows credentials. realm = new ActiveDirectorySecurityRealm(domain, site, bindName, bindPassword, server, lookupStrategy) def instance = Jenkins.getInstance() instance.setSecurityRealm(realm)

            People

              Unassigned Unassigned
              mrose Michael Rose
              Votes:
              5 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: