-
Bug
-
Resolution: Fixed
-
Major
-
Jenkins v1.560; AD plugin v1.37
-
Powered by SuggestiMate
Logging in on version 1.37 of the Active Directory plugin takes too long and I think it is causing occasional proxy timeouts. At first I had blamed it on having too many groups in Active Directory. Then I took a look at the release notes and saw that version 1.37 add some extra group specific logic. I decided to try downgrading the plugin and sure enough I was able to log in almost immediately. Which made me very very happy!
Version 1.37 (I've seen this take up to a minute)
Apr 30, 2014 7:38:17 PM FINE hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider
Stage 2: looking up via memberOf
Apr 30, 2014 7:38:44 PM FINE hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider
Version 1.36
Apr 30, 2014 7:53:34 PM FINE hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider
Stage 2: looking up via memberOf
Apr 30, 2014 7:53:34 PM FINER hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider
[JENKINS-22830] Slow login w/ Active directory plugin
I didn't try version 1.35. We are using version 1.36 right now. Did you try this version? Does 1.35 perform better?
I just tried 1.36 and it works fine. I also tried upgrading from 1.36 to 1.37 with no change in the error message and login problems. I get the following message: Caused by: javax.naming.TimeLimitExceededException: [LDAP: error code 3 - Timelimit Exceeded]; remaining name 'DC=xyx,DC=acme,DC=org'
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3143)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840)
at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:147)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:216)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:189)
at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.parseMembers(ActiveDirectoryUnixAuthenticationProvider.java:456)
at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.resolveGroups(ActiveDirectoryUnixAuthenticationProvider.java:417)
at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:293)
I see the same issue here. Downgrading from 1.37 to 1.36 fixed my issue as well...
(download it from here https://updates.jenkins-ci.org/download/plugins/active-directory/1.36/active-directory.hpi and upload it manually to your server)
I turned on logging for the plugin and it seemed to be spending a lot of time getting the groups from active directory which led me to this issue.
I was getting frustrated comments from several users of the system...so could we bump up the priority? Problems like this (20 seconds to login or load the web page the first time in awhile) tend to deter people from using otherwise-awesome tools.
I would second bumping up the priority as I am experiencing the same errors mentioned above. I am using the same workaround by downgrading to 1.36.
This bug basically killed our LDAP server by making a really bad query. I would bump this as well.
Can you also remove 1.37? It's already caused a big problem in our organization, and may well do so in others.
Oh boy.
The original change was introduced in response to a bug someone filed indicating that 1.36 didn't list all the groups they should be seeing.
I've just spent good time googling the performance of this operator. There are some that reports this is faster than recursive lookup, and others that report this is slower, as in the case of this bug report.
Overall, people do seem to favor this recursive search operator over other options. Does anyone know any AD person who can shed some lights on this?
Fow now I'm going to introduce a switch that can be used to force one way or the other.
For my internal book-keeping, the original request for this change came from ZD-16865.
Kohsuke: Also see ZD-16562. It seems this just won't work in large directories.
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java
src/main/java/hudson/plugins/active_directory/ActiveDirectoryUnixAuthenticationProvider.java
src/main/java/hudson/plugins/active_directory/GroupLookupStrategy.java
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/config.jelly
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/help-groupLookupStrategy.html
src/main/resources/hudson/plugins/active_directory/Messages.properties
http://jenkins-ci.org/commit/active-directory-plugin/ba12cf0afd57cc643d5237c8be04d5a7e44296ee
Log:
[FIXED JENKINS-22830]
Providing the option for the user to pick the lesser of two evils. Performance vs Correctness.
Looking forward to collect more information in the future to improve this situation, but in the mean tim we need to make this plugin usable.
I've released 1.38 that contains the fix to this problem.
You can go to "Configure global security" and under the "advanced" tab, you can choose "Group Membership Lookup Strategy".
You basically have two choices.
- "Recursive queries", which is the pre-1.37 behavior. This involves issuing a number of simple LDAP queries to recursively find all the groups that you belong to. Reported by some people not to find all groups, and also reported by others to be slow (presumably due to the number of queries it has to issue)
- "LDAP_MATCHING_RULE_IN_CHAIN", which is what 1.37 added. People who are hit by
JENKINS-22830is likely not want to use this, but otherwise the consensus in the developer community appears to be that this is the way to go. After all, why would Microsoft go out of its way to add a custom extension to LDAP if it's not supposed to be used!?
The default is "automatic" and it tries to do the right thing the best it can.
The main challenge for me to work on a problem like this is that I do not have direct access to any large AD deployment. Looking forward to your feedback on relative merits/problems so that we can improve the situation over time.
The LDAP_MATCHING_RULE_IN_CHAIN mechanism can be very intensive on large servers. The reference to this on Microsoft's site is:
http://msdn.microsoft.com/en-us/library/aa746475%28v=vs.85%29.aspx
It mentioned this specifically:
"Note that when using LDAP_MATCHING_RULE_IN_CHAIN, scope is not limited—it can be base, one-level, or subtree. Some such queries on subtrees may be more processor intensive, such as chasing links with a high fan-out; that is, listing all the groups that a user is a member of. Inefficient searches will log appropriate event log messages, as with any other type of query."
From what I can see, the search via the chain lookup in the plugin is only using the subtree scope, so it is unlikely that any developer with access to a large AD deployment would want this at all as it can easily kill the AD if there are sufficient queries to accounts with large number of group memberships.
queries to accounts with large number of group memberships.
I tried this a while ago in an LDAP tool against AD global catalog. The request timed out for me with an account with only (recursively) two group memberships. One of them was "All Employees Worldwide" though, with 10k+ accounts in it. So it seems the support article doesn't have the full story.
This is the timed out query for us:
member<==>CN=myid,OU=People,OU=eProfile,DC=core,DC=dir,DC=domain,DC=com
I have just installed version 1.38 of this extension on a new Jenkins instance and I'm running into the issue of AD authentication taking a very long time. I have tried to follow the instructions KohSuke provided here a few comments up (https://issues.jenkins-ci.org/browse/JENKINS-22830?focusedCommentId=203052&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-203052) but I'm only seeing the Domain Name and Domain Controller text fields under advanced. I'm not seeing the Group Membership Lookup Strategy. Can anyone else see this option in their install?
Russ: This option is only available if your Jenkins master doesn't run on Windows, or com4j isn't working.
Daniel,
Thanks for the response. I looked at the commit and was able to determine that the value is stored in the config file as <groupLookupStrategy> and updated my config.xml to include <groupLookupStrategy>Recursive</groupLookupStrategy> which fixes my slow login issue.
Is there a technical reason why displaying this option in the Jenkins Global Security configuration will not work if Jenkins is running on Windows? Curious why the other advance settings for the AD plugin work but not this one?
I use 1.38 and I got this error "AD chain lookup is taking too long (41ms). Falling back to recursive lookup". But I input the domain name in the configuration, I'm not sure will it impact the performance. But in 1.36 it only takes < 1 second.
the workaround "<groupLookupStrategy>Recursive</groupLookupStrategy>" is ok for me
The problem has been fixed at least since 2.4 of the plugin.
I'm experiencing slow lookups with version 2.6:
Aug 09, 2017 1:48:23 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider resolveGroups WARNING: Group lookup via Active Directorys LDAP_MATCHING_RULE_IN_CHAIN extension matched users groups but took 24 seconds to run. Switching to recursive lookup for future group lookup queries
I was configuring Active Directory as follows:
import jenkins.model.* import hudson.security.* import hudson.plugins.active_directory.* String domain = 'ad.example.com' String site = 'FOO' String server = 'bar.ad.example.com' String bindName = 'service-jenkins@ad.example.com' String bindPassword = 'password' // Set the security realm to AD, so users can log in with their windows credentials. realm = new ActiveDirectorySecurityRealm(domain, site, bindName, bindPassword, server) def instance = Jenkins.getInstance() instance.setSecurityRealm(realm)
I worked around this by setting Token Groups as my lookup strategy:
import jenkins.model.* import hudson.security.* import hudson.plugins.active_directory.* String domain = 'ad.example.com' String site = 'FOO' String server = 'bar.ad.example.com' String bindName = 'service-jenkins@ad.example.com' String bindPassword = 'password' // AD lookups can be little slow due to looking up groups. Use Token // Groups, which is the fastest way of looking groups, although it // returns fewer groups. def lookupStrategy = GroupLookupStrategy.TOKENGROUPS // Set the security realm to AD, so users can log in with their windows credentials. realm = new ActiveDirectorySecurityRealm(domain, site, bindName, bindPassword, server, lookupStrategy) def instance = Jenkins.getInstance() instance.setSecurityRealm(realm)
I have the same issue. I am not even able to authenticate using the 1.37 of the plugin. I see timeout errors in the jenkins log, with LTS release 1.532.2. I am also seeing the same problem with LTS 1.554.1 of Jenkins. I had to use version 1.35 of the plugin to resolve the login issues.