-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
Powered by SuggestiMate
The permission lookups gets slow when having (too) many registered AD-groups for the users.
For many authorization strategies, there is only a small set of groups that are actually being used. It is therefore unnecessary to iterate over all registered AD-groups when we know that no rules for most groups will be found anyway.
We are implementing a feature to make it optional for Jenkins to ignore Active Directory groups that are not being used by the active Authorization Strategy.
[JENKINS-24195] Faster permission lookups when having many unused AD-groups
This may be the cause of our issues. We have two Windows Server 2012 masters, both configured with the Active Directory plugin and we intermittently experience login issues. Sometimes the permission lookup works, but occasionally (when no one has logged into either server for a while, less than a few hours) both servers will take quite a bit of time to login and either work or fail with the following exception (logged on the Jenkins server):
Aug 20, 2014 8:41:26 AM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser WARNING: Credential exception tying to authenticate against [domain].net domain org.acegisecurity.BadCredentialsException: Failed to retrieve user information for [userdev]; nested exception is javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: [domain].net:389 [Root exception is java.net.ConnectException: Connection timed out: connect]] at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:309) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:193) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:137) at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:122) at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:200) at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:47) at org.acegisecurity.ui.webapp.AuthenticationProcessingFilter.attemptAuthentication(AuthenticationProcessingFilter.java:74) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:252) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:74) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: [domain].net:389 [Root exception is java.net.ConnectException: Connection timed out: connect]] at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(Unknown Source) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(Unknown Source) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(Unknown Source) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(Unknown Source) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(Unknown Source) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(Unknown Source) at hudson.plugins.active_directory.LDAPSearchBuilder.searchOne(LDAPSearchBuilder.java:44) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:263) ... 45 more Caused by: javax.naming.CommunicationException: [domain].net:389 [Root exception is java.net.ConnectException: Connection timed out: connect] at com.sun.jndi.ldap.LdapReferralContext.<init>(Unknown Source) at com.sun.jndi.ldap.LdapReferralException.getReferralContext(Unknown Source) at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(Unknown Source) ... 53 more Caused by: java.net.ConnectException: Connection timed out: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at java.net.Socket.<init>(Unknown Source) at java.net.Socket.<init>(Unknown Source) at com.sun.jndi.ldap.Connection.createSocket(Unknown Source) at com.sun.jndi.ldap.Connection.<init>(Unknown Source) at com.sun.jndi.ldap.LdapClient.<init>(Unknown Source) at com.sun.jndi.ldap.LdapClient.getInstance(Unknown Source) at com.sun.jndi.ldap.LdapCtx.connect(Unknown Source) at com.sun.jndi.ldap.LdapCtx.<init>(Unknown Source) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(Unknown Source) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(Unknown Source) at com.sun.jndi.url.ldap.ldapURLContextFactory.getObjectInstance(Unknown Source) at javax.naming.spi.NamingManager.getURLObject(Unknown Source) at javax.naming.spi.NamingManager.processURL(Unknown Source) at javax.naming.spi.NamingManager.processURLAddrs(Unknown Source) at javax.naming.spi.NamingManager.getObjectInstance(Unknown Source) ... 56 more
I should mention we are using Amazon EC2 instances as Jenkins servers and logging in through a BOVPN.
Code changed in jenkins
User: Fredrik Persson
Path:
pom.xml
src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java
src/main/java/hudson/plugins/active_directory/ActiveDirectoryUserDetail.java
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/config.jelly
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/help-removeIrrelevantGroups.html
src/test/java/hudson/plugins/active_directory/RemoveIrrelevantGroupsTest.java
http://jenkins-ci.org/commit/active-directory-plugin/a2561997d3570642415ef1cf880aa29d12b2a17b
Log:
JENKINS-24195 Ignoring irrelevant Active Directory groups
Added an option to make Jenkins ignore Active Directory
groups that are not being used by the active
Authorization Strategy. This can significantly improve
performance in environments with a large number of groups
but a small number of corresponding rules defined by the
Authorization Strategy.
Performance tests have been made in an enviroment with
Role Based Authorization Strategy, 7000 jobs and
280 AD groups (where only one of the last groups is
used by the AuthorizationStrategy). The average loading
time of an empty List View went from 6.2 seconds
to 0.2 seconds. An sectioned view with 15 sections
of text went from 36 seconds to 0.6 seconds.
When Matrix Based Authorization was used instead,
loading time of the sectioned view went down from
1.2 seconds to 0.1 seconds.
Code changed in jenkins
User: Fredrik Persson
Path:
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/config.jelly
http://jenkins-ci.org/commit/active-directory-plugin/962f2c9aab34549c51bc5a093034670b2ef4b385
Log:
JENKINS-24195 Added irrelevant group removal for Windows
The checkbox for enabling the "Remove irrelevant groups"
has been added for Windows based masters as well.
Code changed in jenkins
User: Robert Sandell
Path:
pom.xml
src/main/java/hudson/plugins/active_directory/ActiveDirectorySecurityRealm.java
src/main/java/hudson/plugins/active_directory/ActiveDirectoryUserDetail.java
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/config.jelly
src/main/resources/hudson/plugins/active_directory/ActiveDirectorySecurityRealm/help-removeIrrelevantGroups.html
src/test/java/hudson/plugins/active_directory/RemoveIrrelevantGroupsTest.java
http://jenkins-ci.org/commit/active-directory-plugin/90c0688d265eb5ff60e570f4403ea23137a6c92a
Log:
Merge pull request #10 from fredrikpersson/removegroups
JENKINS-24195 Ignoring irrelevant Active Directory groups
Compare: https://github.com/jenkinsci/active-directory-plugin/compare/56796545a3e7...90c0688d265e
The downside of this change is that if you update permissions and start referencing a new group, people need to log out and come back into to see the effect.
I think the performance improvement is great, and I get that this is a practical fix to work around the problem at hand for the author, but at the same time, I feel that this is exposing a complexity to users, when a problem should have been solved by us.
Role-based Authorization Strategy plugin has known performance issues, and as for the matrix auth plugin, I think SparseACL.hasPermission() implementation should be improved to reject uninteresting {{Sid}}s more quickly.
This is fixed in ad side since active-directory-1.39
As KK said, more things can be done at Auth Strategy level.
Pull request:
https://github.com/jenkinsci/active-directory-plugin/pull/10