Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37737

Intermittent login failures with Active Directory / Matrix-based security

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • Jenkins ver. 1.653.
      Active Directory Plugin 1.47 (Windows AD, Matrix-based security)
      Windows Server 2012 R2
      Chrome

      Helllo! We are experiencing intermittent login issues since early August, 2016 for all users from any browser or workstation (location does not seem to be an issue). We have a cross domain - VPN tunnel, which has not experienced recent outages to cause failed logons or AD lookups. Other systems relying on the VPN tunnel are not experiencing authentication issues. Successful manual telnet tests between the Domain Controllers were successful during Jenkins failed logins. We are not ruling out a network issue but we can't see any problems. We have not recently upgraded Jenkins or the Active Directory Plugin.

      Looking forward to any help to resolve our issue.

      Output from log:

      Aug 27, 2016 7:11:51 AM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
      WARNING: Credential exception trying to authenticate against ####### domain
      org.acegisecurity.BadCredentialsException: Failed to retrieve user information for ##############; nested exception is javax.naming.PartialResultException Root exception is javax.naming.CommunicationException: DomainDnsZones.######## [Root exception is java.net.ConnectException: Connection timed out: connect]
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$1.call(ActiveDirectoryUnixAuthenticationProvider.java:332)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$1.call(ActiveDirectoryUnixAuthenticationProvider.java:235)
      at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
      at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
      at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:235)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:200)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:142)
      at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:122)
      at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:200)
      at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:47)
      at org.acegisecurity.ui.webapp.AuthenticationProcessingFilter.attemptAuthentication(AuthenticationProcessingFilter.java:74)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:252)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
      at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      at org.eclipse.jetty.server.Server.handle(Server.java:370)
      at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
      at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
      at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
      at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
      at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: javax.naming.PartialResultException Root exception is javax.naming.CommunicationException: DomainDnsZones.####### [Root exception is java.net.ConnectException: Connection timed out: connect]
      at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(Unknown Source)
      at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(Unknown Source)
      at hudson.plugins.active_directory.LDAPSearchBuilder.searchOne(LDAPSearchBuilder.java:86)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$1.call(ActiveDirectoryUnixAuthenticationProvider.java:280)
      ... 55 more

          [JENKINS-37737] Intermittent login failures with Active Directory / Matrix-based security

          Jens Runge added a comment -

          Hello!

          We have the same problems here.
          In our case i supect the reason in offline locations (DomainControllers listet in DomainDnsZones.<domain>).
          Until i convince our administrators to remove this offline locations from DNS, my workaround is to clear the name resolving cache on the jenkins server.

          IMO the LDAP-plugin ignores the configured LDAP-servers and resolve the Domain Controllers over DNS lookup DomainDnsZones.<domain> and runs into the problem with not available servers listet in DNS.

          Greetings
          JR

          Jens Runge added a comment - Hello! We have the same problems here. In our case i supect the reason in offline locations (DomainControllers listet in DomainDnsZones.<domain>). Until i convince our administrators to remove this offline locations from DNS, my workaround is to clear the name resolving cache on the jenkins server. IMO the LDAP-plugin ignores the configured LDAP-servers and resolve the Domain Controllers over DNS lookup DomainDnsZones.<domain> and runs into the problem with not available servers listet in DNS. Greetings JR

          Derek Sakauye added a comment -

          JR,

          Thanks for your reply. We have yet to resolve our issue. However, I wanted to note that we are using the AD Plugin (not LDAP-plugin). Does your workaround still apply? If so, how do we clear the name resolving cache on the jenkins server?

          Thanks!

          • Derek

          Derek Sakauye added a comment - JR, Thanks for your reply. We have yet to resolve our issue. However, I wanted to note that we are using the AD Plugin (not LDAP-plugin). Does your workaround still apply? If so, how do we clear the name resolving cache on the jenkins server? Thanks! Derek

          Jens Runge added a comment -

          Hi Derek,

          i meant the Active Directory plugin too. It was only a mistake in writing.

          Our Jenkins runs on a Windows server. On console a "ipconfig /flushdns" should help you.

          For Linux i found this page: http://www.cyberciti.biz/faq/rhel-debian-ubuntu-flush-clear-dns-cache/
          Maybe you have to search a other solution for your Linux-Distribution.

          Greetings
          JR

          Jens Runge added a comment - Hi Derek, i meant the Active Directory plugin too. It was only a mistake in writing. Our Jenkins runs on a Windows server. On console a "ipconfig /flushdns" should help you. For Linux i found this page: http://www.cyberciti.biz/faq/rhel-debian-ubuntu-flush-clear-dns-cache/ Maybe you have to search a other solution for your Linux-Distribution. Greetings JR

          Derek Sakauye added a comment -

          Hi JR,

          Is your workaround to prevent the issue from occurring? Or do you apply the ipconfig /flushdns only when you experience the logon authentication issue?

          Thanks for the help!

          Best regards,

          • Derek

          Derek Sakauye added a comment - Hi JR, Is your workaround to prevent the issue from occurring? Or do you apply the ipconfig /flushdns only when you experience the logon authentication issue? Thanks for the help! Best regards, Derek

          Jens Runge added a comment -

          Hi Derek,

          I admit its only a dirty workaround. Our Teams isn't that big, so I have done it every time somone can't login.
          Meanwhile our Administrators cleaned up the DNS, so that DomainDnsZones.<domain> only points to available systems.

          I assume, your Jenkins Service is running with a local server account.
          Then you can try to use a domain account as Jenkins service account on your Jenkins Server.
          Maybe then the windows native authentication for directory listing should not fail and would not fallback to non native authentication.

          Jens Runge added a comment - Hi Derek, I admit its only a dirty workaround. Our Teams isn't that big, so I have done it every time somone can't login. Meanwhile our Administrators cleaned up the DNS, so that DomainDnsZones.<domain> only points to available systems. I assume, your Jenkins Service is running with a local server account. Then you can try to use a domain account as Jenkins service account on your Jenkins Server. Maybe then the windows native authentication for directory listing should not fail and would not fallback to non native authentication.

          Derek Sakauye added a comment -

          Hi JR,

          Our team is pretty small too. So we're in a similar pickle. We actually are running Jenkins with a domain service account. I'll try your "flushdns" workaround next time we can't login. I'll check with Domain Admins about cleaning up the DNS too.

          Thanks,

          • Derek

          Derek Sakauye added a comment - Hi JR, Our team is pretty small too. So we're in a similar pickle. We actually are running Jenkins with a domain service account. I'll try your "flushdns" workaround next time we can't login. I'll check with Domain Admins about cleaning up the DNS too. Thanks, Derek

          Hello!

          We have the same problems with Jenkins ver. 2.7.1. Any ETA for the fix?

          Vijaya Bhaskar Kadiri added a comment - Hello! We have the same problems with Jenkins ver. 2.7.1. Any ETA for the fix?

          Hugo added a comment -

          Hello,

          I have the same issue, Jenkins can only reach 2 controller i my domain, so I use the servers option to point to the server it can reach (don't want to use round robin from Active directory).
          But I can see request to other domain controller which are not configured in my list.
          This cause login failure and various timeout.

          Here is my configuration :

            <securityRealm class="hudson.plugins.active_directory.ActiveDirectorySecurityRealm" plugin="active-directory@2.0">
              <domains>
                <hudson.plugins.active__directory.ActiveDirectoryDomain>
                  <!-- <name>domain.local</name> -->
                  <servers>192.168.1.2:636,192.168.1.3:636</servers>
                </hudson.plugins.active__directory.ActiveDirectoryDomain>
              </domains>
              <bindName>CN=LDAP,OU=Users,DC=domain,DC=local</bindName>
              <bindPassword>DDHCNCNCCC/r9Rxf0HvCqt0QVuU=</bindPassword>
              <groupLookupStrategy>AUTO</groupLookupStrategy>
              <removeIrrelevantGroups>false</removeIrrelevantGroups>
            </securityRealm>
          

          plugin : 2.0
          jenkins : 2.34

          Hugo added a comment - Hello, I have the same issue, Jenkins can only reach 2 controller i my domain, so I use the servers option to point to the server it can reach (don't want to use round robin from Active directory). But I can see request to other domain controller which are not configured in my list. This cause login failure and various timeout. Here is my configuration : <securityRealm class= "hudson.plugins.active_directory.ActiveDirectorySecurityRealm" plugin= "active-directory@2.0" > <domains> <hudson.plugins.active__directory.ActiveDirectoryDomain> <!-- <name>domain.local</name> --> <servers>192.168.1.2:636,192.168.1.3:636</servers> </hudson.plugins.active__directory.ActiveDirectoryDomain> </domains> <bindName>CN=LDAP,OU=Users,DC=domain,DC=local</bindName> <bindPassword>DDHCNCNCCC/r9Rxf0HvCqt0QVuU=</bindPassword> <groupLookupStrategy>AUTO</groupLookupStrategy> <removeIrrelevantGroups> false </removeIrrelevantGroups> </securityRealm> plugin : 2.0 jenkins : 2.34

          Agustin Munoz added a comment -

          Hello,

          I am on the same situation, I was able to perform a small workaround by setting the environment variable com.sun.jndi.ldap.connect.timeout to 100 on advanced properties, now instead of getting stuck in the login screen for several minutes it only takes a few seconds to authenticate the user.

          plugin: 2.4

          jenkins: 2.46.2

          Agustin Munoz added a comment - Hello, I am on the same situation, I was able to perform a small workaround by setting the environment variable com.sun.jndi.ldap.connect.timeout to 100 on advanced properties, now instead of getting stuck in the login screen for several minutes it only takes a few seconds to authenticate the user. plugin: 2.4 jenkins: 2.46.2

            fbelzunc FĂ©lix Belzunce Arcos
            dsakauye Derek Sakauye
            Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: