-
Bug
-
Resolution: Fixed
-
Critical
-
Jenkins 1.609.2 with the following plugins:
active-directory 1.41
analysis-core 1.72
ansicolor 0.4.1
ant 1.2
antisamy-markup-formatter 1.1
audit-trail 2.1
build-token-root 1.1
build-user-vars-plugin 1.4
credentials 1.18
credentials-binding 1.5
cvs 2.11
email-ext 2.40.5
envinject 1.91.4
extensible-choice-parameter 1.3.1
external-monitor-job 1.4
gatling 1.1.1
git 2.4.0
git-client 1.15.0
groovy 1.27
javadoc 1.1
jobConfigHistory 2.12
jquery 1.11.2-0
junit 1.2-beta-4
ldap 1.11
log-parser 1.0.8
mailer 1.11
matrix-auth 1.1
matrix-project 1.4.1
maven-plugin 2.7.1
metrics 3.0.11
nodelabelparameter 1.5.1
ownership 0.7
pam-auth 1.1
plain-credentials 1.0
postbuildscript 0.17
role-strategy 2.2.0
scm-api 0.2
scm-sync-configuration 0.0.8
script-security 1.13
slack 1.8
ssh-credentials 1.10
ssh-slaves 1.9
subversion 1.54
support-core 2.27
token-macro 1.10
translation 1.10
view-job-filters 1.26
warnings 4.48
windows-slaves 1.0
workflow-step-api 1.5Jenkins 1.609.2 with the following plugins: active-directory 1.41 analysis-core 1.72 ansicolor 0.4.1 ant 1.2 antisamy-markup-formatter 1.1 audit-trail 2.1 build-token-root 1.1 build-user-vars-plugin 1.4 credentials 1.18 credentials-binding 1.5 cvs 2.11 email-ext 2.40.5 envinject 1.91.4 extensible-choice-parameter 1.3.1 external-monitor-job 1.4 gatling 1.1.1 git 2.4.0 git-client 1.15.0 groovy 1.27 javadoc 1.1 jobConfigHistory 2.12 jquery 1.11.2-0 junit 1.2-beta-4 ldap 1.11 log-parser 1.0.8 mailer 1.11 matrix-auth 1.1 matrix-project 1.4.1 maven-plugin 2.7.1 metrics 3.0.11 nodelabelparameter 1.5.1 ownership 0.7 pam-auth 1.1 plain-credentials 1.0 postbuildscript 0.17 role-strategy 2.2.0 scm-api 0.2 scm-sync-configuration 0.0.8 script-security 1.13 slack 1.8 ssh-credentials 1.10 ssh-slaves 1.9 subversion 1.54 support-core 2.27 token-macro 1.10 translation 1.10 view-job-filters 1.26 warnings 4.48 windows-slaves 1.0 workflow-step-api 1.5
Our Jenkins instance is chef managed, meaning a process runs on the box every 30 minutes that enforces certain configuration options in Jenkins. It does this by executing jenkins-cli commands. The Jenkins cli authenticates using SSH keys. The ssh key in question is associated with a user that is a full admin on the master. This user is also in our AD since we use the active-directory plugin for authentication. This has been working fine for a long time. (I think we stood this box up in January 2015).
As of 1500 EDT on Aug 3, we started seeing the below error in our Jenkins logs. From the time the error occur until we restart Jenkins, all attempts to use the jenkins-cli by this user fail, effectively disabling our chef management.
Initially this was happening about once a day, it is now typically occurring within one to two hours after a the last corrective restart. However this may simply be due to our more aggressive restarting as soon as we see the error.
The initial error only appears once in the log, after that each subsequent authentication attempt results in a different error, also shown below. The jenkins-cli, for its part, returns the cryptic error reported in JENKINS-22346, which we have come to understand as an authentication failure.
The only notable changes made in this timeframe were the addition of the extensible-choice-parameter and nodelabelparameter plugins which were added on the 4th and 6th of August respectively.
Initial Error that Initiates the Problem
Aug 07, 2015 4:56:25 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser WARNING: Credential exception trying to authenticate against **REDACTED** domain org.acegisecurity.BadCredentialsException: Empty password at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:254) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:219) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:163) at org.acegisecurity.providers.dao.AbstractUserDetailsAuthenticationProvider.authenticate(AbstractUserDetailsAuthenticationProvider.java:122) at org.acegisecurity.providers.ProviderManager.doAuthentication(ProviderManager.java:200) at org.acegisecurity.AbstractAuthenticationManager.authenticate(AbstractAuthenticationManager.java:47) at jenkins.security.BasicHeaderRealPasswordAuthenticator.authenticate(BasicHeaderRealPasswordAuthenticator.java:55) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:79) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Error seen on Subsequent cli authentication attempts
Aug 07, 2015 5:01:36 PM hudson.TcpSlaveAgentListener$ConnectionHandler run INFO: Accepted connection #9 from /127.0.0.1:48680 Exception in thread "Thread-1234" org.acegisecurity.userdetails.UsernameNotFoundException: Authentication was successful but cannot locate the user information for at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:295) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:219) at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:163) at hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:53) at jenkins.security.ImpersonatingUserDetailsService.loadUserByUsername(ImpersonatingUserDetailsService.java:32) at hudson.model.User.impersonate(User.java:309) at org.jenkinsci.main.modules.cli.auth.ssh.SshCliAuthenticator.authenticate(SshCliAuthenticator.java:44) at hudson.cli.CliManagerImpl$2.run(CliManagerImpl.java:109)
- depends on
-
-
- In Progress
-
- links to
