Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11990

After updating Active Directory plugin the Jenkins People database is experiencing massive performance issues synching because it is trying to authenticate users that are no longer in company Active Directory but are part of the Jenkins Database

      After the issues with the 1.23 identified here https://issues.jenkins-ci.org/browse/JENKINS-11720 and upgrading to 1.23 I am experiencing serious performance problems because the Jenkins internal database contains users that are no longer part of the company active directory and the system is trying to validate all the users. It now takes the "People" page over 10 mins to load every time it is clicked. Attempts to go there produce the following error:

      Dec 5, 2011 2:09:37 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
      WARNING: Exhausted all configured domains and could not authenticate against any.
      Dec 5, 2011 2:09:37 PM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
      WARNING: Credential exception tying to authenticate against xxxxxxxxxxx domain
      hudson.security.UserMayOrMayNotExistException: Unable to retrieve the user information without bind DN/password configured
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:129)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:105)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:64)
      at hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:23)
      at hudson.plugins.active_directory.ActiveDirectoryMailAddressResolverImpl.findMailAddressFor(ActiveDirectoryMailAddressResolverImpl.java:31)
      at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
      at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:488)
      at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:315)
      at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:235)
      at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:227)
      at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:189)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:692)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:667)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:645)
      at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
      at hudson.model.Run.run(Run.java:1448)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)

      I have removed the company's domain name and replaced it with the xxxxxxxx entry in the error message for obvious reasons. Not sure if these users that no longer are part of the active directory can just be deleted without causing other issues/problems. Jenkis display information is maintained in this People database so it is needed. Not sure if deleting members no longer in AD will cause Jenkins to have display issues or other problems.

      It was nice to be able to use the Jenkins internal People Database to configure the display of users which when using active directory only show as numerical IDs. It seems that this capability is now lost because of the new AD plug-in.

      NEW SIDE EFFECT:

      I am seeing an additional issue I believe is related to this problem. A job will build successfully but never complete as the system continues to try and validate the person that checked in the code and fails:

      WARNING: RegExMailAddressResolver configuration for regex null and email pattern null is not valid.
      Jan 5, 2012 2:32:40 AM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
      WARNING: Credential exception tying to authenticate against xxxxxxx domain
      hudson.security.UserMayOrMayNotExistException: Unable to retrieve the user information without bind DN/password configured
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:129)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:105)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:64)
      at hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:23)
      at hudson.plugins.active_directory.ActiveDirectoryMailAddressResolverImpl.findMailAddressFor(ActiveDirectoryMailAddressResolverImpl.java:31)
      at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
      at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:488)
      at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:315)
      at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:235)
      at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:227)
      at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:189)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:697)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:672)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:650)
      at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
      at hudson.model.Run.run(Run.java:1448)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:238)
      Jan 5, 2012 2:32:40 AM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser
      WARNING: Credential exception tying to authenticate against xxxxxxxxx domain
      hudson.security.UserMayOrMayNotExistException: Unable to retrieve the user information without bind DN/password configured
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:129)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:105)
      at hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:64)
      at hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:23)
      at hudson.plugins.active_directory.ActiveDirectoryMailAddressResolverImpl.findMailAddressFor(ActiveDirectoryMailAddressResolverImpl.java:31)
      at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
      at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:488)
      at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:315)
      at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:235)
      at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:227)
      at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:189)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:697)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:672)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:650)
      at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
      at hudson.model.Run.run(Run.java:1448)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:238)
      Jan 5, 2012 2:32:40 AM hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider retrieveUser

      I have to restart Jenkins to clear out the locked job because it blocks any other jobs from running.

        1. Thread Dump1.doc
          263 kB
        2. Thread Dump2.doc
          254 kB
        3. Thread Dump3.doc
          296 kB
        4. Thread Dump4.doc
          308 kB
        5. Thread Dump5.doc
          322 kB

          [JENKINS-11990] After updating Active Directory plugin the Jenkins People database is experiencing massive performance issues synching because it is trying to authenticate users that are no longer in company Active Directory but are part of the Jenkins Database

          Thanks, but this stack trace isn't actually from the rendering of the people page. It was coming from a build that ended just around the same time you were requesting the page.

          To help us fix this problem, please request the people page, let it sit, then use access http://server/jenkins/threadDump several times to obtain the thread dump. This will help us identify where the time is spent.

          It was nice to be able to use the Jenkins internal People Database to configure the display of users which when using active directory only show as numerical IDs. It seems that this capability is now lost because of the new AD plug-in.
          

          Also, I'm not sure if I understand above. The display name that Jenkins uses (or more precisely, the text that appears under "Your name" in the user configuration page in Jenkins) comes from Active Directory's "display name" attribute, which is the combination of first name and the last name (whereas "user principal name" tends to be alpha numeric ID (for example, my user principal name was something like "kk123456" in my previous life.)

          Is your AD correctly maintained? Does it have other fields that store user's display name?

          Kohsuke Kawaguchi added a comment - Thanks, but this stack trace isn't actually from the rendering of the people page. It was coming from a build that ended just around the same time you were requesting the page. To help us fix this problem, please request the people page, let it sit, then use access http://server/jenkins/threadDump several times to obtain the thread dump. This will help us identify where the time is spent. It was nice to be able to use the Jenkins internal People Database to configure the display of users which when using active directory only show as numerical IDs. It seems that this capability is now lost because of the new AD plug-in. Also, I'm not sure if I understand above. The display name that Jenkins uses (or more precisely, the text that appears under "Your name" in the user configuration page in Jenkins) comes from Active Directory's "display name" attribute , which is the combination of first name and the last name (whereas "user principal name" tends to be alpha numeric ID (for example, my user principal name was something like "kk123456" in my previous life.) Is your AD correctly maintained? Does it have other fields that store user's display name?

          alexlombardi added a comment -

          I will take the dumps. Do you only need the People section or the entire dump?

          I am not sure if our Active Directory is properly maintained, but it is maintained. I will need to talk to the people that own it to get those details.

          The issue I have is that whenever someone drills in the build, then tries to see the details for the changes the display is what you refer to as the principal name. I have been using the People database to change that to show the persons actual name which is far more useful and descriptive. When new people come on board I change the display so instead of the principal name the full name is shown.

          After the update I mentioned I spend between 10-25 minutes for the People page to load, and just as long to open the configuration of a specific person for editing. This is painful and not workable, but leaving the principal name instead of the actual name in the details is not well received by my users.

          alexlombardi added a comment - I will take the dumps. Do you only need the People section or the entire dump? I am not sure if our Active Directory is properly maintained, but it is maintained. I will need to talk to the people that own it to get those details. The issue I have is that whenever someone drills in the build, then tries to see the details for the changes the display is what you refer to as the principal name. I have been using the People database to change that to show the persons actual name which is far more useful and descriptive. When new people come on board I change the display so instead of the principal name the full name is shown. After the update I mentioned I spend between 10-25 minutes for the People page to load, and just as long to open the configuration of a specific person for editing. This is painful and not workable, but leaving the principal name instead of the actual name in the details is not well received by my users.

          alexlombardi added a comment -

          Any update on this issue?

          alexlombardi added a comment - Any update on this issue?

          alexlombardi added a comment -

          Any update on this issue?

          alexlombardi added a comment - Any update on this issue?

          The thread dumps indicate that the slow down was caused by massive loading of the change sets in the build history (which is, in a way, a known problem), and this appears to be unrelated to AD.

          This is known to happen once during the start-up, but if you constantly see this problem, I suggest you monitor the heap and try increasing it. Those changeset records are weakly referenced, so maybe they are getting thrown away and reparsed.

          Kohsuke Kawaguchi added a comment - The thread dumps indicate that the slow down was caused by massive loading of the change sets in the build history (which is, in a way, a known problem), and this appears to be unrelated to AD. This is known to happen once during the start-up, but if you constantly see this problem, I suggest you monitor the heap and try increasing it. Those changeset records are weakly referenced, so maybe they are getting thrown away and reparsed.

          alexlombardi added a comment -

          I attempted to correct this issue by uninstalling and re-installing the AD plug in before you made your post on Jan 10, and the problem got worse. Jobs would reach the end of the build script, declare a success, but never terminate off the master queue (despite being done when you drilled into the project and looked at the queue there) forcing a restart to clear the issue (which only caused it again as soon as all the slaves where hit and locked). I was left with no option but to blow away and rebuild the entire Jenkins environment.

          I renamed the directory where the master was, created a new one with the same name, and reinstalled and reconfigured Jenkins infrastructure from scratch. I then copied the jobs folder structure and configuration files (but not build history) and restarted the system. The problem is now gone. I have limited the number of builds to keep on all jobs.

          When you recommend a heap size increase I already had that set at 1024 which I tought was quite large. What is recommended? 2048? And what's the recommended minimum RAM for a Jenkins master box? My VM might be too small.

          alexlombardi added a comment - I attempted to correct this issue by uninstalling and re-installing the AD plug in before you made your post on Jan 10, and the problem got worse. Jobs would reach the end of the build script, declare a success, but never terminate off the master queue (despite being done when you drilled into the project and looked at the queue there) forcing a restart to clear the issue (which only caused it again as soon as all the slaves where hit and locked). I was left with no option but to blow away and rebuild the entire Jenkins environment. I renamed the directory where the master was, created a new one with the same name, and reinstalled and reconfigured Jenkins infrastructure from scratch. I then copied the jobs folder structure and configuration files (but not build history) and restarted the system. The problem is now gone. I have limited the number of builds to keep on all jobs. When you recommend a heap size increase I already had that set at 1024 which I tought was quite large. What is recommended? 2048? And what's the recommended minimum RAM for a Jenkins master box? My VM might be too small.

          evernat added a comment -

          @alex
          This issue with build history may have been fixed with lazy loading in recent Jenkins versions.
          Anyway, the recommended heap size depends on the number of jobs, build history, etc (and 1024M is not large nowadays).

          Is the issue reproduced or can we close this?

          evernat added a comment - @alex This issue with build history may have been fixed with lazy loading in recent Jenkins versions. Anyway, the recommended heap size depends on the number of jobs, build history, etc (and 1024M is not large nowadays). Is the issue reproduced or can we close this?

          alexlombardi added a comment -

          Problem still persists evermat.

          alexlombardi added a comment - Problem still persists evermat.

            Unassigned Unassigned
            alexlombardi alexlombardi
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: