Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39340

High CPU in doRename

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
      Jenkins: 2.23
      OS: OS Linux, 4.4.19-29.55.amzn1.x86_64 , amd64/64 (2 cores)
      Java: OpenJDK Runtime Environment, 1.7.0_111-mockbuild_2016_07_29_20_37-b00
      JVM: OpenJDK 64-Bit Server VM, 24.111-b01, mixed mode
    • Similar Issues:

      Description

      High CPU and not responsive every time when someone attempts to rename an existing job.

      Here's the thread stack from the JavaMelody

      Handling POST /job/[job_name]/doRename from [client ip address] : RequestHandlerThread368
      java.io.UnixFileSystem.list(Native Method)
      java.io.File.list(File.java:1116)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1257)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1287)
      org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1211)
      org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:1030)
      org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909)
      hudson.plugins.scm_sync_configuration.strategies.model.PatternsEntityMatcher.matchingFilesFrom(PatternsEntityMatcher.java:77)
      hudson.plugins.scm_sync_configuration.strategies.model.JobOrFolderConfigurationEntityMatcher.matchingFilesFrom(JobOrFolderConfigurationEntityMatcher.java:103)
      hudson.plugins.scm_sync_configuration.strategies.AbstractScmSyncStrategy.collect(AbstractScmSyncStrategy.java:114)
      hudson.plugins.scm_sync_configuration.ScmSyncConfigurationPlugin$3.apply(ScmSyncConfigurationPlugin.java:261)
      hudson.plugins.scm_sync_configuration.ScmSyncConfigurationPlugin$3.apply(ScmSyncConfigurationPlugin.java:258)
      com.google.common.collect.Iterators$8.next(Iterators.java:812)
      com.google.common.collect.Iterables$3.next(Iterables.java:508)
      com.google.common.collect.Iterables$3.next(Iterables.java:501)
      com.google.common.collect.Iterators$5.hasNext(Iterators.java:544)
      hudson.plugins.scm_sync_configuration.transactions.ScmTransaction.registerRenamedPath(ScmTransaction.java:56)
      hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationItemListener.onLocationChanged(ScmSyncConfigurationItemListener.java:94)
      hudson.model.listeners.ItemListener$6.apply(ItemListener.java:238)
      hudson.model.listeners.ItemListener$6.apply(ItemListener.java:236)
      hudson.model.listeners.ItemListener.forAll(ItemListener.java:167)
      hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:236)
      hudson.model.AbstractItem.renameTo(AbstractItem.java:335)
      hudson.model.Job.renameTo(Job.java:659)
      hudson.model.Job.doDoRename(Job.java:1502)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      java.lang.reflect.Method.invoke(Method.java:606)
      org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:324)
      org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:52)
      org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
      org.kohsuke.stapler.Function.bindAndInvoke(Function.java:167)
      org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:100)
      org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
      org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
      org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
      org.kohsuke.stapler.Stapler.service(Stapler.java:238)
      javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
      hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:201)
      net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178)
      net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)
      org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:104)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
      hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
      hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
      hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
      hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
      hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
      org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      org.eclipse.jetty.server.Server.handle(Server.java:499)
      org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      java.lang.Thread.run(Thread.java:745)

        Attachments

          Activity

          Hide
          brendan_shape Brendan Mannix added a comment -

          I'm seeing the same behavior with OpenJDK 8.

          openjdk version "1.8.0_102"
          OpenJDK Runtime Environment (build 1.8.0_102-b14)
          OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode)
          
          Show
          brendan_shape Brendan Mannix added a comment - I'm seeing the same behavior with OpenJDK 8. openjdk version "1.8.0_102" OpenJDK Runtime Environment (build 1.8.0_102-b14) OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode)
          Hide
          brendan_shape Brendan Mannix added a comment -

          The Jenkins Monitoring Plugin provides a convenient way to kill that particular java thread. https://wiki.jenkins-ci.org/display/JENKINS/Monitoring

          Problem threads (e.g. doRenames) appear in the "Current Requests" list

          Show
          brendan_shape Brendan Mannix added a comment - The Jenkins Monitoring Plugin provides a convenient way to kill that particular java thread. https://wiki.jenkins-ci.org/display/JENKINS/Monitoring Problem threads (e.g. doRenames) appear in the "Current Requests" list
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Smells like a bug in SCM Sync Configuration, which performs heavy directory scans during this call

          Show
          oleg_nenashev Oleg Nenashev added a comment - Smells like a bug in SCM Sync Configuration, which performs heavy directory scans during this call
          Hide
          brendan_shape Brendan Mannix added a comment -

          Can confirm that we also have scm_sync running.

          Show
          brendan_shape Brendan Mannix added a comment - Can confirm that we also have scm_sync running.
          Hide
          poklau Pok Lau added a comment -

          Yes, we have SCM Sync Configuration Plugin 0.0.10 in the system.

          Show
          poklau Pok Lau added a comment - Yes, we have SCM Sync Configuration Plugin 0.0.10 in the system.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            poklau Pok Lau
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: