Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: jobconfighistory-plugin
Labels:
None
Environment:
Jenkins 2.121.3
jobConfigHistory-2.23.1
docker-plugin-1.1.4

Similar Issues:

Show

After attempting to save a unrelated Global configuration change, confronted with Jenkins Devil - Ooops! and the complete stack trace below.

Caused by: java.lang.RuntimeException: Could not create rootDir /path/to/jenkins/config-history/nodes/docker-206db4b09bf37b/2021-09-03_06-56-32
        at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryDir(FileHistoryDao.java:261)

After some investigation, it turns out a misconfiguration was creating new docker agents on demand every 30 seconds and in turn being deleted 10 minutes later.

After ~ 2.5 months of this, it seems we exceeded the File-system limit of 64K entries per directory (in config-history/nodes). Unfortunately, we did not actually make any changes via UI for another 3 weeks after the limit was exceeded, at which time we saw the error and investigated.

If was not clear at all for reading error message what the cause was; only when I checked the FS and performed an ls and tried to create another entry, did I figure this out.

I don't know what actions you can recommend to alert the user or what the plugin should do; arbitrary removing oldest may not be appropriate.

This was also the result of a misconfiguration of another plugin (docker), but similar could manifest itself anytime on long-lived systems with significant changes of any type.

Perhaps this was also the cause for JENKINS-35421 ?

Sep 03, 2021 6:56:32 AM org.eclipse.jetty.server.handler.ContextHandler$Context log
WARNING: Error while serving http://jenkins2.tsl.telus.com/configSubmit
java.lang.reflect.InvocationTargetException
  at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)
  at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
  at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
  at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
  at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
  at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
  at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
  at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
  at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
  at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:860)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
  at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
  at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
  at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
  at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
  at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
  at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
  at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
  at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
  at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
  at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
  at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
  at org.eclipse.jetty.server.Server.handle(Server.java:530)
  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
  at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
  at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
  at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
  at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:823)
Caused by: java.lang.RuntimeException: Could not create rootDir /path/to/jenkins/config-history/nodes/docker-206db4b09bf37b/2021-09-03_06-56-32
  at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryDir(FileHistoryDao.java:261)
  at hudson.plugins.jobConfigHistory.FileHistoryDao.getRootDir(FileHistoryDao.java:1004)
  at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryEntry(FileHistoryDao.java:1011)
  at hudson.plugins.jobConfigHistory.FileHistoryDao.deleteNode(FileHistoryDao.java:917)
  at hudson.plugins.jobConfigHistory.ComputerHistoryListener.onRemove(ComputerHistoryListener.java:113)
  at hudson.plugins.jobConfigHistory.ComputerHistoryListener.onConfigurationChange(ComputerHistoryListener.java:70)
  at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:231)
  at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1551)
  at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3698)
  at java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_V(VirtualHandle.java:85)
  at java.lang.invoke.FilterReturnHandle.invokeExact_thunkArchetype_X(FilterReturnHandle.java:59)
  at java.lang.invoke.AsTypeHandle.invokeExact_thunkArchetype_X(AsTypeHandle.java:49)
  at java.lang.invoke.InvokeGenericHandle.invokeExact_thunkArchetype_X(InvokeGenericHandle.java:88)
  at java.lang.invoke.SpreadHandle.invokeExact_thunkArchetype_X(SpreadHandle.java:100)
  at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:473)
  at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)

is caused by

JENKINS-62281 Exclude of node config does not work

Open

is related to

JENKINS-35421 Exception: "Could not create rootDir"

Reopened

JENKINS-66566 Job DSL / Job Configuration History plugin creates tons of xml files

Open

Details

Description

Attachments

Issue Links

Activity

People

Dates