• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • Jenkins 2.121.3
      jobConfigHistory-2.23.1
      docker-plugin-1.1.4

      After attempting to save a unrelated Global configuration change, confronted with Jenkins Devil - Ooops! and the complete stack trace below.

      Caused by: java.lang.RuntimeException: Could not create rootDir /path/to/jenkins/config-history/nodes/docker-206db4b09bf37b/2021-09-03_06-56-32
              at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryDir(FileHistoryDao.java:261)
      

       
      After some investigation, it turns out a misconfiguration was creating new docker agents on demand every 30 seconds and in turn being deleted 10 minutes later.

      After ~ 2.5 months of this, it seems we exceeded the File-system limit of 64K entries per directory (in config-history/nodes). Unfortunately, we did not actually make any changes via UI for another 3 weeks after the limit was exceeded, at which time we saw the error and investigated.

      If was not clear at all for reading error message what the cause was; only when I checked the FS and performed an ls and tried to create another entry, did I figure this out.

      I don't know what actions you can recommend to alert the user or what the plugin should do; arbitrary removing oldest may not be appropriate.

      This was also the result of a misconfiguration of another plugin (docker), but similar could manifest itself anytime on long-lived systems with significant changes of any type.

      Perhaps this was also the cause for JENKINS-35421 ?

       

      Sep 03, 2021 6:56:32 AM org.eclipse.jetty.server.handler.ContextHandler$Context log
      WARNING: Error while serving http://jenkins2.tsl.telus.com/configSubmit
      java.lang.reflect.InvocationTargetException
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:347)
        at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
        at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
        at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
        at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
        at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:860)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
        at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
        at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:99)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:530)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:823)
      Caused by: java.lang.RuntimeException: Could not create rootDir /path/to/jenkins/config-history/nodes/docker-206db4b09bf37b/2021-09-03_06-56-32
        at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryDir(FileHistoryDao.java:261)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.getRootDir(FileHistoryDao.java:1004)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryEntry(FileHistoryDao.java:1011)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.deleteNode(FileHistoryDao.java:917)
        at hudson.plugins.jobConfigHistory.ComputerHistoryListener.onRemove(ComputerHistoryListener.java:113)
        at hudson.plugins.jobConfigHistory.ComputerHistoryListener.onConfigurationChange(ComputerHistoryListener.java:70)
        at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:231)
        at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1551)
        at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:3698)
        at java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_V(VirtualHandle.java:85)
        at java.lang.invoke.FilterReturnHandle.invokeExact_thunkArchetype_X(FilterReturnHandle.java:59)
        at java.lang.invoke.AsTypeHandle.invokeExact_thunkArchetype_X(AsTypeHandle.java:49)
        at java.lang.invoke.InvokeGenericHandle.invokeExact_thunkArchetype_X(InvokeGenericHandle.java:88)
        at java.lang.invoke.SpreadHandle.invokeExact_thunkArchetype_X(SpreadHandle.java:100)
        at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:473)
        at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
      

       

       

          [JENKINS-66533] Exception: "Could not create rootDir"

          pjdarton added a comment -

          If this plugin records the history of countless docker nodes then, yes, that'll fill up the disk.

          Unfortunately, the plugin has hard-coded assumptions about what nodes should/shouldn't have their history recorded and it can't e.g. be told "ignore any nodes whos name starts with docker".

          pjdarton added a comment - If this plugin records the history of countless docker nodes then, yes, that'll fill up the disk. Unfortunately, the plugin has hard-coded assumptions about what nodes should/shouldn't have their history recorded and it can't e.g. be told "ignore any nodes whos name starts with docker".

          Ian Williams added a comment -

          It seems there's a number of issues about how Docker and jobConfigHistory interact. But I believe this issue not not limited to just Docker.

          ANY object type tracked by jobConfigHistory could end up filling an object's directory with entries, not just "config-history/nodes". The plugin should either know what to do when it can't write to directory or provide better feedback to the user that breaking ERROR condition exists.

          Admittedly, the next closest history count on my instance is in the 1000's, so it would be way in the future, but YMMV.

          Ian Williams added a comment - It seems there's a number of issues about how Docker and jobConfigHistory interact. But I believe this issue not not limited to just Docker. ANY object type tracked by jobConfigHistory could end up filling an object's directory with entries, not just "config-history/nodes". The plugin should either know what to do when it can't write to directory or provide better feedback to the user that breaking ERROR condition exists. Admittedly, the next closest history count on my instance is in the 1000's, so it would be way in the future, but YMMV.

          pjdarton added a comment -

          I agree that the "disk full" handling on this plugin is a bit sucky but it's not alone in that - very few bits of Jenkins fare well when the disk fills up. Where this plugin stands out in its ability to fill the disk up so easily, especially as it can record endless changes at some speed and it doesn't prune the nodes at all.
          Worse, such files are often small and so a unix filesystem can run out of inodes before the free disk space gets worryingly low, meaning that the filesystem can be "too full for Jenkins to work" even when it's got free disk space remaining.

          However, I have some good news: I've made a PR for the docker-plugin that should stop the docker-plugin from causing the jobConfigHistory plugin to fill the disk. The docker-plugin's nodes didn't declare themselves as instanceof AbstractCloudSlave and hence were being recorded by the jobConfigHistory. This PR refactors the code so they do, meaning that the jobConfigHistory plugin then ignores them.

          I would welcome opinions on this PR - while it passes the automated unit-tests, I'd be much happier with some "real world" approval before getting it merged in to the official codebase.
          i.e. Please help by testing my docker-plugin code.

          pjdarton added a comment - I agree that the "disk full" handling on this plugin is a bit sucky but it's not alone in that - very few bits of Jenkins fare well when the disk fills up. Where this plugin stands out in its ability to fill the disk up so easily, especially as it can record endless changes at some speed and it doesn't prune the nodes at all. Worse, such files are often small and so a unix filesystem can run out of inodes before the free disk space gets worryingly low, meaning that the filesystem can be "too full for Jenkins to work" even when it's got free disk space remaining. However, I have some good news: I've made a PR for the docker-plugin that should stop the docker-plugin from causing the jobConfigHistory plugin to fill the disk. The docker-plugin's nodes didn't declare themselves as instanceof AbstractCloudSlave and hence were being recorded by the jobConfigHistory. This PR refactors the code so they do, meaning that the jobConfigHistory plugin then ignores them. I would welcome opinions on this PR - while it passes the automated unit-tests, I'd be much happier with some "real world" approval before getting it merged in to the official codebase. i.e. Please help by testing my docker-plugin code .

          Ian Williams added a comment -

          We no longer have the docker scenario configured that would cause massive entries to be logged (don't want to reintroduce that either), so not sure I can provide any real feedback, other than validate it does not spuriously log any more. I will see how I can help validate.

          I suggest to post a note on https://community.jenkins.io/ which explains not just your changes and ask for some wider feedback, including general note as to why it occurred (ie: didn't declare themselves as instanceof ).

          It's conceivable many plugins could cause excessive jobConfigHistory logging, just not as fast as this did. A note on the Jenkins Developer Mailing list and an update for the Plugin Development Guide regarding potential negative plugin interactions and developer guidance as to how to avoid could be useful for others. Perhaps stefanbrausch / jochenafuerbacher /robinschulz might add such a note in a "Developer's section of the README.md

          Ian Williams added a comment - We no longer have the docker scenario configured that would cause massive entries to be logged (don't want to reintroduce that either), so not sure I can provide any real feedback, other than validate it does not spuriously log any more. I will see how I can help validate. I suggest to post a note on https://community.jenkins.io/ which explains not just your changes and ask for some wider feedback, including general note as to why it occurred (ie: didn't declare themselves as instanceof ). It's conceivable many plugins could cause excessive jobConfigHistory logging, just not as fast as this did. A note on the Jenkins Developer Mailing list and an update for the Plugin Development Guide regarding potential negative plugin interactions and developer guidance as to how to avoid could be useful for others. Perhaps stefanbrausch / jochenafuerbacher / robinschulz might add such a note in a "Developer's section of the README.md

            stefanbrausch Stefan Brausch
            ianw Ian Williams
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: