While we weren't paying close attention, the changelog-history-plugin created 1 million entries in the config-history/nodes folder. The reason is apparently that we have been allocating slaves from a cloud/swarm that live for a few minutes and then are reclaimed.
For example here is a snippet of some of the directories created:
The release notes for this plugin contain the following comments:
- Avoid calling User.current() during Jenkins initialization (Thanks to Thomas de Grenier de Latour)
- Avoid tracking changes for cloud slave as well (Thanks to Ryan Campbel)
- Fix proposal for NPE in ComputerHistoryListener.onConfigurationChange method (Thanks to William Bernardet)
JENKINS-22639) Don't record changes to AbstractCloudSlaves or Ephemeral Nodes (Thanks to Ryan Campbell and Jesse Click)
I suspect either these changes were reverted somehow, or it is not covering our case.
We are worried that this behavior may be one of the reasons why:
1. Our lockable resources / throttling is no longer working and
2. we end up with apparent scheduler deadlocks (spun up slaves cannot be connected to before they get reclaimed).
These last two items are pure conjecture, but based on these JIRAs: