We ran into an issue where a single pipeline, attempting to run on an agent that had an issue and errored, caused a thread and memory leak on the controller, eventually resulting in an OOMKill of the controller.

I'm not sure it's relevant, but the agent errored due to trying to access root level directories, after having pod settings applied to restrict that.

I've attached graphs which show the ever increasing thread count and memory usage. I've also included screen shots of a thread dump analysis showing the thread counts before and after this issue occurred.