Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71696

Failing agents cause controller memory leak

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • kubernetes-plugin
    • None
    • Jenkins version: 2.346.3
      Plugin version: 3743.v1fa_4c724c3b_7

      We ran into an issue where a single pipeline, attempting to run on an agent that had an issue and errored, caused a thread and memory leak on the controller, eventually resulting in an OOMKill of the controller.

      I'm not sure it's relevant, but the agent errored due to trying to access root level directories, after having pod settings applied to restrict that.

      I've attached graphs which show the ever increasing thread count and memory usage. I've also included screen shots of a thread dump analysis showing the thread counts before and after this issue occurred.

       

        1. Memory Usage.png
          68 kB
          Adam Dougal
        2. Thread Count.png
          34 kB
          Adam Dougal
        3. Thread Dump After-1.png
          99 kB
          Adam Dougal
        4. Thread Dump After 2.png
          85 kB
          Adam Dougal
        5. Thread Dump Before.png
          91 kB
          Adam Dougal
        6. Thread Dump Before 2.png
          84 kB
          Adam Dougal

            Unassigned Unassigned
            adamdougal Adam Dougal
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: