Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55219

Memory leak in logfilesizechecker-plugin due to its multi-instantiation of SafeTimerTasks

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
      build log file size checker: 1.4
      Jenkins ver. 2.121.2 (latest probably affected as well)
      Free-style job
    • Similar Issues:

      Description

      The log file size checker plugin uses an internal class LogSizeTimerTask that inherits from SafeTimerTask. The LogSizeTimerTask have references to build and listener.
      Unfortunantly, the LogSizeTimerTask is only cancelled, never completely cleaned up, resulting in one instance of the LogSizeTimerTask being present in the thread pool execution queue for every build that make use of the plugin.
      Due to the reference to the build and the listener, those objects are never garbage collected, resulting in every build that was ever built with this plugin active will be kept in memory.
      This becomes even more problematic if the build has references to other large objects preventing them as well from being GCed.

      Our jenkins master had +10GB of objects that could not be GCed due to this issue, with hundred of thousands of LogSizeTimerTasks in the thread pool execution queue.

        Attachments

          Activity

          per_bohlin Per Böhlin created issue -
          per_bohlin Per Böhlin made changes -
          Field Original Value New Value
          Description The log file size checker plugin uses an internal class LogSizeTimerTask that inherits from SafeTimerTask. The LogSizeTimerTask have references to {{build}} and {{lister}}.
          Unfortunantly, the LogSizeTimerTask is only cancelled, never completely cleaned up, resulting in one instance of the LogSizeTimerTask being present in the thread pool execution queue for every build that make use of the plugin.
          Due to the reference to the build and the listener, those objects are never garbage collected, resulting in every build that was ever built with this plugin active will be kept in memory.
          This becomes even more problematic if the build has references to other large objects preventing them as well from being GCed.

          Our jenkins master had +10GB of objects that could not be GCed due to this issue, with hundred of thousands of LogSizeTimerTasks in the thread pool execution queue.
          The log file size checker plugin uses an internal class LogSizeTimerTask that inherits from SafeTimerTask. The LogSizeTimerTask have references to {{build}} and {{listener}}.
          Unfortunantly, the LogSizeTimerTask is only cancelled, never completely cleaned up, resulting in one instance of the LogSizeTimerTask being present in the thread pool execution queue for every build that make use of the plugin.
          Due to the reference to the build and the listener, those objects are never garbage collected, resulting in every build that was ever built with this plugin active will be kept in memory.
          This becomes even more problematic if the build has references to other large objects preventing them as well from being GCed.

          Our jenkins master had +10GB of objects that could not be GCed due to this issue, with hundred of thousands of LogSizeTimerTasks in the thread pool execution queue.
          Hide
          per_bohlin Per Böhlin added a comment -

          I believe that the root cause of this issue is that the plugin instantiate SafeTimerTask (LogSizeTimerTask) for every build. The SafeTimerTask was probably not designed for that usecase since it is never completely cleaned up.
          The quick fix is to remove the references to build and listener once the task is cancelled, minimizing the damage, but the correct fix would be to rewrite the plugin to only use one global instance of the SafeTimerTask and in that loop over all affected builds and check for log-size.

          Show
          per_bohlin Per Böhlin added a comment - I believe that the root cause of this issue is that the plugin instantiate SafeTimerTask (LogSizeTimerTask) for every build. The SafeTimerTask was probably not designed for that usecase since it is never completely cleaned up. The quick fix is to remove the references to build and listener once the task is cancelled, minimizing the damage, but the correct fix would be to rewrite the plugin to only use one global instance of the SafeTimerTask and in that loop over all affected builds and check for log-size.

            People

            Assignee:
            stefanbrausch Stefan Brausch
            Reporter:
            per_bohlin Per Böhlin
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: