Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61841

PathRemover methods can throw enormous CompositeIOExceptions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:
    • Released As:
      Jenkins 2.235

      Description

      Calls to Util.deleteRecursive and other methods that end up calling PathRemover.forceRemoveRecursive on large directories can end up throwing instances of CompositeIOException with a very large number of nested exceptions, leading to excessive memory usage.

      For example, I recently examined a heap dump where 40% of the ~12GB heap was occupied by 200MB instances of CompositeIOException, where each exception had ~4000 nested exceptions. In that case, the exceptions were being stored in Pipeline flow nodes via ErrorAction, so they were also being written to the file system, compounding the problem. The cause of the huge exceptions in that case was an attempt to delete a workspace with a large node_modules directory that Jenkins did not have permission to delete.

      I think it would make sense for PathRemover.forceDeleteRecursive to abort early after hitting 10 or so exceptions to avoid this kind of issue.

      Alternatively, we could keep all of the nested exception's messages, but drop the stack traces to reduce the memory footprint, but it seems very unlikely to me that additional exceptions beyond the first ~10 are going to provide any useful information that helps a user diagnose and fix the problem. Another drawback of this approach is that the stack traces for all of the exceptions would still be generated when the exceptions are created, which is a waste of CPU since we would null them out right after.

        Attachments

          Issue Links

            Activity

            Hide
            dnusbaum Devin Nusbaum added a comment -
            Show
            dnusbaum Devin Nusbaum added a comment - PR is up:  https://github.com/jenkinsci/jenkins/pull/4639 .
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Daniel Beck Recommended a different approach: Instead of aborting the deletion early, just truncate the list of exceptions if it gets too large. The benefits of this approach are that the actual deletion behavior is unchanged, and it avoids issues where one problematic subdirectory could result in a deletion operation not being able to remove a significant number of files because the problematic subdirectory is visited first.

            Show
            dnusbaum Devin Nusbaum added a comment - Daniel Beck Recommended a different approach: Instead of aborting the deletion early, just truncate the list of exceptions if it gets too large. The benefits of this approach are that the actual deletion behavior is unchanged, and it avoids issues where one problematic subdirectory could result in a deletion operation not being able to remove a significant number of files because the problematic subdirectory is visited first.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            A PR using the new approach is up: https://github.com/jenkinsci/jenkins/pull/4655.

            Show
            dnusbaum Devin Nusbaum added a comment - A PR using the new approach is up:  https://github.com/jenkinsci/jenkins/pull/4655 .

              People

              Assignee:
              dnusbaum Devin Nusbaum
              Reporter:
              dnusbaum Devin Nusbaum
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: