Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61841

PathRemover methods can throw enormous CompositeIOExceptions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:
    • Released As:
      Jenkins 2.235

      Description

      Calls to Util.deleteRecursive and other methods that end up calling PathRemover.forceRemoveRecursive on large directories can end up throwing instances of CompositeIOException with a very large number of nested exceptions, leading to excessive memory usage.

      For example, I recently examined a heap dump where 40% of the ~12GB heap was occupied by 200MB instances of CompositeIOException, where each exception had ~4000 nested exceptions. In that case, the exceptions were being stored in Pipeline flow nodes via ErrorAction, so they were also being written to the file system, compounding the problem. The cause of the huge exceptions in that case was an attempt to delete a workspace with a large node_modules directory that Jenkins did not have permission to delete.

      I think it would make sense for PathRemover.forceDeleteRecursive to abort early after hitting 10 or so exceptions to avoid this kind of issue.

      Alternatively, we could keep all of the nested exception's messages, but drop the stack traces to reduce the memory footprint, but it seems very unlikely to me that additional exceptions beyond the first ~10 are going to provide any useful information that helps a user diagnose and fix the problem. Another drawback of this approach is that the stack traces for all of the exceptions would still be generated when the exceptions are created, which is a waste of CPU since we would null them out right after.

        Attachments

          Issue Links

            Activity

            dnusbaum Devin Nusbaum created issue -
            dnusbaum Devin Nusbaum made changes -
            Field Original Value New Value
            Status Open [ 1 ] In Progress [ 3 ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/jenkins#4639 (Web Link)" [ 24828 ]
            Hide
            dnusbaum Devin Nusbaum added a comment -
            Show
            dnusbaum Devin Nusbaum added a comment - PR is up:  https://github.com/jenkinsci/jenkins/pull/4639 .
            dnusbaum Devin Nusbaum made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            dnusbaum Devin Nusbaum made changes -
            Summary PathRemover should abort early to avoid creating enormous CompositeIOExceptions PathRemover methods can throw enormous CompositeIOExceptions
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Daniel Beck Recommended a different approach: Instead of aborting the deletion early, just truncate the list of exceptions if it gets too large. The benefits of this approach are that the actual deletion behavior is unchanged, and it avoids issues where one problematic subdirectory could result in a deletion operation not being able to remove a significant number of files because the problematic subdirectory is visited first.

            Show
            dnusbaum Devin Nusbaum added a comment - Daniel Beck Recommended a different approach: Instead of aborting the deletion early, just truncate the list of exceptions if it gets too large. The benefits of this approach are that the actual deletion behavior is unchanged, and it avoids issues where one problematic subdirectory could result in a deletion operation not being able to remove a significant number of files because the problematic subdirectory is visited first.
            dnusbaum Devin Nusbaum made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            Hide
            dnusbaum Devin Nusbaum added a comment -

            A PR using the new approach is up: https://github.com/jenkinsci/jenkins/pull/4655.

            Show
            dnusbaum Devin Nusbaum added a comment - A PR using the new approach is up:  https://github.com/jenkinsci/jenkins/pull/4655 .
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/jenkins#4655 (Web Link)" [ 24846 ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/jenkins#4639 (Web Link)" [ 24828 ]
            dnusbaum Devin Nusbaum made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            danielbeck Daniel Beck made changes -
            Released As Jenkins 2.235
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Closed [ 6 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            danielbeck Daniel Beck made changes -
            Labels lts-candidate

              People

              Assignee:
              dnusbaum Devin Nusbaum
              Reporter:
              dnusbaum Devin Nusbaum
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: