Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61841

PathRemover methods can throw enormous CompositeIOExceptions

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:
    • Released As:
      Jenkins 2.235

      Description

      Calls to Util.deleteRecursive and other methods that end up calling PathRemover.forceRemoveRecursive on large directories can end up throwing instances of CompositeIOException with a very large number of nested exceptions, leading to excessive memory usage.

      For example, I recently examined a heap dump where 40% of the ~12GB heap was occupied by 200MB instances of CompositeIOException, where each exception had ~4000 nested exceptions. In that case, the exceptions were being stored in Pipeline flow nodes via ErrorAction, so they were also being written to the file system, compounding the problem. The cause of the huge exceptions in that case was an attempt to delete a workspace with a large node_modules directory that Jenkins did not have permission to delete.

      I think it would make sense for PathRemover.forceDeleteRecursive to abort early after hitting 10 or so exceptions to avoid this kind of issue.

      Alternatively, we could keep all of the nested exception's messages, but drop the stack traces to reduce the memory footprint, but it seems very unlikely to me that additional exceptions beyond the first ~10 are going to provide any useful information that helps a user diagnose and fix the problem. Another drawback of this approach is that the stack traces for all of the exceptions would still be generated when the exceptions are created, which is a waste of CPU since we would null them out right after.

        Attachments

          Issue Links

            Activity

            dnusbaum Devin Nusbaum created issue -
            dnusbaum Devin Nusbaum made changes -
            Field Original Value New Value
            Status Open [ 1 ] In Progress [ 3 ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/jenkins#4639 (Web Link)" [ 24828 ]
            dnusbaum Devin Nusbaum made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            dnusbaum Devin Nusbaum made changes -
            Summary PathRemover should abort early to avoid creating enormous CompositeIOExceptions PathRemover methods can throw enormous CompositeIOExceptions
            dnusbaum Devin Nusbaum made changes -
            Status In Review [ 10005 ] In Progress [ 3 ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/jenkins#4655 (Web Link)" [ 24846 ]
            dnusbaum Devin Nusbaum made changes -
            Remote Link This issue links to "jenkinsci/jenkins#4639 (Web Link)" [ 24828 ]
            dnusbaum Devin Nusbaum made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            danielbeck Daniel Beck made changes -
            Released As Jenkins 2.235
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Closed [ 6 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            danielbeck Daniel Beck made changes -
            Labels lts-candidate

              People

              Assignee:
              dnusbaum Devin Nusbaum
              Reporter:
              dnusbaum Devin Nusbaum
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: