Jenkins discards directories that are not needed any longer all the time. Aside from the fact it is subject of different random failures (on windows, on NFS, etc.), it is also a time consuming operation.
Instead of deleting such files synchronously, I propose to create Jenkins-wide service where directories can be registered for later disposal. The service will rename the directory so it will appear deleted immediately and delete the renamed one later.
Similar approach is implemented in Workspace cleanup plugin though the solution leaks renamed directories in case the deletion fails.
Such service will track registered directories even after the deletion failed and retry as many times as necessary or possibly report to admin that there is a problem Jenkins itself cannot handle despite repeated attempts.
There are several areas that can benefit from this aside from Workspace cleanup. though, I am not sure if this behaviour should replace what FilePath#deleteRecursive currently does.