This issue has been converted to EPIC, because there are reports of various independent issues inside.

Issue:

• Remoting threadPool is being widely used in Jenkins: https://github.com/search?q=org%3Ajenkinsci+threadPoolForRemoting&type=Code
• Not all usages of Computer.threadPoolForRemoting are valid for starters
• Computer.threadPoolForRemoting has downscaling logic, threads get killed after 60-second timeout
• The pool has no thread limit by default, so it may grow infinitely until number of threads kills JVM or causes OOM
• Some Jenkins use-cases cause burst Computer.threadPoolForRemoting load by design (e.g. Jenkins startup or agent reconnection after the issue)
• Deadlocks or waits in the threadpool may also make it to grow infinitely

Proposed fixes:

• Define usage policy for this thread pool in the documentation
• Limit number of threads being created depending on the system scale, make the limit configurable (256 by default?)
• Fix the most significant issues where the thread pool gets misused or blocked
   
  Original report (tracked as JENKINS-47012):

> After some period of time the Jenkins master will have up to ten thousand or so threads most of which are Computer.theadPoolForRemoting threads that have leaked. This forces us to restart the Jenkins master.

> We do add and delete slave nodes frequently (thousands per day per master) which I think may be part of the problem.

> I thought https://github.com/jenkinsci/ssh-slaves-plugin/commit/b5f26ae3c685496ba942a7c18fc9659167293e43 may be the fix because stacktraces indicated threads are hanging in the plugins afterDisconnect() method. I have updated half of our Jenkins masters to ssh-slaves plugin version 1.9 which includes that change, but early today we had a master with ssh-slaves plugin fall over from this issue.

> Unfortunately I don't have any stacktraces handy (we had to force reboot the master today), but will update this bug if we get another case of this problem. Hoping that by filing it with as much info as I can we can at least start to diagnose the problem.