There is a support for Direct connection in Remoting: https://github.com/jenkinsci/remoting/blob/master/docs/inbound-agent.md#connect-directly-to-tcp-port , This connection does not use REST API call, and hence it offers more performance.

The problem is that there are multiple parameters to be configured (host/port, instance identity, etc.). Kubernetes plugin automates it starting from 2.20.2: https://github.com/jenkinsci/kubernetes-plugin/releases/tag/kubernetes-1.20.2 . It would be great to have a similar feature in Docker Plugin