If you try to bind to a used port, you see this in the manage jenkins page:

Failed to listen to incoming slave connection. <a href='configure'>Change the port number</a> to solve the problem.See the log for more details.

It should link to configureSecurity instead.