Upgrading ant - and more importantly, removing older versions of ant - isn't entirely clear to me. Does removing an instance of the Apache Ant installer remove associated binaries from the system?
For the other installers (extract archive, batch script, shell script), I can see "uninstall" instructions to be difficult or impossible to implement.
I think they're cached in a Jenkins specific directory and overwritten when replaced when the same "tool name" is changed to point to a different URL, and used in a build. (It's a lot less work for a Jenkins admin to figure out in their existing setup than me to replicate now and see just to be able to answer you).
If you're fine with them not being used anymore, reconfigure (to safe versions) or delete tool installations. This may break builds that are configured to use them and the version is incompatible (former) or tool name not found (the latter).
If you want them to be gone from filesystems, a Groovy script may work to delete on all connected agents. https://www.jenkins.io/doc/book/managing/script-console/ for docs and numerous caveats.
Attached a screenshot of what admins see. It seems you haven't tried using the plugin?