I'm sorry, I keep finding bugs in my own code...
I noticed that the swarm client wouldn't recognize all tools, some were added as Ant installations. That's because Ant is the first item in the dropdown.
Turns out the way I generate the toolinstallation key is not entirely correct. The keys that I have encountered that don't work are those for Ant and Maven. In my tests I always used Groovy, JDK and Ant...Ant seemed to work because it default to the first item in the listbox, being Ant..

I have asked for some help here: https://groups.google.com/forum/#!topic/jenkinsci-dev/QwYKkMvtw9s