Code on the Jenkins plugin side responsible for label removing has been commented out during cleanup.
https://github.com/jenkinsci/swarm-plugin/blame/master/plugin/src/main/java/hudson/plugins/swarm/PluginImpl.java#L144

To reproduce:
run swarm jar with -labelsFile option, modify label string in the file (add a new label and remove some existing label), wait for 10 seconds, go to the agent's page in Jenkins.
The new label will appear, but the deleted one will remain.