There is a migration process on startup that checks for all items in Jenkins and add a Notifier if it does not exist yet: https://github.com/jenkinsci/hipchat-plugin/blob/master/src/main/java/jenkins/plugins/hipchat/upgrade/ConfigurationMigrator.java#L29

This process can take a very long time especially on large instances. For example, for an instance of about 6000 Abstract projects / ~ 3000 folders, the log startup performance loggers show a time of almost 5 minutes:

> 296654:for item listener jenkins.plugins.hipchat.upgrade.ConfigurationMigrator startup

Removing the plugin saves a bit less than 5 minutes on startup.

Note: This could be worth with plugins that implements ItemListener - like the Job Config History plugin that records any changes on Jenkins items.