I changed the Gerrit hostname and path of the SSH private key and noted successful connection when clicking the 'Test Connection' button. However, after saving the new configuration, jenkins.log started to get spammed (see below). I was able to add a new server entry and remove the old one to work around this. I assume that if I restarted Jenkins the new configuration would have been used.

Feb 20, 2016 8:42:51 PM com.sonymobile.tools.gerrit.gerritevents.GerritConnection connect
SEVERE: Could not connect to Gerrit server! Host: <old_hostname> Port: 22
Feb 20, 2016 8:42:51 PM com.sonymobile.tools.gerrit.gerritevents.GerritConnection connect
SEVERE: Proxy:
Feb 20, 2016 8:42:51 PM com.sonymobile.tools.gerrit.gerritevents.GerritConnection connect
SEVERE: User: <old user> KeyFile: <old key file>
Feb 20, 2016 8:42:51 PM com.sonymobile.tools.gerrit.gerritevents.GerritConnection connect
SEVERE: IOException:
com.sonymobile.tools.gerrit.gerritevents.ssh.SshException: com.jcraft.jsch.JSchException: java.io.FileNotFoundException: <old key file> (No such file or directory)
at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionImpl.connect(SshConnectionImpl.java:181)
at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionFactory.getConnection(SshConnectionFactory.java:98)
at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.connect(GerritConnection.java:380)
at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.run(GerritConnection.java:306)
Caused by: com.jcraft.jsch.JSchException: java.io.FileNotFoundException: <old key file> (No such file or directory)
at com.jcraft.jsch.KeyPair.load(KeyPair.java:524)
at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40)
at com.jcraft.jsch.JSch.addIdentity(JSch.java:389)
at com.jcraft.jsch.JSch.addIdentity(JSch.java:370)
at com.sonymobile.tools.gerrit.gerritevents.ssh.SshConnectionImpl.connect(SshConnectionImpl.java:149)
... 3 more
Caused by: java.io.FileNotFoundException: <old key file> (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at com.jcraft.jsch.Util.fromFile(Util.java:492)
at com.jcraft.jsch.KeyPair.load(KeyPair.java:521)
... 7 more