jenkey do you have a backup of that instance, in order to try to reproduce the behavior ?

If it's the case, could you try to restart the instance after the plugin update ?

Could you also give us the version of your credentials plugin ?

In order to reproduce also ourself the problem, did you upgrade the plugin using the update-center ? and then you restarted your instance ?

wfollonier, sorry I don't have a backup for this instance. In the meantime, I simply copied the private key from the file, pasted it into the input box and saved the credentials.

I did install the plugin using the update-center, and I restarted Jenkins using the "Restart Jenkins if installation complete and no jobs are running" checkbox. Is that enough or should I ensure to restart the service next time?

After the upgrade and realizing that all my jobs did not work anymore, I downgraded the ssh-credentials-plugin to version 1.13, restarted and all was working again.

BTW, I am using the jenkins RPM from pkg.jenkins.io/redhat-stable.

The credentials-plugin is at version 2.1.17.

We had exactly the same issue, we are on jenkins 2.107.3, we upgraded the plugin to 1.14 and found the only option we had was 'enter directly'. Downgrading the plugin to 1.13 restored the functionality.

In [SECURITY-440 It is stated that 

SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.

Thus you need to enter your ssh-key directly. We fixed this by entering our ssh-key into an Environment Variable and loading that into the credentials at boot time.

BTW: The scope of the Credentials need to be SYSTEM not GLOBAL for this version (1.14) to work in "GitHub Organization" type jobs. Maybe this is required on other types of jobs as well.

wfollonier took a look but was not able to reproduce this issue. As far as we know the migration should work correctly. If anyone can reproduce the migration not working please reopen the ticket and comment with the steps to reproduce the issue.

Looking at JENKINS-54746 I think the exception in this comment (users report seeing this in the old data monitor) is the root cause. The migration is gated by the RunScripts permission being active when that code runs, and that code is expected to run as ACL.System, but for some reason it ran as ACL.anonymous. When the exception is thrown in readResolve, perhaps the effect is that the serialized data is totally lost and it is as if no private keys were ever entered. Perhaps this code should be modified to instead just return if that permission is not found to avoid the conversion exception.

I had the same symptom - just been watching the issues here, have not tried re-updating since the last failure. In the first iteration, I didn't pay any active attention to log during upgrade itself, only noticed the resolve errors afterwards. 

In my case, using project matrix auth. Will try to see if symptom reproduces and capture some additional logs for you if so. 

nneul That could be very very appreciated! I tried with project matrix auth without success (=migration was smooth). Could you give me insights about the structure of your instance ? Multiple folders ? restricted ones that are hiding some projects ? Credentials used at folders level ? Do you use internal credential provider or external ones ? Any information about your setup that is not "out of the box" will be useful

Gotta love when failure = working fine.

No folders, all jobs visible to anonymous, but have a handful of jobs that are restricted from execution. Didn't look at credentials related to anything within the jobs - only for the agent connections. It's possible there was impact inside the jobs as well - didn't get that far when I saw the upgrade fail. 

Aside from sitting behind apache, and being in a different directory structure the instance should be pretty vanilla. 

For the credentials for me - it's almost entirely pre-existing use of ~jenkins/.ssh/id_rsa that was affected. Very little use of built-in creds - and none at all of that for agent connections. 

This no longer happens for me, so I assumed it was found and fixed; I didn't do anything to try to fix it. You keep mentioning SYSTEM, so I guess I should point out that I was only seeing this on Linux nodes; the SSH private key used to login to them wasn't getting migrated correctly, so none of them would come up

wfollonier sorry I don't have much to contribute, as noted in the other ticket, I got it working and went on my merry way. As nneul noted above - my setup was pretty much the same. The Jenkins user on the Linux server had the ssh keys outside of Jenkins itself (in a standard Unix manner) and I had to manually copy them into the GUI.

Edit: For some reason it stripped the link from "other ticket" to https://issues.jenkins-ci.org/browse/JENKINS-54746?focusedCommentId=357252

We usually don't do upgrades.
By leveraging Docker, and JCasC, we rebuild from scratch, every time we need a new version Jenkins or a plugin. Having no state inside Jenkins, we can treat the containers as cattle. If it gets sick, here comes the bolt-pistol. As mentioned in my previous comment comment-343088, we fixed our issue by entering the SSH key "directly" and setting the proper scope.

Had a chance to try this again, and cannot reproduce now - upgrade processed smoothly with no issues. Sorry I can't provide anything further. 

I've just installed fresh Jenkins and found I can't use SSH private key from Jenkins home directory, ~/.ssh/id_rsa. As workaround, I put SSH key into Jenkins Credentials, it works.

• SSH Slaves v1.29.4,
• SSH Credentials Plugin v1.16,
• Jenkins v2.164.3,
• host and slave OS: Ubuntu 18.04.2 with all updates,
• OpenSSH v7.6p1.

Update: found this:

SSH Credentials Plugin no longer supports SSH credentials from files on the Jenkins master file system, neither user-specified file paths nor ~/.ssh. Existing SSH credentials of these kinds are migrated to "directly entered" SSH credentials.