The Jenkins SSH agent plugin uses trilead-ssh which only supports the legacy PEM and new SSH2 private key file formats. In certain hardened environments, SSH keys are mandated to be encoded in PKCS#8 (presumably because they're more universal and allow for password protection and encrypting the key) to the point where OpenSSH and ssh-keygen are patched to only support PKCS#8 private keys.

Attached to this ticket are various encodings of private keys as we troubleshooted the initial issue described below. Note that private keys that have a header line with BEGIN PRIVATE KEY are indicative of the file being encoded in PKCS#8. If the file header contains BEGIN RSA PRIVATE KEY (or EC or whatever other algorithm), that is the legacy PEM format. If the file header contains BEGIN OPENSSH PRIVATE KEY, then that is the new SSH2 format. The latter two encoding formats are already supported in trilead-ssh.

Original Details

• My Jenkins exists on an Amazon EC2 instance.
• I'm trying to add Red Hat 7.7 slave via ssh.
• I can ssh directly using terminal from my master to slave and vice versa using that key file

jenkins@master:/var/lib/jenkins
$ ssh -i .ssh/id_rsa -q 10.193.177.232

jenkins@slave:/var/lib/jenkins
$ ssh -i .ssh/id_rsa -q 10.193.177.209

I’m also able to make a successful SSH connection when I perform a SSH connection test from the Manage Jenkins -> Configure System section of the Master (FYI - see attached screenshot).

But when I try to configure my slave via launch agent method, it gives me the following error:

caused by: java.io.IOException: PEM problem: it is of unknown type
        at com.trilead.ssh2.crypto.PEMDecoder.decpdeKeyPair(PEMDecoder.java:500)

• How I've configured the slave (one of the method)?
  • I've saved slave machine credentials as "ssh key username with private key"
  • Launch method: Launch agents via SSH
  • Hostname: slave machine private ip address.
  • Host Key Verification Strategy: Manually trusted key Verification Strategy

Note:

• I cannot regenerate ssh keys as I'll not be able to access my machines again and the infrastructure is very strict. 
• All SSH key are 4096 bits in length and RSA.
• I cannot use credentials as username/password as either there is no password.
• I've tried everything whatever mentioned in other jenkins tickets for same issue or on internet, but my issue is not getting resolved.

Please see the following attachments for log details and screenshots.

This is a total blocker. Please advise. Thank you.