While working on JENKINS-70000 I found (and verified with released version of the plugin) that it fails to decipher the certificate credential if used with a remote agent (the PR for JENKINS-70000 gets it to work on Jenkins Controller nicely).

I am not sure however if it is a flaw in http-request-plugin issue or that of remoting – I believe the SecretBytes with the credential contents arrive garbled but did not prove that yet.

Copying grittier details from gitter discussion:

Reproduction:

In Jenkins, register a "cert-credential-name" as an Uploaded Certificate File with the user's private key, its issuer chain, and the Trusted keys if a private CA was used (PR for JENKINS-70000 is needed to actually use this, but for the issue at hand – it fails before looking at that).

The gist of the pipeline is:

pipeline {
    agent {label "worker"}
    stages {
        stage("HTTP Req") {
            script {
                httpRequest([url: 'https://some.site.com/api/v1/login', authentication: 'cert-credential-name'])
            }
        }
    }
} 

so when the agent is "master" (or fancy new "builtin"), all is ok; when it points to another environment, it fails (deciphering the SecretBytes as supposed below).

In the plugin:

This line just at start of CertificateAuthentication.authenticate() works well with practical runs (and self-test builds) that happen on the Jenkins controller:

KeyStore keyStore = credentials.getKeyStore(); 

However when a build agent is used, I see java.io.IOException: Remote call on worker failed with these lines from the long trace seeming relevant: 

Caused by: java.lang.Error: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
    at com.cloudbees.plugins.credentials.SecretBytes.getPlainData(SecretBytes.java:142)
    at com.cloudbees.plugins.credentials.SecretBytes.getPlainData(SecretBytes.java:233)
    at com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl$UploadedKeyStoreSource.getKeyStoreBytes(CertificateCredentialsImpl.java:507)
    at com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl.getKeyStore(CertificateCredentialsImpl.java:157)
    at jenkins.plugins.http_request.auth.CertificateAuthentication.authenticate(CertificateAuthentication.java:48)
    at jenkins.plugins.http_request.HttpRequestExecution.auth(HttpRequestExecution.java:436)
    at jenkins.plugins.http_request.HttpRequestExecution.authAndRequest(HttpRequestExecution.java:378)
    at jenkins.plugins.http_request.HttpRequestExecution.call(HttpRequestExecution.java:285)
    at jenkins.plugins.http_request.HttpRequestExecution.call(HttpRequestExecution.java:81)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
...
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
    at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:859)
    at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:939)
    at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:735)
    at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2205)
    at com.cloudbees.plugins.credentials.SecretBytes.getPlainData(SecretBytes.java:140)
    ... 16 more 

My guess is that the credentials somehow did not arrive intact to the remote (an SSH Agent). Or rather, per source-reading, content of the SecretBytes object is not readable :\
Code looks at keystore password a bit later (same line, different arg).

In pipeline I pass authentication as a string (name of credential). In http-request-plugin it is looked up, and  looks at instanceof to make one of credential object types (e.g. user/pass or cert)
https://github.com/jenkinsci/http-request-plugin/blob/master/src/main/java/jenkins/plugins/http_request/HttpRequestExecution.java#L235-L247

Then https://github.com/jenkinsci/http-request-plugin/blob/master/src/main/java/jenkins/plugins/http_request/HttpRequestExecution.java#L436 passes them to an appropriate "authenticator" – originally to https://github.com/jenkinsci/http-request-plugin/blob/master/src/main/java/jenkins/plugins/http_request/auth/CertificateAuthentication.java#L28 and in my PRed case a largely expanded https://github.com/jenkinsci/http-request-plugin/blob/06dbf1272a28759484a9a2a5c1e64accc346c96e/src/main/java/jenkins/plugins/http_request/auth/CertificateAuthentication.java#L43
Actually both call credentials.getKeyStore() and fail with remote agents in it.

According to my poor-man's tracing (prints and stacktraces) so far, it fails at
https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/impl/CertificateCredentialsImpl.java#L157
to complete keyStoreSource.getKeyStoreBytes() which for the UploadedKeyStoreSource instance in question https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/impl/CertificateCredentialsImpl.java#L507 is to simply:

return SecretBytes.getPlainData(uploadedKeystoreBytes); 

and this works fine on Jenkins controller or a JenkinsRule mock, but fails on (at least) an SSH Agent with javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. around https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/SecretBytes.java#L140 as far as plugin trail goes

I did not yet get to dig out how many bytes the remote agent looks at, or if there is some CR/LF issue (should not be, this particular agent is an SSH shell into another account on same Linux box as the controller).

While SELinux exists on both controller and agent (same CentOS machine in this test), having it enforced or not has no effect.