Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70101

http-request-plugin fails using Certificate authentication from a remote (SSH) agent

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None

      While working on JENKINS-70000 I found (and verified with released version of the plugin) that it fails to decipher the certificate credential if used with a remote agent (the PR for JENKINS-70000 gets it to work on Jenkins Controller nicely).

      I am not sure however if it is a flaw in http-request-plugin issue or that of remoting – I believe the SecretBytes with the credential contents arrive garbled but did not prove that yet.

      Copying grittier details from gitter discussion:

      Reproduction:

      In Jenkins, register a "cert-credential-name" as an Uploaded Certificate File with the user's private key, its issuer chain, and the Trusted keys if a private CA was used (PR for JENKINS-70000 is needed to actually use this, but for the issue at hand – it fails before looking at that).

      The gist of the pipeline is:

      pipeline {
          agent {label "worker"}
          stages {
              stage("HTTP Req") {
                  script {
                      httpRequest([url: 'https://some.site.com/api/v1/login', authentication: 'cert-credential-name'])
                  }
              }
          }
      } 

      so when the agent is "master" (or fancy new "builtin"), all is ok; when it points to another environment, it fails (deciphering the SecretBytes as supposed below).

      In the plugin:

      This line just at start of CertificateAuthentication.authenticate() works well with practical runs (and self-test builds) that happen on the Jenkins controller:

      KeyStore keyStore = credentials.getKeyStore(); 

      However when a build agent is used, I see java.io.IOException: Remote call on worker failed with these lines from the long trace seeming relevant: 

      Caused by: java.lang.Error: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
          at com.cloudbees.plugins.credentials.SecretBytes.getPlainData(SecretBytes.java:142)
          at com.cloudbees.plugins.credentials.SecretBytes.getPlainData(SecretBytes.java:233)
          at com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl$UploadedKeyStoreSource.getKeyStoreBytes(CertificateCredentialsImpl.java:507)
          at com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl.getKeyStore(CertificateCredentialsImpl.java:157)
          at jenkins.plugins.http_request.auth.CertificateAuthentication.authenticate(CertificateAuthentication.java:48)
          at jenkins.plugins.http_request.HttpRequestExecution.auth(HttpRequestExecution.java:436)
          at jenkins.plugins.http_request.HttpRequestExecution.authAndRequest(HttpRequestExecution.java:378)
          at jenkins.plugins.http_request.HttpRequestExecution.call(HttpRequestExecution.java:285)
          at jenkins.plugins.http_request.HttpRequestExecution.call(HttpRequestExecution.java:81)
          at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      ...
      Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
          at java.base/com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:859)
          at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:939)
          at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:735)
          at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
          at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2205)
          at com.cloudbees.plugins.credentials.SecretBytes.getPlainData(SecretBytes.java:140)
          ... 16 more 

      My guess is that the credentials somehow did not arrive intact to the remote (an SSH Agent). Or rather, per source-reading, content of the SecretBytes object is not readable :\
      Code looks at keystore password a bit later (same line, different arg).

      In pipeline I pass authentication as a string (name of credential). In http-request-plugin it is looked up, and  looks at instanceof to make one of credential object types (e.g. user/pass or cert)
      https://github.com/jenkinsci/http-request-plugin/blob/master/src/main/java/jenkins/plugins/http_request/HttpRequestExecution.java#L235-L247

      Then https://github.com/jenkinsci/http-request-plugin/blob/master/src/main/java/jenkins/plugins/http_request/HttpRequestExecution.java#L436 passes them to an appropriate "authenticator" – originally to https://github.com/jenkinsci/http-request-plugin/blob/master/src/main/java/jenkins/plugins/http_request/auth/CertificateAuthentication.java#L28 and in my PRed case a largely expanded https://github.com/jenkinsci/http-request-plugin/blob/06dbf1272a28759484a9a2a5c1e64accc346c96e/src/main/java/jenkins/plugins/http_request/auth/CertificateAuthentication.java#L43
      Actually both call credentials.getKeyStore() and fail with remote agents in it.

      According to my poor-man's tracing (prints and stacktraces) so far, it fails at
      https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/impl/CertificateCredentialsImpl.java#L157
      to complete keyStoreSource.getKeyStoreBytes() which for the UploadedKeyStoreSource instance in question https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/impl/CertificateCredentialsImpl.java#L507 is to simply:

      return SecretBytes.getPlainData(uploadedKeystoreBytes); 

      and this works fine on Jenkins controller or a JenkinsRule mock, but fails on (at least) an SSH Agent with javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. around https://github.com/jenkinsci/credentials-plugin/blob/master/src/main/java/com/cloudbees/plugins/credentials/SecretBytes.java#L140 as far as plugin trail goes

      I did not yet get to dig out how many bytes the remote agent looks at, or if there is some CR/LF issue (should not be, this particular agent is an SSH shell into another account on same Linux box as the controller).

      While SELinux exists on both controller and agent (same CentOS machine in this test), having it enforced or not has no effect.

          [JENKINS-70101] http-request-plugin fails using Certificate authentication from a remote (SSH) agent

          Jim Klimov added a comment -

          According to research hinted by slide_o_mix the problem might actually be in credentials-plugin. The http-request-plugin diligently makes a snapshot() of the credential it would use, which supposedly goes to find if there is a suitable CredentialsSnapshotTaker at https://github.com/jenkinsci/credentials-plugin/blob/e05618c41e623cd974eef36b59e7c5bbda2a23c1/src/main/java/com/cloudbees/plugins/credentials/CredentialsProvider.java#L807

          But it seems that for CertificateCredentialsImpl => UploadedKeyStoreSource there is no such taker, so it returns the original credential.

          My random guess would be that its SecretBytes of the keyStore rely on something known to the Jenkins controller but not to its agents (e.g. likes of the master.key file or some such)?

          Jim Klimov added a comment - According to research hinted by slide_o_mix the problem might actually be in credentials-plugin. The http-request-plugin diligently makes a snapshot() of the credential it would use, which supposedly goes to find if there is a suitable CredentialsSnapshotTaker at https://github.com/jenkinsci/credentials-plugin/blob/e05618c41e623cd974eef36b59e7c5bbda2a23c1/src/main/java/com/cloudbees/plugins/credentials/CredentialsProvider.java#L807 But it seems that for CertificateCredentialsImpl => UploadedKeyStoreSource there is no such taker, so it returns the original credential. My random guess would be that its SecretBytes of the keyStore rely on something known to the Jenkins controller but not to its agents (e.g. likes of the master.key file or some such)?

          Jim Klimov added a comment - - edited

          FWIW, with some instrumentation got data dumps of SecretBytes on local and remote agents. Indeed, the latter is garbled although seems to be the same size of array: https://pastebin.com/VhsYmGcg and https://pastebin.com/krreh5bL (hex version)

          Wondering if there is any pattern to it, or just the bytes are not copied at all from controller to agent.

          Jim Klimov added a comment - - edited FWIW, with some instrumentation got data dumps of SecretBytes on local and remote agents. Indeed, the latter is garbled although seems to be the same size of array: https://pastebin.com/VhsYmGcg  and https://pastebin.com/krreh5bL (hex version) Wondering if there is any pattern to it, or just the bytes are not copied at all from controller to agent.

          Jim Klimov added a comment - - edited

          So after a few rounds at that, I found that every run creates a different encrypted content so comparing bytes from two runs is not useful - only sizes (and padding?) match.

          Augmented the test pipelines to report the SecretBytes contents while querying "directly" (on the controller as pipeline code goes) and via `httpRequest` step, and the contents are indeed the same. There's just one uneasy part of the equation: private static final CredentialsConfidentialKey KEY = new CredentialsConfidentialKey(SecretBytes.class, "KEY");this field of SecretBytes may be tied to the controller (its master.key file, etc.) and different on the agent, something I expected and overlooked...

          UPDATE: not master key, but still an individual secret generated for each JVM this class is instantiated in.

          Jim Klimov added a comment - - edited So after a few rounds at that, I found that every run creates a different encrypted content so comparing bytes from two runs is not useful - only sizes (and padding?) match. Augmented the test pipelines to report the SecretBytes contents while querying "directly" (on the controller as pipeline code goes) and via `httpRequest` step, and the contents are indeed the same. There's just one uneasy part of the equation: private static final CredentialsConfidentialKey KEY = new CredentialsConfidentialKey(SecretBytes. class, "KEY" ) ; – this field of SecretBytes may be tied to the controller (its master.key file, etc.) and different on the agent, something I expected and overlooked... UPDATE: not master key, but still an individual secret generated for each JVM this class is instantiated in.

          Jim Klimov added a comment -

          Further thanks to mawinter69 for ideas and analysis, the problem was in credentials-plugin. Namely, it (IMHO overly zealously) treated SECURITY-1322 by not only neutering the FileOnMasterKeyStoreSource, but also removing the snapshot taker implementation.

          Reviving this ability (and passing snapshots as Secret rather than SecretBytes if snapshotting for a Channel) fixes the issue even with the currently released http-request-plugin 1.16 so is independent of my improvements pursued in JENKINS-70000.

          Preparing a PR...

          Jim Klimov added a comment - Further thanks to mawinter69 for ideas and analysis, the problem was in credentials-plugin. Namely, it (IMHO overly zealously) treated SECURITY-1322 by not only neutering the FileOnMasterKeyStoreSource , but also removing the snapshot taker implementation. Reviving this ability (and passing snapshots as Secret rather than SecretBytes if snapshotting for a Channel ) fixes the issue even with the currently released http-request-plugin 1.16 so is independent of my improvements pursued in JENKINS-70000 . Preparing a PR...

          Jim Klimov added a comment -

          Jim Klimov added a comment - https://github.com/jenkinsci/credentials-plugin/pull/391 posted

            jimklimov Jim Klimov
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: