Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58339

Support new types of credentials

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      0.1.0

      Description

      Hello!

      I've created a [fork|https://github.com/Quaiks/aws-secrets-manager-credentials-provider-plugin] and I am adding new type of credentials.

      The principle is simple: add a tag in AWS with the secret type and store a JSON as value. I am looking for some feedback of the implementation. Right now only happy paths are implemented but I would like to know if I am following the proper path.

       

      Thanks in advance!

        Attachments

          Activity

          quaiks Enrique Fernández-Polo created issue -
          quaiks Enrique Fernández-Polo made changes -
          Field Original Value New Value
          Description Hello!

          I've created a [fork|[https://github.com/Quaiks/aws-secrets-manager-credentials-provider-plugin]] and I am adding new type of credentials.

          The principle is simple: add a tag in AWS with the secret type and store a JSON as value. I am looking for some feedback of the implementation. Right now only happy paths are implemented but I would like to know if I am following the proper path.

           

          Thanks in advance!
          Hello!

          I've created a [fork|[https://github.com/Quaiks/aws-secrets-manager-credentials-provider-plugin]] and I am adding new type of credentials.

          The principle is simple: add a tag in AWS with the secret type and store a JSON as value. I am looking for some feedback of the implementation. Right now only happy paths are implemented but I would like to know if I am following the proper path.

           

          Thanks in advance!
          Hide
          chriskilding Chris Kilding added a comment -

          Hi Enrique,

          This is a feature I’ve been musing on for a little while. As I see it, there’s a few different solutions possible.

          I initially developed the plugin with just the Secret Text credential Type for the following reasons:

          1. It was easier to build for an initial release (obviously).
          2. Travis CI, Circle CI, and some other CI products only have a notion of environment variables and encrypted environment variables (equivalent to Jenkins Secret Text credentials). They’ve been able to get along fine with these, and haven’t needed to add extra credential types, which suggests that maybe Jenkins is over-complicating credentials (when Secret Text alone would do the job). (Of course, if you really need an SSH key, it’s still possible in Travis/Circle - you can base64-encode the SSH key and store it in an encrypted env var.)
          3. It makes storage and retrieval of credentials a lot easier if you know that a credential can only be a simple arbitrary string (or byte array).
          4. It enables a secret to be saved once in AWS and then shared across any consuming application - not just Jenkins.

          Point 4 is probably the strongest reason not to support multiple credential types in the AWS Secrets Manager provider. If we invent a Jenkins-proprietary schema for complex credential types eg SSH key (username, key, passphrase (optional)), only Jenkins will understand it, and we’ll have to either duplicate that credential for our other applications, or update them all to understand the Jenkins credential schema.

          However... this is a tricky architectural problem. There’s probably not a single perfect answer to it. I’d be interested to hear other views on this

          Show
          chriskilding Chris Kilding added a comment - Hi Enrique, This is a feature I’ve been musing on for a little while. As I see it, there’s a few different solutions possible. I initially developed the plugin with just the Secret Text credential Type for the following reasons: It was easier to build for an initial release (obviously). Travis CI, Circle CI, and some other CI products only have a notion of environment variables and encrypted environment variables (equivalent to Jenkins Secret Text credentials). They’ve been able to get along fine with these, and haven’t needed to add extra credential types, which suggests that maybe Jenkins is over-complicating credentials (when Secret Text alone would do the job). (Of course, if you really need an SSH key, it’s still possible in Travis/Circle - you can base64-encode the SSH key and store it in an encrypted env var.) It makes storage and retrieval of credentials a lot easier if you know that a credential can only be a simple arbitrary string (or byte array). It enables a secret to be saved once  in AWS and then shared across any consuming application - not just Jenkins. Point 4 is probably the strongest reason not to support multiple credential types in the AWS Secrets Manager provider. If we invent a Jenkins-proprietary schema for complex credential types eg SSH key (username, key, passphrase (optional)), only Jenkins will understand it, and we’ll have to either duplicate that credential for our other applications, or update them all to understand the Jenkins credential schema. However... this is a tricky architectural problem. There’s probably not a single perfect answer to it. I’d be interested to hear other views on this
          Hide
          mathewpeterson Mathew Peterson added a comment - - edited

          Hello,

          I first wanted to say thank you for putting this plugin together.

          Secondly, this implementation is different than how we have tried to solve this so, maybe my suggestion is incompatible.

          We are using a very crude method to handle pulling secrets from AWS Secrets Manager and creating the Jenkins Credentials for them. Our solution is to create a file ("init.groovy.d/credential-x.groovy") which executes aws cli command to retrieve the secret value for the supplied secret ARN. This file is generated from an Ansible template where we pass in the credential name and ARN, such that there exists one file per credential. In Ansible we have different templates for the different credential implementations, e.g. "AWSCredentialsImpl", "StringCredentialsImpl", "BasicSSHUserPrivateKey", etc. The downside to this is that the secret is only updated once Jenkins reboots or the script gets ran manual via Script Console or similar. 

          To restate, we have a specific ARN mapped to a specific Credential Implementation.

          Again, I know this is somewhat at odds to this plugin implementation but I think it could be beneficial to allow the end user to create the mappings of ARN and then have a schema for each Implementation. 

          Show
          mathewpeterson Mathew Peterson added a comment - - edited Hello, I first wanted to say thank you for putting this plugin together. Secondly, this implementation is different than how we have tried to solve this so, maybe my suggestion is incompatible. We are using a very crude method to handle pulling secrets from AWS Secrets Manager and creating the Jenkins Credentials for them. Our solution is to create a file ("init.groovy.d/credential-x.groovy") which executes aws cli command to retrieve the secret value for the supplied secret ARN. This file is generated from an Ansible template where we pass in the credential name and ARN, such that there exists one file per credential. In Ansible we have different templates for the different credential implementations, e.g. "AWSCredentialsImpl", "StringCredentialsImpl", "BasicSSHUserPrivateKey", etc. The downside to this is that the secret is only updated once Jenkins reboots or the script gets ran manual via Script Console or similar.  To restate, we have a specific ARN mapped to a specific Credential Implementation. Again, I know this is somewhat at odds to this plugin implementation but I think it could be beneficial to allow the end user to create the mappings of ARN and then have a schema for each Implementation. 
          Hide
          quaiks Enrique Fernández-Polo added a comment - - edited

          Hello!

          Chris Kilding We like to use the different types of credentials that Jenkins support because we think Jenkins provides some security measures when using their secrets natively. Avoid accessing files, avoid printing strings and so on. There are some plugins which are expecting a specific type of credential, user and password for bitbucket, Slack token and some others. We have a lot of processes sharing credentials with Jenkins, we are using [summon|https://cyberark.github.io/summon/] so everything is just an environment variable. I am going to continue working in the plugin supporting a lot of different credentials just for us, or creating a new plugin.

           

          Mathew Peterson We we want to avoid to store the credentials in Jenkins. We prefer to retrieve the credential every time we need it. 

           

          We are still playing around with the credentials. Our project is in a highly regulated market and we are still analyzing the best way to store our credentials and how to retrieve them.

          Show
          quaiks Enrique Fernández-Polo added a comment - - edited Hello! Chris Kilding  We like to use the different types of credentials that Jenkins support because we think Jenkins provides some security measures when using their secrets natively. Avoid accessing files, avoid printing strings and so on. There are some plugins which are expecting a specific type of credential, user and password for bitbucket, Slack token and some others. We have a lot of processes sharing credentials with Jenkins, we are using [summon| https://cyberark.github.io/summon/ ] so everything is just an environment variable. I am going to continue working in the plugin supporting a lot of different credentials just for us, or creating a new plugin.   Mathew Peterson We we want to avoid to store the credentials in Jenkins. We prefer to retrieve the credential every time we need it.    We are still playing around with the credentials. Our project is in a highly regulated market and we are still analyzing the best way to store our credentials and how to retrieve them.
          Hide
          chriskilding Chris Kilding added a comment - - edited

          Hi Enrique,

          Ah yes, I had overlooked that some plugins might only be able to handle certain credential types. The weight of legacy I suppose.

          I’m thinking of a way of supporting multiple credential types that would at least keep the important part of the credential (the Secret) compatible with non-Jenkins apps like it is today.

          Something like the following:

          • The secretString remains like it is now - a simple string (or byte array). Not JSON.
          • The extra fields (eg username for SSH key or X509 cert) are added as tags on the AWS Secret. This is fine, because most of those extra fields should be non-secret anyway.
          • The only secret extra field I’m aware of is a passphrase on a private key or cert. This, however, is only really relevant for local on-disk credentials stores - in a remote store like Secrets Manager the passphrase loses its protective power, as it would have to be stored inside... the key it’s protecting, so if an attacker got the key out of SM then they’d have got the passphrase with it. Therefore we’d just populate the passphrase field of the Credential constructor with blank or null.
          • When a Secret is loaded we determine the type by parsing the secretString and tags, maybe with a chain pattern or ADT: If the secretString contains the SSH ASCII armor construct an SSH key credential (get username from tags), if it contains the X509 header etc then construct a client cert credential (get username from tags), and so on.

          With this approach, the core bit of the Secret is still readable by any application using Secrets Manager (not just Jenkins), but the extra non-secret bits can be stored there for Jenkins’ sake. And we don’t need an artificial ‘type’ field/string to be stored either - the chain decoder doesn’t need it.

          Off the top of my head this should work for:

          • SSH key (secretString format test + AWS tags)
          • Client cert (secretString format test + AWS tags)
          • Username / password (if doesn’t pass any secretString format tests, and there’s a username field in the AWS tags)
          • Secret text (if didn’t match any of the above criteria)

          Thoughts? (This is only a first take on the idea, and there might be other credentials types that I’ve overlooked.)

          Show
          chriskilding Chris Kilding added a comment - - edited Hi Enrique, Ah yes, I had overlooked that some plugins might only be able to handle certain credential types. The weight of legacy I suppose. I’m thinking of a way of supporting multiple credential types that would at least keep the important part of the credential (the Secret) compatible with non-Jenkins apps like it is today. Something like the following: The secretString remains like it is now - a simple string (or byte array). Not JSON. The extra fields (eg username for SSH key or X509 cert) are added as tags on the AWS Secret. This is fine, because most of those extra fields should be non-secret anyway. The only secret extra field I’m aware of is a passphrase on a private key or cert. This, however, is only really relevant for local on-disk credentials stores - in a remote store like Secrets Manager the passphrase loses its protective power, as it would have to be stored inside... the key it’s protecting, so if an attacker got the key out of SM then they’d have got the passphrase with it. Therefore we’d just populate the passphrase field of the Credential constructor with blank or null. When a Secret is loaded we determine the type by parsing the secretString and tags, maybe with a chain pattern or ADT: If the secretString contains the SSH ASCII armor construct an SSH key credential (get username from tags), if it contains the X509 header etc then construct a client cert credential (get username from tags), and so on. With this approach, the core bit of the Secret is still readable by any application using Secrets Manager (not just Jenkins), but the extra non-secret bits can be stored there for Jenkins’ sake. And we don’t need an artificial ‘type’ field/string to be stored either - the chain decoder doesn’t need it. Off the top of my head this should work for: SSH key (secretString format test + AWS tags) Client cert (secretString format test + AWS tags) Username / password (if doesn’t pass any secretString format tests, and there’s a username field in the AWS tags) Secret text (if didn’t match any of the above criteria) Thoughts? (This is only a first take on the idea, and there might be other credentials types that I’ve overlooked.)
          chriskilding Chris Kilding made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          chriskilding Chris Kilding made changes -
          Assignee Enrique Fernández-Polo [ quaiks ] Chris Kilding [ chriskilding ]
          Hide
          chriskilding Chris Kilding added a comment -

          I have started working on this auto-sensing implementation at https://github.com/jenkinsci/aws-secrets-manager-credentials-provider-plugin/pull/8

          Thus far I have got a WIP implementation down that supports Secret Text, Username with Password, SSH Private Key credential types.

          I’m looking to support the client certificate type but there is a problem - I can’t find any decent examples of:

          • How to use client credentials via withCredentials()
          • Plugins that operate on certificate credentials instances directly (to which we only pass the cert credential ID as a string).

          Can someone point me to a simple of using Jenkins Certificate credentials to do something in a build?

          Show
          chriskilding Chris Kilding added a comment - I have started working on this auto-sensing implementation at https://github.com/jenkinsci/aws-secrets-manager-credentials-provider-plugin/pull/8 Thus far I have got a WIP implementation down that supports Secret Text, Username with Password, SSH Private Key credential types. I’m looking to support the client certificate type but there is a problem - I can’t find any decent examples of: How to use client credentials via withCredentials() Plugins that operate on certificate credentials instances directly (to which we only pass the cert credential ID as a string). Can someone point me to a simple of using Jenkins Certificate credentials to do something in a build?
          Hide
          chriskilding Chris Kilding added a comment -

          (I have seen https://jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials but the certificate example does not say what format the certificate’s KeyStore is written out in, eg JKS, PEM, P12, and it does not provide an example of an application actually consuming the bound credential.)

          Show
          chriskilding Chris Kilding added a comment - (I have seen https://jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials but the certificate example does not say what format the certificate’s KeyStore is written out in, eg JKS, PEM, P12, and it does not provide an example of an application actually consuming the bound credential.)
          Hide
          chriskilding Chris Kilding added a comment -

          I’ve now completed a first draft implementation of multiple credential types support over at https://github.com/jenkinsci/aws-secrets-manager-credentials-provider-plugin/pull/8

          Your questions and comments are welcome

          Show
          chriskilding Chris Kilding added a comment - I’ve now completed a first draft implementation of multiple credential types support over at https://github.com/jenkinsci/aws-secrets-manager-credentials-provider-plugin/pull/8 Your questions and comments are welcome
          Hide
          ntalbot Nat Talbot added a comment - - edited

          Hi Chris,

          Thanks for doing this implementation! I started using the plugin with AWS Secrets Manager this week it's a feature that is definitely really really helpful. I'm going to try to manually install the plugin from the built .hpi and test the new features, can provide feedback on the PR if there's anything worth mentioning.

          Show
          ntalbot Nat Talbot added a comment - - edited Hi Chris, Thanks for doing this implementation! I started using the plugin with AWS Secrets Manager this week it's a feature that is definitely really really helpful. I'm going to try to manually install the plugin from the built .hpi and test the new features, can provide feedback on the PR if there's anything worth mentioning.
          chriskilding Chris Kilding made changes -
          Remote Link This issue links to "GitHub PR with finished implementation (Web Link)" [ 23943 ]
          Hide
          chriskilding Chris Kilding added a comment - - edited

          Enrique Fernández-Polo I have found a way to add support for this feature in this GitHub PR . Since you originally requested this feature I would be interested to hear your feedback and whether it works for your company.

          We have some Jenkins deployments coming up at work soon, so if I don’t hear anything by the 22nd November I will merge that PR to master. And then publish a Release Candidate artifact to Jenkins Experimentals (this will contain the final functionality, and any changes after will only be bug fixes).

          Show
          chriskilding Chris Kilding added a comment - - edited Enrique Fernández-Polo I have found a way to add support for this feature in this GitHub PR . Since you originally requested this feature I would be interested to hear your feedback and whether it works for your company. We have some Jenkins deployments coming up at work soon, so if I don’t hear anything by the 22nd November I will merge that PR to master. And then publish a Release Candidate artifact to Jenkins Experimentals (this will contain the final functionality, and any changes after will only be bug fixes).
          Hide
          stradenko C added a comment -

          Chris Kilding - I'll open another issue on it, but curious what you think about using parameter store as a backend.  Do you think that having that functionality in this plugin (so the plugin can scan either secrets manager, or parameter store) is a good idea?

          Show
          stradenko C added a comment - Chris Kilding - I'll open another issue on it, but curious what you think about using parameter store as a backend.  Do you think that having that functionality in this plugin (so the plugin can scan either secrets manager, or parameter store) is a good idea?
          Hide
          chriskilding Chris Kilding added a comment -

          I don’t have an intuition for that off the top of my head, but it’s certainly worth a discussion as I know of a couple of projects in our place that put their secrets in parameter store.

          Open a ticket and we’ll discuss

          Show
          chriskilding Chris Kilding added a comment - I don’t have an intuition for that off the top of my head, but it’s certainly worth a discussion as I know of a couple of projects in our place that put their secrets in parameter store. Open a ticket and we’ll discuss
          chriskilding Chris Kilding made changes -
          Released As 0.1.0
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]
          chriskilding Chris Kilding made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            Assignee:
            chriskilding Chris Kilding
            Reporter:
            quaiks Enrique Fernández-Polo
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: