Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60652

Plugin breaks credential list views when it cannot contact AWS

    • 0.1.3

      When this plugin cannot contact AWS it breaks screens that list credentials.

      Examples:

      This can easily reproduced by just installing the plugin on a local instance and not configuring anything more.  

      The exception looks like this:

      com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
      	at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462)
      	at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
      	at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
      	at io.jenkins.plugins.credentials.secretsmanager.AwsCredentialsProvider.fetchCredentials(AwsCredentialsProvider.java:103)
      	at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:173)
      	at io.jenkins.plugins.credentials.secretsmanager.AwsCredentialsProvider.getCredentials(AwsCredentialsProvider.java:61)
      	at com.cloudbees.plugins.credentials.CredentialsProvider.getCredentials(CredentialsProvider.java:1147)
      	at com.cloudbees.plugins.credentials.CredentialsProvider.getCredentials(CredentialsProvider.java:1222)
      	at com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(CredentialsProvider.java:549)
      	at com.cloudbees.plugins.credentials.CredentialsProvider.findCredentialById(CredentialsProvider.java:906)
      	at com.cloudbees.plugins.credentials.CredentialsProvider.findCredentialById(CredentialsProvider.java:850)
      	at org.jenkinsci.plugins.credentialsbinding.MultiBinding.getCredentials(MultiBinding.java:144)
      	at org.jenkinsci.plugins.credentialsbinding.impl.UsernamePasswordMultiBinding.bind(UsernamePasswordMultiBinding.java:75)
      	at org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Execution2.doStart(BindingStep.java:135) 

       

          [JENKINS-60652] Plugin breaks credential list views when it cannot contact AWS

          Chris Kilding added a comment -

          You see this error on the global configuration page because it may attempt to use credential providers (not just this one, others too) before they are ready, where the mechanism for configuring the provider is also on the global configuration page. I reported this issue here.

          Chris Kilding added a comment - You see this error on the global configuration page because it may attempt to use credential providers (not just this one, others too) before they are ready, where the mechanism for configuring the provider is also on the global configuration page. I reported this issue here.

          Chris Kilding added a comment - - edited

          From what you've said I think there are 3 broad error cases to look at:

          • Listing credentials: there is indeed a bug here. Instead of throwing the exception it should probably return an uncached empty credential list, and log a warning in the Jenkins logs.
          • Get credential value within this provider: The credentials provider standard document insists that remote providers must throw an exception if they are asked to provide the value but can't contact their backing store. This will crash the build, but there is no choice here.
          • Get credential value only from other providers, on a Jenkins server where this plugin just happens to be installed: Yep, sounds like a bug too. I'm still working out what the behaviour should be here.

          We should also clarify the correct behaviour in the credentials provider standard document, which currently only specifies what should happen when getting the value fails (throw exception).

          Chris Kilding added a comment - - edited From what you've said I think there are 3 broad error cases to look at: Listing credentials: there is indeed a bug here. Instead of throwing the exception it should probably return an uncached empty credential list, and log a warning in the Jenkins logs. Get credential value within this provider: The credentials provider standard document insists that remote providers must throw an exception if they are asked to provide the value but can't contact their backing store. This will crash the build, but there is no choice here. Get credential value only from other providers, on a Jenkins server where this plugin just happens to be installed: Yep, sounds like a bug too. I'm still working out what the behaviour should be here. We should also clarify the correct behaviour in the credentials provider standard document, which currently only specifies what should happen when getting the value fails (throw exception).

          Chris Kilding added a comment -

          This will take time to work out, so in the meantime if you're wondering how we make this work at our place...

          • We manage our AWS accounts almost entirely with Terraform
          • We use either immutable infrastructure tools (AMI builders, Docker etc plus Jenkins Configuration As Code) or configuration management tools (Puppet etc) to provision servers, so while we try to support manual plugin configuration through the Global Configuration Web UI, it's not a code path we actively use ourselves.
          • We have non-production AWS accounts where we can test all infrastructure changes before rolling them out to production.

          This combination of things probably meant we were able to catch misconfiguration bugs in the credential provider before the bugs ever got near our production Jenkins.

          Chris Kilding added a comment - This will take time to work out, so in the meantime if you're wondering how we make this work at our place... We manage our AWS accounts almost entirely with Terraform We use either immutable infrastructure tools (AMI builders, Docker etc plus Jenkins Configuration As Code) or configuration management tools (Puppet etc) to provision servers, so while we try to support manual plugin configuration through the Global Configuration Web UI, it's not a code path we actively use ourselves. We have non-production AWS accounts where we can test all infrastructure changes before rolling them out to production. This combination of things probably meant we were able to catch misconfiguration bugs in the credential provider before the bugs ever got near our production Jenkins.

          Thanks for the info - I have my setup similar to yours (with CloudFormation, JasC), but until now, the Jenkins Docker Image I create also used to work on my local env (my mac) - but now this unfortunately is not the case anymore and its harder to do some first local testing of the image.

          The most important part for me right now is your first part "Listing credentials" - this effectively hinders my workflow right now.

          I don't know the internals of the credential providers, but from a users point of view: I have a job configured e.g. a pipeline and the only thing to reference credentials are by a string-id. For me as a pipeline admin, I don't care which provider actually holds the credentials. If it is not found by any of the providers, it should throw an exception. So there is no way for me to say that I want this credentials retrieved rather from AWS and not from the internal jenkins provider. Saying this, I don't understand why a provider should throw an exception if it can't resolve credentials for a given ID (unless the credentials framework swallows the exceptions and interprets it as a "NotFound" to hand over to the next provider installed until one can resolve it).

          Dominik Bartholdi added a comment - Thanks for the info - I have my setup similar to yours (with CloudFormation, JasC), but until now, the Jenkins Docker Image I create also used to work on my local env (my mac) - but now this unfortunately is not the case anymore and its harder to do some first local testing of the image. The most important part for me right now is your first part "Listing credentials" - this effectively hinders my workflow right now. I don't know the internals of the credential providers, but from a users point of view: I have a job configured e.g. a pipeline and the only thing to reference credentials are by a string-id. For me as a pipeline admin, I don't care which provider actually holds the credentials. If it is not found by any of the providers, it should throw an exception. So there is no way for me to say that I want this credentials retrieved rather from AWS and not from the internal jenkins provider. Saying this, I don't understand why a provider should throw an exception if it can't resolve credentials for a given ID (unless the credentials framework swallows the exceptions and interprets it as a "NotFound" to hand over to the next provider installed until one can resolve it).

          maybe its worth breaking this issue up into multiple, meaning: fix the low hanging fruits. After all, I think a plugin should never make jenkins unusable just because it is installed but not configured and not used.

          Dominik Bartholdi added a comment - maybe its worth breaking this issue up into multiple, meaning: fix the low hanging fruits. After all, I think a plugin should never make jenkins unusable just because it is installed but not configured and not used.

          Chris Kilding added a comment -

          Yes, we'll break it up, and address the list views first.

          Chris Kilding added a comment - Yes, we'll break it up, and address the list views first.

          Chris Kilding added a comment -

          Started work on the pull request to fix the bug.

          Chris Kilding added a comment - Started work on the pull request to fix the bug.

          Tural Aliyev added a comment - - edited

          Hi chriskilding,

          I am using AWS Jenkins 2.303.3 and plugin AWS pipeline steps 1.12.101 . Still receiving the same exception wile doing assume role. Added region to env and also to withAWS also still the same. Do you think my issue related to this? 

          Tural Aliyev added a comment - - edited Hi chriskilding , I am using AWS  Jenkins 2.303.3  and plugin AWS pipeline steps  1.12.101  . Still receiving the same exception wile doing assume role. Added region to env and also to withAWS also still the same. Do you think my issue related to this? 

            chriskilding Chris Kilding
            imod Dominik Bartholdi
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: