Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59633

EC2 Plugin: Windows EC2 instances are not launching

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • ec2-plugin
    • None
    • ec2 1.47

      Hi,

      Getting below exception on windows slave. It is working fine in 1.43 version

      Unexpected Exception: com.amazonaws.services.ec2.model.AmazonEC2Exception: You are not authorized to perform this operation. (Service: AmazonEC2; Status Code: 403; Error Code: UnauthorizedOperation; Request ID: 071f3254-51f1-4577-810f-132237add295)

       

          [JENKINS-59633] EC2 Plugin: Windows EC2 instances are not launching

          What version are you on and can you show a screenshot of your settings? This might be because you used auto retrieve password from amazon which requires a new permission.

          Raihaan Shouhell added a comment - What version are you on and can you show a screenshot of your settings? This might be because you used auto retrieve password from amazon which requires a new permission.

          tapvir virk added a comment - - edited

          Attaching the screenshot of the ec2 slave setting. Do you know which permission is required? 

          Though I can see the instance getting launched on aws account.

          tapvir virk added a comment - - edited Attaching the screenshot of the ec2 slave setting. Do you know which permission is required?  Though I can see the instance getting launched on aws account.

          You need to check specify password and specify your admin password or give getpassworddata to your appropriate role

          Raihaan Shouhell added a comment - You need to check specify password and specify your admin password or give getpassworddata to your appropriate role

          tapvir virk added a comment -

          I'm running Jenkins 2.476.3 version. Using latest ec2 plugin.

           

          Do you know which permission needs to be added?

           

          tapvir virk added a comment - I'm running Jenkins 2.476.3 version. Using latest ec2 plugin.   Do you know which permission needs to be added?  

          tapvir virk added a comment - - edited

          I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception
          ERROR: Unable to decode password:
          java.lang.IllegalArgumentException: Illegal base64 character d
          com.amazonaws.AmazonClientException: Unable to decode password:
          java.lang.IllegalArgumentException: Illegal base64 character d
          at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141)
          at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156)
          at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39)
          at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
          at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
          at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
          at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)

          tapvir virk added a comment - - edited I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception ERROR: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d com.amazonaws.AmazonClientException: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

          tapvir virk added a comment -

          I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call.
          I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception,

          Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

          tapvir virk added a comment - I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call. I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception, Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

          tapvir virk added a comment - - edited

          Hi raihaan
          As per your comment in JENKINS-58193 ,

          I can see the getPasswordData cli returning \r\n prefixing the content.

          I haven't set any password in the ami. I'm just running below script in the userdata,

          <powershell>

          1. Disable Credssp for Jenkins master to login via WinRM.
            winrm set winrm/config/service/auth '@{CredSSP="false"}'

          $osString = wmic os get caption | Out-String
          echo "Windows OS: $osString"
          </powershell>

          tapvir virk added a comment - - edited Hi raihaan As per your comment in  JENKINS-58193 , I can see the getPasswordData cli returning \r\n prefixing the content. I haven't set any password in the ami. I'm just running below script in the userdata, <powershell> Disable Credssp for Jenkins master to login via WinRM. winrm set winrm/config/service/auth '@{CredSSP="false"}' $osString = wmic os get caption | Out-String echo "Windows OS: $osString" </powershell>

          tapvir virk added a comment -

          Below is the cli request and the response

          aws ec2 get-password-data --instance-id i-xxxxxx --region ca-central-1

          {
          "InstanceId": "i-xxxxx",
          "Timestamp": "2019-10-10T15:48:21.000Z",
          "PasswordData": "\r\nPdbgYc5e0hJZAxqgrcGSPdEGmrno+6kCXej8hApQC2f05pfjCnUZVb/vG/57K42oN8skN6OLE5WGGBaKsYzjAl1qoXq0pw1WBANbsqJgeZWJLKG6O4B9XKog2YPHtb4+aaxnbWK+gU84cBGbGmrEbs5XSnpt54/WA5MOqi0iNV4LTnYc3DfEpGHR4Ir6Zjl0qo9FFOUD5KbA0BMiiRXD9/JguIKGHlES12G/0ko8pJhD3g7GMykWWjt2AAL52UlvUd/pxqzQZHbszrQ1xqKD12XDVs1ewJcl7pzCPKbRYg0hOrAeih4gVCcWNneTf27VObj0v4PSOQAFv3lfTB2jMA==\r\n"
          }

          tapvir virk added a comment - Below is the cli request and the response aws ec2 get-password-data --instance-id i-xxxxxx --region ca-central-1 { "InstanceId": "i-xxxxx", "Timestamp": "2019-10-10T15:48:21.000Z", "PasswordData": "\r\nPdbgYc5e0hJZAxqgrcGSPdEGmrno+6kCXej8hApQC2f05pfjCnUZVb/vG/57K42oN8skN6OLE5WGGBaKsYzjAl1qoXq0pw1WBANbsqJgeZWJLKG6O4B9XKog2YPHtb4+aaxnbWK+gU84cBGbGmrEbs5XSnpt54/WA5MOqi0iNV4LTnYc3DfEpGHR4Ir6Zjl0qo9FFOUD5KbA0BMiiRXD9/JguIKGHlES12G/0ko8pJhD3g7GMykWWjt2AAL52UlvUd/pxqzQZHbszrQ1xqKD12XDVs1ewJcl7pzCPKbRYg0hOrAeih4gVCcWNneTf27VObj0v4PSOQAFv3lfTB2jMA==\r\n" }

          tapvir virk added a comment -

          raihaan - Did you get a chance to look at the above issue?

          tapvir virk added a comment - raihaan - Did you get a chance to look at the above issue?

          tapvir I couldn't reproduce this issue. What ami are you using, if its public?

           

          Raihaan Shouhell added a comment - tapvir I couldn't reproduce this issue. What ami are you using, if its public?  

          Jason Boyd added a comment -

          I am experiencing the same issue using plugin version 1.46.1. AMI is ami-0a6b38f2d62c0cc94: Microsoft Windows Server 2019 Base with Containers.

          aws ec2 get-password-data --instance-id i-044a30c5ada9d3eb6 --region us-west-2

          {
          "InstanceId": "i-044a30c5ada9d3eb6",
          "PasswordData": "\r\nmG0KvA0Ll65mN0PiKbA6DKVwWD8mUcGRgfIqaCeXOh/evAcEpPY2DRTPYBF8IdVJeTfsZWM5bC6a1rgca9006gl6GKEKAIScfYcBxzdtZtSUVN1/xSBLdxuQSd6BZoqG9lAnhdZL6sWs8AWFgZle+7/1G8/k2GZx7R1lz59oMcOeHao+ggcofF8pJ+cyt2ytHcOYLhdNRX23xIM1OfaAjRtrkOYxV4r+c5en+Bisns79iC9LC700nbkSiGPHVdM0eLor78wiAwjmFVj3ukMVylEOMsTcDIIPjnF0Cq7eyssgblZd1v/rVlHS13iMMvYuoFs+ALBMmuOQlzeuYxScnw==\r\n",
          "Timestamp": "2019-11-01T21:36:18.000Z"
          }

          Jason Boyd added a comment - I am experiencing the same issue using plugin version 1.46.1. AMI is ami-0a6b38f2d62c0cc94: Microsoft Windows Server 2019 Base with Containers. aws ec2 get-password-data --instance-id i-044a30c5ada9d3eb6 --region us-west-2 { "InstanceId": "i-044a30c5ada9d3eb6", "PasswordData": "\r\nmG0KvA0Ll65mN0PiKbA6DKVwWD8mUcGRgfIqaCeXOh/evAcEpPY2DRTPYBF8IdVJeTfsZWM5bC6a1rgca9006gl6GKEKAIScfYcBxzdtZtSUVN1/xSBLdxuQSd6BZoqG9lAnhdZL6sWs8AWFgZle+7/1G8/k2GZx7R1lz59oMcOeHao+ggcofF8pJ+cyt2ytHcOYLhdNRX23xIM1OfaAjRtrkOYxV4r+c5en+Bisns79iC9LC700nbkSiGPHVdM0eLor78wiAwjmFVj3ukMVylEOMsTcDIIPjnF0Cq7eyssgblZd1v/rVlHS13iMMvYuoFs+ALBMmuOQlzeuYxScnw==\r\n", "Timestamp": "2019-11-01T21:36:18.000Z" }

          i submitted a pull request, which fixes the problem: https://github.com/jenkinsci/ec2-plugin/pull/418

          Raphael Pionke added a comment - i submitted a pull request, which fixes the problem: https://github.com/jenkinsci/ec2-plugin/pull/418

          tapvir virk added a comment -

          rpionke - Can you give me the hpi file to test on my jenkins instance?

          tapvir virk added a comment - rpionke - Can you give me the hpi file to test on my jenkins instance?

          Raphael Pionke added a comment - tapvir you can find the hpi file here: https://ci.jenkins.io/job/Plugins/job/ec2-plugin/job/PR-418/1/artifact/org/jenkins-ci/plugins/ec2/1.47-rc1073.62885629c57c/ec2-1.47-rc1073.62885629c57c.hpi

          tapvir virk added a comment -

          rpionke Fix is working for me.
          Thanks

          tapvir virk added a comment - rpionke Fix is working for me. Thanks

          Thanks for the fix.

          Raihaan Shouhell added a comment - Thanks for the fix.

            raihaan Raihaan Shouhell
            tapvir tapvir virk
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: