Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58193

EC2 Plugin: EC2 instances are getting terminated right away on initializing in 1.44 version

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • ec2-plugin
    • None

      EC2 instances are getting terminated right away on initializing in 1.44 version

      Below is the error message
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching instance: i-0dedca71f82852276
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: bootstrap()
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Getting keypair...
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Using private key jenkins-slave (SHA-1 fingerprint e7:f4)
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Authenticating as ec2-user
      Jun 25, 2019 3:49:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.188.11.241 on port 22, with timeout 10000.
      Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      Jun 25, 2019 3:49:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connected via SSH.
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Creating tmp directory (/tmp) if it does not exist
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Verifying: java -fullversion
      openjdk full version "1.8.0_201-b09"
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Verifying: which scp
      /usr/bin/scp
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Copying remoting.jar to: /tmp
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir
      ERROR: unexpected stream termination
      java.io.EOFException: unexpected stream termination
      at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
      at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
      at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:431)
      at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:262)
      at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

       

      It's working fine in 1.43 version

          [JENKINS-58193] EC2 Plugin: EC2 instances are getting terminated right away on initializing in 1.44 version

          the fix for windows will be in 1.44.2, can you test the build from master ? 

          FABRIZIO MANFREDI added a comment - the fix for windows will be in 1.44.2, can you test the build from master ? 

          tapvir virk added a comment -

          I dont have setup to build the branch. Do you know when 1.44.2 will be released? If it is at later point, I will try to build from master

          tapvir virk added a comment - I dont have setup to build the branch. Do you know when 1.44.2 will be released? If it is at later point, I will try to build from master

          Our master branch has been failing due to some infra issues. If you don't mind you can test from one of my other PRs that reduces calls to describeInstances but contains the above fix
          https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1008.3d3aaabf683e/

          Raihaan Shouhell added a comment - Our master branch has been failing due to some infra issues. If you don't mind you can test from one of my other PRs that reduces calls to describeInstances but contains the above fix https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1008.3d3aaabf683e/

          tapvir thoulen, was this released? I'm encountering it on version 1.46.1 as well.

          George Cimpoies added a comment - tapvir thoulen , was this released? I'm encountering it on version 1.46.1 as well.

          gcimpoies can you provide a stacktrace if you have one?

          Raihaan Shouhell added a comment - gcimpoies can you provide a stacktrace if you have one?

          raihaan sure, I've documented everything here: https://issues.jenkins-ci.org/browse/JENKINS-59682

          George Cimpoies added a comment - raihaan sure, I've documented everything here:  https://issues.jenkins-ci.org/browse/JENKINS-59682

          tapvir virk added a comment -

          Windows ec2 slave is not working. Getting below exception
          I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception
          ERROR: Unable to decode password:
          java.lang.IllegalArgumentException: Illegal base64 character d
          com.amazonaws.AmazonClientException: Unable to decode password:
          java.lang.IllegalArgumentException: Illegal base64 character d
          at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141)
          at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156)
          at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39)
          at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
          at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
          at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
          at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)

          tapvir virk added a comment - Windows ec2 slave is not working. Getting below exception I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception ERROR: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d com.amazonaws.AmazonClientException: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

          tapvir virk added a comment -

          I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call.
          I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception,
          Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

          tapvir virk added a comment - I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call. I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception, Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

          get password data cannot return newlines according to RFC 4648. Are you setting the password on the AMI or leaving it random? Also can you open a new issue for this the issue is unrelated

          Raihaan Shouhell added a comment - get password data cannot return newlines according to RFC 4648. Are you setting the password on the AMI or leaving it random? Also can you open a new issue for this the issue is unrelated

          tapvir virk added a comment -

          I already created below ticket

          1. JENKINS-59633

          tapvir virk added a comment - I already created below ticket JENKINS-59633

            raihaan Raihaan Shouhell
            tapvir tapvir virk
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: