Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58193

EC2 Plugin: EC2 instances are getting terminated right away on initializing in 1.44 version

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Fixed
    • ec2-plugin
    • None

    Description

      EC2 instances are getting terminated right away on initializing in 1.44 version

      Below is the error message
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching instance: i-0dedca71f82852276
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: bootstrap()
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Getting keypair...
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Using private key jenkins-slave (SHA-1 fingerprint e7:f4)
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Authenticating as ec2-user
      Jun 25, 2019 3:49:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.188.11.241 on port 22, with timeout 10000.
      Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      Jun 25, 2019 3:49:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connected via SSH.
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Creating tmp directory (/tmp) if it does not exist
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Verifying: java -fullversion
      openjdk full version "1.8.0_201-b09"
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Verifying: which scp
      /usr/bin/scp
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Copying remoting.jar to: /tmp
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir
      ERROR: unexpected stream termination
      java.io.EOFException: unexpected stream termination
      at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
      at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
      at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:431)
      at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:262)
      at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

       

      It's working fine in 1.43 version

      Attachments

        Issue Links

          Activity

            tapvir tapvir virk added a comment -

            This is the exception getting logged in the jenkins logs

            tapvir tapvir virk added a comment - This is the exception getting logged in the jenkins logs

            Hey Tapvir can you show us your agent configuration?

            raihaan Raihaan Shouhell added a comment - Hey Tapvir can you show us your agent configuration?
            raihaan Raihaan Shouhell added a comment - tapvir can you try the build from  https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc980.7e1580e3f8d5/
            tapvir tapvir virk added a comment - - edited

            Hi Raihaan,
            It is working in 1.45-rc980.7e1580e3f8d5 version

            tapvir tapvir virk added a comment - - edited Hi Raihaan, It is working in 1.45-rc980.7e1580e3f8d5 version

            Is this a duplicate of JENKINS-58163 ?

            jbochenski Jakub Bochenski added a comment - Is this a duplicate of JENKINS-58163 ?
            tapvir tapvir virk added a comment -

            When this fix will be released?

            tapvir tapvir virk added a comment - When this fix will be released?

            Released in 1.44.1

            thoulen FABRIZIO MANFREDI added a comment - Released in 1.44.1
            tapvir tapvir virk added a comment -

            Hi Raihaan,

             

            I just observed, same exception is thrown on WIndows slave. Linux slaves are working fine.

             

            Can you please check and provide the fix?

             

            Thanks

            Tapvir

            tapvir tapvir virk added a comment - Hi Raihaan,   I just observed, same exception is thrown on WIndows slave. Linux slaves are working fine.   Can you please check and provide the fix?   Thanks Tapvir

            the fix for windows will be in 1.44.2, can you test the build from master ? 

            thoulen FABRIZIO MANFREDI added a comment - the fix for windows will be in 1.44.2, can you test the build from master ? 
            tapvir tapvir virk added a comment -

            I dont have setup to build the branch. Do you know when 1.44.2 will be released? If it is at later point, I will try to build from master

            tapvir tapvir virk added a comment - I dont have setup to build the branch. Do you know when 1.44.2 will be released? If it is at later point, I will try to build from master

            Our master branch has been failing due to some infra issues. If you don't mind you can test from one of my other PRs that reduces calls to describeInstances but contains the above fix
            https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1008.3d3aaabf683e/

            raihaan Raihaan Shouhell added a comment - Our master branch has been failing due to some infra issues. If you don't mind you can test from one of my other PRs that reduces calls to describeInstances but contains the above fix https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1008.3d3aaabf683e/

            tapvir thoulen, was this released? I'm encountering it on version 1.46.1 as well.

            gcimpoies George Cimpoies added a comment - tapvir thoulen , was this released? I'm encountering it on version 1.46.1 as well.

            gcimpoies can you provide a stacktrace if you have one?

            raihaan Raihaan Shouhell added a comment - gcimpoies can you provide a stacktrace if you have one?

            raihaan sure, I've documented everything here: https://issues.jenkins-ci.org/browse/JENKINS-59682

            gcimpoies George Cimpoies added a comment - raihaan sure, I've documented everything here:  https://issues.jenkins-ci.org/browse/JENKINS-59682
            tapvir tapvir virk added a comment -

            Windows ec2 slave is not working. Getting below exception
            I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception
            ERROR: Unable to decode password:
            java.lang.IllegalArgumentException: Illegal base64 character d
            com.amazonaws.AmazonClientException: Unable to decode password:
            java.lang.IllegalArgumentException: Illegal base64 character d
            at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141)
            at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156)
            at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39)
            at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
            at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
            at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

            tapvir tapvir virk added a comment - Windows ec2 slave is not working. Getting below exception I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception ERROR: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d com.amazonaws.AmazonClientException: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
            tapvir tapvir virk added a comment -

            I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call.
            I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception,
            Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

            tapvir tapvir virk added a comment - I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call. I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception, Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

            get password data cannot return newlines according to RFC 4648. Are you setting the password on the AMI or leaving it random? Also can you open a new issue for this the issue is unrelated

            raihaan Raihaan Shouhell added a comment - get password data cannot return newlines according to RFC 4648. Are you setting the password on the AMI or leaving it random? Also can you open a new issue for this the issue is unrelated
            tapvir tapvir virk added a comment -

            I already created below ticket

            1. JENKINS-59633
            tapvir tapvir virk added a comment - I already created below ticket JENKINS-59633

            People

              raihaan Raihaan Shouhell
              tapvir tapvir virk
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: