Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58193

EC2 Plugin: EC2 instances are getting terminated right away on initializing in 1.44 version

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Fixed
    • ec2-plugin
    • None

    Description

      EC2 instances are getting terminated right away on initializing in 1.44 version

      Below is the error message
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching instance: i-0dedca71f82852276
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: bootstrap()
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Getting keypair...
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Using private key jenkins-slave (SHA-1 fingerprint e7:f4)
      Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
      INFO: Authenticating as ec2-user
      Jun 25, 2019 3:49:23 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connecting to 10.188.11.241 on port 22, with timeout 10000.
      Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
      INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
      Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
      INFO: Waiting for SSH to come up. Sleeping 5.
      Jun 25, 2019 3:49:38 PM hudson.plugins.ec2.EC2Cloud
      INFO: Connected via SSH.
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Creating tmp directory (/tmp) if it does not exist
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Verifying: java -fullversion
      openjdk full version "1.8.0_201-b09"
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Verifying: which scp
      /usr/bin/scp
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Copying remoting.jar to: /tmp
      Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
      INFO: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir
      ERROR: unexpected stream termination
      java.io.EOFException: unexpected stream termination
      at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
      at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
      at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:431)
      at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:262)
      at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

       

      It's working fine in 1.43 version

      Attachments

        Issue Links

          Activity

            tapvir tapvir virk created issue -
            tapvir tapvir virk made changes -
            Field Original Value New Value
            Description EC2 instances are getting terminated right away on initializing in 1.44 version EC2 instances are getting terminated right away on initializing in 1.44 version

            Below is the error message
            java.io.EOFException: unexpected stream termination
            tapvir tapvir virk made changes -
            Description EC2 instances are getting terminated right away on initializing in 1.44 version

            Below is the error message
            java.io.EOFException: unexpected stream termination
            EC2 instances are getting terminated right away on initializing in 1.44 version

            Below is the error message
             java.io.EOFException: unexpected stream termination

             

            It's working fine in 1.43 version
            tapvir tapvir virk made changes -
            Description EC2 instances are getting terminated right away on initializing in 1.44 version

            Below is the error message
             java.io.EOFException: unexpected stream termination

             

            It's working fine in 1.43 version
            EC2 instances are getting terminated right away on initializing in 1.44 version

            Below is the error message
            Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
            INFO: Launching instance: i-0dedca71f82852276
            Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
            INFO: bootstrap()
            Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
            INFO: Getting keypair...
            Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
            INFO: Using private key jenkins-slave (SHA-1 fingerprint e7:f4)
            Jun 25, 2019 3:49:22 PM hudson.plugins.ec2.EC2Cloud
            INFO: Authenticating as ec2-user
            Jun 25, 2019 3:49:23 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connecting to 10.188.11.241 on port 22, with timeout 10000.
            Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
            Jun 25, 2019 3:49:33 PM hudson.plugins.ec2.EC2Cloud
            INFO: Waiting for SSH to come up. Sleeping 5.
            Jun 25, 2019 3:49:38 PM hudson.plugins.ec2.EC2Cloud
            INFO: Connected via SSH.
            Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
            INFO: Creating tmp directory (/tmp) if it does not exist
            Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
            INFO: Verifying: java -fullversion
            openjdk full version "1.8.0_201-b09"
            Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
            INFO: Verifying: which scp
            /usr/bin/scp
            Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
            INFO: Copying remoting.jar to: /tmp
            Jun 25, 2019 3:50:46 PM hudson.plugins.ec2.EC2Cloud
            INFO: Launching remoting agent (via Trilead SSH2 Connection): java -jar /tmp/remoting.jar -workDir
            ERROR: unexpected stream termination
            java.io.EOFException: unexpected stream termination
             at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:415)
             at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:360)
             at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:431)
             at hudson.plugins.ec2.ssh.EC2UnixLauncher.launchScript(EC2UnixLauncher.java:262)
             at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
             at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
             at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
             at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
             at java.util.concurrent.FutureTask.run(FutureTask.java:266)
             at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
             at java.lang.Thread.run(Thread.java:748)

             

            It's working fine in 1.43 version
            jvz Matt Sicker made changes -
            Comment [ Do you have more of a stack trace? ]
            tapvir tapvir virk added a comment -

            This is the exception getting logged in the jenkins logs

            tapvir tapvir virk added a comment - This is the exception getting logged in the jenkins logs

            Hey Tapvir can you show us your agent configuration?

            raihaan Raihaan Shouhell added a comment - Hey Tapvir can you show us your agent configuration?
            raihaan Raihaan Shouhell made changes -
            Assignee Matt Sicker [ jvz ] Raihaan Shouhell [ raihaan ]
            raihaan Raihaan Shouhell made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            raihaan Raihaan Shouhell added a comment - tapvir can you try the build from  https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc980.7e1580e3f8d5/
            raihaan Raihaan Shouhell made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            tapvir tapvir virk added a comment - - edited

            Hi Raihaan,
            It is working in 1.45-rc980.7e1580e3f8d5 version

            tapvir tapvir virk added a comment - - edited Hi Raihaan, It is working in 1.45-rc980.7e1580e3f8d5 version

            Is this a duplicate of JENKINS-58163 ?

            jbochenski Jakub Bochenski added a comment - Is this a duplicate of JENKINS-58163 ?
            tapvir tapvir virk added a comment -

            When this fix will be released?

            tapvir tapvir virk added a comment - When this fix will be released?

            Released in 1.44.1

            thoulen FABRIZIO MANFREDI added a comment - Released in 1.44.1
            thoulen FABRIZIO MANFREDI made changes -
            Link This issue blocks JENKINS-58163 [ JENKINS-58163 ]
            thoulen FABRIZIO MANFREDI made changes -
            Resolution Duplicate [ 3 ]
            Status In Review [ 10005 ] Closed [ 6 ]
            tapvir tapvir virk added a comment -

            Hi Raihaan,

             

            I just observed, same exception is thrown on WIndows slave. Linux slaves are working fine.

             

            Can you please check and provide the fix?

             

            Thanks

            Tapvir

            tapvir tapvir virk added a comment - Hi Raihaan,   I just observed, same exception is thrown on WIndows slave. Linux slaves are working fine.   Can you please check and provide the fix?   Thanks Tapvir
            tapvir tapvir virk made changes -
            Resolution Duplicate [ 3 ]
            Status Closed [ 6 ] Reopened [ 4 ]

            the fix for windows will be in 1.44.2, can you test the build from master ? 

            thoulen FABRIZIO MANFREDI added a comment - the fix for windows will be in 1.44.2, can you test the build from master ? 
            tapvir tapvir virk added a comment -

            I dont have setup to build the branch. Do you know when 1.44.2 will be released? If it is at later point, I will try to build from master

            tapvir tapvir virk added a comment - I dont have setup to build the branch. Do you know when 1.44.2 will be released? If it is at later point, I will try to build from master
            raihaan Raihaan Shouhell made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]

            Our master branch has been failing due to some infra issues. If you don't mind you can test from one of my other PRs that reduces calls to describeInstances but contains the above fix
            https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1008.3d3aaabf683e/

            raihaan Raihaan Shouhell added a comment - Our master branch has been failing due to some infra issues. If you don't mind you can test from one of my other PRs that reduces calls to describeInstances but contains the above fix https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1008.3d3aaabf683e/

            tapvir thoulen, was this released? I'm encountering it on version 1.46.1 as well.

            gcimpoies George Cimpoies added a comment - tapvir thoulen , was this released? I'm encountering it on version 1.46.1 as well.

            gcimpoies can you provide a stacktrace if you have one?

            raihaan Raihaan Shouhell added a comment - gcimpoies can you provide a stacktrace if you have one?

            raihaan sure, I've documented everything here: https://issues.jenkins-ci.org/browse/JENKINS-59682

            gcimpoies George Cimpoies added a comment - raihaan sure, I've documented everything here:  https://issues.jenkins-ci.org/browse/JENKINS-59682
            tapvir tapvir virk added a comment -

            Windows ec2 slave is not working. Getting below exception
            I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception
            ERROR: Unable to decode password:
            java.lang.IllegalArgumentException: Illegal base64 character d
            com.amazonaws.AmazonClientException: Unable to decode password:
            java.lang.IllegalArgumentException: Illegal base64 character d
            at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141)
            at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156)
            at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39)
            at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
            at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
            at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

            tapvir tapvir virk added a comment - Windows ec2 slave is not working. Getting below exception I have given the permission to retrieve the password. Now, I'm not getting the above exception. But instance/slave node is getting terminated with the below exception ERROR: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d com.amazonaws.AmazonClientException: Unable to decode password: java.lang.IllegalArgumentException: Illegal base64 character d at hudson.plugins.ec2.EC2PrivateKey.decryptWindowsPassword(EC2PrivateKey.java:141) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:156) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
            tapvir tapvir virk added a comment -

            I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call.
            I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception,
            Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

            tapvir tapvir virk added a comment - I think it is related to newline characters returned in the PasswordData property from "getPasswordData" api call. I tried to replicate on my local machine with standalone application. With newline characters, it is giving me below exception, Exception in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 dataException in thread "main" org.bouncycastle.util.encoders.DecoderException: unable to decode base64 string: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64.decode(Unknown Source) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.decryptWindowsPassword(AwsEC2GetPasswordTest.java:51) at com.telushealth.thcp.pipeline.gradle.aws.ec2.AwsEC2GetPasswordTest.main(AwsEC2GetPasswordTest.java:19)Caused by: java.io.IOException: invalid characters encountered in base64 data at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source) ... 3 more

            get password data cannot return newlines according to RFC 4648. Are you setting the password on the AMI or leaving it random? Also can you open a new issue for this the issue is unrelated

            raihaan Raihaan Shouhell added a comment - get password data cannot return newlines according to RFC 4648. Are you setting the password on the AMI or leaving it random? Also can you open a new issue for this the issue is unrelated
            raihaan Raihaan Shouhell made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            raihaan Raihaan Shouhell made changes -
            Summary EC2 Plguin: EC2 instances are getting terminated right away on initializing in 1.44 version EC2 Plugin: EC2 instances are getting terminated right away on initializing in 1.44 version
            tapvir tapvir virk added a comment -

            I already created below ticket

            1. JENKINS-59633
            tapvir tapvir virk added a comment - I already created below ticket JENKINS-59633

            People

              raihaan Raihaan Shouhell
              tapvir tapvir virk
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: