Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55515

windows GCE instance connects but then report "java.io.EOFException: unexpected stream termination"

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • Jenkins ver. 2.107.2
      Google Compute Engine Plugin 1.0.7

      I can't use windows instances in GCE because of launching agent problem. Please help me to debug/resolve the issue.

       

      Steps to reproduce: 

      1. Create windows instance in GCE
      2. Login to the instance by RDP and add user tester with Administrator role
      3. Login to the instance by RDP as tester
      4. Install java8, cygwin with openssh, configure openssh (see how to here: https://docs.oracle.com/cd/E24628_01/install.121/e22624/preinstall_req_cygwin_ssh.htm#EMBSC281)
      5. Check you are able to connect by ssh with tester user and its password
      6. Create private/public rsa keypair (using ssh-keygen), put public key to the /home/tester/.ssh/authorized_keys file
      7. Copy generated private key to your computer to ~/key.txt
      8. Check you are able to connect to the instance with private key without password:
        ssh -i ~/key.txt tester@<ip_address>
      9. Stop the instance and create an image from the instance
      10. Goto http://<your_jenkins_address>/credentials/ page and add new credentials "SSH Username with private key", choose "enter directly" for private key and put generated private key here
      11. Add new "Instance configuration" for created image on http://<your_jenkins_address>/configure page: set "Windows?" checkbox, set "Windows Username"=tester, set "Windows SSH Private Key Credentials" to credentials that were created on prev step, set Labels=windows-gce-test, set "Remote Location"=C:\jenkins
      12. Run job with a label "windows-gce-test"

      Expected: new GCE instance and jenkins slave are created, the slave is successfully connected and job is successfully ended

      Actual: new GCE instance and jenkins slave are created, but slave can't connect

      Slave output is the following:
      INFO: Connecting to 35.233.217.99 on port 22, with timeout 10000.
      Jan 10, 2019 4:42:07 AM null
      INFO: Connected via SSH.
      Jan 10, 2019 4:42:08 AM null
      INFO: Copying slave.jar to: C:\
      Jan 10, 2019 4:42:11 AM null
      INFO: Verifying: java -fullversion
      openjdk full version "1.8.0_181-b02"
      Jan 10, 2019 4:42:12 AM null
      INFO: Launching Jenkins agent via plugin SSH: java -jar C:\slave.jar
      Jan 10, 2019 4:42:12 AM null
      WARNING: Error: Exception: java.io.EOFException: unexpected stream termination

      Jenkins log:
      Connected via SSH.
      Jan 10, 2019 5:04:02 AM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
      Copying slave.jar to: C:\
      Jan 10, 2019 5:04:03 AM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
      Verifying: java -fullversion
      Jan 10, 2019 5:04:03 AM INFO com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
      Launching Jenkins agent via plugin SSH: java -jar C:\slave.jar
      Jan 10, 2019 5:04:03 AM WARNING com.google.jenkins.plugins.computeengine.ComputeEngineCloud log
      Error:
      java.io.EOFException: unexpected stream termination
      at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:408)
      at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:353)
      at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:415)
      at com.google.jenkins.plugins.computeengine.ComputeEngineWindowsLauncher.launch(ComputeEngineWindowsLauncher.java:128)
      at com.google.jenkins.plugins.computeengine.ComputeEngineComputerLauncher.launch(ComputeEngineComputerLauncher.java:127)
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:288)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      Jan 10, 2019 5:04:03 AM WARNING com.google.jenkins.plugins.computeengine.ComputeEngineCloud$1 call
      Exception waiting for node zulu-win2016-tests-gce-enn0eu to connect
      java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details.
      at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:312)
      Caused: java.util.concurrent.ExecutionException
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      at com.google.jenkins.plugins.computeengine.ComputeEngineCloud$1.call(ComputeEngineCloud.java:171)
      at com.google.jenkins.plugins.computeengine.ComputeEngineCloud$1.call(ComputeEngineCloud.java:161)
      at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      I'm not sure that GCE uses "Remote Location" for windows slaves (because the log says "INFO: Copying slave.jar to: C:\"). Could it be a cause of problem?

      I'm unable to create GCE node because of JENKINS-55380 but tried to create permanent aget node and agent starts normally for it, I used the following parameters:
      Permanent Agent
      Remote root directory=.
      Launch method=Launch agent agents via SSH
      Credentails=<created_credentials_with_private_key>

      Node log:
      [01/10/19 05:14:27] [SSH] Checking java version of ./jdk/bin/java
      Couldn't figure out the Java version of ./jdk/bin/java
      bash: ./jdk/bin/java: No such file or directory

      [01/10/19 05:14:27] [SSH] Checking java version of java
      [01/10/19 05:14:27] [SSH] java -version returned 1.8.0_181.
      [01/10/19 05:14:27] [SSH] Starting sftp client.
      [01/10/19 05:14:28] [SSH] Copying latest remoting.jar...
      [01/10/19 05:14:30] [SSH] Copied 762,466 bytes.
      Expanded the channel window size to 4MB
      [01/10/19 05:14:30] [SSH] Starting agent process: cd "." && java -jar remoting.jar -workDir .
      Jan 10, 2019 1:14:30 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using .\remoting as a remoting work directory
      Both error and output logs will be printed to .\remoting
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Remoting version: 3.17
      This is a Windows agent
      NOTE: Relative remote path resolved to: C:\cygwin64\home\tester\.
      Agent successfully connected and online

          [JENKINS-55515] windows GCE instance connects but then report "java.io.EOFException: unexpected stream termination"

          Rachel Yen added a comment -

          I'm running it now and it's fine. verusikk, do you have more information on this issue? Have you tried the 2.0.0 release.

          Rachel Yen added a comment - I'm running it now and it's fine. verusikk , do you have more information on this issue? Have you tried the 2.0.0 release.

          Vera Akulova added a comment -

          Hi Rachel, I tried with 2.0.0 (org.jenkins-ci.plugins:google-compute-engine:2.0.0) and now I have another issue - jenkins can't connect because uses wrong user (Build instead of tester while tester is set in instance configuration), so I can't check if the problem "java.io.EOFException" is still reproducible

          Now error output is the following:
          Mar 06, 2019 4:03:46 AM null
          FINEST: Instance zulu-win2016-tests-gce-rju6z5 is running and ready...
          Mar 06, 2019 4:03:46 AM null
          INFO: Launching instance: zulu-win2016-tests-gce-rju6z5
          Mar 06, 2019 4:03:46 AM null
          INFO: bootstrap
          Mar 06, 2019 4:03:46 AM null
          INFO: Authenticating as Build
          Mar 06, 2019 4:03:47 AM null
          INFO: Connecting to 35.247.53.220 on port 22, with timeout 10000.
          Mar 06, 2019 4:03:57 AM null
          INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
          Mar 06, 2019 4:03:57 AM null
          INFO: Waiting for SSH to come up. Sleeping 5.
          Mar 06, 2019 4:04:02 AM null
          INFO: Connecting to 35.247.53.220 on port 22, with timeout 10000.
          Mar 06, 2019 4:04:02 AM null
          INFO: Connected via SSH.
          ERROR: Server rejected the 1 private key(s) for Build (credentialId:tester-gce-ssh/method:publickey)
          Mar 06, 2019 4:04:02 AM null
          WARNING: Authentication failed. Trying again...

          Vera Akulova added a comment - Hi Rachel, I tried with 2.0.0 (org.jenkins-ci.plugins:google-compute-engine:2.0.0) and now I have another issue - jenkins can't connect because uses wrong user (Build instead of tester while tester is set in instance configuration), so I can't check if the problem "java.io.EOFException" is still reproducible Now error output is the following: Mar 06, 2019 4:03:46 AM null FINEST: Instance zulu-win2016-tests-gce-rju6z5 is running and ready... Mar 06, 2019 4:03:46 AM null INFO: Launching instance: zulu-win2016-tests-gce-rju6z5 Mar 06, 2019 4:03:46 AM null INFO: bootstrap Mar 06, 2019 4:03:46 AM null INFO: Authenticating as Build Mar 06, 2019 4:03:47 AM null INFO: Connecting to 35.247.53.220 on port 22, with timeout 10000. Mar 06, 2019 4:03:57 AM null INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. Mar 06, 2019 4:03:57 AM null INFO: Waiting for SSH to come up. Sleeping 5. Mar 06, 2019 4:04:02 AM null INFO: Connecting to 35.247.53.220 on port 22, with timeout 10000. Mar 06, 2019 4:04:02 AM null INFO: Connected via SSH. ERROR: Server rejected the 1 private key(s) for Build (credentialId:tester-gce-ssh/method:publickey) Mar 06, 2019 4:04:02 AM null WARNING: Authentication failed. Trying again...

          Rachel Yen added a comment -

          It doesn't work when you enter tester as the windows username in the instance configuration?

          Rachel Yen added a comment - It doesn't work when you enter tester as the windows username in the instance configuration?

          Rachel Yen added a comment -

          I'm going to release our username change tomorrow. Hopefully that will resolve the issues.

          Rachel Yen added a comment - I'm going to release our username change tomorrow. Hopefully that will resolve the issues.

          Rachel Yen added a comment -

          username change has been released. Did it resolve your issues? It's release 3.0.0

          Rachel Yen added a comment - username change has been released. Did it resolve your issues? It's release 3.0.0

          Vera Akulova added a comment -

          It's still not working. Upgraded plugin to "org.jenkins-ci.plugins:google-compute-engine:3.0.0": "configure" jenkins page has "Run as user"=tester, but log for automatically started instance contains the following (uses Build user name instead of tester):

          Mar 27, 2019 12:34:25 PM null
          FINEST: Instance zulu-win2016-tests-gce-ulumed is running and ready...
          Mar 27, 2019 12:34:25 PM null
          INFO: Launching instance: zulu-win2016-tests-gce-ulumed
          Mar 27, 2019 12:34:25 PM null
          INFO: bootstrap
          Mar 27, 2019 12:34:25 PM null
          INFO: Authenticating as Build
          Mar 27, 2019 12:34:25 PM null
          INFO: Connecting to 35.197.99.246 on port 22, with timeout 10000.
          Mar 27, 2019 12:34:35 PM null
          INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired.
          Mar 27, 2019 12:34:35 PM null
          INFO: Waiting for SSH to come up. Sleeping 5.
          Mar 27, 2019 12:34:41 PM null
          INFO: Connecting to 35.197.99.246 on port 22, with timeout 10000.
          Mar 27, 2019 12:34:41 PM null
          INFO: Connected via SSH.
          ERROR: Server rejected the 1 private key(s) for Build (credentialId:tester-gce-ssh/method:publickey)
          Mar 27, 2019 12:34:41 PM null
          WARNING: Authentication failed. Trying again...
          Mar 27, 2019 12:34:56 PM null
          INFO: Authenticating as Build
          Mar 27, 2019 12:34:56 PM null
          INFO: Connecting to 35.197.99.246 on port 22, with timeout 10000.
          Mar 27, 2019 12:34:57 PM null
          INFO: Connected via SSH.
          ERROR: Server rejected the 1 private key(s) for Build (credentialId:tester-gce-ssh/method:publickey)
          Mar 27, 2019 12:34:57 PM null
          WARNING: Authentication failed. Trying again...

          Vera Akulova added a comment - It's still not working. Upgraded plugin to "org.jenkins-ci.plugins:google-compute-engine:3.0.0": "configure" jenkins page has "Run as user"=tester, but log for automatically started instance contains the following (uses Build user name instead of tester): Mar 27, 2019 12:34:25 PM null FINEST: Instance zulu-win2016-tests-gce-ulumed is running and ready... Mar 27, 2019 12:34:25 PM null INFO: Launching instance: zulu-win2016-tests-gce-ulumed Mar 27, 2019 12:34:25 PM null INFO: bootstrap Mar 27, 2019 12:34:25 PM null INFO: Authenticating as Build Mar 27, 2019 12:34:25 PM null INFO: Connecting to 35.197.99.246 on port 22, with timeout 10000. Mar 27, 2019 12:34:35 PM null INFO: Failed to connect via ssh: The kexTimeout (10000 ms) expired. Mar 27, 2019 12:34:35 PM null INFO: Waiting for SSH to come up. Sleeping 5. Mar 27, 2019 12:34:41 PM null INFO: Connecting to 35.197.99.246 on port 22, with timeout 10000. Mar 27, 2019 12:34:41 PM null INFO: Connected via SSH. ERROR: Server rejected the 1 private key(s) for Build (credentialId:tester-gce-ssh/method:publickey) Mar 27, 2019 12:34:41 PM null WARNING: Authentication failed. Trying again... Mar 27, 2019 12:34:56 PM null INFO: Authenticating as Build Mar 27, 2019 12:34:56 PM null INFO: Connecting to 35.197.99.246 on port 22, with timeout 10000. Mar 27, 2019 12:34:57 PM null INFO: Connected via SSH. ERROR: Server rejected the 1 private key(s) for Build (credentialId:tester-gce-ssh/method:publickey) Mar 27, 2019 12:34:57 PM null WARNING: Authentication failed. Trying again...

          Rachel Yen added a comment -

          I'll have to dive into the issue with the username, but it looks like you're also having issues with your keys. Did you put the wrong key into Jenkins for your credentials?

          Rachel Yen added a comment - I'll have to dive into the issue with the username, but it looks like you're also having issues with your keys. Did you put the wrong key into Jenkins for your credentials?

          Rachel Yen added a comment -

          Regarding the username issue, can you show me your instance configuration set up?

          Rachel Yen added a comment - Regarding the username issue, can you show me your instance configuration set up?

          Rachel Yen added a comment -

          Rachel Yen added a comment - If possible, can we also migrate to:  https://github.com/jenkinsci/google-compute-engine-plugin/issues/

          Craig Barber added a comment -

          Craig Barber added a comment - Migrated to GitHub: https://github.com/jenkinsci/google-compute-engine-plugin/issues/75

            zombiemoose Rachel Yen
            verusikk Vera Akulova
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: