Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42856

Unable to launch Windows slaves using Microsoft OpenSSH: Unexpected termination of the channel

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Minor Minor
    • ssh-slaves-plugin
    • None
    • Windows 2012 R2 master and slaves, Jenkins 2.32.3 LTS, Win32-OpenSSH 0.0.10.0, SSH Slaves plugin 1.14

      As a first step to being able to use the VMware plugin to spawn slaves automatically, I'm trying to get ssh slave connectivity to work. Normally I use Java Web Start for permanent slaves, which works fine, but I have no way to use that with a dynamic slave (without turning off the connect security and forgoing having a secret in the JNLP command).

      I've installed the Microsoft SSH port (Win32-OpenSSH) for sshd, and Cygwin to get the "true" utility that Jenkins expects to call, but my attempts to launch the agent fail almost immediately (after copying the slave.jar file), with the following log:

      [03/16/17 14:35:26] [SSH] Opening SSH connection to myserver:22.
[03/16/17 14:35:31] [SSH] Authentication successful.
[03/16/17 14:35:34] [SSH] The remote users environment is:
[03/16/17 14:35:36] [SSH] Starting sftp client.
[03/16/17 14:35:39] [SSH] Copying latest slave.jar...
[03/16/17 14:36:00] [SSH] Copied 717,563 bytes.
Expanded the channel window size to 4MB
[03/16/17 14:36:00] [SSH] Starting slave process: cd "C:\Jenkins" && "C:\Program Files\Java\jre1.8.0_121\bin\java"  -jar slave.jar
<===[JENKINS REMOTING CAPACITY]===><===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Request.abort(Request.java:307)
	at hudson.remoting.Channel.terminate(Channel.java:888)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:92)
	at ......remote call to myserver(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
	at hudson.remoting.Request.call(Request.java:172)
	at hudson.remoting.Channel.call(Channel.java:821)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:516)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
	at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:989)
	at hudson.plugins.sshslaves.SSHLauncher.access$400(SSHLauncher.java:139)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:738)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:719)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
	at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
	at java.io.ObjectInputStream.<init>(Unknown Source)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
[03/16/17 14:36:09] Launch failed - cleaning up connection
ERROR: Connection terminated
java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused by: java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
	at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
	at java.io.ObjectInputStream.<init>(Unknown Source)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
[03/16/17 14:36:09] [SSH] Connection closed.

      I've searched and found a number of cases of the "This is probably a bug in Jenkins." error in launching a slave, but seemingly not this exact one. It seems to be that the connection or process is being terminated on the slave side but I can't figure out by what.

      How can this be fixed or worked around?

      EDIT (March 17, 2017): Per stephenconnolly and my own testing, this appears to be n incompatibility between the Jenkins SSH Slaves plugin and a bug in the Microsoft's Win32-OpenSSH port specifically. When I switched the slave node to run Cygwin's OpenSSH, it worked perfectly.

        1. image-2017-03-16-22-26-53-920.png
          image-2017-03-16-22-26-53-920.png
          71 kB
        2. Screen Shot 2017-03-17 at 20.59.21.png
          Screen Shot 2017-03-17 at 20.59.21.png
          44 kB
        3. sshlogMaster.log
          5 kB
        4. sshlogMaster.log
          5 kB

          [JENKINS-42856] Unable to launch Windows slaves using Microsoft OpenSSH: Unexpected termination of the channel

          Ivan Fernandez Calvo added a comment - - edited

          emixam Java 10 is not supported jet see JENKINS-4541

          [09/21/18 09:17:05] [SSH] java -version returned 10.0.2.

          I think you error is not related with the "true" command, the error is in this line, it is executed after the test, so no related.
          Are log files in C:\Jenkins\remoting? could you attach them?

          Did you said that making a workaround it works? What do you exactly change? if you attach a node log with the configuration that works I could see the configuration in the first line.

          Ivan Fernandez Calvo added a comment - - edited emixam Java 10 is not supported jet see JENKINS-4541 [09/21/18 09:17:05] [SSH] java -version returned 10.0.2. I think you error is not related with the "true" command, the error is in this line , it is executed after the test, so no related. Are log files in C:\Jenkins\remoting? could you attach them? Did you said that making a workaround it works? What do you exactly change? if you attach a node log with the configuration that works I could see the configuration in the first line.

          Maxime Philippe added a comment -

          Oh, I did not know about Java 10.

          The issue I'm having is this one (JENKINS-42856) with an "Unexpected termination of the channel". I linked JENKINS-37152 only due to it's (misleading) title "Support Win32-OpenSSH".

          The fix is this one above:

           I was able to circumvent it using the following SSH prefix and postfix commands:  

          • Prefix: powershell -Command "cd C:\jenkins ; java -jar slave.jar" ; exit 0 ; rem '
          • Postfix: '

          Sorry, I'm not having much time to answer you.

          Maxime Philippe added a comment - Oh, I did not know about Java 10. The issue I'm having is this one ( JENKINS-42856 ) with an "Unexpected termination of the channel". I linked JENKINS-37152 only due to it's (misleading) title "Support Win32-OpenSSH". The fix is this one above:  I was able to circumvent it using the following SSH prefix and postfix commands:   Prefix: powershell -Command "cd C:\jenkins ; java -jar slave.jar" ; exit 0 ; rem ' Postfix: ' Sorry, I'm not having much time to answer you.

          Henti Smith added a comment -

          For the record.

          Jenkins 2.121.3 with Windows 7 machines and OpenSSH as per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

          java version "1.8.0_162"

          Using Launch agent via execution of command on the master with command as follows

          ssh -T machine.fqdn.name 'powershell -Command "cd f:\jenkins ; java -jar slave.jar"'

          works as expected.

          Henti Smith added a comment - For the record. Jenkins 2.121.3 with Windows 7 machines and OpenSSH as per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH java version "1.8.0_162" Using  Launch agent via execution of command on the master with command as follows ssh -T machine.fqdn.name 'powershell -Command "cd f:\jenkins ; java -jar slave.jar" ' works as expected.

          Ivan Fernandez Calvo added a comment -

          It seems related with the shell used, probably if you set the "Prefix start agent command" and the "Suffix start agent command" to those two values you fix the issue. If it works, we could try to find a way to launch the remoting process with powershell.

          "Prefix start agent command" - 

          powershell -Command "


          "Suffix start agent command" - 

          "

          Ivan Fernandez Calvo added a comment - It seems related with the shell used, probably if you set the "Prefix start agent command" and the "Suffix start agent command" to those two values you fix the issue. If it works, we could try to find a way to launch the remoting process with powershell. "Prefix start agent command" - powershell -Command " "Suffix start agent command" - "

          Mark Waite added a comment - - edited

          ifernandezcalvo I am running several Windows 10 machines as agents with the Microsoft OpenSSH server by using:

          Prefix Start Agent Command - Works

          powershell -Command "cd C:\J\S ; C:\J\S\jdk\bin\java.exe -jar remoting.jar" ; exit 0 ; rem '

          Suffix Start Agent Command - Works

          '

          Simpler Prefix Start Agent Command - Fails

          powershell -Command "

          Simpler Suffix Start Agent Command - Fails

          "

          When I attempt the simpler strings suggested by ifernandezcalvo, PowerShell reports the following error message:

          [12/07/18 08:24:08] [SSH] Checking java version of C:\J\V/jdk/bin/java
[12/07/18 08:24:09] [SSH] C:\J\V/jdk/bin/java -version returned 1.8.0_181.
[12/07/18 08:24:09] [SSH] Starting sftp client.
[12/07/18 08:24:09] [SSH] Copying latest remoting.jar...
[12/07/18 08:24:09] [SSH] Copied 776,717 bytes.
Expanded the channel window size to 4MB
[12/07/18 08:24:09] [SSH] Starting agent process: powershell -Command "cd "C:\J\V" && C:\J\V/jdk/bin/java  -jar remoting.jar -workDir C:\J\V"
At line:1 char:11
+ cd C:\J\V && C:\J\V/jdk/bin/java  -jar remoting.jar -workDir C:\J\V
+           ~~
The token '&&' is not a valid statement separator in this version.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : InvalidEndOfLine
 
Slave JVM has terminated. Exit code=1

          I believe that message means that the command used by ssh-slaves plugin to start the remote agent is not understood by PowerShell (though it is understood by the Windows Command Processor cmd.exe).

          I'm quite pleased that there is a working technique to use Windows OpenSSH server for Jenkins agents. I'll explore further and report more of what I learn.

          Mark Waite added a comment - - edited ifernandezcalvo I am running several Windows 10 machines as agents with the Microsoft OpenSSH server by using: Prefix Start Agent Command - Works powershell -Command "cd C:\J\S ; C:\J\S\jdk\bin\java.exe -jar remoting.jar" ; exit 0 ; rem ' Suffix Start Agent Command - Works ' Simpler Prefix Start Agent Command - Fails powershell -Command " Simpler Suffix Start Agent Command - Fails " When I attempt the simpler strings suggested by ifernandezcalvo , PowerShell reports the following error message: [12/07/18 08:24:08] [SSH] Checking java version of C:\J\V/jdk/bin/java [12/07/18 08:24:09] [SSH] C:\J\V/jdk/bin/java -version returned 1.8.0_181. [12/07/18 08:24:09] [SSH] Starting sftp client. [12/07/18 08:24:09] [SSH] Copying latest remoting.jar... [12/07/18 08:24:09] [SSH] Copied 776,717 bytes. Expanded the channel window size to 4MB [12/07/18 08:24:09] [SSH] Starting agent process: powershell -Command "cd "C:\J\V" && C:\J\V/jdk/bin/java -jar remoting.jar -workDir C:\J\V" At line:1 char:11 + cd C:\J\V && C:\J\V/jdk/bin/java -jar remoting.jar -workDir C:\J\V + ~~ The token '&&' is not a valid statement separator in this version. + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : InvalidEndOfLine Slave JVM has terminated. Exit code=1 I believe that message means that the command used by ssh-slaves plugin to start the remote agent is not understood by PowerShell (though it is understood by the Windows Command Processor cmd.exe). I'm quite pleased that there is a working technique to use Windows OpenSSH server for Jenkins agents. I'll explore further and report more of what I learn.

          Gerry Weißbach added a comment -

          Please note our most recent version of the command is the following:

          markewaite your command shows a second quote after 'cd' which breaks the command.

          Prefix:

          cd C:/jenkins ; java -jar remoting.jar -workDir C:/jenkins ; exit 0 ; # "

          Postfix:

          "

          Gerry Weißbach added a comment - Please note our most recent version of the command is the following: markewaite your command shows a second quote after 'cd' which breaks the command. Prefix: cd C:/jenkins ; java -jar remoting.jar -workDir C:/jenkins ; exit 0 ; # " Postfix: "

          Mark Waite added a comment -

          gamma I don't understand your comment. The text I've placed in the "Prefix Start Agent Command" is a verbatim copy of the text that is working on my Jenkins server right now, connecting an ssh agent on a Windows machine with a Docker master running on Debian slim. It has been working reasonably enough that I've replaced two of my JNLP Windows agents with ssh agents and have used them for all my testing the last 2 days.

          Can you explain further?

          Mark Waite added a comment - gamma I don't understand your comment. The text I've placed in the "Prefix Start Agent Command" is a verbatim copy of the text that is working on my Jenkins server right now, connecting an ssh agent on a Windows machine with a Docker master running on Debian slim. It has been working reasonably enough that I've replaced two of my JNLP Windows agents with ssh agents and have used them for all my testing the last 2 days. Can you explain further?

          Ivan Fernandez Calvo added a comment -

          I am working on some kind of commands profiles that allow you to select different commands for different OS (bash, cmd, PowerShell, Python, whatever)

          Ivan Fernandez Calvo added a comment - I am working on some kind of commands profiles that allow you to select different commands for different OS (bash, cmd, PowerShell, Python, whatever)

          Ivan Fernandez Calvo added a comment - - edited

          I open an improvement JENKINS-55362 and add Mark Waite's comments to the Configuration Guide, thus I am going to close this issue.

          Ivan Fernandez Calvo added a comment - - edited I open an improvement JENKINS-55362 and add Mark Waite's comments to the Configuration Guide , thus I am going to close this issue.

          Dirk Heinrichs added a comment -

          BTW, markewaite, gamma, I think both your commands are correct. It just depends on whether the default shell has been changed to powershell or not:

          New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

          Dirk Heinrichs added a comment - BTW, markewaite , gamma , I think both your commands are correct. It just depends on whether the default shell has been changed to powershell or not: New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force

            ifernandezcalvo Ivan Fernandez Calvo
            medianick Nick Jones
            Votes:
            5 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: