Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42856

Unable to launch Windows slaves using Microsoft OpenSSH: Unexpected termination of the channel

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Priority: Minor
    • Resolution: Done
    • Component/s: ssh-slaves-plugin
    • Labels:
      None
    • Environment:
      Windows 2012 R2 master and slaves, Jenkins 2.32.3 LTS, Win32-OpenSSH 0.0.10.0, SSH Slaves plugin 1.14
    • Similar Issues:

      Description

      As a first step to being able to use the VMware plugin to spawn slaves automatically, I'm trying to get ssh slave connectivity to work. Normally I use Java Web Start for permanent slaves, which works fine, but I have no way to use that with a dynamic slave (without turning off the connect security and forgoing having a secret in the JNLP command).

      I've installed the Microsoft SSH port (Win32-OpenSSH) for sshd, and Cygwin to get the "true" utility that Jenkins expects to call, but my attempts to launch the agent fail almost immediately (after copying the slave.jar file), with the following log:

      [03/16/17 14:35:26] [SSH] Opening SSH connection to myserver:22.
      [03/16/17 14:35:31] [SSH] Authentication successful.
      [03/16/17 14:35:34] [SSH] The remote users environment is:
      [03/16/17 14:35:36] [SSH] Starting sftp client.
      [03/16/17 14:35:39] [SSH] Copying latest slave.jar...
      [03/16/17 14:36:00] [SSH] Copied 717,563 bytes.
      Expanded the channel window size to 4MB
      [03/16/17 14:36:00] [SSH] Starting slave process: cd "C:\Jenkins" && "C:\Program Files\Java\jre1.8.0_121\bin\java"  -jar slave.jar
      <===[JENKINS REMOTING CAPACITY]===><===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
      hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      	at hudson.remoting.Request.abort(Request.java:307)
      	at hudson.remoting.Channel.terminate(Channel.java:888)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:92)
      	at ......remote call to myserver(Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1537)
      	at hudson.remoting.Request.call(Request.java:172)
      	at hudson.remoting.Channel.call(Channel.java:821)
      	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:516)
      	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
      	at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:989)
      	at hudson.plugins.sshslaves.SSHLauncher.access$400(SSHLauncher.java:139)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:738)
      	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:719)
      	at java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: java.io.IOException: Unexpected termination of the channel
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
      Caused by: java.io.EOFException
      	at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
      	at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
      	at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
      	at java.io.ObjectInputStream.<init>(Unknown Source)
      	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
      [03/16/17 14:36:09] Launch failed - cleaning up connection
      ERROR: Connection terminated
      java.io.IOException: Unexpected termination of the channel
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
      Caused by: java.io.EOFException
      	at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
      	at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
      	at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
      	at java.io.ObjectInputStream.<init>(Unknown Source)
      	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
      [03/16/17 14:36:09] [SSH] Connection closed.

      I've searched and found a number of cases of the "This is probably a bug in Jenkins." error in launching a slave, but seemingly not this exact one. It seems to be that the connection or process is being terminated on the slave side but I can't figure out by what.

      How can this be fixed or worked around?

      EDIT (March 17, 2017): Per Stephen Connolly and my own testing, this appears to be n incompatibility between the Jenkins SSH Slaves plugin and a bug in the Microsoft's Win32-OpenSSH port specifically. When I switched the slave node to run Cygwin's OpenSSH, it worked perfectly.

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment - - edited

            Ivan Fernandez Calvo I am running several Windows 10 machines as agents with the Microsoft OpenSSH server by using:

            Prefix Start Agent Command - Works

            powershell -Command "cd C:\J\S ; C:\J\S\jdk\bin\java.exe -jar remoting.jar" ; exit 0 ; rem '
            

            Suffix Start Agent Command - Works

            '
            

            Simpler Prefix Start Agent Command - Fails

            powershell -Command "
            

            Simpler Suffix Start Agent Command - Fails

            "
            

            When I attempt the simpler strings suggested by Ivan Fernandez Calvo, PowerShell reports the following error message:

            [12/07/18 08:24:08] [SSH] Checking java version of C:\J\V/jdk/bin/java
            [12/07/18 08:24:09] [SSH] C:\J\V/jdk/bin/java -version returned 1.8.0_181.
            [12/07/18 08:24:09] [SSH] Starting sftp client.
            [12/07/18 08:24:09] [SSH] Copying latest remoting.jar...
            [12/07/18 08:24:09] [SSH] Copied 776,717 bytes.
            Expanded the channel window size to 4MB
            [12/07/18 08:24:09] [SSH] Starting agent process: powershell -Command "cd "C:\J\V" && C:\J\V/jdk/bin/java  -jar remoting.jar -workDir C:\J\V"
            At line:1 char:11
            + cd C:\J\V && C:\J\V/jdk/bin/java  -jar remoting.jar -workDir C:\J\V
            +           ~~
            The token '&&' is not a valid statement separator in this version.
                + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
                + FullyQualifiedErrorId : InvalidEndOfLine
             
            Slave JVM has terminated. Exit code=1
            

            I believe that message means that the command used by ssh-slaves plugin to start the remote agent is not understood by PowerShell (though it is understood by the Windows Command Processor cmd.exe).

            I'm quite pleased that there is a working technique to use Windows OpenSSH server for Jenkins agents. I'll explore further and report more of what I learn.

            Show
            markewaite Mark Waite added a comment - - edited Ivan Fernandez Calvo I am running several Windows 10 machines as agents with the Microsoft OpenSSH server by using: Prefix Start Agent Command - Works powershell -Command "cd C:\J\S ; C:\J\S\jdk\bin\java.exe -jar remoting.jar" ; exit 0 ; rem ' Suffix Start Agent Command - Works ' Simpler Prefix Start Agent Command - Fails powershell -Command " Simpler Suffix Start Agent Command - Fails " When I attempt the simpler strings suggested by Ivan Fernandez Calvo , PowerShell reports the following error message: [12/07/18 08:24:08] [SSH] Checking java version of C:\J\V/jdk/bin/java [12/07/18 08:24:09] [SSH] C:\J\V/jdk/bin/java -version returned 1.8.0_181. [12/07/18 08:24:09] [SSH] Starting sftp client. [12/07/18 08:24:09] [SSH] Copying latest remoting.jar... [12/07/18 08:24:09] [SSH] Copied 776,717 bytes. Expanded the channel window size to 4MB [12/07/18 08:24:09] [SSH] Starting agent process: powershell -Command "cd "C:\J\V" && C:\J\V/jdk/bin/java -jar remoting.jar -workDir C:\J\V" At line:1 char:11 + cd C:\J\V && C:\J\V/jdk/bin/java -jar remoting.jar -workDir C:\J\V + ~~ The token '&&' is not a valid statement separator in this version. + CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : InvalidEndOfLine Slave JVM has terminated. Exit code=1 I believe that message means that the command used by ssh-slaves plugin to start the remote agent is not understood by PowerShell (though it is understood by the Windows Command Processor cmd.exe). I'm quite pleased that there is a working technique to use Windows OpenSSH server for Jenkins agents. I'll explore further and report more of what I learn.
            Hide
            gamma Gerry Weißbach added a comment -

            Please note our most recent version of the command is the following:

            Mark Waite your command shows a second quote after 'cd' which breaks the command.

            Prefix:

            cd C:/jenkins ; java -jar remoting.jar -workDir C:/jenkins ; exit 0 ; # "

            Postfix:

            "

            Show
            gamma Gerry Weißbach added a comment - Please note our most recent version of the command is the following: Mark Waite your command shows a second quote after 'cd' which breaks the command. Prefix: cd C:/jenkins ; java -jar remoting.jar -workDir C:/jenkins ; exit 0 ; # " Postfix: "
            Hide
            markewaite Mark Waite added a comment -

            Gerry Weißbach I don't understand your comment. The text I've placed in the "Prefix Start Agent Command" is a verbatim copy of the text that is working on my Jenkins server right now, connecting an ssh agent on a Windows machine with a Docker master running on Debian slim. It has been working reasonably enough that I've replaced two of my JNLP Windows agents with ssh agents and have used them for all my testing the last 2 days.

            Can you explain further?

            Show
            markewaite Mark Waite added a comment - Gerry Weißbach I don't understand your comment. The text I've placed in the "Prefix Start Agent Command" is a verbatim copy of the text that is working on my Jenkins server right now, connecting an ssh agent on a Windows machine with a Docker master running on Debian slim. It has been working reasonably enough that I've replaced two of my JNLP Windows agents with ssh agents and have used them for all my testing the last 2 days. Can you explain further?
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment -

            I am working on some kind of commands profiles that allow you to select different commands for different OS (bash, cmd, PowerShell, Python, whatever)

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - I am working on some kind of commands profiles that allow you to select different commands for different OS (bash, cmd, PowerShell, Python, whatever)
            Hide
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited

            I open an improvement JENKINS-55362 and add Mark Waite's comments to the Configuration Guide, thus I am going to close this issue.

            Show
            ifernandezcalvo Ivan Fernandez Calvo added a comment - - edited I open an improvement JENKINS-55362 and add Mark Waite's comments to the Configuration Guide , thus I am going to close this issue.

              People

              Assignee:
              ifernandezcalvo Ivan Fernandez Calvo
              Reporter:
              medianick Nick Jones
              Votes:
              5 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: