I just updated to the latest version of the SSH Slaves plugin, 1.14, and tried again. The only difference is that now it correctly outputs the remote user's environment. It still fails immediately after slave.jar copying with:

ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.Request.abort(Request.java:307)
	at hudson.remoting.Channel.terminate(Channel.java:888)

etc.

CC stephenconnolly since he likely has some context

The Microsoft OpenSSH port has some issue that I - after 2 days fighting - was unable to get around.

There is currently no way to make this work with Jenkins.

It does not work even using the `use a command on the command line` option to fire a native ssh client.

I am not sure if it is CHCP code page related mangling of the streams (though ssh -T should avoid that, and that didn't work)

I wish I had even a reproducible test case that I could provide to Microsoft to raise the issue.

The state of affairs results in me having to post this here: 

Thanks stephenc. Taking a step back, is there any sshd that has been found to work with Jenkins on Windows, if not the Microsoft OpenSSH one?

I switched to Cygwin's OpenSSH and got this working.

medianick that's great news.

Just to clarify, this is not an issue with the ssh-slaves plugin.

There is something about the Microsoft port of OpenSSH that prevents it from working.

If it was an issue with the ssh-slaves plugin then using a launch method such as:

would connect the agent.

However, one cannot get that to work with Microsoft's port of OpenSSH.

Thus I conclude that the issue is with Microsoft's port of OpenSSH in general.

As this appears to be a bug in Microsoft's OpenSSH port, I'd agree with stephenconnolly that this is not a bug in Jenkins or its SSH Slaves plugin. I'm happy to have this issue out there documenting that the Cygwin OpenSSH works instead.

I am using cygwin sshd and I face the same issue, but only if I use "noacl" in /etc/fstab on the slave. If I remove "noacl" from the /etc/fstab file, the slave works.

It seems to be some issue when the slave.jar tries to replace "jenkins-slave.exe" with a newer one. To sum up, neither of these mount lines work: (Default path: c:\cygwin\etc\fstab)
 
none /cygdrive cygdrive binary,posix=0,user,noacl,exec 0 0
none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

This one works:
none /cygdrive cygdrive binary,posix=0,user 0 0

The reported error can be re-created using "cygwin sshd" if the "fstab" file is modified. Test environment:

master: Windows Server 2008 R2 - Jenkins ver 2.62
slave: Windows 7, cygwin sshd -

The slave runs the "CYGWIN sshd" service logged in as a domain user with admin rights. I've created SSH keys for password free login.

I've shown 3 ways to modify the file c:\cygwin\etc\fstab in the comments, 2 ways that fail and 1 way that works. Perhaps this can be useful in re-creating the problem.

I'm also trying to use the Win32-OpenSSH implementation, because getting our build system to run via the environment of the msys2/cygwin openssh server was just too complicated.

(The main problem was that even with MSYS2_PATH_TYPE=inherit my (quite long) path was truncated by msys2. On top of that, the build scripts needed a specific perl version etc.).

Now, I can connect to a slave with Launch agent via execution of command on master with ssh user@foo.corp.com java -jar jenkins/slave.jar, but connecting through the ssh slave plugin fails.

The first problem - mentioned above - is that the ssh slave plugin runs the true command, which does not natively exist on Windows. That problem can be easily worked around.

But after that, I also get the error:

<===[JENKINS REMOTING CAPACITY]===><===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.

If this is a bug in the Win32-OpenSSH implementation, could you give details? I'd file a bug at https://github.com/PowerShell/Win32-OpenSSH/issues if I knew more.

I have the same issue as the previous comment, i.e. major issues seem to be resolved in Win32-OpenSSH. It works when running the ssh command manually from the server, but not when using ssh-slaves-plugin.

I wonder if it could be related to the channel window size ("Expanded the channel window size to 4MB") triggering a bug in Win32-OpenSSH?

It works when running the ssh command manually from the server, but not when using ssh-slaves-plugin.

patrickm Do you mean where you have configured the agent to launch like

Or do you mean where you just type ssh in a shell on the server

I thought I had it working yesterday using the 'launch agent via execution of command on the master', but whatever I try now it doesn't work . Using (or omitting) -T gives similar results as ssh-slaves plugin (Unexpected termination of the channel). Using ssh -tt triggers a different exception (java.io.IOException: Invalid encoded sequence encountered: 3B 33 36 48).

When I run it on the command-line it works and starts the slave:

jenkins@patrick:~$ ssh -i ~/.ssh/jenkins_key patrick-win10 "cd c:\Jenkins && java -jar slave.jar"
<===[JENKINS REMOTING CAPACITY]===>rO0ABXNyABpodWRzb24ucmVtb3RpbmcuQ2FwYWJpbGl0eQAAAAAAAAABAgABSgAEbWFza3hwAAAAAAAAAP4=

I'm using default 0.0.22.0 Win32-OpenSSH (64-bit). So shell is set to cmd.exe, I may try a different shell (bash, powershell) when I have some time.

patrickm I spent 3 weeks banging my head against that, it happens with both Trilead SSH and Apache MINA SSH java libraries.

Good luck, but I wouldn't recommend holding your breath... just use Cygwin

I've opened an issue over at PowerShell/Win32-OpenSSH:

https://github.com/PowerShell/Win32-OpenSSH/issues/927

I have a similar problem with the channel being terminated too early. I was able to circumvent it using the following SSH prefix and postfix commands:

• Prefix: powershell -Command "cd C:\jenkins ; java -jar slave.jar" ; exit 0 ; rem '
• Postfix: '

gamma, your method worked for me with a permanent slave node running Windows Server 2016 with OpenSSH 0.0.22. I installed Cygwin and added it to the Path simply to have a true.exe utility but was able to use OpenSSH's sshd rather than Cygwin's. Both OpenSSH and Cygwin were installed via Chocolatey.

medianick I faked the true.exe to not have the cygwin dependency. I created a file "/WINDOWS/system32/true.bat" wit the following content for that:

@%COMSPEC% /C exit 1 >nul

Thanks gamma! That same trick worked for me, and I've removed Cygwin.

Next on my list is to see if this setup can work with the Azure VM Agents plugin (for dynamic agents from a base image), although that doesn't appear to have the same Prefix and Postfix capabilities. I'm suspecting I may need Cygwin in that case, although I'm pleased to have a non-Cygwin solution that works for permanent nodes.

JENKINS-37152 adds support for Win32-OpenSSH

Just wanted to signal that this issue still exists as JENKINS-37152 fix does not correct this behavior, at least for me.

I had to use gamma's postfix & prefix tweak to make my node work on SSH Slaves plugin 1.28.1 (latest).

emixam Could you attach the log of the node to see the exact error? Also this info Common info needed to troubleshooting a bug

But in any case, it is weird the 'true' command it is not used any more, now it uses the command 'exit 0'

https://github.com/jenkinsci/ssh-slaves-plugin/commit/bf5488b09029b4cf153dbc0b46a89094f249872f

• Jenkins core version : 2.138 / SSH Slaves plugin 1.28.1
• OS you use on your SSH agents : CentOS7 master/ Windows 10 Pro slave.
• OpenSSH version you have installed on your SSH agents?  OpenSSH_7.4p1 on CentOS / OpenSSH_7.6p1for Windows on Windows 10

sshlogMaster.log

emixam Java 10 is not supported jet see JENKINS-4541

[09/21/18 09:17:05] [SSH] java -version returned 10.0.2.

I think you error is not related with the "true" command, the error is in this line, it is executed after the test, so no related.
Are log files in C:\Jenkins\remoting? could you attach them?

Did you said that making a workaround it works? What do you exactly change? if you attach a node log with the configuration that works I could see the configuration in the first line.

For the record.

Jenkins 2.121.3 with Windows 7 machines and OpenSSH as per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

java version "1.8.0_162"

Using Launch agent via execution of command on the master with command as follows

ssh -T machine.fqdn.name 'powershell -Command "cd f:\jenkins ; java -jar slave.jar"' 

works as expected.

It seems related with the shell used, probably if you set the "Prefix start agent command" and the "Suffix start agent command" to those two values you fix the issue. If it works, we could try to find a way to launch the remoting process with powershell.

"Prefix start agent command" -

powershell -Command "

"Suffix start agent command" -

"

ifernandezcalvo I am running several Windows 10 machines as agents with the Microsoft OpenSSH server by using:

Prefix Start Agent Command - Works

powershell -Command "cd C:\J\S ; C:\J\S\jdk\bin\java.exe -jar remoting.jar" ; exit 0 ; rem '

Suffix Start Agent Command - Works

'

Simpler Prefix Start Agent Command - Fails

powershell -Command "

Simpler Suffix Start Agent Command - Fails

"

When I attempt the simpler strings suggested by ifernandezcalvo, PowerShell reports the following error message:

[12/07/18 08:24:08] [SSH] Checking java version of C:\J\V/jdk/bin/java
[12/07/18 08:24:09] [SSH] C:\J\V/jdk/bin/java -version returned 1.8.0_181.
[12/07/18 08:24:09] [SSH] Starting sftp client.
[12/07/18 08:24:09] [SSH] Copying latest remoting.jar...
[12/07/18 08:24:09] [SSH] Copied 776,717 bytes.
Expanded the channel window size to 4MB
[12/07/18 08:24:09] [SSH] Starting agent process: powershell -Command "cd "C:\J\V" && C:\J\V/jdk/bin/java  -jar remoting.jar -workDir C:\J\V"
At line:1 char:11
+ cd C:\J\V && C:\J\V/jdk/bin/java  -jar remoting.jar -workDir C:\J\V
+           ~~
The token '&&' is not a valid statement separator in this version.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : InvalidEndOfLine
 
Slave JVM has terminated. Exit code=1

I believe that message means that the command used by ssh-slaves plugin to start the remote agent is not understood by PowerShell (though it is understood by the Windows Command Processor cmd.exe).

I'm quite pleased that there is a working technique to use Windows OpenSSH server for Jenkins agents. I'll explore further and report more of what I learn.

Please note our most recent version of the command is the following:

markewaite your command shows a second quote after 'cd' which breaks the command.

Prefix:

cd C:/jenkins ; java -jar remoting.jar -workDir C:/jenkins ; exit 0 ; # "

Postfix:

"

gamma I don't understand your comment. The text I've placed in the "Prefix Start Agent Command" is a verbatim copy of the text that is working on my Jenkins server right now, connecting an ssh agent on a Windows machine with a Docker master running on Debian slim. It has been working reasonably enough that I've replaced two of my JNLP Windows agents with ssh agents and have used them for all my testing the last 2 days.

Can you explain further?

I am working on some kind of commands profiles that allow you to select different commands for different OS (bash, cmd, PowerShell, Python, whatever)

I open an improvement JENKINS-55362 and add Mark Waite's comments to the Configuration Guide, thus I am going to close this issue.

BTW, markewaite, gamma, I think both your commands are correct. It just depends on whether the default shell has been changed to powershell or not:

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force