Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45755

Unable to launch SSH Slave since 2.68 when HOME is not writable on Master

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Critical
    • Resolution: Fixed
    • Single master and single slave setup, both running Debian and using SSH slave where master user is not the same as the slave user. Jenkins 2.71 / Remoting 3.10

    Description

      Last working for me Jenkins version 2.67

      Versions not working 2.68 - 2.78

      Issue: SSH Slave will not launch

      Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

      I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

      There appears to be 2 issues:

      • The slave is not using the correct location for the default jar cache because it seems to be using the home directory of the master user.
      • Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.
      [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
      [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
      [07/24/17 10:42:11] [SSH] Authentication successful.
      [07/24/17 10:42:11] [SSH] The remote users environment is:
      BASH=/bin/bash
      BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
      BASH_ALIASES=()
      BASH_ARGC=()
      BASH_ARGV=()
      BASH_CMDS=()
      BASH_EXECUTION_STRING=set
      BASH_LINENO=()
      BASH_SOURCE=()
      BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
      BASH_VERSION='4.3.30(1)-release'
      DIRSTACK=()
      EUID=10169
      GROUPS=()
      HOME=/home/SPALDING/jenkinsbuildserver
      HOSTNAME=SLAVEHOST
      HOSTTYPE=x86_64
      IFS=$' \t\n'
      LANG=en_US.UTF-8
      LOGNAME=jenkinsbuildserver
      MACHTYPE=x86_64-pc-linux-gnu
      MAIL=/var/mail/jenkinsbuildserver
      OPTERR=1
      OPTIND=1
      OSTYPE=linux-gnu
      PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
      PIPESTATUS=([0]="0")
      PPID=21458
      PS4='+ '
      PWD=/home/SPALDING/jenkinsbuildserver
      SHELL=/bin/bash
      SHELLOPTS=braceexpand:hashall:interactive-comments
      SHLVL=1
      SSH_CLIENT='10.10.1.179 57938 22'
      SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
      TERM=dumb
      UID=10169
      USER=jenkinsbuildserver
      _=']'
      [07/24/17 10:42:11] [SSH] Checking java version of java
      [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
      [07/24/17 10:42:11] [SSH] Starting sftp client.
      [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
      [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
      Expanded the channel window size to 4MB
      [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
      Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
      Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
      <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
      java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
              at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
              at hudson.remoting.JarCache.getDefault(JarCache.java:32)
              at hudson.remoting.Channel.<init>(Channel.java:505)
              at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
              at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
              at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
              at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
              at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
              at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
      [07/24/17 10:42:12] Launch failed - cleaning up connection
      [07/24/17 10:42:12] [SSH] Connection closed.
      
      
      

      Reproducer using Docker

      Launch Jenkins using the following

      mkdir jenkins_home
      sudo chown 2000 jenkins_home
      sudo chmod 777 jenkins_home
      docker run -ti -v $(pwd)/jenkins_home:/var/jenkins_home  -u 2000 --rm -p 8080:8080 -p 50000:50000 jenkinsci/jenkins:2.73
      

      Create a jnlp agent, then try to connect. You will then get the following sequence:

      master

      Sep 12, 2017 1:06:16 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
      INFO: Accepted JNLP4-connect connection #1 from /172.17.0.1:38552
      Sep 12, 2017 1:06:16 PM org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer onRecv
      WARNING: [JNLP4-connect connection from 172.17.0.1/172.17.0.1:38552]
      java.lang.RuntimeException: Root directory not writable: ?/.jenkins/cache/jars
      	at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
      	at hudson.remoting.JarCache.getDefault(JarCache.java:32)
      	at hudson.remoting.Channel.<init>(Channel.java:505)
      	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:339)
      	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:149)
      	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
      	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
      	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      agent

      INFOS: Locating server among [http://localhost:8080/]
      sept. 12, 2017 1:06:16 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFOS: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Agent discovery successful
        Agent address: localhost
        Agent port:    50000
        Identity:      80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Handshaking
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Connecting to localhost:50000
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Trying protocol: JNLP4-connect
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Remote identity confirmed: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Connected
      sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
      INFOS: Terminated
      

      Attachments

        Issue Links

          Activity

            jmccormick Jesse McCormick created issue -
            jmccormick Jesse McCormick made changes -
            Field Original Value New Value
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SCIPAX-MGT-V06:22.[07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.[07/24/17 10:42:11] [SSH] Authentication successful.[07/24/17 10:42:11] [SSH] The remote users environment is:BASH=/bin/bashBASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepathBASH_ALIASES=()BASH_ARGC=()BASH_ARGV=()BASH_CMDS=()BASH_EXECUTION_STRING=setBASH_LINENO=()BASH_SOURCE=()BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")BASH_VERSION='4.3.30(1)-release'DIRSTACK=()EUID=10169GROUPS=()HOME=/home/SPALDING/jenkinsbuildserverHOSTNAME=SCIPAX-MGT-V06HOSTTYPE=x86_64IFS=$' \t\n'LANG=en_US.UTF-8LOGNAME=jenkinsbuildserverMACHTYPE=x86_64-pc-linux-gnuMAIL=/var/mail/jenkinsbuildserverOPTERR=1OPTIND=1OSTYPE=linux-gnuPATH=/usr/local/bin:/usr/bin:/bin:/usr/gamesPIPESTATUS=([0]="0")PPID=21458PS4='+ 'PWD=/home/SPALDING/jenkinsbuildserverSHELL=/bin/bashSHELLOPTS=braceexpand:hashall:interactive-commentsSHLVL=1SSH_CLIENT='10.10.1.179 57938 22'SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'TERM=dumbUID=10169USER=jenkinsbuildserver_=']'[07/24/17 10:42:11] [SSH] Checking java version of java[07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.[07/24/17 10:42:11] [SSH] Starting sftp client.[07/24/17 10:42:11] [SSH] Copying latest slave.jar...[07/24/17 10:42:11] [SSH] Copied 730,299 bytes.Expanded the channel window size to 4MB[07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissingJul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directoryBoth error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting<===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars        at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)        at hudson.remoting.JarCache.getDefault(JarCache.java:32)        at hudson.remoting.Channel.<init>(Channel.java:505)        at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)        at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)        at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)        at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)        at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)        at java.util.concurrent.FutureTask.run(FutureTask.java:266)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)        at java.lang.Thread.run(Thread.java:748)[07/24/17 10:42:12] Launch failed - cleaning up connection[07/24/17 10:42:12] [SSH] Connection closed.

             {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SCIPAX-MGT-V06:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SCIPAX-MGT-V06
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.

            {code}
            jmccormick Jesse McCormick made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SCIPAX-MGT-V06:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SCIPAX-MGT-V06
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.

            {code}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.
            jmccormick Jesse McCormick made changes -
            Description [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SCIPAX-MGT-V06:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SCIPAX-MGT-V06
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.

            {code}
            jmccormick Jesse McCormick made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SCIPAX-MGT-V06:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SCIPAX-MGT-V06
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.

            {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            jmccormick Jesse McCormick made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

             

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            jmccormick Jesse McCormick made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.
            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            oleg_nenashev Oleg Nenashev made changes -
            Component/s ssh-slaves-plugin [ 15578 ]
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Priority Minor [ 4 ] Critical [ 2 ]
            jmccormick Jesse McCormick made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the default jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            oleg_nenashev Oleg Nenashev added a comment -

            It seems to be related to the recent Remoting 3.10 upgrade, which became more protective in terms of cache management. IWill try to investigate the issue on Wednesday

            oleg_nenashev Oleg Nenashev added a comment - It seems to be related to the recent Remoting 3.10 upgrade, which became more protective in terms of cache management. IWill try to investigate the issue on Wednesday
            psunjohnny Peng Sun added a comment -

            Yeah, I have the same problem.

            psunjohnny Peng Sun added a comment - Yeah, I have the same problem.
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels regression
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Not in progress, got switched to other tasks.

            Anybody can pick it and propose a PR. I am still keeping it in my task queue, but no ETA

            oleg_nenashev Oleg Nenashev added a comment - - edited Not in progress, got switched to other tasks. Anybody can pick it and propose a PR. I am still keeping it in my task queue, but no ETA

            The exception comes from the Master. My bet is /usr/share/tomcat8/ is readonly on master.

            You can workaround the problem by setting a home directory for the user running jenkins to a writable directory, or set the system property user.home when launching the master.

            vlatombe Vincent Latombe added a comment - The exception comes from the Master. My bet is /usr/share/tomcat8/ is readonly on master. You can workaround the problem by setting a home directory for the user running jenkins to a writable directory, or set the system property user.home when launching the master.
            vlatombe Vincent Latombe made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the default jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the default jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}

            *Reproducer using Docker*

            Launch Jenkins using the following
            {code}
            mkdir jenkins_home
            sudo chown 2000 jenkins_home
            sudo chmod 777 jenkins_home
            docker run -ti -v $(pwd)/jenkins_home:/var/jenkins_home -u 2000 --rm -p 8080:8080 -p 50000:50000 jenkinsci/jenkins:2.73
            {code}

            Create a jnlp agent, then try to connect. You will then get the following sequence:

            master
            {code}
            Sep 12, 2017 1:06:16 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
            INFO: Accepted JNLP4-connect connection #1 from /172.17.0.1:38552
            Sep 12, 2017 1:06:16 PM org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer onRecv
            WARNING: [JNLP4-connect connection from 172.17.0.1/172.17.0.1:38552]
            java.lang.RuntimeException: Root directory not writable: ?/.jenkins/cache/jars
            at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
            at hudson.remoting.JarCache.getDefault(JarCache.java:32)
            at hudson.remoting.Channel.<init>(Channel.java:505)
            at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:339)
            at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:149)
            at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207)
            at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
            at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
            at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
            at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
            at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
            at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
            at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            {code}

            agent
            {code}
            INFOS: Locating server among [http://localhost:8080/]
            sept. 12, 2017 1:06:16 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
            INFOS: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Agent discovery successful
              Agent address: localhost
              Agent port: 50000
              Identity: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Handshaking
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Connecting to localhost:50000
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Trying protocol: JNLP4-connect
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Remote identity confirmed: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Connected
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Terminated
            {code}
            vlatombe Vincent Latombe added a comment - Culprit commit: https://github.com/jenkinsci/remoting/commit/589275ea8d2198f86aa9fc78e68b024446c6124b
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev added a comment -

            After some review I am going to revert the default value. Although it is generally not correct, there are multiple ChannelBuilder usage cases in Jenkins core, which I cannot fix easily in a backportable way.

            E.g. SlaveComputer:

            public void setChannel(InputStream in, OutputStream out, OutputStream launchLog, Channel.Listener listener) throws IOException, InterruptedException {
            ChannelBuilder cb = new ChannelBuilder(nodeName,threadPoolForRemoting)
            .withMode(Channel.Mode.NEGOTIATE)
            .withHeaderStream(launchLog);
            oleg_nenashev Oleg Nenashev added a comment - After some review I am going to revert the default value. Although it is generally not correct, there are multiple ChannelBuilder usage cases in Jenkins core, which I cannot fix easily in a backportable way. E.g. SlaveComputer: public void setChannel(InputStream in, OutputStream out, OutputStream launchLog, Channel.Listener listener) throws IOException, InterruptedException { ChannelBuilder cb = new ChannelBuilder(nodeName,threadPoolForRemoting) .withMode(Channel.Mode.NEGOTIATE) .withHeaderStream(launchLog);
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "https://github.com/jenkinsci/remoting/pull/197 (Web Link)" [ 17670 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            oleg_nenashev Oleg Nenashev made changes -
            Summary Unable to launch SSH Slave since 2.68 Unable to launch SSH Slave since 2.68 when HOME is not writable
            oleg_nenashev Oleg Nenashev made changes -
            Summary Unable to launch SSH Slave since 2.68 when HOME is not writable Unable to launch SSH Slave since 2.68 when HOME is not writable on Master
            jglick Jesse Glick added a comment -

            I suspect the Docker image could be fixed with merely

            ENV HOME $JENKINS_HOME
            

            since we already did useradd -d "$JENKINS_HOME" and the error suggests that $HOME or System.getProperty("user.home") is resolving to ?. This is something we should fix anyway as there can be other code which would innocently try to use the home directory for something. I have seen it happen for Maven caches, etc.

            jglick Jesse Glick added a comment - I suspect the Docker image could be fixed with merely ENV HOME $JENKINS_HOME since we already did useradd -d "$JENKINS_HOME" and the error suggests that $HOME or System.getProperty("user.home") is resolving to ? . This is something we should fix anyway as there can be other code which would innocently try to use the home directory for something. I have seen it happen for Maven caches, etc.
            jglick Jesse Glick added a comment -

            $HOME does not work but user.home does. Working on a patch.

            jglick Jesse Glick added a comment - $HOME does not work but user.home does. Working on a patch.
            jglick Jesse Glick added a comment -

            docker PR 572 fixes the issue for the reproduction case kindly supplied by jmccormick.

            jglick Jesse Glick added a comment - docker PR 572 fixes the issue for the reproduction case kindly supplied by jmccormick .
            jglick Jesse Glick made changes -
            Remote Link This issue links to "docker PR 572 (Web Link)" [ 17674 ]
            jglick Jesse Glick added a comment -

            As noted in my PR, arguably the reproduction case is simply incorrect: the Jenkins user is supposed to be 1000, not 2000, and if you keep the standard user, user.home is writable already and there is no problem connecting JNLP agents.

            jglick Jesse Glick added a comment - As noted in my PR, arguably the reproduction case is simply incorrect: the Jenkins user is supposed to be 1000, not 2000, and if you keep the standard user, user.home is writable already and there is no problem connecting JNLP agents.
            jglick Jesse Glick added a comment -

            IOW the code fix here is valuable as a robustness measure but the supposed use case was invalid to begin with, so this is a low-priority issue.

            jglick Jesse Glick added a comment - IOW the code fix here is valuable as a robustness measure but the supposed use case was invalid to begin with, so this is a low-priority issue.
            danielbeck Daniel Beck added a comment - - edited

            This also affects the remoting connection to Maven builds:

            java.lang.RuntimeException: Root directory not writable: /var/jenkins_home/.jenkins/cache/jars
             at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
             at hudson.remoting.JarCache.getDefault(JarCache.java:32)
             at hudson.remoting.Channel.<init>(Channel.java:520)
             at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
             at hudson.slaves.Channels.forProcess(Channels.java:115)
             at hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:294)
             at hudson.maven.ProcessCache.get(ProcessCache.java:236)
             at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:803)
             at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
             at hudson.model.Run.execute(Run.java:1729)
             at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542)
             at hudson.model.ResourceController.execute(ResourceController.java:97)
             at hudson.model.Executor.run(Executor.java:419)
            danielbeck Daniel Beck added a comment - - edited This also affects the remoting connection to Maven builds: java.lang.RuntimeException: Root directory not writable: / var /jenkins_home/.jenkins/cache/jars at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57) at hudson.remoting.JarCache.getDefault(JarCache.java:32) at hudson.remoting.Channel.<init>(Channel.java:520) at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323) at hudson.slaves.Channels.forProcess(Channels.java:115) at hudson.maven.AbstractMavenProcessFactory.newProcess(AbstractMavenProcessFactory.java:294) at hudson.maven.ProcessCache.get(ProcessCache.java:236) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:803) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496) at hudson.model.Run.execute(Run.java:1729) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:419)
            danielbeck Daniel Beck added a comment - - edited

            Legacy remoting-based CLI connections are also affected:

            Sep 14, 2017 4:11:14 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
            SEVERE: A thread (TCP agent connection handler #6 with /192.168.99.1:65128/107) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
            java.lang.RuntimeException: Root directory not writable: /.jenkins/cache/jars
                at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                at hudson.remoting.Channel.<init>(Channel.java:520)
                at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                at hudson.cli.CliProtocol$Handler.runCli(CliProtocol.java:106)
                at hudson.cli.CliProtocol2$Handler2.run(CliProtocol2.java:101)
                at hudson.cli.CliProtocol2.handle(CliProtocol2.java:57)
                at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:235)
            danielbeck Daniel Beck added a comment - - edited Legacy remoting-based CLI connections are also affected: Sep 14, 2017 4:11:14 AM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException SEVERE: A thread (TCP agent connection handler #6 with /192.168.99.1:65128/107) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. java.lang.RuntimeException: Root directory not writable: /.jenkins/cache/jars     at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)     at hudson.remoting.JarCache.getDefault(JarCache.java:32)     at hudson.remoting.Channel.<init>(Channel.java:520)     at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)     at hudson.cli.CliProtocol$Handler.runCli(CliProtocol.java:106)     at hudson.cli.CliProtocol2$Handler2.run(CliProtocol2.java:101)     at hudson.cli.CliProtocol2.handle(CliProtocol2.java:57)     at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:235)
            oleg_nenashev Oleg Nenashev added a comment -

            Yes. There are many valid use-cases when the home directory should not be writable at all, so I disagree with the priority assessment by jglick . IMHO it is still a high-priority regression we need to fix, at least in .2.

            oleg_nenashev Oleg Nenashev added a comment - Yes. There are many valid use-cases when the home directory should not be writable at all, so I disagree with the priority assessment by jglick . IMHO it is still a high-priority regression we need to fix, at least in .2.
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "Upgrade Guide Notice (Web Link)" [ 17687 ]
            oleg_nenashev Oleg Nenashev made changes -
            Component/s core [ 15593 ]
            jmccormick Jesse McCormick added a comment - - edited

            In my specific case, the home directory for each user is writable, just not by the other user. Here is more detail on the setup:

            Master Node:

            • Master User: tomcat8
            • Master Home: /var/lib/tomcat8 (writable by user tomcat8)
            • Slave User: jenkinsbuildserver
            • Slave Home: Does not exist on master

            Slave Node:

            • Master User: Does not exist on slave
            • Master Home: Does not exist on slave
            • Slave User: jenkinsbuildserver
            • Slave Home: /home/SPALDING/jenkinsbuildserver (writable by user jenkinsbuildserver )
            jmccormick Jesse McCormick added a comment - - edited In my specific case, the home directory for each user is writable, just not by the other user. Here is more detail on the setup: Master Node: Master User: tomcat8 Master Home: /var/lib/tomcat8 (writable by user tomcat8) Slave User: jenkinsbuildserver Slave Home: Does not exist on master Slave Node: Master User: Does not exist on slave Master Home: Does not exist on slave Slave User: jenkinsbuildserver Slave Home: /home/SPALDING/jenkinsbuildserver (writable by user jenkinsbuildserver )

            Hi jmccormick, from your report it looks like the home directory for your tomcat8 user on the machine running master is /usr/share/tomcat8/ which is not writable. If it is not the user home, maybe the system property -Duser.home=/usr/share/tomcat8 is specified when launching your tomcat instance, this would have the same effect.

            vlatombe Vincent Latombe added a comment - Hi jmccormick , from your report it looks like the home directory for your tomcat8 user on the machine running master is /usr/share/tomcat8/ which is not writable. If it is not the user home, maybe the system property -Duser.home=/usr/share/tomcat8 is specified when launching your tomcat instance, this would have the same effect.

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            pom.xml
            http://jenkins-ci.org/commit/jenkins/018f9875ca58230afc4eb52ac66b3195f00128ef
            Log:
            [JENKINS-45755, JENKINS-46140] - Update Remoting to 3.12

            https://github.com/jenkinsci/remoting/blob/master/CHANGELOG.md#312

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/018f9875ca58230afc4eb52ac66b3195f00128ef Log: [JENKINS-45755, JENKINS-46140] - Update Remoting to 3.12 https://github.com/jenkinsci/remoting/blob/master/CHANGELOG.md#312

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            pom.xml
            http://jenkins-ci.org/commit/jenkins/e8b2f5a59c0b075f62bea9b6a45ef35a1c2ca1bb
            Log:
            Merge pull request #3025 from oleg-nenashev/remoting/3.12

            [JENKINS-45755, JENKINS-46140] - Update Remoting to 3.12

            Compare: https://github.com/jenkinsci/jenkins/compare/2343909f0240...e8b2f5a59c0b

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/e8b2f5a59c0b075f62bea9b6a45ef35a1c2ca1bb Log: Merge pull request #3025 from oleg-nenashev/remoting/3.12 [JENKINS-45755, JENKINS-46140] - Update Remoting to 3.12 Compare: https://github.com/jenkinsci/jenkins/compare/2343909f0240...e8b2f5a59c0b
            jmccormick Jesse McCormick made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ] Jesse McCormick [ jmccormick ]
            jmccormick Jesse McCormick made changes -
            Assignee Jesse McCormick [ jmccormick ] Oleg Nenashev [ oleg_nenashev ]
            jmccormick Jesse McCormick added a comment - - edited

            vlatombe - Tomcat was started with -Dcatalina.home=/usr/share/tomcat8. And yes, the directory is not writable by the tomcat8 user. This has caused other problems for Jenkins before. For instance, when the SSH client starts, it is also looking in /usr/share/tomcat8/, which is not the user's home directory (for the .ssh directory access). Is it possible we are using catalina.home instead of the actual user's home directory?

            Full parameters:

            /usr/lib/jvm/default-java/bin/java 
            -Djava.util.logging.config.file=/var/lib/tomcat8/conf/logging.properties 
            -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
            -Djava.awt.headless=true
            -Xmx1280m 
            -XX:+UseConcMarkSweepGC 
            -DJENKINS_HOME=/var/lib/tomcat8/webapps/jenkins/
            -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
            -Djdk.tls.ephemeralDHKeySize=2048
            -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
            -classpath /usr/share/tomcat8/bin/bootstrap.jar:/usr/share/tomcat8/bin/tomcat-juli.jar
            -Dcatalina.base=/var/lib/tomcat8
            -Dcatalina.home=/usr/share/tomcat8
            -Djava.io.tmpdir=/tmp/tomcat8-tomcat8-tmp 
            org.apache.catalina.startup.Bootstrap start

             

            jmccormick Jesse McCormick added a comment - - edited vlatombe - Tomcat was started with -Dcatalina.home=/usr/share/tomcat8. And yes, the directory is not writable by the tomcat8 user. This has caused other problems for Jenkins before. For instance, when the SSH client starts, it is also looking in /usr/share/tomcat8/, which is not the user's home directory (for the .ssh directory access). Is it possible we are using catalina.home instead of the actual user's home directory? Full parameters: /usr/lib/jvm/ default -java/bin/java -Djava.util.logging.config.file=/ var /lib/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless= true -Xmx1280m -XX:+UseConcMarkSweepGC -DJENKINS_HOME=/ var /lib/tomcat8/webapps/jenkins/ -Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH= true -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -classpath /usr/share/tomcat8/bin/bootstrap.jar:/usr/share/tomcat8/bin/tomcat-juli.jar -Dcatalina.base=/ var /lib/tomcat8 -Dcatalina.home=/usr/share/tomcat8 -Djava.io.tmpdir=/tmp/tomcat8-tomcat8-tmp org.apache.catalina.startup.Bootstrap start  
            jmccormick Jesse McCormick made changes -
            Description Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.71

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the default jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}

            *Reproducer using Docker*

            Launch Jenkins using the following
            {code}
            mkdir jenkins_home
            sudo chown 2000 jenkins_home
            sudo chmod 777 jenkins_home
            docker run -ti -v $(pwd)/jenkins_home:/var/jenkins_home -u 2000 --rm -p 8080:8080 -p 50000:50000 jenkinsci/jenkins:2.73
            {code}

            Create a jnlp agent, then try to connect. You will then get the following sequence:

            master
            {code}
            Sep 12, 2017 1:06:16 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
            INFO: Accepted JNLP4-connect connection #1 from /172.17.0.1:38552
            Sep 12, 2017 1:06:16 PM org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer onRecv
            WARNING: [JNLP4-connect connection from 172.17.0.1/172.17.0.1:38552]
            java.lang.RuntimeException: Root directory not writable: ?/.jenkins/cache/jars
            at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
            at hudson.remoting.JarCache.getDefault(JarCache.java:32)
            at hudson.remoting.Channel.<init>(Channel.java:505)
            at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:339)
            at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:149)
            at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207)
            at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
            at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
            at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
            at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
            at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
            at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
            at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            {code}

            agent
            {code}
            INFOS: Locating server among [http://localhost:8080/]
            sept. 12, 2017 1:06:16 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
            INFOS: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Agent discovery successful
              Agent address: localhost
              Agent port: 50000
              Identity: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Handshaking
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Connecting to localhost:50000
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Trying protocol: JNLP4-connect
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Remote identity confirmed: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Connected
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Terminated
            {code}
            Last working for me Jenkins version 2.67

            Versions not working 2.68 - 2.78

            Issue: SSH Slave will not launch

            Some part of the release in 2.68 seems to have changed how the environment is setup for slaves. The user my slave connects as has a home directory of: /home/SPALDING/jenkinsbuildserver however I can see in the error message that jarCache is attempting to look in master user's home directory which is /usr/share/tomcat8, which doesn't exist on the slave.

            I figured I could at least work around this issue if I set the new "workDir" parameter, but the launch will still fail. Even if I create the /usr/share/tomcat8/.jenkins/cache/jars directory and ensure it is writable by the slave user,  it still fails to launch.

            There appears to be 2 issues:
             * The slave is not using the correct location for the default jar cache because it seems to be using the home directory of the master user.
             * Even when the 'workDir' parameter is specified, Jenkins is still trying to validate the default location.

            {code:java}
            [07/24/17 10:42:11] [SSH] Opening SSH connection to SLAVEHOST:22.
            [07/24/17 10:42:11] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
            [07/24/17 10:42:11] [SSH] Authentication successful.
            [07/24/17 10:42:11] [SSH] The remote users environment is:
            BASH=/bin/bash
            BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
            BASH_ALIASES=()
            BASH_ARGC=()
            BASH_ARGV=()
            BASH_CMDS=()
            BASH_EXECUTION_STRING=set
            BASH_LINENO=()
            BASH_SOURCE=()
            BASH_VERSINFO=([0]="4" [1]="3" [2]="30" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
            BASH_VERSION='4.3.30(1)-release'
            DIRSTACK=()
            EUID=10169
            GROUPS=()
            HOME=/home/SPALDING/jenkinsbuildserver
            HOSTNAME=SLAVEHOST
            HOSTTYPE=x86_64
            IFS=$' \t\n'
            LANG=en_US.UTF-8
            LOGNAME=jenkinsbuildserver
            MACHTYPE=x86_64-pc-linux-gnu
            MAIL=/var/mail/jenkinsbuildserver
            OPTERR=1
            OPTIND=1
            OSTYPE=linux-gnu
            PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
            PIPESTATUS=([0]="0")
            PPID=21458
            PS4='+ '
            PWD=/home/SPALDING/jenkinsbuildserver
            SHELL=/bin/bash
            SHELLOPTS=braceexpand:hashall:interactive-comments
            SHLVL=1
            SSH_CLIENT='10.10.1.179 57938 22'
            SSH_CONNECTION='10.10.1.179 57938 10.10.0.251 22'
            TERM=dumb
            UID=10169
            USER=jenkinsbuildserver
            _=']'
            [07/24/17 10:42:11] [SSH] Checking java version of java
            [07/24/17 10:42:11] [SSH] java -version returned 1.8.0_131.
            [07/24/17 10:42:11] [SSH] Starting sftp client.
            [07/24/17 10:42:11] [SSH] Copying latest slave.jar...
            [07/24/17 10:42:11] [SSH] Copied 730,299 bytes.
            Expanded the channel window size to 4MB
            [07/24/17 10:42:11] [SSH] Starting slave process: cd "/home/SPALDING/jenkinsbuildserver/jenkins-agent" && java  -jar slave.jar -workDir /home/SPALDING/jenkinsbuildserver/jenkins-agent/work -failIfWorkDirIsMissing
            Jul 24, 2017 10:42:12 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
            INFO: Using /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting as a remoting work directory
            Both error and output logs will be printed to /home/SPALDING/jenkinsbuildserver/jenkins-agent/work/remoting
            <===[JENKINS REMOTING CAPACITY]===>ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
            java.lang.RuntimeException: Root directory not writable: /usr/share/tomcat8/.jenkins/cache/jars
                    at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
                    at hudson.remoting.JarCache.getDefault(JarCache.java:32)
                    at hudson.remoting.Channel.<init>(Channel.java:505)
                    at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:323)
                    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:389)
                    at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1070)
                    at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:144)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:817)
                    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:792)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                    at java.lang.Thread.run(Thread.java:748)
            [07/24/17 10:42:12] Launch failed - cleaning up connection
            [07/24/17 10:42:12] [SSH] Connection closed.


            {code}
            *Reproducer using Docker*

            Launch Jenkins using the following
            {code:java}
            mkdir jenkins_home
            sudo chown 2000 jenkins_home
            sudo chmod 777 jenkins_home
            docker run -ti -v $(pwd)/jenkins_home:/var/jenkins_home -u 2000 --rm -p 8080:8080 -p 50000:50000 jenkinsci/jenkins:2.73
            {code}
            Create a jnlp agent, then try to connect. You will then get the following sequence:

            master
            {code:java}
            Sep 12, 2017 1:06:16 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
            INFO: Accepted JNLP4-connect connection #1 from /172.17.0.1:38552
            Sep 12, 2017 1:06:16 PM org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer onRecv
            WARNING: [JNLP4-connect connection from 172.17.0.1/172.17.0.1:38552]
            java.lang.RuntimeException: Root directory not writable: ?/.jenkins/cache/jars
            at hudson.remoting.FileSystemJarCache.<init>(FileSystemJarCache.java:57)
            at hudson.remoting.JarCache.getDefault(JarCache.java:32)
            at hudson.remoting.Channel.<init>(Channel.java:505)
            at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:339)
            at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:149)
            at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207)
            at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
            at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
            at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
            at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
            at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
            at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
            at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:721)
            at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            {code}
            agent
            {code:java}
            INFOS: Locating server among [http://localhost:8080/]
            sept. 12, 2017 1:06:16 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
            INFOS: Remoting server accepts the following protocols: [JNLP4-connect, JNLP-connect, Ping, JNLP2-connect]
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Agent discovery successful
              Agent address: localhost
              Agent port: 50000
              Identity: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Handshaking
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Connecting to localhost:50000
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Trying protocol: JNLP4-connect
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Remote identity confirmed: 80:e1:c5:f6:d5:96:cf:1d:6a:58:45:48:2b:fe:67:76
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Connected
            sept. 12, 2017 1:06:16 PM hudson.remoting.jnlp.Main$CuiListener status
            INFOS: Terminated
            {code}

            Thanks for the fix! I can confirm all is well in the Jenkins 2.79 release.

            jmccormick Jesse McCormick added a comment - Thanks for the fix! I can confirm all is well in the Jenkins 2.79 release.
            danielbeck Daniel Beck made changes -
            Labels regression lts-candidate regression
            danielbeck Daniel Beck added a comment -

            Fixed in 2.79.

            danielbeck Daniel Beck added a comment - Fixed in 2.79.
            danielbeck Daniel Beck made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            olivergondza Oliver Gondža made changes -
            Labels lts-candidate regression 2.73.2-fixed regression

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/remoting/Channel.java
            src/main/java/hudson/remoting/ChannelBuilder.java
            src/main/java/hudson/remoting/Engine.java
            src/main/java/hudson/remoting/FileSystemJarCache.java
            src/main/java/hudson/remoting/JarCache.java
            src/main/java/hudson/remoting/Launcher.java
            src/main/java/hudson/remoting/ResourceImageBoth.java
            src/main/java/hudson/remoting/ResourceImageInJar.java
            src/main/java/org/jenkinsci/remoting/nio/NioChannelBuilder.java
            src/test/java/hudson/remoting/PrefetchingTest.java
            http://jenkins-ci.org/commit/remoting/8f550052f803bcb9ad708881b3534e0227d91150
            Log:
            JENKINS-45755 - Make JARCache nullable in Channel

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/ChannelBuilder.java src/main/java/hudson/remoting/Engine.java src/main/java/hudson/remoting/FileSystemJarCache.java src/main/java/hudson/remoting/JarCache.java src/main/java/hudson/remoting/Launcher.java src/main/java/hudson/remoting/ResourceImageBoth.java src/main/java/hudson/remoting/ResourceImageInJar.java src/main/java/org/jenkinsci/remoting/nio/NioChannelBuilder.java src/test/java/hudson/remoting/PrefetchingTest.java http://jenkins-ci.org/commit/remoting/8f550052f803bcb9ad708881b3534e0227d91150 Log: JENKINS-45755 - Make JARCache nullable in Channel
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2469 (Web Link)" [ 18282 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CLTS-2229 (Web Link)" [ 19078 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-47977 [ JENKINS-47977 ]

            People

              oleg_nenashev Oleg Nenashev
              jmccormick Jesse McCormick
              Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: