Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54897

Unexpected error in launching a agent after restart

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • ssh-slaves-plugin
    • None
    • Jenkins ver. 2.138.2
      ssh-slaves-plugin ver: 1.29.1
      Linux RHEL 7.5
    • ssh-slaves-1.30.2

      Launch agent has issues after restarting Jenkins. If you try to launch several times, it will eventually work.

       

      ssh rules: no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

       

      Node config.xml

      <?xml version='1.1' encoding='UTF-8'?>
<slave>
  <name>cac-slave-01</name>
  <description></description>
  <remoteFS>/app/var/lib/jenkins/slave_node</remoteFS>
  <numExecutors>5</numExecutors>
  <mode>NORMAL</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
  <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.29.1">
    <host>jenkins-slave.example.com</host>
    <port>22</port>
    <credentialsId>jenkins-master-ssh-key</credentialsId>
    <launchTimeoutSeconds>210</launchTimeoutSeconds>
    <maxNumRetries>10</maxNumRetries>
    <retryWaitTime>15</retryWaitTime>
    <sshHostKeyVerificationStrategy class="hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"/>
  </launcher>
  <label></label>
  <nodeProperties/>
</slave>

       

      Error log 1:

      SSHLauncher{host='jenkins-slave.example.com', port=22, credentialsId='jenkins-master-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[11/27/18 16:46:51] [SSH] Opening SSH connection to jenkins-slave.example.com:22.
ERROR: null
java.util.concurrent.CancellationException
	at java.util.concurrent.FutureTask.report(FutureTask.java:121)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:902)
	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[11/27/18 16:50:21] Launch failed - cleaning up connection
[11/27/18 16:50:21] SSH Launch of cac-slave-01 on jenkins-slave.example.com failed in 210,000 ms

      Error log 2:

      SSHLauncher{host='jenkins-slave.example.com', port=22, credentialsId='jenkins-master-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[11/27/18 16:18:08] [SSH] Opening SSH connection to jenkins-slave.example.com:22.
[11/27/18 16:18:29] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
[11/27/18 16:18:29] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
[11/27/18 16:18:29] [SSH] Authentication successful.
[11/27/18 16:18:29] [SSH] Authentication successful.
[11/27/18 16:18:29] [SSH] The remote user's environment is:
[11/27/18 16:18:29] [SSH] The remote user's environment is:
BASH=/usr/bin/bash
BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="2" [2]="46" [3]="2" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='4.2.46(2)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOME=/var/lib/jenkins_slave
HOSTNAME=jenkins-slave.example.com
HOSTTYPE=x86_64
ID=1000
IFS=$' \t\n'
LANG=en_US.UTF-8
LESSOPEN='||/usr/bin/lesspipe.sh %s'
LOGNAME=jenkins_slave_local
MACHTYPE=x86_64-redhat-linux-gnu
MAIL=/var/mail/jenkins_slave_local
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin
PIPESTATUS=([0]="0")
PPID=15365
PS4='+ '
PWD=/var/lib/jenkins_slave
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='165.115.33.181 36584 22'
SSH_CONNECTION='165.115.33.181 36584 165.115.33.182 22'
TERM=dumb
TMOUT=1800
UID=1000
USER=jenkins_slave_local
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_ID=3293
_=/etc/bashrc
[11/27/18 16:18:29] [SSH] Checking java version of /app/var/lib/jenkins/slave_node/jdk/bin/java
BASH=/usr/bin/bash
BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="2" [2]="46" [3]="2" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='4.2.46(2)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOME=/var/lib/jenkins_slave
HOSTNAME=jenkins-slave.example.com
HOSTTYPE=x86_64
ID=1000
IFS=$' \t\n'
LANG=en_US.UTF-8
LESSOPEN='||/usr/bin/lesspipe.sh %s'
LOGNAME=jenkins_slave_local
MACHTYPE=x86_64-redhat-linux-gnu
MAIL=/var/mail/jenkins_slave_local
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin
PIPESTATUS=([0]="0")
PPID=15366
PS4='+ '
PWD=/var/lib/jenkins_slave
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='165.115.33.181 36878 22'
SSH_CONNECTION='165.115.33.181 36878 165.115.33.182 22'
TERM=dumb
TMOUT=1800
UID=1000
USER=jenkins_slave_local
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_ID=3292
_=/etc/bashrc
[11/27/18 16:18:29] [SSH] Checking java version of /app/var/lib/jenkins/slave_node/jdk/bin/java
Couldn't figure out the Java version of /app/var/lib/jenkins/slave_node/jdk/bin/java
bash: /app/var/lib/jenkins/slave_node/jdk/bin/java: No such file or directory[11/27/18 16:18:29] [SSH] Checking java version of java
Couldn't figure out the Java version of /app/var/lib/jenkins/slave_node/jdk/bin/java
bash: /app/var/lib/jenkins/slave_node/jdk/bin/java: No such file or directory[11/27/18 16:18:29] [SSH] Checking java version of java
[11/27/18 16:18:29] [SSH] java -version returned 1.8.0_181.
[11/27/18 16:18:29] [SSH] Starting sftp client.
[11/27/18 16:18:29] [SSH] java -version returned 1.8.0_181.
[11/27/18 16:18:29] [SSH] Starting sftp client.
[11/27/18 16:18:29] [SSH] Copying latest remoting.jar...
[11/27/18 16:18:29] [SSH] Copying latest remoting.jar...
[11/27/18 16:18:29] [SSH] Copied 776,265 bytes.
Expanded the channel window size to 4MB
[11/27/18 16:18:29] [SSH] Starting agent process: cd "/app/var/lib/jenkins/slave_node" && java  -jar remoting.jar -workDir /app/var/lib/jenkins/slave_node
[11/27/18 16:18:29] [SSH] Copied 776,265 bytes.
Expanded the channel window size to 4MB
[11/27/18 16:18:29] [SSH] Starting agent process: cd "/app/var/lib/jenkins/slave_node" && java  -jar remoting.jar -workDir /app/var/lib/jenkins/slave_node
Nov 27, 2018 4:18:30 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /app/var/lib/jenkins/slave_node/remoting as a remoting work directory
Both error and output logs will be printed to /app/var/lib/jenkins/slave_node/remoting
Nov 27, 2018 4:18:30 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /app/var/lib/jenkins/slave_node/remoting as a remoting work directory
Both error and output logs will be printed to /app/var/lib/jenkins/slave_node/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.25
Remoting version: 3.25
This is a Unix agent
This is a Unix agent
Evacuated stdout
Evacuated stdout
ERROR: Unexpected error in launching a agent. This is probably a bug in Jenkins.
java.lang.IllegalStateException: Already connected
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:678)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:432)
	at hudson.plugins.sshslaves.SSHLauncher.startAgent(SSHLauncher.java:1032)
	at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:128)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:866)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:831)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[11/27/18 16:18:33] Launch failed - cleaning up connection
[11/27/18 16:18:33] [SSH] Connection closed.
Connection terminated
channel stopped
Connection terminated
ERROR: Unexpected error in launching a agent. This is probably a bug in Jenkins.
java.lang.NullPointerException
	at org.jenkinsci.modules.systemd_slave_installer.SlaveInstallerFactoryImpl.createIfApplicable(SlaveInstallerFactoryImpl.java:33)
	at org.jenkinsci.modules.slave_installer.SlaveInstallerFactory.createIfApplicable(SlaveInstallerFactory.java:29)
	at org.jenkinsci.modules.slave_installer.SlaveInstallerFactory.createFor(SlaveInstallerFactory.java:46)
	at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:30)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:693)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:432)
	at hudson.plugins.sshslaves.SSHLauncher.startAgent(SSHLauncher.java:1032)
	at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:128)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:866)
	at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:831)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[11/27/18 16:18:33] Launch failed - cleaning up connection
[11/27/18 16:18:33] [SSH] Connection closed.
SSHLauncher{host='jenkins-slave.example.com', port=22, credentialsId='jenkins-master-ssh-key', jvmOptions='', javaPath='', prefixStartSlaveCmd='', suffixStartSlaveCmd='', launchTimeoutSeconds=210, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[11/27/18 16:19:03] [SSH] Opening SSH connection to jenkins-slave.example.com:22.
[11/27/18 16:19:03] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
[11/27/18 16:19:03] [SSH] Authentication successful.
[11/27/18 16:19:03] [SSH] The remote user's environment is:
BASH=/usr/bin/bash
BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_EXECUTION_STRING=set
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="2" [2]="46" [3]="2" [4]="release" [5]="x86_64-redhat-linux-gnu")
BASH_VERSION='4.2.46(2)-release'
DIRSTACK=()
EUID=1000
GROUPS=()
HOME=/var/lib/jenkins_slave
HOSTNAME=jenkins-slave.example.com
HOSTTYPE=x86_64
ID=1000
IFS=$' \t\n'
LANG=en_US.UTF-8
LESSOPEN='||/usr/bin/lesspipe.sh %s'
LOGNAME=jenkins_slave_local
MACHTYPE=x86_64-redhat-linux-gnu
MAIL=/var/mail/jenkins_slave_local
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/bin:/usr/bin
PIPESTATUS=([0]="0")
PPID=15645
PS4='+ '
PWD=/var/lib/jenkins_slave
SHELL=/bin/bash
SHELLOPTS=braceexpand:hashall:interactive-comments
SHLVL=1
SSH_CLIENT='165.115.33.181 37012 22'
SSH_CONNECTION='165.115.33.181 37012 165.115.33.182 22'
TERM=dumb
TMOUT=1800
UID=1000
USER=jenkins_slave_local
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_ID=3294
_=/etc/bashrc
[11/27/18 16:19:03] [SSH] Checking java version of /app/var/lib/jenkins/slave_node/jdk/bin/java
Couldn't figure out the Java version of /app/var/lib/jenkins/slave_node/jdk/bin/java
bash: /app/var/lib/jenkins/slave_node/jdk/bin/java: No such file or directory[11/27/18 16:19:03] [SSH] Checking java version of java
[11/27/18 16:19:03] [SSH] java -version returned 1.8.0_181.
[11/27/18 16:19:03] [SSH] Starting sftp client.
[11/27/18 16:19:03] [SSH] Copying latest remoting.jar...
[11/27/18 16:19:03] [SSH] Copied 776,265 bytes.
Expanded the channel window size to 4MB
[11/27/18 16:19:03] [SSH] Starting agent process: cd "/app/var/lib/jenkins/slave_node" && java  -jar remoting.jar -workDir /app/var/lib/jenkins/slave_node
Nov 27, 2018 4:19:03 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /app/var/lib/jenkins/slave_node/remoting as a remoting work directory
Both error and output logs will be printed to /app/var/lib/jenkins/slave_node/remoting
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 3.25
This is a Unix agent
Evacuated stdout
Agent successfully connected and online

            ifernandezcalvo Ivan Fernandez Calvo
            kpsimoulis Konstantinos Psimoulis
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: