Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46601

error in log files when Unix Jenkins slaves are shutdown

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ec2-plugin
    • Jenkins Version: 2.60.3
      Amazon EC2 plugins: 1.36
      Idle termination time: -5

      We are seeing sporadic stack traces in the Jenkins master log file and Jenkins slave log file when Unix Jenkins slave are successfully shutdown at the end of the billing period. Could this be due to the AWS instance being shutdown before it was disconnected from Jenkins master?  In a local branch we changed the logic to disconnect the agent before stopping the instance.

      From Jenkins slave log file:

      Sep 01, 2017 12:18:35 PM null
INFO: Launching slave agent (via Trilead SSH2 Connection): java -Xms128m -Xmx1g -jar /tmp/slave.jar
<===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 3.7
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
ERROR: Connection terminated
ha:////xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)

      From Jenkins master log file:

      Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 23 idle minutes, with 4 minutes remaining in billing period
Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
Sep 01, 2017 1:13:34 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel jenkins-slave-name (i-xxxxxxxxxxxxxxxx)
java.io.IOException: Unexpected termination of the channel
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused by: java.io.EOFException
        at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
        at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
        at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Sep 01, 2017 1:13:35 PM hudson.plugins.ec2.EC2AbstractSlave stop
INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx

      From SSHD log file on Jenkins slave:

      Sep  1 13:13:34 ip-xx-xxx-xxx-xxx sshd[2220]: pam_unix(sshd:session): session closed for user centos
Sep  1 13:13:34 ip-xx-xxx-xxx-xxx sshd[798]: Received signal 15; terminating.

      From messages log file on Jenkins slave:

      Sep  1 13:13:34 ip-xx-xxx-xxx-xxx systemd: Stopping OpenSSH server daemon...

            francisu Francis Upton
            mkozell Mike Kozell
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: