Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46601

error in log files when Unix Jenkins slaves are shutdown

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ec2-plugin
    • Jenkins Version: 2.60.3
      Amazon EC2 plugins: 1.36
      Idle termination time: -5

      We are seeing sporadic stack traces in the Jenkins master log file and Jenkins slave log file when Unix Jenkins slave are successfully shutdown at the end of the billing period. Could this be due to the AWS instance being shutdown before it was disconnected from Jenkins master?  In a local branch we changed the logic to disconnect the agent before stopping the instance.

      From Jenkins slave log file:

      Sep 01, 2017 12:18:35 PM null
      INFO: Launching slave agent (via Trilead SSH2 Connection): java -Xms128m -Xmx1g -jar /tmp/slave.jar
      <===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 3.7
      This is a Unix agent
      Evacuated stdout
      Agent successfully connected and online
      ERROR: Connection terminated
      ha:////xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==java.io.EOFException
              at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
              at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
              at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
              at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
              at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
              at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
      Caused: java.io.IOException: Unexpected termination of the channel
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
      

      From Jenkins master log file:

      Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
      INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 23 idle minutes, with 4 minutes remaining in billing period
      Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
      Sep 01, 2017 1:13:34 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
      SEVERE: I/O error in channel jenkins-slave-name (i-xxxxxxxxxxxxxxxx)
      java.io.IOException: Unexpected termination of the channel
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
      Caused by: java.io.EOFException
              at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
              at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
              at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
              at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
              at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
              at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
              at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
      Sep 01, 2017 1:13:35 PM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx
      

      From SSHD log file on Jenkins slave:

      Sep  1 13:13:34 ip-xx-xxx-xxx-xxx sshd[2220]: pam_unix(sshd:session): session closed for user centos
      Sep  1 13:13:34 ip-xx-xxx-xxx-xxx sshd[798]: Received signal 15; terminating.
      

      From messages log file on Jenkins slave:

      Sep  1 13:13:34 ip-xx-xxx-xxx-xxx systemd: Stopping OpenSSH server daemon...
      

          [JENKINS-46601] error in log files when Unix Jenkins slaves are shutdown

          Mike Kozell created issue -
          Mike Kozell made changes -
          Description Original: We are seeing sporadic stack traces in the Jenkins master log file and Jenkins slave log file when Unix Jenkins slave are successfully shutdown at the end of the billing period. Could this be due to the AWS instance being shutdown before it was disconnected from Jenkins master?

          From Jenkins slave log file:
          {code}
          Sep 01, 2017 12:18:35 PM null
          INFO: Launching slave agent (via Trilead SSH2 Connection): java -Xms128m -Xmx1g -jar /tmp/slave.jar
          <===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 3.7
          This is a Unix agent
          Evacuated stdout
          Agent successfully connected and online
          ERROR: Connection terminated
          ha:////xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==java.io.EOFException
                  at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
                  at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
                  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
                  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
                  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
                  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
          Caused: java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
          {code}

          From Jenkins master log file:
          {code}
          Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
          INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 23 idle minutes, with 4 minutes remaining in billing period
          Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
          INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
          Sep 01, 2017 1:13:34 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel jenkins-slave-name (i-xxxxxxxxxxxxxxxx)
          java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
          Caused by: java.io.EOFException
                  at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
                  at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
                  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
                  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
                  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
                  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
          Sep 01, 2017 1:13:35 PM hudson.plugins.ec2.EC2AbstractSlave stop
          INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx
          {code}

          From SSHD log file on Jenkins slave:
          {code}
          Sep 1 13:13:34 ip-xx-xxx-xxx-xxx sshd[2220]: pam_unix(sshd:session): session closed for user centos
          Sep 1 13:13:34 ip-xx-xxx-xxx-xxx sshd[798]: Received signal 15; terminating.
          {code}

          From messages log file on Jenkins slave:
          {code}
          Sep 1 13:13:34 ip-xx-xxx-xxx-xxx systemd: Stopping OpenSSH server daemon...
          {code}
          New: We are seeing sporadic stack traces in the Jenkins master log file and Jenkins slave log file when Unix Jenkins slave are successfully shutdown at the end of the billing period. Could this be due to the AWS instance being shutdown before it was disconnected from Jenkins master?  In a local branch we changed the logic to disconnect the agent before stopping the instance.

          From Jenkins slave log file:
          {code:java}
          Sep 01, 2017 12:18:35 PM null
          INFO: Launching slave agent (via Trilead SSH2 Connection): java -Xms128m -Xmx1g -jar /tmp/slave.jar
          <===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 3.7
          This is a Unix agent
          Evacuated stdout
          Agent successfully connected and online
          ERROR: Connection terminated
          ha:////xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==java.io.EOFException
                  at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
                  at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
                  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
                  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
                  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
                  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
          Caused: java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
          {code}
          From Jenkins master log file:
          {code:java}
          Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
          INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 23 idle minutes, with 4 minutes remaining in billing period
          Sep 01, 2017 1:13:34 PM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
          INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
          Sep 01, 2017 1:13:34 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel jenkins-slave-name (i-xxxxxxxxxxxxxxxx)
          java.io.IOException: Unexpected termination of the channel
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
          Caused by: java.io.EOFException
                  at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
                  at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
                  at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
                  at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
                  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
                  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
                  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
          Sep 01, 2017 1:13:35 PM hudson.plugins.ec2.EC2AbstractSlave stop
          INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx
          {code}
          From SSHD log file on Jenkins slave:
          {code:java}
          Sep 1 13:13:34 ip-xx-xxx-xxx-xxx sshd[2220]: pam_unix(sshd:session): session closed for user centos
          Sep 1 13:13:34 ip-xx-xxx-xxx-xxx sshd[798]: Received signal 15; terminating.
          {code}
          From messages log file on Jenkins slave:
          {code:java}
          Sep 1 13:13:34 ip-xx-xxx-xxx-xxx systemd: Stopping OpenSSH server daemon...
          {code}

            francisu Francis Upton
            mkozell Mike Kozell
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: