Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46837

Jenkins slave (EC2 AWS plugin) fails occasionally when build takes too long with error "Caused: java.io.IOException: Unexpected termination of the channel"

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • ec2-plugin
    • None

      I have Jenkins configured to launch EC2 instances (via AWS plugin) to execute a build.
      The actual build steps use the Execute Shell method, and launch ansible scripts.
       
      I find that if the ansible script runs for too long, the Jenkins slave on the EC2 system goes down.
      The Jenkins GUI shows this error:
       
      Connection was broken
      java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)   
      Some googling lead to this page: https://wiki.jenkins.io/display/JENKINS/Remoting+issue
      It suggests I check these log files:
      1) connection logs on master - I checked these, and they have the same information as stack trace.
      2) slave logs - I do not know where to find these.  The locations suggested do not exist on my slave.  
       
      I even inspected the files that the slave java process opens, and the only file that seems like a log file ends up being empty:
       
      ubuntu@ip-172-31-93-175:~/support$ ps auxww | grep java | grep slave
      ubuntu    25235  8.0  0.5 12211452 157616 ?     Ssl  22:23   0:05 java -jar /tmp/slave.jar
      ubuntu@ip-172-31-93-175:~/support$ sudo lsof | grep 25235  | grep -i log | awk '{print $NF}' | sort | uniq
      /home/ubuntu/support/all_2017-09-11_22.23.07.log
      ubuntu@ip-172-31-93-175:~/support$ cat /home/ubuntu/support/all_2017-09-11_22.23.07.log 
      ubuntu@ip-172-31-93-175:~/support$ 
       
       
       
      Has anyone run into this issue or know how or where to find the java slave logs? 

      Thanks so much.
       
       
      VERSION INFO:
      Master:
      Jenkins ver. 2.60.2
      Java:  openjdk version "1.8.0_131"
       
      Build node:
      Launched via: AWS plugin: https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin (latest version)
       
      Ec2 instance running Ubuntu 14.04 and Java 8:
      ubuntu@ip-172-31-93-175:~$ cat /etc/release | grep VERSION
      VERSION="14.04.5 LTS, Trusty Tahr"
      VERSION_ID="14.04"
      ubuntu@ip-172-31-93-175:~$ java -version
      java version "1.8.0_144"
      Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
      Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

          [JENKINS-46837] Jenkins slave (EC2 AWS plugin) fails occasionally when build takes too long with error "Caused: java.io.IOException: Unexpected termination of the channel"

          Any suggestions?

           

          Litmus Automation added a comment - Any suggestions?  

          Shubham Dhoka added a comment -

          I am running into similar issue with version 1.42 of the ec2-plugin and java 1.8

          Shubham Dhoka added a comment - I am running into similar issue with version 1.42 of the ec2-plugin and java 1.8

          Bruno Queiros added a comment -

          Same problem with version 1.49 of ec2-plugin and java 1.8

          Bruno Queiros added a comment - Same problem with version 1.49 of ec2-plugin and java 1.8

          Bruno Queiros added a comment -

          Do we have an update on this? Is someone working on it?

          Bruno Queiros added a comment - Do we have an update on this? Is someone working on it?

          brunoqueiros Are you using STS?

          Scott Sutherland added a comment - brunoqueiros Are you using STS?

          Bruno Queiros added a comment -

          suthsc No

          Bruno Queiros added a comment - suthsc No

            francisu Francis Upton
            litmus Litmus Automation
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: