-
Bug
-
Resolution: Unresolved
-
Blocker
-
None
I have Jenkins configured to launch EC2 instances (via AWS plugin) to execute a build.
The actual build steps use the Execute Shell method, and launch ansible scripts.
I find that if the ansible script runs for too long, the Jenkins slave on the EC2 system goes down.
The Jenkins GUI shows this error:
Connection was broken
java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Some googling lead to this page: https://wiki.jenkins.io/display/JENKINS/Remoting+issue
It suggests I check these log files:
1) connection logs on master - I checked these, and they have the same information as stack trace.
2) slave logs - I do not know where to find these. The locations suggested do not exist on my slave.
I even inspected the files that the slave java process opens, and the only file that seems like a log file ends up being empty:
ubuntu@ip-172-31-93-175:~/support$ ps auxww | grep java | grep slave
ubuntu 25235 8.0 0.5 12211452 157616 ? Ssl 22:23 0:05 java -jar /tmp/slave.jar
ubuntu@ip-172-31-93-175:~/support$ sudo lsof | grep 25235 | grep -i log | awk '{print $NF}' | sort | uniq
/home/ubuntu/support/all_2017-09-11_22.23.07.log
ubuntu@ip-172-31-93-175:~/support$ cat /home/ubuntu/support/all_2017-09-11_22.23.07.log
ubuntu@ip-172-31-93-175:~/support$
Has anyone run into this issue or know how or where to find the java slave logs?
Thanks so much.
VERSION INFO:
Master:
Jenkins ver. 2.60.2
Java: openjdk version "1.8.0_131"
Build node:
Launched via: AWS plugin: https://wiki.jenkins.io/display/JENKINS/Amazon+EC2+Plugin (latest version)
Ec2 instance running Ubuntu 14.04 and Java 8:
ubuntu@ip-172-31-93-175:~$ cat /etc/release | grep VERSION
VERSION="14.04.5 LTS, Trusty Tahr"
VERSION_ID="14.04"
ubuntu@ip-172-31-93-175:~$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
Any suggestions?