-
Bug
-
Resolution: Not A Defect
-
Blocker
-
Jenkins version=1.580.1, OS=Linux, Flavor=SLES11, Java version=1.6 Container=Tomcat 7.0.47
-
Powered by SuggestiMate
Our Jenkins master instance got crashed around 3 times in one week and it is getting shutdown by itself once in a while(once or twice per a day). I had mentioned the Environment details in the Environment Field. I am also attaching the last three Jenkins crash logs. This is a serious blocker for our Team, Please let us know what is the issue and remedy.
Please do let me know if you need further details
Quick response will be much useful !! Thanks in advance
Regards,
Vamsi Hari
[JENKINS-25869] Jenkins is getting crashed and shutting down once in a while
That said, I'd try to figure out why it crashes in the same proxy related code and investigate the proxy configuration in plugin manager, proxy related environment variables, and/or system properties.
Thanks a lot Daniel for the prompt response,
Adding to what you said, we are also facing another issue (https://issues.jenkins-ci.org/browse/JENKINS-25870) which is related to proxy settings not working from plugin manager, anything related?
More Information: Recent config changes that we have done with our Jenkins master is we had added a couple of windows slaves to run selenium Tests on them and we already have a linux slave.
My apologies, if this issue is not related to this Tracker, we just want to corner the issue, It would be so grateful if you can help us in cornering the issue
Please let me know if you need further details,
Below is the part of Tomcat log on the day when earlier crash happened. Below errors are repetitively found in logs (for better understanding i had renamed the hostnames of windows and linux slaves as "windowsslave" & "Linuxslave" )
Nov 19, 2014 2:35:23 PM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed
WARNING: Channel reader thread: WindowsSlave for + WindowsSlave terminated
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2576)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1295)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
at hudson.remoting.Command.readFrom(Command.java:92)
at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Nov 19, 2014 2:35:33 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
INFO: Accepted connection #12 from /51.16.108.187:61593
Nov 19, 2014 2:35:34 PM hudson.TcpSlaveAgentListener$ConnectionHandler run
WARNING: Connection #12 failed
java.io.EOFException: unexpected stream termination
at hudson.remoting.ClassicCommandTransport.create(ClassicCommandTransport.java:135)
at hudson.remoting.Channel.<init>(Channel.java:406)
at hudson.remoting.Channel.<init>(Channel.java:402)
at hudson.remoting.Channel.<init>(Channel.java:363)
at hudson.remoting.Channel.<init>(Channel.java:359)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:344)
at jenkins.slaves.JnlpSlaveAgentProtocol$Handler.jnlpConnect(JnlpSlaveAgentProtocol.java:122)
at jenkins.slaves.JnlpSlaveAgentProtocol2$Handler2.run(JnlpSlaveAgentProtocol2.java:99)
at jenkins.slaves.JnlpSlaveAgentProtocol2.handle(JnlpSlaveAgentProtocol2.java:43)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:154)
Nov 19, 2014 2:39:45 PM hudson.model.Run execute
INFO: servicegateway-static-build-git-claimsource #520 main build action completed: SUCCESS
Nov 19, 2014 2:39:54 PM hudson.remoting.Channel$2 handle
SEVERE: Failed to execute command Pipe.EOF(272) (channel Linuxslave)
java.lang.NullPointerException
at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177)
at hudson.remoting.Channel$2.handle(Channel.java:461)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Nov 19, 2014 2:39:54 PM hudson.remoting.Channel$2 handle
SEVERE: This command is created here
Command Pipe.EOF(272) created at
at hudson.remoting.Command.<init>(Command.java:67)
at hudson.remoting.Command.<init>(Command.java:50)
at hudson.remoting.ProxyWriter$EOF.<init>(ProxyWriter.java:169)
at hudson.remoting.ProxyWriter.close(ProxyWriter.java:124)
at hudson.remoting.ProxyWriter.finalize(ProxyWriter.java:132)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:172)
Nov 19, 2014 2:40:22 PM hudson.remoting.Channel$2 handle
SEVERE: Failed to execute command Pipe.EOF(275) (channel Linuxslave)
java.lang.NullPointerException
at hudson.remoting.ProxyWriter$EOF.execute(ProxyWriter.java:177)
at hudson.remoting.Channel$2.handle(Channel.java:461)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Regards,
Vamsi Hari
That looks like a completely different issue. What is the relation of that to the earlier reported JRE crashes?
I am not sure, I had added that as you said you saw some proxy related code in the error log. But from every JVM crash log there are some logs related svn and proxy. BTW We are still having our Jenkins instance crashed regularly, we had recent crash on Jan 1. I am attaching the recent crash log too.
Please provide the configuration of the job PortalEXT-EAR-deployment-PEO and PortalEXT-myts-static-STATIC-deployment and the output on the /systemInfo URL. Or better yet, install https://wiki.jenkins-ci.org/display/JENKINS/Support+Core+Plugin and attach a support bundle to this issue (in addition to the two job configurations, since they're not in the bundle).
Daniel - I had attached all the info requested by you. FYI, In the job config files i had commented host names and mentioned respective details over there.
I'd try to make sure to unset the http_proxy and related environment variables before starting the Jenkins process (or rather Tomcat in your case?), instead (if necessary) using proxy configuration in Jenkins (Manage Jenkins » Manage Plugins » Advanced). This may result in child processes (e.g. shell scripts and the tools they run) not inheriting the proxy configuration by default though.
The bug may well be in the gconf (Gnome) library or the JDK, as that's where the crash occurs.
http://hg.openjdk.java.net/jdk7u/jdk7u/jdk/file/54acd5cd0485/src/solaris/native/sun/net/spi/DefaultProxySelector.c#l155
Please also note that the user and password to the proxy are included in plain text in the support bundle (nodes/master/environment.txt), so you should change those now.
Thanks Daniel for your help,
I will try removing all the proxy related environment variables and restarting the Jenkins container(Tomcat) and let you know about the result.
Hey Daniel - We had tried by commenting the proxy related env variables in the host. we had only specified the proxy settings in the Jenkins UI, but the Jenkins crashes still persists. I am attaching the most recent crash log. Please verify,
We truly appreciate your help,
Thanks and Regards,
Vamsi Hari
This appears to be a bug in your JDK, and has nothing to do with Jenkins (except that it runs code that causes the JDK code to run).
See:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=7188755
https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/885195
http://mail.openjdk.java.net/pipermail/jdk7u-dev/2012-August/003911.html
Thanks Daniel for your help,
We will keep posted if we find any solution for this java bug.
Cheers,
Vamsi Hari
This is a bug tracker, not a support site.