-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Powered by SuggestiMate
We had the problem that all our builds did hang, because the XMPP server was not responding (it had an OOME)
[JENKINS-9233] Builds hang on jabber notification when XMPP server is not responding
Same happening here. In this state the cross icon next to the build progress indicator to cancel the build does not work. There seems to be no way to recover besides killing the whole Jenkins service and restarting.
I'd like to look into this, but so far I failed to reproduce the issue.
I'm using:
jenkins master
jabber-plugin master
prosody-0.8.2
I have set up a test build that waits 20 seconds, then tries to send a XMPP notification. Before the build completes I'd like to simulate a non-responding XMPP server. When I send prosody SIGKILL, jenkins notices that the connection has closed. If I send it a SIGSTOP, jenkins doesn't report any error on sending the notification.
Any idea how to reproduce this behaviour?
We have the same problem here. If javver server down, our builds freezes forever (can't even terminate them by clicking the cross, only by terminating the server - tomcat)
This is a huge problem! Timeout is required or this in unusable. Please fix it, we like this plugin ..
We use
windows server 2008 and with Tomcat 7
jenkins 1.473 as war
Jenkins Jabber notifier plugin 1.22
Is there any news about this bug? Our company IM server went down last night and brought production to a standstill. Seems IM messages should be queued and threaded as to not halt the build(s) if there's a hiccup in the network, or an unstable IM server specified. I personally wouldn't consider a failed IM notification a "build breaker". I would consider a hung Jenkins server due to a deadlock in a low priority notification routine a much bigger issue.
Jenkins 1.504
Instant Messenger Plugin 1.25
Jabber Plugin 1.25
We have been hit by this issue today as well. Our Jabber server has been having issues all day. Our builds in Jenkins are setup to send a notification after a build but they hang indefinitely and the abort build button does not work.
Restarting Tomcat/Jenkins is an issue as well. Jenkins is not coming back up. Tailing catalina.out shows that the plugin is in an endless loop trying to re-connect.
2013-09-05 18:37:47 INFO [onnector-Thread] - hudson.plugins.im.IMConnectionProvider - Reconnect failed. Next connection attempt in 1 minutes
2013-09-05 18:38:47 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Trying to connect to XMPP on jabber.highwinds.com:5222/jabber.highwinds.com with SASL
2013-09-05 18:39:52 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying connection with legacy SSL
2013-09-05 18:53:48 WARN [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying with legacy SSL failed: Connection failed. No response from server.
2013-09-05 18:53:48 WARN [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Exception of original (without legacy SSL) connection attempt:
– caused by: Connection failed. No response from server.:
at hudson.plugins.jabber.im.transport.JabberIMConnection.retryConnectionWithLegacySSL(JabberIMConnection.java:349)
at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:315)
at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:162)
at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42)
at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:65)
at hudson.plugins.im.IMConnectionProvider.access$600(IMConnectionProvider.java:22)
at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:183)
at java.lang.Thread.run(Thread.java:662)
Nested Exception:
Connection failed. No response from server.:
at org.jivesoftware.smack.PacketReader.startup(PacketReader.java:119)
at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:568)
at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection.java:527)
at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:953)
at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:305)
at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:162)
at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42)
at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:65)
at hudson.plugins.im.IMConnectionProvider.access$600(IMConnectionProvider.java:22)
at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:183)
at java.lang.Thread.run(Thread.java:662)
2013-09-05 18:53:48 INFO [onnector-Thread] - hudson.plugins.im.IMConnectionProvider - Reconnect failed. Next connection attempt in 2 minutes
2013-09-05 18:55:48 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Trying to connect to XMPP on jabber.highwinds.com:5222/jabber.highwinds.com with SASL
2013-09-05 18:56:09 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying connection with legacy SSL
@cforce: as I already stated earlier, I didn't find any way to set timeouts for this - so I'm limited by the Smack API to fix this.
Unless youself find a way, increasing the priority of this bug or assigning it to me, again, won't help a bit to get this fixed.
BTW: pull request are highly welcome
What about setPacketReplyTimeout() ??
public static void setPacketReplyTimeout(int timeout)
Sets the number of milliseconds to wait for a response from the server.
Parameters:
timeout - the milliseconds to wait for a response from the server
e.g.
SmackConfiguration.setPacketReplyTimeout(15000);
ConnectionConfiguration connectionConfig = new ConnectionConfiguration(host, port);
connectionConfig.setRosterLoadedAtLogin(true);
connectionConfig.setSendPresence(true);
connectionConfig.setSASLAuthenticationEnabled(false);
connectionConfig.setReconnectionAllowed(true);
connection = new XMPPConnection(connectionConfig);
Still happening here.
@kutzi: what do you think about @cforce suggestion?
I don't remember the details anymore, but as far as I remember setting the packet reply timeout won't help in this case.
We would need to set a connection timeout, but that is not possible. Or at least was not possible at that time.
As stated earlier: pull requests are welcome!
Thanks for the feedback. You believe we would need a patch in Smack lib?
I've released jabber-plugin 1.36 which bumps Smack to 4.1.9. I'll try to reproduce the situation and see if the issue still exists, but I can't guarantee that I will find anytime soon doing so.
Stack trace of Hudson job (excerpt):
"Executor #1 for master : executing CORE_FULL_BUILD_INTEGRATION_LIVE #56" prio=10 tid=0x00002aaad03c6000 nid=0xd67 waiting for monitor entry [0x0000000043e78000]
java.lang.Thread.State: BLOCKED (on object monitor)
at hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:78)
at hudson.plugins.jabber.im.transport.JabberPublisher.getIMConnection(JabberPublisher.java:113)
at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:347)
at hudson.plugins.im.IMPublisher.notifyChats(IMPublisher.java:505)
at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:275)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:675)
at hudson.model.Run.run(Run.java:1408)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:349)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
"Executor #0 for master : executing BO_TRUNK #33" prio=10 tid=0x00002aaad0913800 nid=0xd66 waiting for monitor entry [0x000000004356f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:78)
at hudson.plugins.jabber.im.transport.JabberPublisher.getIMConnection(JabberPublisher.java:113)
at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:347)
at hudson.plugins.im.IMPublisher.notifyChats(IMPublisher.java:505)
at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:275)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:675)
at hudson.model.Run.run(Run.java:1408)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:349)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
"Smack Packet Writer (1)" daemon prio=10 tid=0x00002aaac816f800 nid=0xd35 runnable [0x0000000044d87000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at java.io.BufferedWriter.flush(BufferedWriter.java:236)
at java.io.FilterWriter.flush(FilterWriter.java:83)
at org.jivesoftware.smack.PacketWriter.openStream(PacketWriter.java:352)
at org.jivesoftware.smack.PacketWriter.writePackets(PacketWriter.java:252)
at org.jivesoftware.smack.PacketWriter.access$000(PacketWriter.java:40)
at org.jivesoftware.smack.PacketWriter$1.run(PacketWriter.java:87)
"IM-Reconnector-Thread" daemon prio=10 tid=0x0000000051b86800 nid=0xd27 waiting for monitor entry [0x0000000044882000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.io.BufferedReader.close(BufferedReader.java:495)
at java.io.FilterReader.close(FilterReader.java:104)
at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:985)
at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection.java:904)
at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:1415)
at hudson.plugins.jabber.im.transport.JabberIMConnection.retryConnectionWithLegacySSL(JabberIMConnection.java:286)
at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:259)
at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:133)
at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42)
at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:60)
at hudson.plugins.im.IMConnectionProvider.access$500(IMConnectionProvider.java:17)
at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:173)
at java.lang.Thread.run(Thread.java:662)