Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9233

Builds hang on jabber notification when XMPP server is not responding

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • jabber-plugin
    • None

      We had the problem that all our builds did hang, because the XMPP server was not responding (it had an OOME)

          [JENKINS-9233] Builds hang on jabber notification when XMPP server is not responding

          cforce added a comment -

          openfire 3.71

          cforce added a comment - openfire 3.71

          Jeff Stewart added a comment -

          Is there any news about this bug? Our company IM server went down last night and brought production to a standstill. Seems IM messages should be queued and threaded as to not halt the build(s) if there's a hiccup in the network, or an unstable IM server specified. I personally wouldn't consider a failed IM notification a "build breaker". I would consider a hung Jenkins server due to a deadlock in a low priority notification routine a much bigger issue.

          Jenkins 1.504
          Instant Messenger Plugin 1.25
          Jabber Plugin 1.25

          Jeff Stewart added a comment - Is there any news about this bug? Our company IM server went down last night and brought production to a standstill. Seems IM messages should be queued and threaded as to not halt the build(s) if there's a hiccup in the network, or an unstable IM server specified. I personally wouldn't consider a failed IM notification a "build breaker". I would consider a hung Jenkins server due to a deadlock in a low priority notification routine a much bigger issue. Jenkins 1.504 Instant Messenger Plugin 1.25 Jabber Plugin 1.25

          Derek Gallo added a comment -

          We have been hit by this issue today as well. Our Jabber server has been having issues all day. Our builds in Jenkins are setup to send a notification after a build but they hang indefinitely and the abort build button does not work.

          Restarting Tomcat/Jenkins is an issue as well. Jenkins is not coming back up. Tailing catalina.out shows that the plugin is in an endless loop trying to re-connect.

          2013-09-05 18:37:47 INFO [onnector-Thread] - hudson.plugins.im.IMConnectionProvider - Reconnect failed. Next connection attempt in 1 minutes
          2013-09-05 18:38:47 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Trying to connect to XMPP on jabber.highwinds.com:5222/jabber.highwinds.com with SASL
          2013-09-05 18:39:52 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying connection with legacy SSL
          2013-09-05 18:53:48 WARN [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying with legacy SSL failed: Connection failed. No response from server.
          2013-09-05 18:53:48 WARN [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Exception of original (without legacy SSL) connection attempt:
          – caused by: Connection failed. No response from server.:
          at hudson.plugins.jabber.im.transport.JabberIMConnection.retryConnectionWithLegacySSL(JabberIMConnection.java:349)
          at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:315)
          at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:162)
          at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42)
          at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:65)
          at hudson.plugins.im.IMConnectionProvider.access$600(IMConnectionProvider.java:22)
          at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:183)
          at java.lang.Thread.run(Thread.java:662)
          Nested Exception:
          Connection failed. No response from server.:
          at org.jivesoftware.smack.PacketReader.startup(PacketReader.java:119)
          at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:568)
          at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection.java:527)
          at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:953)
          at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:305)
          at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:162)
          at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42)
          at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:65)
          at hudson.plugins.im.IMConnectionProvider.access$600(IMConnectionProvider.java:22)
          at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:183)
          at java.lang.Thread.run(Thread.java:662)

          2013-09-05 18:53:48 INFO [onnector-Thread] - hudson.plugins.im.IMConnectionProvider - Reconnect failed. Next connection attempt in 2 minutes
          2013-09-05 18:55:48 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Trying to connect to XMPP on jabber.highwinds.com:5222/jabber.highwinds.com with SASL
          2013-09-05 18:56:09 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying connection with legacy SSL

          Derek Gallo added a comment - We have been hit by this issue today as well. Our Jabber server has been having issues all day. Our builds in Jenkins are setup to send a notification after a build but they hang indefinitely and the abort build button does not work. Restarting Tomcat/Jenkins is an issue as well. Jenkins is not coming back up. Tailing catalina.out shows that the plugin is in an endless loop trying to re-connect. 2013-09-05 18:37:47 INFO [onnector-Thread] - hudson.plugins.im.IMConnectionProvider - Reconnect failed. Next connection attempt in 1 minutes 2013-09-05 18:38:47 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Trying to connect to XMPP on jabber.highwinds.com:5222/jabber.highwinds.com with SASL 2013-09-05 18:39:52 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying connection with legacy SSL 2013-09-05 18:53:48 WARN [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying with legacy SSL failed: Connection failed. No response from server. 2013-09-05 18:53:48 WARN [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Exception of original (without legacy SSL) connection attempt: – caused by: Connection failed. No response from server.: at hudson.plugins.jabber.im.transport.JabberIMConnection.retryConnectionWithLegacySSL(JabberIMConnection.java:349) at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:315) at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:162) at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42) at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:65) at hudson.plugins.im.IMConnectionProvider.access$600(IMConnectionProvider.java:22) at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:183) at java.lang.Thread.run(Thread.java:662) Nested Exception: Connection failed. No response from server.: at org.jivesoftware.smack.PacketReader.startup(PacketReader.java:119) at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:568) at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection.java:527) at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:953) at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:305) at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:162) at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42) at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:65) at hudson.plugins.im.IMConnectionProvider.access$600(IMConnectionProvider.java:22) at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:183) at java.lang.Thread.run(Thread.java:662) 2013-09-05 18:53:48 INFO [onnector-Thread] - hudson.plugins.im.IMConnectionProvider - Reconnect failed. Next connection attempt in 2 minutes 2013-09-05 18:55:48 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Trying to connect to XMPP on jabber.highwinds.com:5222/jabber.highwinds.com with SASL 2013-09-05 18:56:09 INFO [onnector-Thread] - hudson.plugins.jabber.im.transport.JabberIMConnection - Retrying connection with legacy SSL

          kutzi added a comment -

          @cforce: as I already stated earlier, I didn't find any way to set timeouts for this - so I'm limited by the Smack API to fix this.
          Unless youself find a way, increasing the priority of this bug or assigning it to me, again, won't help a bit to get this fixed.

          BTW: pull request are highly welcome

          kutzi added a comment - @cforce: as I already stated earlier, I didn't find any way to set timeouts for this - so I'm limited by the Smack API to fix this. Unless youself find a way, increasing the priority of this bug or assigning it to me, again, won't help a bit to get this fixed. BTW: pull request are highly welcome

          cforce added a comment -

          What about setPacketReplyTimeout() ??

          public static void setPacketReplyTimeout(int timeout)
          Sets the number of milliseconds to wait for a response from the server.
          Parameters:
          timeout - the milliseconds to wait for a response from the server

          http://www.igniterealtime.org/builds/smack/docs/latest/javadoc/org/jivesoftware/smack/SmackConfiguration.html#getPacketReplyTimeout()

          e.g.

          SmackConfiguration.setPacketReplyTimeout(15000);
          ConnectionConfiguration connectionConfig = new ConnectionConfiguration(host, port);
          connectionConfig.setRosterLoadedAtLogin(true);
          connectionConfig.setSendPresence(true);
          connectionConfig.setSASLAuthenticationEnabled(false);
          connectionConfig.setReconnectionAllowed(true);
          connection = new XMPPConnection(connectionConfig);

          cforce added a comment - What about setPacketReplyTimeout() ?? public static void setPacketReplyTimeout(int timeout) Sets the number of milliseconds to wait for a response from the server. Parameters: timeout - the milliseconds to wait for a response from the server http://www.igniterealtime.org/builds/smack/docs/latest/javadoc/org/jivesoftware/smack/SmackConfiguration.html#getPacketReplyTimeout( ) e.g. SmackConfiguration.setPacketReplyTimeout(15000); ConnectionConfiguration connectionConfig = new ConnectionConfiguration(host, port); connectionConfig.setRosterLoadedAtLogin(true); connectionConfig.setSendPresence(true); connectionConfig.setSASLAuthenticationEnabled(false); connectionConfig.setReconnectionAllowed(true); connection = new XMPPConnection(connectionConfig);

          Paul Eipper added a comment -

          Still happening here.

          @kutzi: what do you think about @cforce suggestion?

          Paul Eipper added a comment - Still happening here. @kutzi: what do you think about @cforce suggestion?

          kutzi added a comment -

          I don't remember the details anymore, but as far as I remember setting the packet reply timeout won't help in this case.
          We would need to set a connection timeout, but that is not possible. Or at least was not possible at that time.

          As stated earlier: pull requests are welcome!

          kutzi added a comment - I don't remember the details anymore, but as far as I remember setting the packet reply timeout won't help in this case. We would need to set a connection timeout, but that is not possible. Or at least was not possible at that time. As stated earlier: pull requests are welcome!

          Paul Eipper added a comment -

          Thanks for the feedback. You believe we would need a patch in Smack lib?

          Paul Eipper added a comment - Thanks for the feedback. You believe we would need a patch in Smack lib?

          kutzi added a comment -

          Not sure, but probably yes.

          kutzi added a comment - Not sure, but probably yes.

          I've released jabber-plugin 1.36 which bumps Smack to 4.1.9. I'll try to reproduce the situation and see if the issue still exists, but I can't guarantee that I will find anytime soon doing so.

          Florian Schmaus added a comment - I've released jabber-plugin 1.36 which bumps Smack to 4.1.9. I'll try to reproduce the situation and see if the issue still exists, but I can't guarantee that I will find anytime soon doing so.

            flow Florian Schmaus
            kutzi kutzi
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: