Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9233

Builds hang on jabber notification when XMPP server is not responding

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • jabber-plugin
    • None

      We had the problem that all our builds did hang, because the XMPP server was not responding (it had an OOME)

          [JENKINS-9233] Builds hang on jabber notification when XMPP server is not responding

          kutzi created issue -

          kutzi added a comment -

          Stack trace of Hudson job (excerpt):

          "Executor #1 for master : executing CORE_FULL_BUILD_INTEGRATION_LIVE #56" prio=10 tid=0x00002aaad03c6000 nid=0xd67 waiting for monitor entry [0x0000000043e78000]
          java.lang.Thread.State: BLOCKED (on object monitor)
          at hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:78)

          • waiting to lock <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider)
            at hudson.plugins.jabber.im.transport.JabberPublisher.getIMConnection(JabberPublisher.java:113)
            at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:347)
            at hudson.plugins.im.IMPublisher.notifyChats(IMPublisher.java:505)
            at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:275)
            at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
            at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603)
            at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582)
            at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:675)
            at hudson.model.Run.run(Run.java:1408)
            at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:349)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:145)

          "Executor #0 for master : executing BO_TRUNK #33" prio=10 tid=0x00002aaad0913800 nid=0xd66 waiting for monitor entry [0x000000004356f000]
          java.lang.Thread.State: BLOCKED (on object monitor)
          at hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:78)

          • waiting to lock <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider)
            at hudson.plugins.jabber.im.transport.JabberPublisher.getIMConnection(JabberPublisher.java:113)
            at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:347)
            at hudson.plugins.im.IMPublisher.notifyChats(IMPublisher.java:505)
            at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:275)
            at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
            at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603)
            at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582)
            at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:675)
            at hudson.model.Run.run(Run.java:1408)
            at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:349)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:145)

          "Smack Packet Writer (1)" daemon prio=10 tid=0x00002aaac816f800 nid=0xd35 runnable [0x0000000044d87000]
          java.lang.Thread.State: RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(SocketInputStream.java:129)
          at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
          at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
          at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)

          • locked <0x00000000833fcfd8> (a java.lang.Object)
            at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
          • locked <0x00000000834004a8> (a java.lang.Object)
            at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
            at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
          • locked <0x00000000833fd010> (a com.sun.net.ssl.internal.ssl.AppOutputStream)
            at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
            at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
            at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
            at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
          • locked <0x00000000833ff070> (a java.io.OutputStreamWriter)
            at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
            at java.io.BufferedWriter.flush(BufferedWriter.java:236)
          • locked <0x00000000833ff070> (a java.io.OutputStreamWriter)
            at java.io.FilterWriter.flush(FilterWriter.java:83)
            at org.jivesoftware.smack.PacketWriter.openStream(PacketWriter.java:352)
            at org.jivesoftware.smack.PacketWriter.writePackets(PacketWriter.java:252)
            at org.jivesoftware.smack.PacketWriter.access$000(PacketWriter.java:40)
            at org.jivesoftware.smack.PacketWriter$1.run(PacketWriter.java:87)

          "IM-Reconnector-Thread" daemon prio=10 tid=0x0000000051b86800 nid=0xd27 waiting for monitor entry [0x0000000044882000]
          java.lang.Thread.State: BLOCKED (on object monitor)
          at java.io.BufferedReader.close(BufferedReader.java:495)

          • waiting to lock <0x00000000833f1cb0> (a java.io.InputStreamReader)
            at java.io.FilterReader.close(FilterReader.java:104)
            at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:985)
            at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection.java:904)
            at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:1415)
            at hudson.plugins.jabber.im.transport.JabberIMConnection.retryConnectionWithLegacySSL(JabberIMConnection.java:286)
            at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:259)
            at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:133)
            at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42)
          • locked <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider)
            at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:60)
          • locked <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider)
            at hudson.plugins.im.IMConnectionProvider.access$500(IMConnectionProvider.java:17)
            at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:173)
          • locked <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider)
            at java.lang.Thread.run(Thread.java:662)

          kutzi added a comment - Stack trace of Hudson job (excerpt): "Executor #1 for master : executing CORE_FULL_BUILD_INTEGRATION_LIVE #56" prio=10 tid=0x00002aaad03c6000 nid=0xd67 waiting for monitor entry [0x0000000043e78000] java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:78) waiting to lock <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider) at hudson.plugins.jabber.im.transport.JabberPublisher.getIMConnection(JabberPublisher.java:113) at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:347) at hudson.plugins.im.IMPublisher.notifyChats(IMPublisher.java:505) at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:275) at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582) at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:675) at hudson.model.Run.run(Run.java:1408) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:349) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:145) "Executor #0 for master : executing BO_TRUNK #33" prio=10 tid=0x00002aaad0913800 nid=0xd66 waiting for monitor entry [0x000000004356f000] java.lang.Thread.State: BLOCKED (on object monitor) at hudson.plugins.im.IMConnectionProvider.currentConnection(IMConnectionProvider.java:78) waiting to lock <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider) at hudson.plugins.jabber.im.transport.JabberPublisher.getIMConnection(JabberPublisher.java:113) at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:347) at hudson.plugins.im.IMPublisher.notifyChats(IMPublisher.java:505) at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:275) at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:603) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:582) at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:675) at hudson.model.Run.run(Run.java:1408) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:349) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:145) "Smack Packet Writer (1)" daemon prio=10 tid=0x00002aaac816f800 nid=0xd35 runnable [0x0000000044d87000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798) locked <0x00000000833fcfd8> (a java.lang.Object) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138) locked <0x00000000834004a8> (a java.lang.Object) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632) at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59) locked <0x00000000833fd010> (a com.sun.net.ssl.internal.ssl.AppOutputStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) locked <0x00000000833ff070> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at java.io.BufferedWriter.flush(BufferedWriter.java:236) locked <0x00000000833ff070> (a java.io.OutputStreamWriter) at java.io.FilterWriter.flush(FilterWriter.java:83) at org.jivesoftware.smack.PacketWriter.openStream(PacketWriter.java:352) at org.jivesoftware.smack.PacketWriter.writePackets(PacketWriter.java:252) at org.jivesoftware.smack.PacketWriter.access$000(PacketWriter.java:40) at org.jivesoftware.smack.PacketWriter$1.run(PacketWriter.java:87) "IM-Reconnector-Thread" daemon prio=10 tid=0x0000000051b86800 nid=0xd27 waiting for monitor entry [0x0000000044882000] java.lang.Thread.State: BLOCKED (on object monitor) at java.io.BufferedReader.close(BufferedReader.java:495) waiting to lock <0x00000000833f1cb0> (a java.io.InputStreamReader) at java.io.FilterReader.close(FilterReader.java:104) at org.jivesoftware.smack.XMPPConnection.initConnection(XMPPConnection.java:985) at org.jivesoftware.smack.XMPPConnection.connectUsingConfiguration(XMPPConnection.java:904) at org.jivesoftware.smack.XMPPConnection.connect(XMPPConnection.java:1415) at hudson.plugins.jabber.im.transport.JabberIMConnection.retryConnectionWithLegacySSL(JabberIMConnection.java:286) at hudson.plugins.jabber.im.transport.JabberIMConnection.createConnection(JabberIMConnection.java:259) at hudson.plugins.jabber.im.transport.JabberIMConnection.connect(JabberIMConnection.java:133) at hudson.plugins.jabber.im.transport.JabberIMConnectionProvider.createConnection(JabberIMConnectionProvider.java:42) locked <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider) at hudson.plugins.im.IMConnectionProvider.create(IMConnectionProvider.java:60) locked <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider) at hudson.plugins.im.IMConnectionProvider.access$500(IMConnectionProvider.java:17) at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:173) locked <0x0000000082e288c8> (a hudson.plugins.jabber.im.transport.JabberIMConnectionProvider) at java.lang.Thread.run(Thread.java:662)

          kutzi added a comment -

          Can we add a timeout to let the builds fail at least?

          kutzi added a comment - Can we add a timeout to let the builds fail at least?

          kutzi added a comment -

          As far as I've seen, there's no way to set a timeout there

          kutzi added a comment - As far as I've seen, there's no way to set a timeout there

          lplatypus added a comment -

          Same happening here. In this state the cross icon next to the build progress indicator to cancel the build does not work. There seems to be no way to recover besides killing the whole Jenkins service and restarting.

          lplatypus added a comment - Same happening here. In this state the cross icon next to the build progress indicator to cancel the build does not work. There seems to be no way to recover besides killing the whole Jenkins service and restarting.

          I'd like to look into this, but so far I failed to reproduce the issue.

          I'm using:
          jenkins master
          jabber-plugin master
          prosody-0.8.2

          I have set up a test build that waits 20 seconds, then tries to send a XMPP notification. Before the build completes I'd like to simulate a non-responding XMPP server. When I send prosody SIGKILL, jenkins notices that the connection has closed. If I send it a SIGSTOP, jenkins doesn't report any error on sending the notification.

          Any idea how to reproduce this behaviour?

          Christoph Gysin added a comment - I'd like to look into this, but so far I failed to reproduce the issue. I'm using: jenkins master jabber-plugin master prosody-0.8.2 I have set up a test build that waits 20 seconds, then tries to send a XMPP notification. Before the build completes I'd like to simulate a non-responding XMPP server. When I send prosody SIGKILL, jenkins notices that the connection has closed. If I send it a SIGSTOP, jenkins doesn't report any error on sending the notification. Any idea how to reproduce this behaviour?

          cforce added a comment -

          We have the same problem here. If javver server down, our builds freezes forever (can't even terminate them by clicking the cross, only by terminating the server - tomcat)
          This is a huge problem! Timeout is required or this in unusable. Please fix it, we like this plugin ..

          We use

          windows server 2008 and with Tomcat 7
          jenkins 1.473 as war
          Jenkins Jabber notifier plugin 1.22

          cforce added a comment - We have the same problem here. If javver server down, our builds freezes forever (can't even terminate them by clicking the cross, only by terminating the server - tomcat) This is a huge problem! Timeout is required or this in unusable. Please fix it, we like this plugin .. We use windows server 2008 and with Tomcat 7 jenkins 1.473 as war Jenkins Jabber notifier plugin 1.22

          cforce added a comment -

          openfire 3.71

          cforce added a comment - openfire 3.71

          Jeff Stewart added a comment -

          Is there any news about this bug? Our company IM server went down last night and brought production to a standstill. Seems IM messages should be queued and threaded as to not halt the build(s) if there's a hiccup in the network, or an unstable IM server specified. I personally wouldn't consider a failed IM notification a "build breaker". I would consider a hung Jenkins server due to a deadlock in a low priority notification routine a much bigger issue.

          Jenkins 1.504
          Instant Messenger Plugin 1.25
          Jabber Plugin 1.25

          Jeff Stewart added a comment - Is there any news about this bug? Our company IM server went down last night and brought production to a standstill. Seems IM messages should be queued and threaded as to not halt the build(s) if there's a hiccup in the network, or an unstable IM server specified. I personally wouldn't consider a failed IM notification a "build breaker". I would consider a hung Jenkins server due to a deadlock in a low priority notification routine a much bigger issue. Jenkins 1.504 Instant Messenger Plugin 1.25 Jabber Plugin 1.25
          kutzi made changes -
          Assignee Original: kutzi [ kutzi ]

            flow Florian Schmaus
            kutzi kutzi
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: