Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22042

IRCBot disconnects after long idle

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Critical
    • Resolution: Unresolved
    • ircbot-plugin
    • None
    • CentOS 5.4 x64, ircd-seven (latest), Atheme Services 7.0.x

    Description

      We have an own installation of IRCd which is running the latest version of IRCd Seven with Atheme Services (more or less the same setup as on Freenode).

      Jenkins Bot is able to join and authenticate properly and stays in the channel for some time. It is also correctly publishing notifications and responds to commands. But after a period of inactivity (we have no activity in channel during the night) it dies (basically the bot times out during the night at some point). And the only way for it to re-join is for us to restart Jenkins server which seems bad.

      Attachments

        Issue Links

          Activity

            alexejk Alexej Kubarev created issue -

            I have observed this behavior on irc.gimp.net as well.

            alexrp Alex Rønne Petersen added a comment - I have observed this behavior on irc.gimp.net as well.

            I think we have a similar issue, internally reported as: https://www.drupal.org/node/2159903

            When I restart jenkins I get this RuntimeException:

            IRC notifier plugin: Sending notification to: #aegir
            IRC notifier plugin: There was an error sending notification to: #aegir
            java.lang.RuntimeException: Waiting for mode response interrupted
            at org.pircbotx.Channel.getMode(Channel.java:133)
            at hudson.plugins.ircbot.v2.IRCConnection.send(IRCConnection.java:276)
            at hudson.plugins.ircbot.v2.IRCConnection.send(IRCConnection.java:268)
            at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:379)
            at hudson.plugins.im.IMPublisher.notifyChatsOnBuildEnd(IMPublisher.java:585)
            at hudson.plugins.im.IMPublisher.notifyOnBuildEnd(IMPublisher.java:304)
            at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:291)
            at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
            at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
            at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
            at hudson.model.Build$BuildExecution.post2(Build.java:183)
            at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:725)
            at hudson.model.Run.execute(Run.java:1709)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:88)
            at hudson.model.Executor.run(Executor.java:231)
            Caused by: java.lang.InterruptedException
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:979)
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
            at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
            at org.pircbotx.Channel.getMode(Channel.java:127)
            ... 15 more

            An older RuntimeException from the jenkins log, I guess the first notice of when the connection was lost.

            Aug 6, 2014 2:31:18 AM hudson.plugins.ircbot.v2.IRCConnection send
            WARNING: Bot is configured to use colors, but channel #aegir disallows colors!
            Exception in thread "bot1-input" java.lang.RuntimeException: Waiting for mode response interrupted
            at org.pircbotx.Channel.getMode(Channel.java:133)
            at org.pircbotx.Channel.getModeArgument(Channel.java:182)
            at org.pircbotx.Channel.getChannelKey(Channel.java:239)
            at org.pircbotx.PircBotX.shutdown(PircBotX.java:2872)
            at org.pircbotx.PircBotX.shutdown(PircBotX.java:2833)
            at org.pircbotx.InputThread.run(InputThread.java:115)
            Caused by: java.lang.InterruptedException
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279)
            at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
            at org.pircbotx.Channel.getMode(Channel.java:127)
            ... 5 more

            helmo Herman van Rink added a comment - I think we have a similar issue, internally reported as: https://www.drupal.org/node/2159903 When I restart jenkins I get this RuntimeException: IRC notifier plugin: Sending notification to: #aegir IRC notifier plugin: There was an error sending notification to: #aegir java.lang.RuntimeException: Waiting for mode response interrupted at org.pircbotx.Channel.getMode(Channel.java:133) at hudson.plugins.ircbot.v2.IRCConnection.send(IRCConnection.java:276) at hudson.plugins.ircbot.v2.IRCConnection.send(IRCConnection.java:268) at hudson.plugins.im.IMPublisher.sendNotification(IMPublisher.java:379) at hudson.plugins.im.IMPublisher.notifyChatsOnBuildEnd(IMPublisher.java:585) at hudson.plugins.im.IMPublisher.notifyOnBuildEnd(IMPublisher.java:304) at hudson.plugins.im.IMPublisher.perform(IMPublisher.java:291) at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:725) at hudson.model.Run.execute(Run.java:1709) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:979) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at org.pircbotx.Channel.getMode(Channel.java:127) ... 15 more An older RuntimeException from the jenkins log, I guess the first notice of when the connection was lost. Aug 6, 2014 2:31:18 AM hudson.plugins.ircbot.v2.IRCConnection send WARNING: Bot is configured to use colors, but channel #aegir disallows colors! Exception in thread "bot1-input" java.lang.RuntimeException: Waiting for mode response interrupted at org.pircbotx.Channel.getMode(Channel.java:133) at org.pircbotx.Channel.getModeArgument(Channel.java:182) at org.pircbotx.Channel.getChannelKey(Channel.java:239) at org.pircbotx.PircBotX.shutdown(PircBotX.java:2872) at org.pircbotx.PircBotX.shutdown(PircBotX.java:2833) at org.pircbotx.InputThread.run(InputThread.java:115) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) at org.pircbotx.Channel.getMode(Channel.java:127) ... 5 more

            Got feedback in another issue: https://issues.jenkins-ci.org/browse/JENKINS-26491

            The build-timeout-plugin is not a solution here.

            I therefore would like to request some sort of timeout in the instant-messaging plugin.

            helmo Herman van Rink added a comment - Got feedback in another issue: https://issues.jenkins-ci.org/browse/JENKINS-26491 The build-timeout-plugin is not a solution here. I therefore would like to request some sort of timeout in the instant-messaging plugin.
            helmo Herman van Rink made changes -
            Field Original Value New Value
            Component/s instant-messaging-plugin [ 15637 ]
            helmo Herman van Rink made changes -
            Link This issue is related to JENKINS-26491 [ JENKINS-26491 ]
            helmo Herman van Rink made changes -
            Link This issue is related to JENKINS-16875 [ JENKINS-16875 ]
            kutzi kutzi added a comment -

            Herman, are you sure you are talking about the same issue?
            Why would you even think that the build-timeout plugin could be a solution to the IRC plugin disconnecting after a long idle time?

            kutzi kutzi added a comment - Herman, are you sure you are talking about the same issue? Why would you even think that the build-timeout plugin could be a solution to the IRC plugin disconnecting after a long idle time?

            Kutzi, I agree that it's not a proper solution. Some form of re-connect would probably be needed.

            What I had hoped the build-timeout plugin to solve is to stop a build that is hanging on the notification. A failed build would alert me of an issue, letting me restart Jenkins if needed to get the IRC connection back. The ironic thing is that the notification is an after-build step, that's why the build-timeout plugin did not help.

            Without a timeout it could be a week before anyone noticed that the builds have stalled.

            helmo Herman van Rink added a comment - Kutzi, I agree that it's not a proper solution. Some form of re-connect would probably be needed. What I had hoped the build-timeout plugin to solve is to stop a build that is hanging on the notification. A failed build would alert me of an issue, letting me restart Jenkins if needed to get the IRC connection back. The ironic thing is that the notification is an after-build step, that's why the build-timeout plugin did not help. Without a timeout it could be a week before anyone noticed that the builds have stalled.
            kutzi kutzi made changes -
            Component/s instant-messaging-plugin [ 15637 ]
            Priority Critical [ 2 ] Major [ 3 ]
            kutzi kutzi added a comment -

            Alexej, any exception in the log before Jenkins is restarted?
            Actually, the bot should be able to detect disconnects and reconnect on itself. I wonder, what is not working with this.

            kutzi kutzi added a comment - Alexej, any exception in the log before Jenkins is restarted? Actually, the bot should be able to detect disconnects and reconnect on itself. I wonder, what is not working with this.
            urras Nick Hovaker made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            kutzi kutzi added a comment -

            Has some similarities with JENKINS-28175

            kutzi kutzi added a comment - Has some similarities with JENKINS-28175
            kutzi kutzi made changes -
            Link This issue is related to JENKINS-28175 [ JENKINS-28175 ]
            kutzi kutzi added a comment -

            That is: for the error that Herman is reporting.
            Alexej's error might be a completely different one. No way to be sure until he clarifies what it means that the bot 'dies'

            kutzi kutzi added a comment - That is: for the error that Herman is reporting. Alexej's error might be a completely different one. No way to be sure until he clarifies what it means that the bot 'dies'
            kutzi kutzi added a comment -

            Herman, on further thoughts your issue seems to be pretty different from the originally reported issue - disconnecting and hanging on notifications is actually a totally different thing.
            Isn't it actually what you reported in JENKINS-26491?

            kutzi kutzi added a comment - Herman, on further thoughts your issue seems to be pretty different from the originally reported issue - disconnecting and hanging on notifications is actually a totally different thing. Isn't it actually what you reported in JENKINS-26491 ?
            kutzi kutzi added a comment -

            @urras: as you changed this to Critical: can you provide more information. I.e. how to reproduce, error entries in log file etc?

            kutzi kutzi added a comment - @urras: as you changed this to Critical: can you provide more information. I.e. how to reproduce, error entries in log file etc?
            urras Nick Hovaker added a comment -

            @kutzi I will once it happens again on my instance

            urras Nick Hovaker added a comment - @kutzi I will once it happens again on my instance

            Exception in thread "bot2-input" java.lang.RuntimeException: Waiting for mode response interrupted
            at org.pircbotx.Channel.getMode(Channel.java:133)
            at org.pircbotx.Channel.getModeArgument(Channel.java:182)
            at org.pircbotx.Channel.getChannelKey(Channel.java:239)
            at org.pircbotx.PircBotX.shutdown(PircBotX.java:2872)
            at org.pircbotx.PircBotX.shutdown(PircBotX.java:2833)
            at org.pircbotx.InputThread.run(InputThread.java:115)
            Caused by: java.lang.InterruptedException
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
            at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
            at org.pircbotx.Channel.getMode(Channel.java:127)
            ... 5 more

            then the entire jenkins instance is stuck as no job can finish, stuck indefinitely in "IRC notifier plugin: Sending notification to:...."

            require a hard restart to get the thing moving again.

            shmget Norbert Thiebaud added a comment - Exception in thread "bot2-input" java.lang.RuntimeException: Waiting for mode response interrupted at org.pircbotx.Channel.getMode(Channel.java:133) at org.pircbotx.Channel.getModeArgument(Channel.java:182) at org.pircbotx.Channel.getChannelKey(Channel.java:239) at org.pircbotx.PircBotX.shutdown(PircBotX.java:2872) at org.pircbotx.PircBotX.shutdown(PircBotX.java:2833) at org.pircbotx.InputThread.run(InputThread.java:115) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) at org.pircbotx.Channel.getMode(Channel.java:127) ... 5 more then the entire jenkins instance is stuck as no job can finish, stuck indefinitely in "IRC notifier plugin: Sending notification to:...." require a hard restart to get the thing moving again.

            We just had this exact issue effect us, had to restart to get moving again as well.

            Exception in thread "bot4-input" java.lang.RuntimeException: Waiting for mode response interrupted
            at org.pircbotx.Channel.getMode(Channel.java:133)
            at org.pircbotx.Channel.getModeArgument(Channel.java:182)
            at org.pircbotx.Channel.getChannelKey(Channel.java:239)
            at org.pircbotx.PircBotX.shutdown(PircBotX.java:2872)
            at org.pircbotx.PircBotX.shutdown(PircBotX.java:2833)
            at org.pircbotx.InputThread.run(InputThread.java:115)
            Caused by: java.lang.InterruptedException
            at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
            at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
            at org.pircbotx.Channel.getMode(Channel.java:127)
            ... 5 more

            jimbo jim gedarovich added a comment - We just had this exact issue effect us, had to restart to get moving again as well. Exception in thread "bot4-input" java.lang.RuntimeException: Waiting for mode response interrupted at org.pircbotx.Channel.getMode(Channel.java:133) at org.pircbotx.Channel.getModeArgument(Channel.java:182) at org.pircbotx.Channel.getChannelKey(Channel.java:239) at org.pircbotx.PircBotX.shutdown(PircBotX.java:2872) at org.pircbotx.PircBotX.shutdown(PircBotX.java:2833) at org.pircbotx.InputThread.run(InputThread.java:115) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236) at org.pircbotx.Channel.getMode(Channel.java:127) ... 5 more
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 154121 ] JNJira + In-Review [ 178698 ]
            jimklimov Jim Klimov made changes -
            Link This issue is related to JENKINS-12221 [ JENKINS-12221 ]

            People

              kutzi kutzi
              alexejk Alexej Kubarev
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated: