Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40965

Gerrit trigger connection hangs while WatchDog attempts to reconnect.

    XMLWordPrintable

Details

    Description

      Hi,

      On some occasions when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connection back. Many times, it works!

      Looks like the WatchDog attempting to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

      The only way to recover is to restart jenkins. Could be similar to the issue already report JENKINS-15065

      ThreadDump

      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at java.lang.Thread.join(Thread.java:1245)
      	- locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
      	at java.lang.Thread.join(Thread.java:1319)
      	at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
      	- locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	.
      	.
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
      	- locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
         Locked ownable synchronizers:
      	- <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
      	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
      	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      ..
      .
         Locked ownable synchronizers:
      	- <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
      	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
      	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      .
      .
         Locked ownable synchronizers:
      	- <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      
      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
      stackTrace:
          java.lang.Thread.State: WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at java.lang.Thread.join(Thread.java:1245)
          - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
          at java.lang.Thread.join(Thread.java:1319)
          at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
          - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      
      

      Catalina.out

      Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
      INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
      

      Thanks!
      Dilip Mahadevappa

      Attachments

        Issue Links

          Activity

            People

              hektve Hector David
              dilipm79 Dilip Mahadevappa
              Votes:
              7 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: