Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40965

Gerrit trigger connection hangs while WatchDog attempts to reconnect.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • gerrit-trigger-plugin
    • Gerrit trigger:2.21.1, 2.22.0, 2.23.0
      Jenkins: 1.651.3, 1.651.1

      Hi,

      On some occasions when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connection back. Many times, it works!

      Looks like the WatchDog attempting to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

      The only way to recover is to restart jenkins. Could be similar to the issue already report JENKINS-15065

      ThreadDump

      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1245)
	- locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
	at java.lang.Thread.join(Thread.java:1319)
	at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
	- locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	.
	.
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
	- locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
	- <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
..
.
   Locked ownable synchronizers:
	- <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
.
.
   Locked ownable synchronizers:
	- <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)


"Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
stackTrace:
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1245)
    - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
    at java.lang.Thread.join(Thread.java:1319)
    at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
    at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
    - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
    at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      Catalina.out

      Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.

      Thanks!
      Dilip Mahadevappa

        1. JENKINS-40965.png
          JENKINS-40965.png
          4 kB

          [JENKINS-40965] Gerrit trigger connection hangs while WatchDog attempts to reconnect.

          Dilip Mahadevappa created issue -
          Dilip Mahadevappa made changes -
          Description Original: Hi,

          On some occasions when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connect back. Many times, it works!

          Looks like the WatchDog attempt to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

          The only way to recover is to restart jenkins. Could be similar to the issue already report [JENKINS-15065 | https://issues.jenkins-ci.org/browse/JENKINS-15065]

          *ThreadDump*

          {noformat}
          "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
             java.lang.Thread.State: WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at java.lang.Thread.join(Thread.java:1245)
          - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
          at java.lang.Thread.join(Thread.java:1319)
          at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
          - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          .
          .
          at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
          at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
          at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
          - locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)

             Locked ownable synchronizers:
          - <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

          "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
          - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
          at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
          at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
          ..
          .
             Locked ownable synchronizers:
          - <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)

          "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
          - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
          at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
          at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
          at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
          at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
          .
          .
             Locked ownable synchronizers:
          - <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)


          "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
          stackTrace:
              java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Thread.join(Thread.java:1245)
              - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
              at java.lang.Thread.join(Thread.java:1319)
              at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
              at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
              - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
              at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


          {noformat}


          *Catalina.out*


          {noformat}
          Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
          INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
          {noformat}


          Thanks!
          Dilip Mahadevappa           		New: Hi,

          On _some occasions_ when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connection back. Many times, it works!

          Looks like the WatchDog attempting to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

          The only way to recover is to restart jenkins. Could be similar to the issue already report [JENKINS-15065 | https://issues.jenkins-ci.org/browse/JENKINS-15065]

          *ThreadDump*

          {noformat}
          "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
             java.lang.Thread.State: WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at java.lang.Thread.join(Thread.java:1245)
          - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
          at java.lang.Thread.join(Thread.java:1319)
          at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
          - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          .
          .
          at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
          at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
          at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
          - locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)

             Locked ownable synchronizers:
          - <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

          "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
          - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
          at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
          at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
          ..
          .
             Locked ownable synchronizers:
          - <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)

          "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
             java.lang.Thread.State: BLOCKED (on object monitor)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
          - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
          at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
          at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
          at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
          at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
          .
          .
             Locked ownable synchronizers:
          - <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)


          "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
          stackTrace:
              java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Thread.join(Thread.java:1245)
              - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
              at java.lang.Thread.join(Thread.java:1319)
              at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
              at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
              - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
              at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


          {noformat}


          *Catalina.out*


          {noformat}
          Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
          INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
          {noformat}


          Thanks!
          Dilip Mahadevappa
          Dilip Mahadevappa made changes -
          Environment Original: Gerrit trigger:2.21.1
          Jenkins: 1.651.3           		New: Gerrit trigger:2.21.1, 2.22.0
          Jenkins: 1.651.3, 1.651.1
          Dilip Mahadevappa made changes -
          Environment Original: Gerrit trigger:2.21.1, 2.22.0
          Jenkins: 1.651.3, 1.651.1           		New: Gerrit trigger:2.21.1, 2.22.0, 2.23.0
          Jenkins: 1.651.3, 1.651.1
          Dilip Mahadevappa made changes -
          Attachment New: JENKINS-40965.png [ 35929 ]
          Hector David made changes -
          Assignee Original: rsandell [ rsandell ] New: Hector David [ hektve ]
          Hector David made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          rsandell made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
          Markus Winter made changes -
          Link New: This issue relates to JENKINS-44414 [ JENKINS-44414 ]
          Markus Winter made changes -
          Link Original: This issue relates to JENKINS-44414 [ JENKINS-44414 ]
          James Dumay made changes -
          Remote Link New: This issue links to "CloudBees Internal OSS-2278 (Web Link)" [ 18359 ]

            hektve Hector David
            dilipm79 Dilip Mahadevappa
            Votes:
            7 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: