Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40965

Gerrit trigger connection hangs while WatchDog attempts to reconnect.

    XMLWordPrintable

Details

    Description

      Hi,

      On some occasions when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connection back. Many times, it works!

      Looks like the WatchDog attempting to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

      The only way to recover is to restart jenkins. Could be similar to the issue already report JENKINS-15065

      ThreadDump

      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	at java.lang.Thread.join(Thread.java:1245)
      	- locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
      	at java.lang.Thread.join(Thread.java:1319)
      	at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
      	- locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	.
      	.
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
      	- locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
         Locked ownable synchronizers:
      	- <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
      	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
      	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      ..
      .
         Locked ownable synchronizers:
      	- <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
      	- waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
      	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
      	at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      .
      .
         Locked ownable synchronizers:
      	- <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
      
      
      "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
      stackTrace:
          java.lang.Thread.State: WAITING (on object monitor)
          at java.lang.Object.wait(Native Method)
          at java.lang.Thread.join(Thread.java:1245)
          - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
          at java.lang.Thread.join(Thread.java:1319)
          at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
          - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
          at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      
      
      

      Catalina.out

      Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
      INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
      

      Thanks!
      Dilip Mahadevappa

      Attachments

        Issue Links

          Activity

            dilipm79 Dilip Mahadevappa created issue -
            dilipm79 Dilip Mahadevappa made changes -
            Field Original Value New Value
            Description Hi,

            On some occasions when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connect back. Many times, it works!

            Looks like the WatchDog attempt to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

            The only way to recover is to restart jenkins. Could be similar to the issue already report [JENKINS-15065 | https://issues.jenkins-ci.org/browse/JENKINS-15065]

            *ThreadDump*

            {noformat}
            "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
               java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            at java.lang.Thread.join(Thread.java:1245)
            - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
            at java.lang.Thread.join(Thread.java:1319)
            at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
            - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            .
            .
            at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
            at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
            at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
            - locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)

               Locked ownable synchronizers:
            - <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

            "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
               java.lang.Thread.State: BLOCKED (on object monitor)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
            - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
            at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            ..
            .
               Locked ownable synchronizers:
            - <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)

            "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
               java.lang.Thread.State: BLOCKED (on object monitor)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
            - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
            at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
            at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
            .
            .
               Locked ownable synchronizers:
            - <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)


            "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
            stackTrace:
                java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Thread.join(Thread.java:1245)
                - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
                at java.lang.Thread.join(Thread.java:1319)
                at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
                at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
                - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
                at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


            {noformat}


            *Catalina.out*


            {noformat}
            Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
            INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
            {noformat}


            Thanks!
            Dilip Mahadevappa
            Hi,

            On _some occasions_ when the connection between Jenkins and Gerrit server is down, the watchdog fails to re-establish the connection back. Many times, it works!

            Looks like the WatchDog attempting to shutdown the stream-events connection waits indefinitely, eventually BLOCKING many other Gerrit serverStatuses threads.

            The only way to recover is to restart jenkins. Could be similar to the issue already report [JENKINS-15065 | https://issues.jenkins-ci.org/browse/JENKINS-15065]

            *ThreadDump*

            {noformat}
            "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-914" #1169869 daemon prio=5 os_prio=0 tid=0x00007fb90c3b6800 nid=0x2d2a in Object.wait() [0x00007fb8e4495000]
               java.lang.Thread.State: WAITING (on object monitor)
            at java.lang.Object.wait(Native Method)
            at java.lang.Thread.join(Thread.java:1245)
            - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
            at java.lang.Thread.join(Thread.java:1319)
            at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
            - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            .
            .
            at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
            at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
            at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
            - locked <0x0000000560c9c5f8> (a org.apache.tomcat.util.net.SocketWrapper)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)

               Locked ownable synchronizers:
            - <0x0000000561678500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

            "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-938 GerritManagement/serverStatuses.jelly" #1172250 daemon prio=5 os_prio=0 tid=0x00007fb904856000 nid=0x3e25 waiting for monitor entry [0x00007fb8d6a03000]
               java.lang.Thread.State: BLOCKED (on object monitor)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
            - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
            at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            ..
            .
               Locked ownable synchronizers:
            - <0x00000005c0101218> (a java.util.concurrent.ThreadPoolExecutor$Worker)

            "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-929 GerritManagement/serverStatuses.jelly" #1170917 daemon prio=5 os_prio=0 tid=0x00007fb93a009800 nid=0x6f08 waiting for monitor entry [0x00007fb8dc822000]
               java.lang.Thread.State: BLOCKED (on object monitor)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
            - waiting to lock <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
            at sun.reflect.GeneratedMethodAccessor904.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:497)
            at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
            at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
            .
            .
               Locked ownable synchronizers:
            - <0x000000056ee04ca0> (a java.util.concurrent.ThreadPoolExecutor$Worker)


            "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx: http-bio-8443-exec-914 - priority:5 - threadId:0x00007fb90c3b6800 - nativeId:0x2d2a - state:WAITING
            stackTrace:
                java.lang.Thread.State: WAITING (on object monitor)
                at java.lang.Object.wait(Native Method)
                at java.lang.Thread.join(Thread.java:1245)
                - locked <0x0000000545878ae0> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
                at java.lang.Thread.join(Thread.java:1319)
                at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
                at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
                - locked <0x0000000541cab1f8> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
                at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


            {noformat}


            *Catalina.out*


            {noformat}
            Jan 10, 2017 1:42:49 AM com.sonymobile.tools.gerrit.gerritevents.watchdog.StreamWatchdog run
            INFO: Last lively connection with Gerrit was 60 seconds ago; reconnecting.
            {noformat}


            Thanks!
            Dilip Mahadevappa

            We had another occurrence of this with the same stacktrace.

            dilipm79 Dilip Mahadevappa added a comment - We had another occurrence of this with the same stacktrace.
            dilipm79 Dilip Mahadevappa made changes -
            Environment Gerrit trigger:2.21.1
            Jenkins: 1.651.3
            Gerrit trigger:2.21.1, 2.22.0
            Jenkins: 1.651.3, 1.651.1

            We had another occurrence of the same issue.. there was some 30 threads in BLOCKED waiting to lock states..

            *Gerrit version: 2.22.0 *

            "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-112 GerritManagement/serverStatuses.jelly" daemon prio=10 tid=0x00007f4aa4ac0000 nid=0x6664 waiting for monitor entry [0x00007f4a61a33000]
               java.lang.Thread.State: BLOCKED (on object monitor)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554)
            	- waiting to lock <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219)
            	at sun.reflect.GeneratedMethodAccessor827.invoke(Unknown Source)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
            	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
            	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
            	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
            	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
            	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
            	at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
            	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
            	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
            	at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24)
            	at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66)
            	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
            	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
            	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
            	at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
            	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
            	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
            	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
            	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
            	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
            	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
            	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
            	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
            	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
            	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
            	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
            	- locked <0x000000057d42c0f0> (a org.apache.tomcat.util.net.SocketWrapper)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            
               Locked ownable synchronizers:
            	- <0x000000057d42c240> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            
            
            "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-98" daemon prio=10 tid=0x00007f4a54fb7000 nid=0x170d in Object.wait() [0x00007f4a6294c000]
               java.lang.Thread.State: WAITING (on object monitor)
            	at java.lang.Object.wait(Native Method)
            	at java.lang.Thread.join(Thread.java:1281)
            	- locked <0x0000000547091b00> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection)
            	at java.lang.Thread.join(Thread.java:1355)
            	at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537)
            	- locked <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer)
            	at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            	at java.lang.reflect.Method.invoke(Method.java:606)
            	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:320)
            	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:163)
            	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
            	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
            	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
            	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
            	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
            	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
            	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611)
            	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
            	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
            	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
            	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
            	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
            	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
            	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
            	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
            	- locked <0x000000054ac742f0> (a org.apache.tomcat.util.net.SocketWrapper)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            
               Locked ownable synchronizers:
            	- <0x000000054ac743e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            
            
            
            dilipm79 Dilip Mahadevappa added a comment - We had another occurrence of the same issue.. there was some 30 threads in BLOCKED waiting to lock states.. *Gerrit version: 2.22.0 * "Handling GET /jenkins/gerrit-trigger/serverStatuses from xx.xx.xx.xx : http-bio-8443-exec-112 GerritManagement/serverStatuses.jelly" daemon prio=10 tid=0x00007f4aa4ac0000 nid=0x6664 waiting for monitor entry [0x00007f4a61a33000] java.lang. Thread .State: BLOCKED (on object monitor) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.isConnected(GerritServer.java:554) - waiting to lock <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritManagement.getServerStatuses(GerritManagement.java:219) at sun.reflect.GeneratedMethodAccessor827.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.parser.EscapingExpression.evaluate(EscapingExpression.java:24) at org.apache.commons.jelly.impl.ExpressionScript.run(ExpressionScript.java:66) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) - locked <0x000000057d42c0f0> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Locked ownable synchronizers: - <0x000000057d42c240> (a java.util.concurrent.ThreadPoolExecutor$Worker) "Handling GET /jenkins/gerrit-trigger/server/defaultServer/sleep from xx.xx.xx.xx : http-bio-8443-exec-98" daemon prio=10 tid=0x00007f4a54fb7000 nid=0x170d in Object .wait() [0x00007f4a6294c000] java.lang. Thread .State: WAITING (on object monitor) at java.lang. Object .wait(Native Method) at java.lang. Thread .join( Thread .java:1281) - locked <0x0000000547091b00> (a com.sonymobile.tools.gerrit.gerritevents.GerritConnection) at java.lang. Thread .join( Thread .java:1355) at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:603) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:537) - locked <0x0000000543590590> (a com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer) at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.doSleep(GerritServer.java:1041) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:320) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:163) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:124) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$5.doDispatch(MetaClass.java:233) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:611) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315) - locked <0x000000054ac742f0> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Locked ownable synchronizers: - <0x000000054ac743e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
            dilipm79 Dilip Mahadevappa made changes -
            Environment Gerrit trigger:2.21.1, 2.22.0
            Jenkins: 1.651.3, 1.651.1
            Gerrit trigger:2.21.1, 2.22.0, 2.23.0
            Jenkins: 1.651.3, 1.651.1
            dilipm79 Dilip Mahadevappa made changes -
            Attachment JENKINS-40965.png [ 35929 ]

            Whenever the issue is happening, the Gerrit trigger configure page itself hangs. Like see in this screenshot

            dilipm79 Dilip Mahadevappa added a comment - Whenever the issue is happening, the Gerrit trigger configure page itself hangs. Like see in this screenshot
            hektve Hector David made changes -
            Assignee rsandell [ rsandell ] Hector David [ hektve ]
            hektve Hector David made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            pedersen Björn Pedersen added a comment - Possibly fixed by https://github.com/sonyxperiadev/gerrit-events/pull/62/ ?
            rsandell rsandell added a comment -

            Released in 2.23.3

            rsandell rsandell added a comment - Released in 2.23.3
            rsandell rsandell made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            mawinter69 Markus Winter made changes -
            Link This issue relates to JENKINS-44414 [ JENKINS-44414 ]
            mawinter69 Markus Winter made changes -
            Link This issue relates to JENKINS-44414 [ JENKINS-44414 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2278 (Web Link)" [ 18359 ]

            People

              hektve Hector David
              dilipm79 Dilip Mahadevappa
              Votes:
              7 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: