Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66388

metrics plugin causes infinite freeze jenkins threads

XMLWordPrintable

      We faced problem described in https://github.com/dropwizard/metrics/pull/1654

      Under heavy load web intarface hangs and all threads that handles http requests are stuck in waiting state until restart with the same stacktrace:

      Handling GET /jenkins/job/somejob/1251 from 10.18.172.22 : Jetty (winstone)-815006" #815006 prio=5 os_prio=0 tid=0x00007f44d0128800 nid=0x4fdc waiting on condition [0x00007f43e989a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000242204970> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
	at com.codahale.metrics.ExponentiallyDecayingReservoir.lockForRescale(ExponentiallyDecayingReservoir.java:197)
	at com.codahale.metrics.ExponentiallyDecayingReservoir.rescale(ExponentiallyDecayingReservoir.java:164)
	at com.codahale.metrics.ExponentiallyDecayingReservoir.rescaleIfNeeded(ExponentiallyDecayingReservoir.java:122)
	at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:94)
	at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:84)
	at com.codahale.metrics.Histogram.update(Histogram.java:41)
	at com.codahale.metrics.Timer.update(Timer.java:185)
	at com.codahale.metrics.Timer.update(Timer.java:89)
	at com.codahale.metrics.Timer$Context.stop(Timer.java:36)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:127)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159)
.......

      I hope update of dropwizard metrics library up to 4.1.13 helps

      https://github.com/dropwizard/metrics/releases/tag/v4.1.13

            Unassigned Unassigned
            brainsam Alexander Moiseenko
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: