Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66388

metrics plugin causes infinite freeze jenkins threads

      We faced problem described in https://github.com/dropwizard/metrics/pull/1654

      Under heavy load web intarface hangs and all threads that handles http requests are stuck in waiting state until restart with the same stacktrace:

      Handling GET /jenkins/job/somejob/1251 from 10.18.172.22 : Jetty (winstone)-815006" #815006 prio=5 os_prio=0 tid=0x00007f44d0128800 nid=0x4fdc waiting on condition [0x00007f43e989a000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000242204970> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
      	at com.codahale.metrics.ExponentiallyDecayingReservoir.lockForRescale(ExponentiallyDecayingReservoir.java:197)
      	at com.codahale.metrics.ExponentiallyDecayingReservoir.rescale(ExponentiallyDecayingReservoir.java:164)
      	at com.codahale.metrics.ExponentiallyDecayingReservoir.rescaleIfNeeded(ExponentiallyDecayingReservoir.java:122)
      	at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:94)
      	at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:84)
      	at com.codahale.metrics.Histogram.update(Histogram.java:41)
      	at com.codahale.metrics.Timer.update(Timer.java:185)
      	at com.codahale.metrics.Timer.update(Timer.java:89)
      	at com.codahale.metrics.Timer$Context.stop(Timer.java:36)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:127)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:159)
      .......
      

      I hope update of dropwizard metrics library up to 4.1.13 helps

      https://github.com/dropwizard/metrics/releases/tag/v4.1.13

          [JENKINS-66388] metrics plugin causes infinite freeze jenkins threads

          I hope update of dropwizard metrics library up to 4.1.13 helps

          Version 4.2.10-386.v1c612ef0355c of the Metrics plugin was released in July 2022. Comparing v4.2.10 to v4.1.13 shows that 4.2.10 includes everything that was in 4.1.13. brainsam, can you still reproduce the issue with this plugin version or a later one?

          Kalle Niemitalo added a comment - I hope update of dropwizard metrics library up to 4.1.13 helps Version 4.2.10-386.v1c612ef0355c of the Metrics plugin was released in July 2022. Comparing v4.2.10 to v4.1.13 shows that 4.2.10 includes everything that was in 4.1.13. brainsam , can you still reproduce the issue with this plugin version or a later one?

            Unassigned Unassigned
            brainsam Alexander Moiseenko
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: