Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62039

Jenkins ping thread freezing other threads

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We got a situation on Jenkins ver. 2.204.2 when the ping to the build agent connected via ssh actually started blocking other requests due to delays in the response from the agent, including the web interface gave a timeout, we did not wait for a ping response from the agent and restarted the master

      A problematic thread for which we also saw a large CPU load

       

      "Executor #3 for tkles-jenci0013" #899694 daemon prio=5 os_prio=0 tid=0x000000000179c800 nid=0x4f9 runnable [0x00007f3b7a2b0000]"Executor #3 for tkles-jenci0013" #899694 daemon prio=5 os_prio=0 tid=0x000000000179c800 nid=0x4f9 runnable [0x00007f3b7a2b0000]   java.lang.Thread.State: RUNNABLE at java.lang.Thread.setPriority0(Native Method) at java.lang.Thread.setPriority(Thread.java:1095) at java.lang.Thread.init(Thread.java:417) at java.lang.Thread.init(Thread.java:349) at java.lang.Thread.<init>(Thread.java:678) at java.util.concurrent.Executors$DefaultThreadFactory.newThread(Executors.java:613) at hudson.util.DaemonThreadFactory.newThread(DaemonThreadFactory.java:46) at hudson.util.ExceptionCatchingThreadFactory.newThread(ExceptionCatchingThreadFactory.java:50) at hudson.util.NamingThreadFactory.newThread(NamingThreadFactory.java:52) at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1604) at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1830) at jenkins.metrics.impl.JenkinsMetricProviderImpl$ScheduledRate.onLeft(JenkinsMetricProviderImpl.java:919) at hudson.model.Queue$LeftItem.enter(Queue.java:2788) at hudson.model.Queue.onStartExecuting(Queue.java:1168) at hudson.model.Executor$1.call(Executor.java:359) at hudson.model.Executor$1.call(Executor.java:345) at hudson.model.Queue._withLock(Queue.java:1451) at hudson.model.Queue.withLock(Queue.java:1312) at hudson.model.Executor.run(Executor.java:345)
         Locked ownable synchronizers: - <0x000000025d637d48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      

       

      At this time, the waiting threads looked like this

      "jenkins.util.Timer [#6]" #117 daemon prio=5 os_prio=0 tid=0x00007f3b98002800 nid=0x12f1 waiting on condition [0x00007f3bea1e2000]"jenkins.util.Timer [#6]" #117 daemon prio=5 os_prio=0 tid=0x00007f3b98002800 nid=0x12f1 waiting on condition [0x00007f3bea1e2000]   java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for  <0x000000025d637d48> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue.maintain(Queue.java:1474) at hudson.model.Queue$MaintainTask.doRun(Queue.java:2898) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:70) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
      

      I think this is a global defect connecting to the slave via ssh

       

        Attachments

          Activity

          Hide
          jthompson Jeff Thompson added a comment -

          It doesn't look like there is enough information here to be able to diagnose much. I don't know what to suggest that would be needed.

          Is this a repeated problem? Are you able to reproduce it?

          How did you identify the ping thread as the problem?

          Show
          jthompson Jeff Thompson added a comment - It doesn't look like there is enough information here to be able to diagnose much. I don't know what to suggest that would be needed. Is this a repeated problem? Are you able to reproduce it? How did you identify the ping thread as the problem?
          Hide
          danielbeck Daniel Beck added a comment -

          2.204.2 is an obsolete LTS line.

          Show
          danielbeck Daniel Beck added a comment - 2.204.2 is an obsolete LTS line.
          Hide
          vnenashev Владислав Ненашев added a comment -

          Jeff Thompson it can be seen from jstack

          Show
          vnenashev Владислав Ненашев added a comment - Jeff Thompson it can be seen from jstack

            People

            Assignee:
            jthompson Jeff Thompson
            Reporter:
            vnenashev Владислав Ненашев
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: