Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54187

EC2 Plugin deadlock leaving Jenkins unresponsive

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • ec2-plugin
    • None

      We recently upgraded our Jenkins instance to the latest LTS (2.138.2) and started noticing some irregularities with the EC2 plugin, namely:

      • A lot more build nodes were created than before, for the same amount of incoming items in the build queue
      • This caused an increased amount of AWS API calls, namely StopInstances event. Possibly related to the previous point.

      This led us to try out the latest snapshot, 1.41-SNAPSHOT build on revision d4bdd6b83a7102330fd97ffbbd067edc34e47f97. A few hours later, our Jenkins instance had a deadlock problem that is described in the log I'm attaching.

      Notice how the Gerrit plugin stops processing events after the EC2 plugin deadlock, this ultimately left our Jenkins unresponsive.

      One important bit of information might be that we're using the Stop/Disconnect on Idle Timeout plugin option.

      We're happy to provide more information if needed.

        1. all-jenkins.log
          26 kB
        2. jenkins.log
          10 kB
        3. jenkins.txt
          6 kB

          [JENKINS-54187] EC2 Plugin deadlock leaving Jenkins unresponsive

          Renzo Crisóstomo created issue -

          Oliver Pereira added a comment - - edited

          We are currently facing this same issue but I believe this us caused by Jenkins itself (introduced in build 2.138.2) rather than the version of ec2 plugin.

          We am using an older version of the ec2 plugin based of PR252.

          There is a similar issue reported elsewhere.

          https://serverfault.com/questions/927415/jenkins-web-server-not-responding-to-http-requests-in-ubuntu-dell-poweredge-r640

           

           

          Oliver Pereira added a comment - - edited We are currently facing this same issue but I believe this us caused by Jenkins itself (introduced in build 2.138.2) rather than the version of ec2 plugin. We am using an older version of the ec2 plugin based of PR252. There is a similar issue reported elsewhere. https://serverfault.com/questions/927415/jenkins-web-server-not-responding-to-http-requests-in-ubuntu-dell-poweredge-r640    

          We've downgraded the plugin to a version build on top of revision 67aba04d0c60e8db1134d068ea2c10bc3e961c90 (1.40.0) today, which was stable for us prior our upgrade to the latest LTS (2.138.2).

          So far we haven't experience any deadlocks, but it might be related to the fact that this version is more conservative when spinning up new build nodes therefore less need to handle concurrency. We are back to the same amount of build nodes we used to have.

          I'll report back if this changes.

          Renzo Crisóstomo added a comment - We've downgraded the plugin to a version build on top of revision 67aba04d0c60e8db1134d068ea2c10bc3e961c90 (1.40.0) today, which was stable for us prior our upgrade to the latest LTS (2.138.2). So far we haven't experience any deadlocks, but it might be related to the fact that this version is more conservative when spinning up new build nodes therefore less need to handle concurrency. We are back to the same amount of build nodes we used to have. I'll report back if this changes.

          Are you using the spot instance ? 

          the 1.40 and the 1.41 use the same algorithm for calculate the nodes that is needed.

          FABRIZIO MANFREDI added a comment - Are you using the spot instance ?  the 1.40 and the 1.41 use the same algorithm for calculate the nodes that is needed.
          Danny made changes -
          Attachment New: jenkins.log [ 44850 ]

          Danny added a comment -

          I'm facing the same behavior, running on Jenkins ver. 2.138.2

          In fact, our ec2 plugin is old and rusty, 1.37. 

          Jenkins goes AWOL

          jenkins.log

          Danny added a comment - I'm facing the same behavior, running on Jenkins ver. 2.138.2 In fact, our ec2 plugin is old and rusty, 1.37.  Jenkins goes AWOL jenkins.log

          Oct 24, 2018 4:15:51 PM jenkins.metrics.api.Metrics$HealthChecker executeOct 24, 2018 4:15:51 PM jenkins.metrics.api.Metrics$HealthChecker executeWARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [jenkins.util.Timer [#8] locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@594db2a0 (owned by jenkins.util.Timer [#9]):  at sun.misc.Unsafe.park(Native Method)  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)  at hudson.model.Queue._withLock(Queue.java:1437)  at hudson.model.Queue.withLock(Queue.java:1300)  at jenkins.model.Nodes.updateNode(Nodes.java:193)  at jenkins.model.Jenkins.updateNode(Jenkins.java:2080)  at hudson.model.Node.save(Node.java:140)  at hudson.util.PersistedList.onModified(PersistedList.java:173)  at hudson.util.PersistedList.replaceBy(PersistedList.java:85)  at hudson.model.Slave.<init>(Slave.java:198)  at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:138)  at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49)  at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42)  at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:963)  at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:660)  at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:632)  at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:463)  at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:587)  at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:602)  at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)  at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)  at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)  at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)  at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748), jenkins.util.Timer [#9] locked on hudson.plugins.ec2.AmazonEC2Cloud@755cde3d (owned by jenkins.util.Timer [#8]):  at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748)  at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47)  at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25)  at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:127)  at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:112)  at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:90)  at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:48)  at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)  at hudson.model.Queue._withLock(Queue.java:1380)  at hudson.model.Queue.withLock(Queue.java:1257)  at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)  at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)  at java.lang.Thread.run(Thread.java:748)]] 

          Oliver Pereira added a comment - Oct 24, 2018 4:15:51 PM jenkins.metrics.api.Metrics$HealthChecker executeOct 24, 2018 4:15:51 PM jenkins.metrics.api.Metrics$HealthChecker executeWARNING: Some health checks are reporting as unhealthy: [thread-deadlock : [jenkins.util.Timer [#8] locked on java.util.concurrent.locks.ReentrantLock$NonfairSync@594db2a0 (owned by jenkins.util.Timer [#9]): at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at hudson.model.Queue._withLock(Queue.java:1437) at hudson.model.Queue.withLock(Queue.java:1300) at jenkins.model.Nodes.updateNode(Nodes.java:193) at jenkins.model.Jenkins.updateNode(Jenkins.java:2080) at hudson.model.Node.save(Node.java:140) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.Slave.<init>(Slave.java:198) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:138) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:49) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:42) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:963) at hudson.plugins.ec2.SlaveTemplate.toSlaves(SlaveTemplate.java:660) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:632) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:463) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:587) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:602) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748), jenkins.util.Timer [#9] locked on hudson.plugins.ec2.AmazonEC2Cloud@755cde3d (owned by jenkins.util.Timer [#8]): at hudson.plugins.ec2.EC2Cloud.connect(EC2Cloud.java:748) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:47) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:25) at hudson.plugins.ec2.EC2Computer.getState(EC2Computer.java:127) at hudson.plugins.ec2.EC2RetentionStrategy.internalCheck(EC2RetentionStrategy.java:112) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:90) at hudson.plugins.ec2.EC2RetentionStrategy.check(EC2RetentionStrategy.java:48) at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72) at hudson.model.Queue._withLock(Queue.java:1380) at hudson.model.Queue.withLock(Queue.java:1257) at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)]]

          The above error was reported in our test environment and I was using the the latest snapshot of the ec2 plugin from the master branch and the Jenkins 2.138.2.

          I downgraded Jenkins to 2.138.1 and it's working fine.

          So it seems there are some changes introduced in Jenkins 2.138.2, which are causing the thread locks.

           

          Oliver Pereira added a comment - The above error was reported in our test environment and I was using the the latest snapshot of the ec2 plugin from the master branch and the Jenkins 2.138.2. I downgraded Jenkins to 2.138.1 and it's working fine. So it seems there are some changes introduced in Jenkins 2.138.2, which are causing the thread locks.  

          Danny added a comment -

          So it's been couple of days now that this issue didn't occur on my setup and frankly, I didn't upgrade/downgraded a thing...

          Danny added a comment - So it's been couple of days now that this issue didn't occur on my setup and frankly, I didn't upgrade/downgraded a thing...

          I can confirm that we haven't experienced deadlocks since we downgrade to a version build on top of revision 67aba04d0c60e8db1134d068ea2c10bc3e961c90 (1.40.0).

          thoulen We're not using spot instances.

          Renzo Crisóstomo added a comment - I can confirm that we haven't experienced deadlocks since we downgrade to a version build on top of revision 67aba04d0c60e8db1134d068ea2c10bc3e961c90 (1.40.0). thoulen We're not using spot instances.

            thoulen FABRIZIO MANFREDI
            ruenzuo Renzo Crisóstomo
            Votes:
            6 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: