Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Duplicate
-
None
-
RHEL 6.x
Jenkins LTS 1.596.1
Durable Task Plugin 1.4
Description
Our Jenkins instance are getting locked up every day.
It seems like this is due to the durable task plugin.
Usins JConsole and connecting to the running java process I find dadlocks and gets this stacktrace:
Name: Computer.threadPoolForRemoting [#179] State: BLOCKED on hudson.slaves.RetentionStrategy$Demand@1455cecd owned by: jenkins.util.Timer [#1] Total blocked: 26 Total waited: 522 Stack trace: hudson.slaves.RetentionStrategy$Demand.check(RetentionStrategy.java:212) hudson.slaves.RetentionStrategy$Demand.check(RetentionStrategy.java:172) hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:661) hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120) hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180) - locked java.lang.Object@68ed76f9 jenkins.model.Jenkins.updateComputerList(Jenkins.java:1218) jenkins.model.Jenkins.setNodes(Jenkins.java:1714) jenkins.model.Jenkins.removeNode(Jenkins.java:1709) - locked hudson.model.Hudson@794217b7 hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:65) org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy$1.run(OnceRetentionStrategy.java:125) - locked hudson.model.Queue@5a25192e jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:722)
We are running the Jenkins LTS version 1.596.1 and Durable Task Plugin 1.4.
We also had this problem with Durable Task plugin 1.3.
Running Durable Task plugin 1.2 on Jenkins LTS 1.580.3 seemd to work OK.
Attachments
Issue Links
- depends on
-
JENKINS-27565 Nodes can be removed as idle before the assigned tasks have started
-
- Closed
-
Code changed in jenkins
User: Stephen Connolly
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/46dc6850edb1d7ef52592794b15e69db7dfbed1a
Log:
Noting merges
JENKINS-21618JENKINS-22941JENKINS-25938JENKINS-26391JENKINS-26900JENKINS-27476JENKINS-27563JENKINS-27564JENKINS-27565JENKINS-27566