Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27476

Plugin casue deadlock on Jenkins LTS 1.596.1

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: durable-task-plugin
    • Labels:
      None
    • Environment:
      RHEL 6.x
      Jenkins LTS 1.596.1
      Durable Task Plugin 1.4
    • Similar Issues:

      Description

      Our Jenkins instance are getting locked up every day.
      It seems like this is due to the durable task plugin.

      Usins JConsole and connecting to the running java process I find dadlocks and gets this stacktrace:

      Name: Computer.threadPoolForRemoting [#179]
State: BLOCKED on hudson.slaves.RetentionStrategy$Demand@1455cecd owned by: jenkins.util.Timer [#1]
Total blocked: 26  Total waited: 522

Stack trace: 
hudson.slaves.RetentionStrategy$Demand.check(RetentionStrategy.java:212)
hudson.slaves.RetentionStrategy$Demand.check(RetentionStrategy.java:172)
hudson.slaves.SlaveComputer.setNode(SlaveComputer.java:661)
hudson.model.AbstractCIBase.updateComputer(AbstractCIBase.java:120)
hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:180)
   - locked java.lang.Object@68ed76f9
jenkins.model.Jenkins.updateComputerList(Jenkins.java:1218)
jenkins.model.Jenkins.setNodes(Jenkins.java:1714)
jenkins.model.Jenkins.removeNode(Jenkins.java:1709)
   - locked hudson.model.Hudson@794217b7
hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:65)
org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy$1.run(OnceRetentionStrategy.java:125)
   - locked hudson.model.Queue@5a25192e
jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
java.util.concurrent.FutureTask.run(FutureTask.java:166)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:722)

      We are running the Jenkins LTS version 1.596.1 and Durable Task Plugin 1.4.
      We also had this problem with Durable Task plugin 1.3.

      Running Durable Task plugin 1.2 on Jenkins LTS 1.580.3 seemd to work OK.

        Attachments

          Issue Links

          depends on

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-27565 Nodes can be removed as idle before the assigned tasks have started

          • Major - Major loss of function.
          • Closed

            Activity

            Ascending order - Click to sort in descending order
            10 older comments
            Hide
            Permalink
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            changelog.html
            http://jenkins-ci.org/commit/jenkins/46dc6850edb1d7ef52592794b15e69db7dfbed1a
            Log:
            Noting merges

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: changelog.html http://jenkins-ci.org/commit/jenkins/46dc6850edb1d7ef52592794b15e69db7dfbed1a Log: Noting merges JENKINS-15355 JENKINS-21618 JENKINS-22941 JENKINS-25938 JENKINS-26391 JENKINS-26900 JENKINS-27476 JENKINS-27563 JENKINS-27564 JENKINS-27565 JENKINS-27566 Fixing link text for JENKINS-6167
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            This is filed in a plugin and so by definition cannot be lts-candidate. Stephen Connolly what is its status?

            Show
            jglick Jesse Glick added a comment - This is filed in a plugin and so by definition cannot be lts-candidate . Stephen Connolly what is its status?
            Hide
            Permalink
            danielbeck Daniel Beck added a comment -

            Per Arnold Blaasmo Does this issue still occur in Jenkins 1.607 or higher, or can it be considered resolved?

            Show
            danielbeck Daniel Beck added a comment - Per Arnold Blaasmo Does this issue still occur in Jenkins 1.607 or higher, or can it be considered resolved?
            Hide
            Permalink
            jglick Jesse Glick added a comment -

            Closing as covered by the core fix unless I hear information to the contrary.

            Show
            jglick Jesse Glick added a comment - Closing as covered by the core fix unless I hear information to the contrary.
            Hide
            Permalink
            pablaasmo Per Arnold Blaasmo added a comment -

            I have not seen this issue again. So I think it is ok to close

            Show
            pablaasmo Per Arnold Blaasmo added a comment - I have not seen this issue again. So I think it is ok to close

              People

              Assignee:
              stephenconnolly Stephen Connolly
              Reporter:
              pablaasmo Per Arnold Blaasmo
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: