Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

      Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
      
      The threads are blocked in:
      hudson.model.Run.save(Run.java:1856) (#1568)
      hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
      
      Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
      hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
      jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
      hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
      hudson.model.RunMap.removeValue(RunMap.java:120)
      hudson.model.RunMap.remove(RunMap.java:86)
      jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
      hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
      hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
      hudson.model.Run.removeRunFromParent(Run.java:1478)
      hudson.model.Run.delete(Run.java:1473)
      hudson.tasks.LogRotator.perform(LogRotator.java:124)
      hudson.model.Job.logRotate(Job.java:441)
      hudson.model.Run.execute(Run.java:1751)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:231)
      
      Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
      hudson.model.Run.save(Run.java:1856)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
      hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
      hudson.model.Run.delete(Run.java:1447)
      hudson.tasks.LogRotator.perform(LogRotator.java:124)
      hudson.model.Job.logRotate(Job.java:441)
      hudson.model.Run.execute(Run.java:1751)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:231)
      

          [JENKINS-22560] Deadlocked executors on slaves since 1.556

          Dirk Kuypers created issue -
          Dirk Kuypers made changes -
          Summary Original: Deadlocked executors on lslaves since 1.556 New: Deadlocked executors on slaves since 1.556
          kutzi made changes -
          Link New: This issue depends on JENKINS-22131 [ JENKINS-22131 ]

          kutzi added a comment -

          Quite possible that this was caused by JENKINS-22131

          kutzi added a comment - Quite possible that this was caused by JENKINS-22131
          kutzi made changes -
          Assignee New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Description Original: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)

          <em>
          Executor&nbsp;#13&nbsp;for&nbsp;1sp-slave1&nbsp;:&nbsp;executing&nbsp;CONTEST_Technologies_Tests&nbsp;#1569<br>
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)<br>
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)<br>
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)<br>
          hudson.model.RunMap.removeValue(RunMap.java:120)<br>
          hudson.model.RunMap.remove(RunMap.java:86)<br>
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)<br>
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)<br>
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)<br>
          hudson.model.Run.removeRunFromParent(Run.java:1478)<br>
          hudson.model.Run.delete(Run.java:1473)<br>
          hudson.tasks.LogRotator.perform(LogRotator.java:124)<br>
          hudson.model.Job.logRotate(Job.java:441)<br>
          hudson.model.Run.execute(Run.java:1751)<br>
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)<br>
          hudson.model.ResourceController.execute(ResourceController.java:88)<br>
          hudson.model.Executor.run(Executor.java:231)<br>
          </em>

          <em>
          Executor&nbsp;#0&nbsp;for&nbsp;1sp-slave12&nbsp;:&nbsp;executing&nbsp;CONTEST_Technologies_Tests&nbsp;#1568<br>
          hudson.model.Run.save(Run.java:1856)<br>
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)<br>
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)<br>
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)<br>
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)<br>
          hudson.model.Run.delete(Run.java:1447)<br>
          hudson.tasks.LogRotator.perform(LogRotator.java:124)<br>
          hudson.model.Job.logRotate(Job.java:441)<br>
          hudson.model.Run.execute(Run.java:1751)<br>
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)<br>
          hudson.model.ResourceController.execute(ResourceController.java:88)<br>
          hudson.model.Executor.run(Executor.java:231)<br>
          </em>
          New: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          {code:none}
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
          {code}

          {code:none}
          Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
          hudson.model.RunMap.removeValue(RunMap.java:120)
          hudson.model.RunMap.remove(RunMap.java:86)
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
          hudson.model.Run.removeRunFromParent(Run.java:1478)
          hudson.model.Run.delete(Run.java:1473)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)

          Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
          hudson.model.Run.save(Run.java:1856)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
          hudson.model.Run.delete(Run.java:1447)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)
          {code}
          Labels New: deadlock regression

          Jesse Glick added a comment -

          In the future please attach an actual thread dump (preferably as produced by jstack) as that would indicate the proximate cause of the deadlock, specifically which locks are being held.

          Jesse Glick added a comment - In the future please attach an actual thread dump (preferably as produced by jstack ) as that would indicate the proximate cause of the deadlock, specifically which locks are being held.
          Jesse Glick made changes -
          Description Original: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          {code:none}
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
          {code}

          {code:none}
          Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
          hudson.model.RunMap.removeValue(RunMap.java:120)
          hudson.model.RunMap.remove(RunMap.java:86)
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
          hudson.model.Run.removeRunFromParent(Run.java:1478)
          hudson.model.Run.delete(Run.java:1473)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)

          Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
          hudson.model.Run.save(Run.java:1856)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
          hudson.model.Run.delete(Run.java:1447)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)
          {code}
          New: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          {code:none}
          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)

          Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
          hudson.model.RunMap.removeValue(RunMap.java:120)
          hudson.model.RunMap.remove(RunMap.java:86)
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
          hudson.model.Run.removeRunFromParent(Run.java:1478)
          hudson.model.Run.delete(Run.java:1473)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)

          Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
          hudson.model.Run.save(Run.java:1856)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
          hudson.model.Run.delete(Run.java:1447)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)
          {code}

          Dirk Kuypers added a comment -

          We are a .NET shop. I need more details how I could debug Java applications. I even asked on the user mailing list how to get a thread dump but got no answer. I was really glad when I could extract it from the bubble help of monitoring plugin's html page.

          So if I can provide more info just tell me what I should do.

          Dirk Kuypers added a comment - We are a .NET shop. I need more details how I could debug Java applications. I even asked on the user mailing list how to get a thread dump but got no answer. I was really glad when I could extract it from the bubble help of monitoring plugin's html page. So if I can provide more info just tell me what I should do.

          Jesse Glick added a comment -

          Looks like Run.execute always acquired a lock on an older build via LogRotator and Run.delete. And using the Change Log History plugin means that when deleting an older build Jenkins also acquires a lock on a newer build, which is a bug in that plugin. And the change in 1.556 means that Run.delete could also (via dropLinks) acquire locks on both older and newer builds, which in combination with the bug in CLH means that deadlocks are possible.

          Jesse Glick added a comment - Looks like Run.execute always acquired a lock on an older build via LogRotator and Run.delete . And using the Change Log History plugin means that when deleting an older build Jenkins also acquires a lock on a newer build, which is a bug in that plugin. And the change in 1.556 means that Run.delete could also (via dropLinks ) acquire locks on both older and newer builds, which in combination with the bug in CLH means that deadlocks are possible.

            jglick Jesse Glick
            bruce Dirk Kuypers
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: