Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

      Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
      
      The threads are blocked in:
      hudson.model.Run.save(Run.java:1856) (#1568)
      hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
      
      Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
      hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
      jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
      hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
      hudson.model.RunMap.removeValue(RunMap.java:120)
      hudson.model.RunMap.remove(RunMap.java:86)
      jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
      hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
      hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
      hudson.model.Run.removeRunFromParent(Run.java:1478)
      hudson.model.Run.delete(Run.java:1473)
      hudson.tasks.LogRotator.perform(LogRotator.java:124)
      hudson.model.Job.logRotate(Job.java:441)
      hudson.model.Run.execute(Run.java:1751)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:231)
      
      Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
      hudson.model.Run.save(Run.java:1856)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
      hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
      hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
      hudson.model.Run.delete(Run.java:1447)
      hudson.tasks.LogRotator.perform(LogRotator.java:124)
      hudson.model.Job.logRotate(Job.java:441)
      hudson.model.Run.execute(Run.java:1751)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:231)
      

          [JENKINS-22560] Deadlocked executors on slaves since 1.556

          Dirk Kuypers created issue -
          Dirk Kuypers made changes -
          Summary Original: Deadlocked executors on lslaves since 1.556 New: Deadlocked executors on slaves since 1.556
          kutzi made changes -
          Link New: This issue depends on JENKINS-22131 [ JENKINS-22131 ]
          kutzi made changes -
          Assignee New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Description Original: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)

          <em>
          Executor&nbsp;#13&nbsp;for&nbsp;1sp-slave1&nbsp;:&nbsp;executing&nbsp;CONTEST_Technologies_Tests&nbsp;#1569<br>
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)<br>
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)<br>
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)<br>
          hudson.model.RunMap.removeValue(RunMap.java:120)<br>
          hudson.model.RunMap.remove(RunMap.java:86)<br>
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)<br>
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)<br>
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)<br>
          hudson.model.Run.removeRunFromParent(Run.java:1478)<br>
          hudson.model.Run.delete(Run.java:1473)<br>
          hudson.tasks.LogRotator.perform(LogRotator.java:124)<br>
          hudson.model.Job.logRotate(Job.java:441)<br>
          hudson.model.Run.execute(Run.java:1751)<br>
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)<br>
          hudson.model.ResourceController.execute(ResourceController.java:88)<br>
          hudson.model.Executor.run(Executor.java:231)<br>
          </em>

          <em>
          Executor&nbsp;#0&nbsp;for&nbsp;1sp-slave12&nbsp;:&nbsp;executing&nbsp;CONTEST_Technologies_Tests&nbsp;#1568<br>
          hudson.model.Run.save(Run.java:1856)<br>
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)<br>
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)<br>
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)<br>
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)<br>
          hudson.model.Run.delete(Run.java:1447)<br>
          hudson.tasks.LogRotator.perform(LogRotator.java:124)<br>
          hudson.model.Job.logRotate(Job.java:441)<br>
          hudson.model.Run.execute(Run.java:1751)<br>
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)<br>
          hudson.model.ResourceController.execute(ResourceController.java:88)<br>
          hudson.model.Executor.run(Executor.java:231)<br>
          </em>
          New: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          {code:none}
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
          {code}

          {code:none}
          Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
          hudson.model.RunMap.removeValue(RunMap.java:120)
          hudson.model.RunMap.remove(RunMap.java:86)
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
          hudson.model.Run.removeRunFromParent(Run.java:1478)
          hudson.model.Run.delete(Run.java:1473)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)

          Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
          hudson.model.Run.save(Run.java:1856)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
          hudson.model.Run.delete(Run.java:1447)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)
          {code}
          Labels New: deadlock regression
          Jesse Glick made changes -
          Description Original: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          {code:none}
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)
          {code}

          {code:none}
          Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
          hudson.model.RunMap.removeValue(RunMap.java:120)
          hudson.model.RunMap.remove(RunMap.java:86)
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
          hudson.model.Run.removeRunFromParent(Run.java:1478)
          hudson.model.Run.delete(Run.java:1473)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)

          Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
          hudson.model.Run.save(Run.java:1856)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
          hudson.model.Run.delete(Run.java:1447)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)
          {code}
          New: Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

          {code:none}
          Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

          The threads are blocked in:
          hudson.model.Run.save(Run.java:1856) (#1568)
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)

          Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
          hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
          jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
          hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
          hudson.model.RunMap.removeValue(RunMap.java:120)
          hudson.model.RunMap.remove(RunMap.java:86)
          jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
          hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
          hudson.model.Run.removeRunFromParent(Run.java:1478)
          hudson.model.Run.delete(Run.java:1473)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)

          Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
          hudson.model.Run.save(Run.java:1856)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
          hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
          hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
          hudson.model.Run.delete(Run.java:1447)
          hudson.tasks.LogRotator.perform(LogRotator.java:124)
          hudson.model.Job.logRotate(Job.java:441)
          hudson.model.Run.execute(Run.java:1751)
          hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          hudson.model.ResourceController.execute(ResourceController.java:88)
          hudson.model.Executor.run(Executor.java:231)
          {code}
          Jesse Glick made changes -
          Component/s New: changelog-history [ 15600 ]
          Labels Original: deadlock regression New: deadlock regression threads
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 154662 ] New: JNJira + In-Review [ 194983 ]

            jglick Jesse Glick
            bruce Dirk Kuypers
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: