Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22560

Deadlocked executors on slaves since 1.556

    XMLWordPrintable

Details

    Description

      Could it be that the change in JENKINS-22131 is causing deadlocks? If I downgrade to 1.555, there is no problem, all versions from 1.556 show the same behaviour: More and more executors are deadlocked. To me it looks if two concurrent jobs do something which locks each other.

      Warning, the following threads are deadlocked : Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568, Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569

The threads are blocked in:
hudson.model.Run.save(Run.java:1856) (#1568)
hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178) (#1569)

Executor #13 for 1sp-slave1 : executing CONTEST_Technologies_Tests #1569
hudson.model.AbstractBuild.getRunMixIn(AbstractBuild.java:178)
jenkins.model.lazy.LazyBuildMixIn$RunMixIn.dropLinks(LazyBuildMixIn.java:336)
hudson.model.AbstractBuild.dropLinks(AbstractBuild.java:193)
hudson.model.RunMap.removeValue(RunMap.java:120)
hudson.model.RunMap.remove(RunMap.java:86)
jenkins.model.lazy.LazyBuildMixIn.removeRun(LazyBuildMixIn.java:216)
hudson.model.AbstractProject.removeRun(AbstractProject.java:1002)
hudson.model.AbstractProject.removeRun(AbstractProject.java:145)
hudson.model.Run.removeRunFromParent(Run.java:1478)
hudson.model.Run.delete(Run.java:1473)
hudson.tasks.LogRotator.perform(LogRotator.java:124)
hudson.model.Job.logRotate(Job.java:441)
hudson.model.Run.execute(Run.java:1751)
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
hudson.model.ResourceController.execute(ResourceController.java:88)
hudson.model.Executor.run(Executor.java:231)

Executor #0 for 1sp-slave12 : executing CONTEST_Technologies_Tests #1568
hudson.model.Run.save(Run.java:1856)
hudson.plugins.changelog_history.ChangeLogHistoryRunListener.copyChangeLogs(ChangeLogHistoryRunListener.java:82)
hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:50)
hudson.plugins.changelog_history.ChangeLogHistoryRunListener.onDeleted(ChangeLogHistoryRunListener.java:40)
hudson.model.listeners.RunListener.fireDeleted(RunListener.java:244)
hudson.model.Run.delete(Run.java:1447)
hudson.tasks.LogRotator.perform(LogRotator.java:124)
hudson.model.Job.logRotate(Job.java:441)
hudson.model.Run.execute(Run.java:1751)
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
hudson.model.ResourceController.execute(ResourceController.java:88)
hudson.model.Executor.run(Executor.java:231)

      Attachments

        Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. JENKINS-22131 Extract generic features from AbstractProject/AbstractBuild

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 7 older comments
            jglick Jesse Glick added a comment -

            Or just go back to 1.555 (or whatever) temporarily; I hope the fix will be in 1.560.

            jglick Jesse Glick added a comment - Or just go back to 1.555 (or whatever) temporarily; I hope the fix will be in 1.560.
            bruce Dirk Kuypers added a comment -

            I am back to 1.555. But I was not sure if it is fixable in short time. Then I will change nothing and will be happy to run the ultimative test for you. I think this issue is not easily reproducible.

            bruce Dirk Kuypers added a comment - I am back to 1.555. But I was not sure if it is fixable in short time. Then I will change nothing and will be happy to run the ultimative test for you. I think this issue is not easily reproducible.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/hudson/model/AbstractBuild.java
            core/src/main/java/hudson/model/Run.java
            core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java
            http://jenkins-ci.org/commit/jenkins/a5d848bcec45100ca4f12303a8c619664f2eb945
            Log:
            [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final.
            (Forgot that Run’s are unmarshalled in place after a special constructor is called, so there is no need for readResolve or other tricks.)
            Also calling RunListener.onDeleted outside of the Run lock to avoid problems with things like ChangeLogHistoryRunListener.

            Compare: https://github.com/jenkinsci/jenkins/compare/b35f834b6651...a5d848bcec45

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/hudson/model/Run.java core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java http://jenkins-ci.org/commit/jenkins/a5d848bcec45100ca4f12303a8c619664f2eb945 Log: [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final. (Forgot that Run’s are unmarshalled in place after a special constructor is called, so there is no need for readResolve or other tricks.) Also calling RunListener.onDeleted outside of the Run lock to avoid problems with things like ChangeLogHistoryRunListener. Compare: https://github.com/jenkinsci/jenkins/compare/b35f834b6651...a5d848bcec45
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3292
            [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final. (Revision a5d848bcec45100ca4f12303a8c619664f2eb945)

            Result = SUCCESS
            Jesse Glick : a5d848bcec45100ca4f12303a8c619664f2eb945
            Files :

            • core/src/main/java/hudson/model/AbstractBuild.java
            • core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java
            • changelog.html
            • core/src/main/java/hudson/model/Run.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3292 [FIXED JENKINS-22560] Avoid deadlock by making AbstractBuild.runMixIn final. (Revision a5d848bcec45100ca4f12303a8c619664f2eb945) Result = SUCCESS Jesse Glick : a5d848bcec45100ca4f12303a8c619664f2eb945 Files : core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/jenkins/model/lazy/LazyBuildMixIn.java changelog.html core/src/main/java/hudson/model/Run.java
            bruce Dirk Kuypers added a comment -

            Just a short feedback: Jenkins ver. 1.560-SNAPSHOT (private-04/10/2014 15:41 GMT-jenkins) is running since about 8 hours now without a deadlock, so I think the bug is gone. Thanks a lot for the fast fix!

            bruce Dirk Kuypers added a comment - Just a short feedback: Jenkins ver. 1.560-SNAPSHOT (private-04/10/2014 15:41 GMT-jenkins) is running since about 8 hours now without a deadlock, so I think the bug is gone. Thanks a lot for the fast fix!

            People

              jglick Jesse Glick
              bruce Dirk Kuypers
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: