Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50663

Deadlock on jenkins startup

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: core, sonar-plugin
    • Labels:
    • Environment:
      Jenkins version 2.114
      Plugins versions:
      * sonar: 2.7
      * git 3.8.0
      * branch-api 2.0.19
    • Similar Issues:
    • Released As:
      Jenkins 2.163

      Description

      After plugin update jenkins hangs on startup. Attached thread dump showing deadlock. Here's short version:

      Found one Java-level deadlock:
      =============================
      "CustomOrganizationFolderDescriptor.addSpecificDescriptors":
        waiting to lock monitor 0x00007f5510004f38 (object 0x00000006c6654678, a hudson.ExtensionList$Lock),
        which is held by "GitSCM.onLoaded"
      "GitSCM.onLoaded":
        waiting for ownable synchronizer 0x00000006c6eb7488, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "SonarGlobalConfiguration.migrate"
      "SonarGlobalConfiguration.migrate":
        waiting to lock monitor 0x00007f5510004f38 (object 0x00000006c6654678, a hudson.ExtensionList$Lock),
        which is held by "GitSCM.onLoaded"
      

        Attachments

          Issue Links

            Activity

            jchorko Janusz Chorko created issue -
            jchorko Janusz Chorko made changes -
            Field Original Value New Value
            Component/s branch-api-plugin [ 18621 ]
            jchorko Janusz Chorko made changes -
            Environment Jenkins version 2.114
            Plugins versions:
            * sonar: 2.7
            * git 3.8.0
            Jenkins version 2.114
            Plugins versions:
            * sonar: 2.7
            * git 3.8.0
            * branch-api 2.0.19
            Hide
            edtech Ed Ed added a comment - - edited

            I had the same problem. It's the new Sonar 2.7 plugin that creates the problem. I restored the 2.6.1 and Jenkins started properly.

            Show
            edtech Ed Ed added a comment - - edited I had the same problem. It's the new Sonar 2.7 plugin that creates the problem. I restored the 2.6.1 and Jenkins started properly.
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            markewaite Mark Waite made changes -
            Component/s git-plugin [ 15543 ]
            markewaite Mark Waite made changes -
            Component/s branch-api-plugin [ 18621 ]
            Hide
            martinmajewski Martin Majewski added a comment -

            How to revert back to an older version, when Jenkins is not starting?

             

            Show
            martinmajewski Martin Majewski added a comment - How to revert back to an older version, when Jenkins is not starting?  
            Hide
            giannello Giuseppe Iannello added a comment -

            Same here.

            Martin Majewski go to the jenkins data directory, then `plugins`, you'll find `sonar.bak` and `sonar.jpi`. Remove the latter, and rename the former to `sonar.jpi`. Restart jenkins and you're good to go

            Show
            giannello Giuseppe Iannello added a comment - Same here. Martin Majewski go to the jenkins data directory, then `plugins`, you'll find `sonar.bak` and `sonar.jpi`. Remove the latter, and rename the former to `sonar.jpi`. Restart jenkins and you're good to go
            svanoort Sam Van Oort made changes -
            Priority Major [ 3 ] Blocker [ 1 ]
            svanoort Sam Van Oort made changes -
            Link This issue is related to JENKINS-50671 [ JENKINS-50671 ]
            Hide
            svanoort Sam Van Oort added a comment -

            I have upgraded this to Blocker because it prevents Jenkins from functioning at all. Note that I'm not impacted, but am something of collateral damage since plugins I maintain caught the initial blame until we investigated.

            Show
            svanoort Sam Van Oort added a comment - I have upgraded this to Blocker because it prevents Jenkins from functioning at all. Note that I'm not impacted, but am something of collateral damage since plugins I maintain caught the initial blame until we investigated.
            Hide
            kpop kpop added a comment -

            Same here, workaround by Giuseppe Iannello allowed Jenkins to start again.

            Show
            kpop kpop added a comment - Same here, workaround by Giuseppe Iannello allowed Jenkins to start again.
            Show
            cobexer Ing. Christoph Obexer added a comment - FYI: Apparently a fixed Sonar plugin is already published:  https://plugins.jenkins.io/sonar?why-is-this-caching-so-damn-bad Fix: https://github.com/jenkinsci/sonarqube-plugin/commit/735513927f6b3811cf9ced1a38681913639b6ab0  
            Hide
            henryju Julien HENRY added a comment -

            Fixed in 2.7.1

            Ticket on our side: https://jira.sonarsource.com/browse/SONARJNKNS-294

            Show
            henryju Julien HENRY added a comment - Fixed in 2.7.1 Ticket on our side:  https://jira.sonarsource.com/browse/SONARJNKNS-294
            Hide
            kpop kpop added a comment -

            On my setup, Sonar 2.7.1 fixes the startup issue. Thank you Julien HENRY for the quick fix!

            Show
            kpop kpop added a comment - On my setup, Sonar 2.7.1 fixes the startup issue. Thank you Julien HENRY for the quick fix!
            Hide
            jglick Jesse Glick added a comment -

            While a Sonar plugin change may have triggered this bug, it looks more like a core issue to me. The deadlock is between GuiceFinder and ExtensionList, but possibly OldDataMonitor.report should not be acquiring a lock either since it is involved at a low level in initialization—perhaps it could queue up reports until after startup has completed, or at least extensions have been prepared, or perhaps data could become a static WeakHashMap<Jenkins, …> reference.

            Show
            jglick Jesse Glick added a comment - While a Sonar plugin change may have triggered this bug, it looks more like a core issue to me. The deadlock is between GuiceFinder and ExtensionList , but possibly OldDataMonitor.report should not be acquiring a lock either since it is involved at a low level in initialization—perhaps it could queue up reports until after startup has completed, or at least extensions have been prepared, or perhaps data could become a static WeakHashMap<Jenkins, …> reference.
            jglick Jesse Glick made changes -
            Component/s core [ 15593 ]
            Labels deadlock threads
            Hide
            owenmehegan Owen Mehegan added a comment -

            Julien HENRY please consider submitting a request to remove 2.7.0 from the update center, given that installing it can render Jenkins unable to start at all. See https://jenkins.io/doc/developer/publishing/removing-from-distribution/

            Show
            owenmehegan Owen Mehegan added a comment - Julien HENRY please consider submitting a request to remove 2.7.0 from the update center, given that installing it can render Jenkins unable to start at all. See  https://jenkins.io/doc/developer/publishing/removing-from-distribution/
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/test/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/mf/ReadManifestStepTest.java
            http://jenkins-ci.org/commit/pipeline-utility-steps-plugin/982516b2fc8c2f641fb362da915f6bb9a037d76b
            Log:
            JENKINS-50663 - Restore Remoting manifest checks as requested by @rsandell

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/test/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/mf/ReadManifestStepTest.java http://jenkins-ci.org/commit/pipeline-utility-steps-plugin/982516b2fc8c2f641fb362da915f6bb9a037d76b Log: JENKINS-50663 - Restore Remoting manifest checks as requested by @rsandell
            michaelhpe michael jing made changes -
            Assignee michael jing [ michaelhpe ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            dlouzado Dario Louzado added a comment -

            In my case, updating to version 2.7.1 did not fix the problem.

            2.7.1 did work on a new installation and did not on my staging environment. Jenkins startup freezed again and no logs were written. Cannot use this version on production env.

            Show
            dlouzado Dario Louzado added a comment - In my case, updating to version 2.7.1 did not fix the problem. 2.7.1 did work on a new installation and did not on my staging environment. Jenkins startup freezed again and no logs were written. Cannot use this version on production env.
            dlouzado Dario Louzado made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            greybird Arnaud TAMAILLON made changes -
            Link This issue is related to JENKINS-55361 [ JENKINS-55361 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

             

            JENKINS-55361 should fix it in 2.163 from what I see

            Show
            oleg_nenashev Oleg Nenashev added a comment -   JENKINS-55361 should fix it in 2.163 from what I see
            oleg_nenashev Oleg Nenashev made changes -
            Released As JENKINS
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
            oleg_nenashev Oleg Nenashev made changes -
            Released As JENKINS Jenkins 2.163
            oleg_nenashev Oleg Nenashev made changes -
            Assignee michael jing [ michaelhpe ]
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels deadlock threads deadlock lts-candidate threads
            olivergondza Oliver Gondža made changes -
            Labels deadlock lts-candidate threads deadlock threads

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jchorko Janusz Chorko
              Votes:
              5 Vote for this issue
              Watchers:
              22 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: