Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50663

Deadlock on jenkins startup

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Blocker
    • Resolution: Fixed
    • core, sonar-plugin
    • Jenkins version 2.114
      Plugins versions:
      * sonar: 2.7
      * git 3.8.0
      * branch-api 2.0.19
    • Jenkins 2.163

    Description

      After plugin update jenkins hangs on startup. Attached thread dump showing deadlock. Here's short version:

      Found one Java-level deadlock:
=============================
"CustomOrganizationFolderDescriptor.addSpecificDescriptors":
  waiting to lock monitor 0x00007f5510004f38 (object 0x00000006c6654678, a hudson.ExtensionList$Lock),
  which is held by "GitSCM.onLoaded"
"GitSCM.onLoaded":
  waiting for ownable synchronizer 0x00000006c6eb7488, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "SonarGlobalConfiguration.migrate"
"SonarGlobalConfiguration.migrate":
  waiting to lock monitor 0x00007f5510004f38 (object 0x00000006c6654678, a hudson.ExtensionList$Lock),
  which is held by "GitSCM.onLoaded"

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-50671 Jenkins won't start after upgrading workflow-cps plugin to 2.47

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-55361 DescriptorExtensionList not locking correctly, leading to deadlocks

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            edtech Ed Ed added a comment - - edited

            I had the same problem. It's the new Sonar 2.7 plugin that creates the problem. I restored the 2.6.1 and Jenkins started properly.

            edtech Ed Ed added a comment - - edited I had the same problem. It's the new Sonar 2.7 plugin that creates the problem. I restored the 2.6.1 and Jenkins started properly.
            martinmajewski Martin Majewski added a comment -

            How to revert back to an older version, when Jenkins is not starting?

             

            martinmajewski Martin Majewski added a comment - How to revert back to an older version, when Jenkins is not starting?  
            giannello Giuseppe Iannello added a comment -

            Same here.

            martinmajewski go to the jenkins data directory, then `plugins`, you'll find `sonar.bak` and `sonar.jpi`. Remove the latter, and rename the former to `sonar.jpi`. Restart jenkins and you're good to go

            giannello Giuseppe Iannello added a comment - Same here. martinmajewski go to the jenkins data directory, then `plugins`, you'll find `sonar.bak` and `sonar.jpi`. Remove the latter, and rename the former to `sonar.jpi`. Restart jenkins and you're good to go
            svanoort Sam Van Oort added a comment -

            I have upgraded this to Blocker because it prevents Jenkins from functioning at all. Note that I'm not impacted, but am something of collateral damage since plugins I maintain caught the initial blame until we investigated.

            svanoort Sam Van Oort added a comment - I have upgraded this to Blocker because it prevents Jenkins from functioning at all. Note that I'm not impacted, but am something of collateral damage since plugins I maintain caught the initial blame until we investigated.
            kpop kpop added a comment -

            Same here, workaround by giannello allowed Jenkins to start again.

            kpop kpop added a comment - Same here, workaround by giannello allowed Jenkins to start again.
            cobexer Ing. Christoph Obexer added a comment -

            FYI: Apparently a fixed Sonar plugin is already published: https://plugins.jenkins.io/sonar?why-is-this-caching-so-damn-bad

            Fix: https://github.com/jenkinsci/sonarqube-plugin/commit/735513927f6b3811cf9ced1a38681913639b6ab0

             

            cobexer Ing. Christoph Obexer added a comment - FYI: Apparently a fixed Sonar plugin is already published:  https://plugins.jenkins.io/sonar?why-is-this-caching-so-damn-bad Fix: https://github.com/jenkinsci/sonarqube-plugin/commit/735513927f6b3811cf9ced1a38681913639b6ab0  
            henryju Julien HENRY added a comment -

            Fixed in 2.7.1

            Ticket on our side: https://jira.sonarsource.com/browse/SONARJNKNS-294

            henryju Julien HENRY added a comment - Fixed in 2.7.1 Ticket on our side:  https://jira.sonarsource.com/browse/SONARJNKNS-294
            kpop kpop added a comment -

            On my setup, Sonar 2.7.1 fixes the startup issue. Thank you henryju for the quick fix!

            kpop kpop added a comment - On my setup, Sonar 2.7.1 fixes the startup issue. Thank you henryju for the quick fix!
            jglick Jesse Glick added a comment -

            While a Sonar plugin change may have triggered this bug, it looks more like a core issue to me. The deadlock is between GuiceFinder and ExtensionList, but possibly OldDataMonitor.report should not be acquiring a lock either since it is involved at a low level in initialization—perhaps it could queue up reports until after startup has completed, or at least extensions have been prepared, or perhaps data could become a static WeakHashMap<Jenkins, …> reference.

            jglick Jesse Glick added a comment - While a Sonar plugin change may have triggered this bug, it looks more like a core issue to me. The deadlock is between GuiceFinder and ExtensionList , but possibly OldDataMonitor.report should not be acquiring a lock either since it is involved at a low level in initialization—perhaps it could queue up reports until after startup has completed, or at least extensions have been prepared, or perhaps data could become a static WeakHashMap<Jenkins, …> reference.
            owenmehegan Owen Mehegan added a comment -

            henryju please consider submitting a request to remove 2.7.0 from the update center, given that installing it can render Jenkins unable to start at all. See https://jenkins.io/doc/developer/publishing/removing-from-distribution/

            owenmehegan Owen Mehegan added a comment - henryju please consider submitting a request to remove 2.7.0 from the update center, given that installing it can render Jenkins unable to start at all. See  https://jenkins.io/doc/developer/publishing/removing-from-distribution/
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/test/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/mf/ReadManifestStepTest.java
            http://jenkins-ci.org/commit/pipeline-utility-steps-plugin/982516b2fc8c2f641fb362da915f6bb9a037d76b
            Log:
            JENKINS-50663 - Restore Remoting manifest checks as requested by @rsandell

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/test/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/mf/ReadManifestStepTest.java http://jenkins-ci.org/commit/pipeline-utility-steps-plugin/982516b2fc8c2f641fb362da915f6bb9a037d76b Log: JENKINS-50663 - Restore Remoting manifest checks as requested by @rsandell
            dlouzado Dario Louzado added a comment -

            In my case, updating to version 2.7.1 did not fix the problem.

            2.7.1 did work on a new installation and did not on my staging environment. Jenkins startup freezed again and no logs were written. Cannot use this version on production env.

            dlouzado Dario Louzado added a comment - In my case, updating to version 2.7.1 did not fix the problem. 2.7.1 did work on a new installation and did not on my staging environment. Jenkins startup freezed again and no logs were written. Cannot use this version on production env.
            oleg_nenashev Oleg Nenashev added a comment -

             

            JENKINS-55361 should fix it in 2.163 from what I see

            oleg_nenashev Oleg Nenashev added a comment -   JENKINS-55361 should fix it in 2.163 from what I see

            People

              Unassigned Unassigned
              jchorko Janusz Chorko
              Votes:
              5 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: