Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65091

Jenkins UI unresponsive

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: sonar-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Hello, we upgraded today to 2.283 from 2.280

      After a few hours of running the UI became un-responsive although there was nothing in the logs that looked interesting, jobs were still running

      I've attached a thread dump, would be interested to know if anyone has any thoughts on it

        Attachments

          Activity

          Hide
          raihaan Raihaan Shouhell added a comment -

          Hey Tim, could you collect several consecutive threaddumps? Its useful for analyzing if there are any threads holding onto locks for extended periods. Cloudbees has a nice script for this https://support.cloudbees.com/hc/en-us/articles/360016440131-What-is-jenkinshangWithJstack-sh-and-how-does-it-help-

          Show
          raihaan Raihaan Shouhell added a comment - Hey Tim, could you collect several consecutive threaddumps? Its useful for analyzing if there are any threads holding onto locks for extended periods. Cloudbees has a nice script for this https://support.cloudbees.com/hc/en-us/articles/360016440131-What-is-jenkinshangWithJstack-sh-and-how-does-it-help-
          Hide
          timja Tim Jacomb added a comment -

          Sure if it happens again, already restarted this time

          Show
          timja Tim Jacomb added a comment - Sure if it happens again, already restarted this time
          Hide
          danielbeck Daniel Beck added a comment -

          Seems like Azure Keyvault and/or Sonar, this is the thread holding the lock all others are waiting for:

          "Handling GET /blue/organizations/jenkins/HMCTS_SSCS%2Fsscs-tribunals-case-api/activity/ from 77.100.205.123 : Jetty (winstone)-34723 BlueOceanUI/index.jelly PageStatePreloadDecorator/header.jelly" #34723 prio=5 os_prio=0 cpu=221.64ms elapsed=3843.29s tid=0x00007f5fec9dd000 nid=0x2e84 waiting on condition  [0x00007f5ef8046000]
             java.lang.Thread.State: WAITING (parking)
          	at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method)
          	- parking to wait for  <0x00000004ecc9f5b0> (a java.util.concurrent.CountDownLatch$Sync)
          	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:194)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.10/AbstractQueuedSynchronizer.java:885)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1039)
          	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1345)
          	at java.util.concurrent.CountDownLatch.await(java.base@11.0.10/CountDownLatch.java:232)
          	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
          	at reactor.core.publisher.Mono.block(Mono.java:1663)
          	at com.azure.security.keyvault.secrets.SecretClient.getSecretWithResponse(SecretClient.java:171)
          	at com.azure.security.keyvault.secrets.SecretClient.getSecret(SecretClient.java:150)
          	at com.microsoft.jenkins.keyvault.BaseSecretCredentials$SecretGetter$1.getKeyVaultSecret(BaseSecretCredentials.java:96)
          	at com.microsoft.jenkins.keyvault.BaseSecretCredentials.getKeyVaultSecret(BaseSecretCredentials.java:61)
          	at com.microsoft.jenkins.keyvault.SecretStringCredentials.getSecret(SecretStringCredentials.java:34)
          	at hudson.plugins.sonar.SonarInstallation.getServerAuthenticationToken(SonarInstallation.java:186)
          	at hudson.plugins.sonar.client.SQProjectResolver.resolve(SQProjectResolver.java:56)
          	at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:77)
          	at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:52)
          	at hudson.plugins.sonar.action.SonarProjectActionFactory.createProjectPage(SonarProjectActionFactory.java:118)
          	- locked <0x00000004f44aa630> (a org.jenkinsci.plugins.workflow.job.WorkflowRun)
          	at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:83)
          	at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:42) 

          https://github.com/jenkinsci/sonarqube-plugin/blob/1bade4ab6ad3076bdf66d5537e72ba0c0a728355/src/main/java/hudson/plugins/sonar/action/SonarProjectActionFactory.java#L116-L119 this looks like quite the decision…

          Show
          danielbeck Daniel Beck added a comment - Seems like Azure Keyvault and/or Sonar, this is the thread holding the lock all others are waiting for: "Handling GET /blue/organizations/jenkins/HMCTS_SSCS%2Fsscs-tribunals-case-api/activity/ from 77.100.205.123 : Jetty (winstone)-34723 BlueOceanUI/index.jelly PageStatePreloadDecorator/header.jelly" #34723 prio=5 os_prio=0 cpu=221.64ms elapsed=3843.29s tid=0x00007f5fec9dd000 nid=0x2e84 waiting on condition [0x00007f5ef8046000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method) - parking to wait for <0x00000004ecc9f5b0> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.10/AbstractQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1039) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1345) at java.util.concurrent.CountDownLatch.await(java.base@11.0.10/CountDownLatch.java:232) at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81) at reactor.core.publisher.Mono.block(Mono.java:1663) at com.azure.security.keyvault.secrets.SecretClient.getSecretWithResponse(SecretClient.java:171) at com.azure.security.keyvault.secrets.SecretClient.getSecret(SecretClient.java:150) at com.microsoft.jenkins.keyvault.BaseSecretCredentials$SecretGetter$1.getKeyVaultSecret(BaseSecretCredentials.java:96) at com.microsoft.jenkins.keyvault.BaseSecretCredentials.getKeyVaultSecret(BaseSecretCredentials.java:61) at com.microsoft.jenkins.keyvault.SecretStringCredentials.getSecret(SecretStringCredentials.java:34) at hudson.plugins.sonar.SonarInstallation.getServerAuthenticationToken(SonarInstallation.java:186) at hudson.plugins.sonar.client.SQProjectResolver.resolve(SQProjectResolver.java:56) at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:77) at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:52) at hudson.plugins.sonar.action.SonarProjectActionFactory.createProjectPage(SonarProjectActionFactory.java:118) - locked <0x00000004f44aa630> (a org.jenkinsci.plugins.workflow.job.WorkflowRun) at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:83) at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:42) https://github.com/jenkinsci/sonarqube-plugin/blob/1bade4ab6ad3076bdf66d5537e72ba0c0a728355/src/main/java/hudson/plugins/sonar/action/SonarProjectActionFactory.java#L116-L119 this looks like quite the decision…
          Hide
          timja Tim Jacomb added a comment -
          Show
          timja Tim Jacomb added a comment - Should be fixed by https://github.com/jenkinsci/azure-keyvault-plugin/pull/87

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: