Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65091

Jenkins UI unresponsive

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • sonar-plugin
    • None

    Description

      Hello, we upgraded today to 2.283 from 2.280

      After a few hours of running the UI became un-responsive although there was nothing in the logs that looked interesting, jobs were still running

      I've attached a thread dump, would be interested to know if anyone has any thoughts on it

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          timja Tim Jacomb created issue -
          raihaan Raihaan Shouhell added a comment -

          Hey Tim, could you collect several consecutive threaddumps? Its useful for analyzing if there are any threads holding onto locks for extended periods. Cloudbees has a nice script for this https://support.cloudbees.com/hc/en-us/articles/360016440131-What-is-jenkinshangWithJstack-sh-and-how-does-it-help-

          raihaan Raihaan Shouhell added a comment - Hey Tim, could you collect several consecutive threaddumps? Its useful for analyzing if there are any threads holding onto locks for extended periods. Cloudbees has a nice script for this https://support.cloudbees.com/hc/en-us/articles/360016440131-What-is-jenkinshangWithJstack-sh-and-how-does-it-help-
          timja Tim Jacomb added a comment -

          Sure if it happens again, already restarted this time

          timja Tim Jacomb added a comment - Sure if it happens again, already restarted this time
          danielbeck Daniel Beck added a comment -

          Seems like Azure Keyvault and/or Sonar, this is the thread holding the lock all others are waiting for:

          "Handling GET /blue/organizations/jenkins/HMCTS_SSCS%2Fsscs-tribunals-case-api/activity/ from 77.100.205.123 : Jetty (winstone)-34723 BlueOceanUI/index.jelly PageStatePreloadDecorator/header.jelly" #34723 prio=5 os_prio=0 cpu=221.64ms elapsed=3843.29s tid=0x00007f5fec9dd000 nid=0x2e84 waiting on condition  [0x00007f5ef8046000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method)
	- parking to wait for  <0x00000004ecc9f5b0> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.10/AbstractQueuedSynchronizer.java:885)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1039)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1345)
	at java.util.concurrent.CountDownLatch.await(java.base@11.0.10/CountDownLatch.java:232)
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81)
	at reactor.core.publisher.Mono.block(Mono.java:1663)
	at com.azure.security.keyvault.secrets.SecretClient.getSecretWithResponse(SecretClient.java:171)
	at com.azure.security.keyvault.secrets.SecretClient.getSecret(SecretClient.java:150)
	at com.microsoft.jenkins.keyvault.BaseSecretCredentials$SecretGetter$1.getKeyVaultSecret(BaseSecretCredentials.java:96)
	at com.microsoft.jenkins.keyvault.BaseSecretCredentials.getKeyVaultSecret(BaseSecretCredentials.java:61)
	at com.microsoft.jenkins.keyvault.SecretStringCredentials.getSecret(SecretStringCredentials.java:34)
	at hudson.plugins.sonar.SonarInstallation.getServerAuthenticationToken(SonarInstallation.java:186)
	at hudson.plugins.sonar.client.SQProjectResolver.resolve(SQProjectResolver.java:56)
	at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:77)
	at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:52)
	at hudson.plugins.sonar.action.SonarProjectActionFactory.createProjectPage(SonarProjectActionFactory.java:118)
	- locked <0x00000004f44aa630> (a org.jenkinsci.plugins.workflow.job.WorkflowRun)
	at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:83)
	at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:42)

          https://github.com/jenkinsci/sonarqube-plugin/blob/1bade4ab6ad3076bdf66d5537e72ba0c0a728355/src/main/java/hudson/plugins/sonar/action/SonarProjectActionFactory.java#L116-L119 this looks like quite the decision…

          danielbeck Daniel Beck added a comment - Seems like Azure Keyvault and/or Sonar, this is the thread holding the lock all others are waiting for: "Handling GET /blue/organizations/jenkins/HMCTS_SSCS%2Fsscs-tribunals-case-api/activity/ from 77.100.205.123 : Jetty (winstone)-34723 BlueOceanUI/index.jelly PageStatePreloadDecorator/header.jelly" #34723 prio=5 os_prio=0 cpu=221.64ms elapsed=3843.29s tid=0x00007f5fec9dd000 nid=0x2e84 waiting on condition [0x00007f5ef8046000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method) - parking to wait for <0x00000004ecc9f5b0> (a java.util.concurrent.CountDownLatch$Sync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.10/AbstractQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1039) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(java.base@11.0.10/AbstractQueuedSynchronizer.java:1345) at java.util.concurrent.CountDownLatch.await(java.base@11.0.10/CountDownLatch.java:232) at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:81) at reactor.core.publisher.Mono.block(Mono.java:1663) at com.azure.security.keyvault.secrets.SecretClient.getSecretWithResponse(SecretClient.java:171) at com.azure.security.keyvault.secrets.SecretClient.getSecret(SecretClient.java:150) at com.microsoft.jenkins.keyvault.BaseSecretCredentials$SecretGetter$1.getKeyVaultSecret(BaseSecretCredentials.java:96) at com.microsoft.jenkins.keyvault.BaseSecretCredentials.getKeyVaultSecret(BaseSecretCredentials.java:61) at com.microsoft.jenkins.keyvault.SecretStringCredentials.getSecret(SecretStringCredentials.java:34) at hudson.plugins.sonar.SonarInstallation.getServerAuthenticationToken(SonarInstallation.java:186) at hudson.plugins.sonar.client.SQProjectResolver.resolve(SQProjectResolver.java:56) at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:77) at hudson.plugins.sonar.action.SonarCacheAction.get(SonarCacheAction.java:52) at hudson.plugins.sonar.action.SonarProjectActionFactory.createProjectPage(SonarProjectActionFactory.java:118) - locked <0x00000004f44aa630> (a org.jenkinsci.plugins.workflow.job.WorkflowRun) at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:83) at hudson.plugins.sonar.action.SonarProjectActionFactory.createFor(SonarProjectActionFactory.java:42) https://github.com/jenkinsci/sonarqube-plugin/blob/1bade4ab6ad3076bdf66d5537e72ba0c0a728355/src/main/java/hudson/plugins/sonar/action/SonarProjectActionFactory.java#L116-L119 this looks like quite the decision…
          danielbeck Daniel Beck made changes -
          Field Original Value New Value
          Component/s azure-keyvault-plugin [ 24823 ]
          Component/s sonar-plugin [ 15523 ]
          Component/s core [ 15593 ]
          ircbot Jenkins IRC Bot made changes -
          Component/s _unsorted [ 19622 ]
          Component/s azure-keyvault-plugin [ 24823 ]
          timja Tim Jacomb made changes -
          Component/s _unsorted [ 19622 ]
          timja Tim Jacomb added a comment -

          Should be fixed by https://github.com/jenkinsci/azure-keyvault-plugin/pull/87

          timja Tim Jacomb added a comment - Should be fixed by https://github.com/jenkinsci/azure-keyvault-plugin/pull/87
          timja Tim Jacomb made changes -
          Remote Link This issue links to "https://github.com/jenkinsci/azure-keyvault-plugin/pull/87 (Web Link)" [ 26903 ]
          timja Tim Jacomb made changes -
          Remote Link This issue links to "https://github.com/SonarSource/sonar-scanner-jenkins/pull/182 (Web Link)" [ 26904 ]
          timja Tim Jacomb made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          timja Tim Jacomb made changes -
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]

          People

            Unassigned Unassigned
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: