Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48505

Lock contention in ExtensionList.getExtensionList

      Upgraded Jenkins from 2.66 to 2.89.1 and hit lock contention on ExtensionList.getExtensionList

      maybe caused by the change #24ede5

           @SuppressWarnings({"unchecked"})
           public <T> ExtensionList<T> getExtensionList(Class<T> extensionType) {
      -        return extensionLists.get(extensionType);
      +        return extensionLists.computeIfAbsent(extensionType, key -> ExtensionList.create(this, key));
           }
      
      

      Sample code:
      GlobalConfiguration.all().get(SplunkJenkinsInstallation.class);
      source: https://github.com/splunk/splunkforjenkins/blob/master/splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java#L106

      stack trace

      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#793]" #190991 daemon prio=5 os_prio=0 tid=0x00007fb38f2f8800 nid=0xe54 waiting for monitor entry [0x00007fb3632dc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - waiting to lock <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#673]" #155842 daemon prio=5 os_prio=0 tid=0x00007fb38f3c6800 nid=0xe7d8 waiting for monitor entry [0x00007fb2ee566000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1674)
              - locked <0x0000000342d647c8> (a java.util.concurrent.ConcurrentHashMap$Node)
              at jenkins.model.Jenkins.getDescriptorList(Jenkins.java:2652)
              at jenkins.model.GlobalConfiguration.all(GlobalConfiguration.java:73)
              at com.splunk.splunkjenkins.SplunkJenkinsInstallation.get(SplunkJenkinsInstallation.java:105)
              at com.splunk.splunkjenkins.utils.LogFileCallable.send(LogFileCallable.java:90)
              at com.splunk.splunkjenkins.utils.LogFileCallable.sendFiles(LogFileCallable.java:57)
              at com.splunk.splunkjenkins.utils.LogEventHelper.sendFiles(LogEventHelper.java:282)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:111)
              at com.splunk.splunkjenkins.SplunkLogFileStep$SplunkLogFileStepExecution.run(SplunkLogFileStep.java:92)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
              at hudson.security.ACL.impersonate(ACL.java:260)
              at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      grep "waiting to lock <0x0000000342d647c8>" jstack_dump.txt |wc -l
      139
      

      the lock 0x00007fb2ee566000 was not show up in jstack, maybe a C level lock
      Note: there is no dead lock, it is still functional but slow

          [JENKINS-48505] Lock contention in ExtensionList.getExtensionList

          Ted Xiao added a comment -

          maybe related to the JDK bug: JDK-8184907 : ConcurrentHashMap.computeIfAbsent acquires lock even when key present
          http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8184907
          It seems that it is only fixed in JDK9

          Ted Xiao added a comment - maybe related to the JDK bug: JDK-8184907 : ConcurrentHashMap.computeIfAbsent acquires lock even when key present http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8184907 It seems that it is only fixed in JDK9

          Oleg Nenashev added a comment -

          CC jglick

          Oleg Nenashev added a comment - CC jglick

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/java/jenkins/model/Jenkins.java
          http://jenkins-ci.org/commit/jenkins/3a529636cfc64cc6ea454d55ca88fd7a1f7981a5
          Log:
          Merge pull request #3198 from johnou/contention-JENKINS-48505

          JENKINS-48505 - Invoke optimistic get before computeIfAbsent to avoid contention.

          Compare: https://github.com/jenkinsci/jenkins/compare/09bcc5d6538b...3a529636cfc6

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/jenkins/model/Jenkins.java http://jenkins-ci.org/commit/jenkins/3a529636cfc64cc6ea454d55ca88fd7a1f7981a5 Log: Merge pull request #3198 from johnou/contention- JENKINS-48505 JENKINS-48505 - Invoke optimistic get before computeIfAbsent to avoid contention. Compare: https://github.com/jenkinsci/jenkins/compare/09bcc5d6538b...3a529636cfc6

          Daniel Beck added a comment -

          Merged towards 2.96.

          Daniel Beck added a comment - Merged towards 2.96.

          Code changed in jenkins
          User: Ted Xiao
          Path:
          splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java
          http://jenkins-ci.org/commit/splunk-devops-plugin/10c7ca8fd87a17f68c708e9b49e4b06501d96726
          Log:
          workaround JENKINS-48505 JDK-8184907

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Ted Xiao Path: splunk-devops/src/main/java/com/splunk/splunkjenkins/SplunkJenkinsInstallation.java http://jenkins-ci.org/commit/splunk-devops-plugin/10c7ca8fd87a17f68c708e9b49e4b06501d96726 Log: workaround JENKINS-48505 JDK-8184907

            johnou johnou
            fengxx Ted Xiao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: