Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73360

This plugin was causing slowness for shared library functions.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • build-user-vars-plugin
    • None
    • Plugin version: 166.v52976843b_435
      Jenkins version: 2.452.2

      Took months to find issue. Replicated Jenkins to new environment, No errors that related to this plugin were seen on Jenkins logs, hanging threads etc in monitoring area.

      In the end, started uninstalling all plugins one by one until I found this plugin was the cause.

      It was slowing down my Jenkins pipeline significantly. Every operation in Jenkins was running slowly when this plugin was installed.

          [JENKINS-73360] This plugin was causing slowness for shared library functions.

          Mark Waite added a comment -

          pagoel can you provide more details that describe any ideas you might have that would indicate why this plugin was slowing Pipeline execution? Was there something specific in your configuration that might not be used in other configurations?

          Mark Waite added a comment - pagoel can you provide more details that describe any ideas you might have that would indicate why this plugin was slowing Pipeline execution? Was there something specific in your configuration that might not be used in other configurations?

          Paras Goel added a comment - - edited

          Hi Mark,

          The slowness was particularly noticed when i was running a shared library function. The function is one that validates the pipeline code. Stuff like checking agent name, stage name, ensures declerative pipeline etc.
          It was running very slowly when the above plugin was installed, ran instantly without the plugin.

          Kind Regards,

          Paras

          Paras Goel added a comment - - edited Hi Mark, The slowness was particularly noticed when i was running a shared library function. The function is one that validates the pipeline code. Stuff like checking agent name, stage name, ensures declerative pipeline etc. It was running very slowly when the above plugin was installed, ran instantly without the plugin. Kind Regards, Paras

          Fábio Silva added a comment -

          pagoel to help identify the cause, could you please try reinstalling the plugin and disabling the "Build User Variables > Enabled for all builds" option in the global configurations?

          Additionally, could you provide some information about the type of security setup you're using in Jenkins? Are you using the saml plugin?

          Fábio Silva added a comment - pagoel to help identify the cause, could you please try reinstalling the plugin and disabling the "Build User Variables > Enabled for all builds" option in the global configurations? Additionally, could you provide some information about the type of security setup you're using in Jenkins? Are you using the saml plugin?

          nils m added a comment -

          Hi,
          we are currently expieriencing a simular problem since updating some plugins and our java version (11 -> 17 -> 21).

          Specs:
          Linux Host System
          Jenkins 2.462.2
          OpenJDK Java21
          build user vars plugin Version 176.vb_9c7907fd524
          Active Directory plugin Version 2.37

          "Build User Variables > Enabled for all builds" is activated

          One way of reproducing hanging threads is: Start a couple of Runs for a job so that the job has running Runs while one run is finishing.
          This has a googd chance to deadlock all Jenkins CPS Threads (the once diriving the declarative pipeline code evaluation).

          Here are parts of a Threaddump taken while the jenkins is hanging:

          Hanging CPS Thread

          "Running CpsFlowExecution[Owner[<JOBNAME>/1095:<JOBNAME> #1095]]" Id=3733 Group=main WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 owned by "Thread-1682" Id=5386
          	at java.base@21.0.4/jdk.internal.misc.Unsafe.park(Native Method)
          	-  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4
          	at java.base@21.0.4/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
          	at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754)
          	at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990)
          	at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153)
          	at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1830)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1808)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.shutdown(SSLSocketImpl.java:1756)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:796)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:664)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:584)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl$AppOutputStream.close(SSLSocketImpl.java:1334)
          	at java.naming@21.0.4/com.sun.jndi.ldap.Connection.flushAndCloseOutputStream(Connection.java:697)
          	at java.naming@21.0.4/com.sun.jndi.ldap.Connection.cleanup(Connection.java:650)
          	-  locked com.sun.jndi.ldap.Connection@355fa96d
          	at java.naming@21.0.4/com.sun.jndi.ldap.LdapClient.close(LdapClient.java:450)
          	-  locked com.sun.jndi.ldap.LdapClient@5a4bfeeb
          	at java.naming@21.0.4/com.sun.jndi.ldap.LdapCtx.closeConnection(LdapCtx.java:2961)
          	at java.naming@21.0.4/com.sun.jndi.ldap.LdapCtx.close(LdapCtx.java:2671)
          	-  locked com.sun.jndi.ldap.LdapCtx@756dfdc1
          	at java.naming@21.0.4/javax.naming.InitialContext.close(InitialContext.java:592)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.closeQuietly(ActiveDirectoryUnixAuthenticationProvider.java:584)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.lambda$retrieveUser$0(ActiveDirectoryUnixAuthenticationProvider.java:451)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$$Lambda/0x00007fd0d6f652f0.apply(Unknown Source)
          	at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2716)
          	-  locked com.github.benmanes.caffeine.cache.PSWMS@38c6847c
          	at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda/0x00007fd0d698b490.apply(Unknown Source)
          	at java.base@21.0.4/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940)
          	-  locked java.util.concurrent.ConcurrentHashMap$Node@1f8601f2
          	at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686)
          	at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669)
          	at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
          	at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:454)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:297)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:223)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:47)
          	at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectorySecurityRealm.loadUserByUsername(ActiveDirectorySecurityRealm.java:941)
          	at hudson.security.AbstractPasswordBasedSecurityRealm.loadUserByUsername2(AbstractPasswordBasedSecurityRealm.java:129)
          	at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.varsetter.impl.UserIdCauseDeterminant.getUserGroups(UserIdCauseDeterminant.java:78)
          	at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.varsetter.impl.UserIdCauseDeterminant.setJenkinsUserBuildVars(UserIdCauseDeterminant.java:47)
          	at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.BuildUser.makeUserBuildVariables(BuildUser.java:72)
          	at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.BuildUser.makeUserBuildVariables(BuildUser.java:90)
          	at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.BuildUserVarsEnvironmentContributor.buildEnvironmentFor(BuildUserVarsEnvironmentContributor.java:18)
          	at hudson.model.Run.getEnvironment(Run.java:2432)
          	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:519)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:92)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:108)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsWhitelist.permitsMethod(CpsWhitelist.java:59)
          	at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.ProxyWhitelist.permitsMethod(ProxyWhitelist.java:63)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.GroovyClassLoaderWhitelist.permitsMethod(GroovyClassLoaderWhitelist.java:45)
          	at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:346)
          	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker$7.call(Checker.java:375)
          	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:379)
          	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:355)
          	at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:355)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.getProperty(LoggingInvoker.java:133)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:73)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:65)
          	at java.base@21.0.4/java.lang.invoke.LambdaForm$DMH/0x00007fd0d6859800.invokeVirtual(LambdaForm$DMH)
          	at java.base@21.0.4/java.lang.invoke.LambdaForm$MH/0x00007fd0d6f7c000.invoke(LambdaForm$MH)
          	at java.base@21.0.4/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
          	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154)
          	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
          	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83)
          	at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x00007fd0d6d8d900.call(Unknown Source)
          	at java.base@21.0.4/java.util.concurrent.FutureTask.run(FutureTask.java:317)
          	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
          	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
          	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
          	at jenkins.util.ErrorLoggingExecutorService$$Lambda/0x00007fd0d6b0e210.run(Unknown Source)
          	at java.base@21.0.4/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
          	at java.base@21.0.4/java.util.concurrent.FutureTask.run(FutureTask.java:317)
          	at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
          	at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
          	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
          	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
          	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x00007fd0d6d8db20.run(Unknown Source)
          	at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596)
          	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)
          
          	Number of locked synchronizers = 1
          	- java.util.concurrent.ThreadPoolExecutor$Worker@36b0e380
          

          Thread-1682 holding the lock

          "Thread-1682" Id=5386 Group=main RUNNABLE (in native)
          	at java.base@21.0.4/sun.nio.ch.Net.poll(Native Method)
          	at java.base@21.0.4/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191)
          	at java.base@21.0.4/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201)
          	at java.base@21.0.4/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
          	at java.base@21.0.4/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
          	at java.base@21.0.4/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
          	at java.base@21.0.4/java.net.Socket$SocketInputStream.read(Socket.java:1099)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461)
          	at java.base@21.0.4/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066)
          	at java.naming@21.0.4/com.sun.jndi.ldap.Connection.run(Connection.java:899)
          	at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596)
          	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)
          
          	Number of locked synchronizers = 2
          	- java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4
          	- java.util.concurrent.locks.ReentrantLock$NonfairSync@685da118
          

          all Jobs running at the time the lockup happens (rougly when one of the jobs described above finished) are in state BLOCKED as their are waiting for locked java.util.concurrent.ConcurrentHashMap$Node@1f8601f2 that was previously blocked by the CPS Thread.

          Why is this an Issue:
          The ConcurrentMap that is getting blocked in the CPS execution seems to be used by ALL jobs running on an instance.
          The result of the deadlock isd the following:

          • All jobs on this Jenkins instance stop evaluating their pipeline (e.g. running sh calls finish but then the pipeline is stuck)
          • once a blocked job times out, jenkins starts to notice that something is wrong
          • jenkins will try to stop all blocked jobs for roughly 15min before terminating them
          • jenkins will continue to accept and execute new jobs
            This process blocks all builds on the jenkins for roughly 30+ minutes

          I hope this can help you identifying the issue.

          nils m added a comment - Hi, we are currently expieriencing a simular problem since updating some plugins and our java version (11 -> 17 -> 21). Specs: Linux Host System Jenkins 2.462.2 OpenJDK Java21 build user vars plugin Version 176.vb_9c7907fd524 Active Directory plugin Version 2.37 "Build User Variables > Enabled for all builds" is activated One way of reproducing hanging threads is: Start a couple of Runs for a job so that the job has running Runs while one run is finishing. This has a googd chance to deadlock all Jenkins CPS Threads (the once diriving the declarative pipeline code evaluation). Here are parts of a Threaddump taken while the jenkins is hanging: Hanging CPS Thread "Running CpsFlowExecution[Owner[<JOBNAME>/1095:<JOBNAME> #1095]]" Id=3733 Group=main WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 owned by " Thread -1682" Id=5386 at java.base@21.0.4/jdk.internal.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 at java.base@21.0.4/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221) at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754) at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990) at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153) at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1830) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1808) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.shutdown(SSLSocketImpl.java:1756) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:796) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:664) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:584) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl$AppOutputStream.close(SSLSocketImpl.java:1334) at java.naming@21.0.4/com.sun.jndi.ldap.Connection.flushAndCloseOutputStream(Connection.java:697) at java.naming@21.0.4/com.sun.jndi.ldap.Connection.cleanup(Connection.java:650) - locked com.sun.jndi.ldap.Connection@355fa96d at java.naming@21.0.4/com.sun.jndi.ldap.LdapClient.close(LdapClient.java:450) - locked com.sun.jndi.ldap.LdapClient@5a4bfeeb at java.naming@21.0.4/com.sun.jndi.ldap.LdapCtx.closeConnection(LdapCtx.java:2961) at java.naming@21.0.4/com.sun.jndi.ldap.LdapCtx.close(LdapCtx.java:2671) - locked com.sun.jndi.ldap.LdapCtx@756dfdc1 at java.naming@21.0.4/javax.naming.InitialContext.close(InitialContext.java:592) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.closeQuietly(ActiveDirectoryUnixAuthenticationProvider.java:584) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.lambda$retrieveUser$0(ActiveDirectoryUnixAuthenticationProvider.java:451) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$$Lambda/0x00007fd0d6f652f0.apply(Unknown Source) at PluginClassLoader for caffeine-api //com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2716) - locked com.github.benmanes.caffeine.cache.PSWMS@38c6847c at PluginClassLoader for caffeine-api //com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda/0x00007fd0d698b490.apply(Unknown Source) at java.base@21.0.4/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940) - locked java.util.concurrent.ConcurrentHashMap$Node@1f8601f2 at PluginClassLoader for caffeine-api //com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686) at PluginClassLoader for caffeine-api //com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669) at PluginClassLoader for caffeine-api //com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) at PluginClassLoader for caffeine-api //com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:454) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:297) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:223) at PluginClassLoader for active-directory //hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:47) at PluginClassLoader for active-directory //hudson.plugins.active_directory.ActiveDirectorySecurityRealm.loadUserByUsername(ActiveDirectorySecurityRealm.java:941) at hudson.security.AbstractPasswordBasedSecurityRealm.loadUserByUsername2(AbstractPasswordBasedSecurityRealm.java:129) at PluginClassLoader for build-user-vars-plugin //org.jenkinsci.plugins.builduser.varsetter.impl.UserIdCauseDeterminant.getUserGroups(UserIdCauseDeterminant.java:78) at PluginClassLoader for build-user-vars-plugin //org.jenkinsci.plugins.builduser.varsetter.impl.UserIdCauseDeterminant.setJenkinsUserBuildVars(UserIdCauseDeterminant.java:47) at PluginClassLoader for build-user-vars-plugin //org.jenkinsci.plugins.builduser.BuildUser.makeUserBuildVariables(BuildUser.java:72) at PluginClassLoader for build-user-vars-plugin //org.jenkinsci.plugins.builduser.BuildUser.makeUserBuildVariables(BuildUser.java:90) at PluginClassLoader for build-user-vars-plugin //org.jenkinsci.plugins.builduser.BuildUserVarsEnvironmentContributor.buildEnvironmentFor(BuildUserVarsEnvironmentContributor.java:18) at hudson.model.Run.getEnvironment(Run.java:2432) at PluginClassLoader for workflow-job //org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:519) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:92) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:108) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsWhitelist.permitsMethod(CpsWhitelist.java:59) at PluginClassLoader for script-security //org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.ProxyWhitelist.permitsMethod(ProxyWhitelist.java:63) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.GroovyClassLoaderWhitelist.permitsMethod(GroovyClassLoaderWhitelist.java:45) at PluginClassLoader for script-security //org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:346) at PluginClassLoader for script-security //org.kohsuke.groovy.sandbox.impl.Checker$7.call(Checker.java:375) at PluginClassLoader for script-security //org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:379) at PluginClassLoader for script-security //org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:355) at PluginClassLoader for script-security //org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:355) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.LoggingInvoker.getProperty(LoggingInvoker.java:133) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:73) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:65) at java.base@21.0.4/java.lang.invoke.LambdaForm$DMH/0x00007fd0d6859800.invokeVirtual(LambdaForm$DMH) at java.base@21.0.4/java.lang.invoke.LambdaForm$MH/0x00007fd0d6f7c000.invoke(LambdaForm$MH) at java.base@21.0.4/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154) at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.Next.step(Next.java:83) at PluginClassLoader for workflow-cps //com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x00007fd0d6d8d900.call(Unknown Source) at java.base@21.0.4/java.util.concurrent.FutureTask.run(FutureTask.java:317) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) at jenkins.util.ErrorLoggingExecutorService$$Lambda/0x00007fd0d6b0e210.run(Unknown Source) at java.base@21.0.4/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base@21.0.4/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50) at PluginClassLoader for workflow-cps //org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x00007fd0d6d8db20.run(Unknown Source) at java.base@21.0.4/java.lang. Thread .runWith( Thread .java:1596) at java.base@21.0.4/java.lang. Thread .run( Thread .java:1583) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@36b0e380 Thread-1682 holding the lock " Thread -1682" Id=5386 Group=main RUNNABLE (in native ) at java.base@21.0.4/sun.nio.ch.Net.poll(Native Method) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) at java.base@21.0.4/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) at java.base@21.0.4/java.net.Socket$SocketInputStream.read(Socket.java:1099) at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) at java.naming@21.0.4/com.sun.jndi.ldap.Connection.run(Connection.java:899) at java.base@21.0.4/java.lang. Thread .runWith( Thread .java:1596) at java.base@21.0.4/java.lang. Thread .run( Thread .java:1583) Number of locked synchronizers = 2 - java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 - java.util.concurrent.locks.ReentrantLock$NonfairSync@685da118 all Jobs running at the time the lockup happens (rougly when one of the jobs described above finished) are in state BLOCKED as their are waiting for locked java.util.concurrent.ConcurrentHashMap$Node@1f8601f2 that was previously blocked by the CPS Thread. Why is this an Issue: The ConcurrentMap that is getting blocked in the CPS execution seems to be used by ALL jobs running on an instance. The result of the deadlock isd the following: All jobs on this Jenkins instance stop evaluating their pipeline (e.g. running sh calls finish but then the pipeline is stuck) once a blocked job times out, jenkins starts to notice that something is wrong jenkins will try to stop all blocked jobs for roughly 15min before terminating them jenkins will continue to accept and execute new jobs This process blocks all builds on the jenkins for roughly 30+ minutes I hope this can help you identifying the issue.

            zedasvacas Fábio Silva
            pagoel Paras Goel
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: