-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Plugin version: 166.v52976843b_435
Jenkins version: 2.452.2
-
Powered by SuggestiMate
Took months to find issue. Replicated Jenkins to new environment, No errors that related to this plugin were seen on Jenkins logs, hanging threads etc in monitoring area.
In the end, started uninstalling all plugins one by one until I found this plugin was the cause.
It was slowing down my Jenkins pipeline significantly. Every operation in Jenkins was running slowly when this plugin was installed.
[JENKINS-73360] This plugin was causing slowness for shared library functions.
Hi Mark,
The slowness was particularly noticed when i was running a shared library function. The function is one that validates the pipeline code. Stuff like checking agent name, stage name, ensures declerative pipeline etc.
It was running very slowly when the above plugin was installed, ran instantly without the plugin.
Kind Regards,
Paras
pagoel to help identify the cause, could you please try reinstalling the plugin and disabling the "Build User Variables > Enabled for all builds" option in the global configurations?
Additionally, could you provide some information about the type of security setup you're using in Jenkins? Are you using the saml plugin?
Hi,
we are currently expieriencing a simular problem since updating some plugins and our java version (11 -> 17 -> 21).
Specs:
Linux Host System
Jenkins 2.462.2
OpenJDK Java21
build user vars plugin Version 176.vb_9c7907fd524
Active Directory plugin Version 2.37
"Build User Variables > Enabled for all builds" is activated
One way of reproducing hanging threads is: Start a couple of Runs for a job so that the job has running Runs while one run is finishing.
This has a googd chance to deadlock all Jenkins CPS Threads (the once diriving the declarative pipeline code evaluation).
Here are parts of a Threaddump taken while the jenkins is hanging:
Hanging CPS Thread
"Running CpsFlowExecution[Owner[<JOBNAME>/1095:<JOBNAME> #1095]]" Id=3733 Group=main WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 owned by "Thread-1682" Id=5386 at java.base@21.0.4/jdk.internal.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 at java.base@21.0.4/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221) at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:754) at java.base@21.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:990) at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153) at java.base@21.0.4/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1830) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1808) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.shutdown(SSLSocketImpl.java:1756) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:796) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:664) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:584) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl$AppOutputStream.close(SSLSocketImpl.java:1334) at java.naming@21.0.4/com.sun.jndi.ldap.Connection.flushAndCloseOutputStream(Connection.java:697) at java.naming@21.0.4/com.sun.jndi.ldap.Connection.cleanup(Connection.java:650) - locked com.sun.jndi.ldap.Connection@355fa96d at java.naming@21.0.4/com.sun.jndi.ldap.LdapClient.close(LdapClient.java:450) - locked com.sun.jndi.ldap.LdapClient@5a4bfeeb at java.naming@21.0.4/com.sun.jndi.ldap.LdapCtx.closeConnection(LdapCtx.java:2961) at java.naming@21.0.4/com.sun.jndi.ldap.LdapCtx.close(LdapCtx.java:2671) - locked com.sun.jndi.ldap.LdapCtx@756dfdc1 at java.naming@21.0.4/javax.naming.InitialContext.close(InitialContext.java:592) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.closeQuietly(ActiveDirectoryUnixAuthenticationProvider.java:584) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.lambda$retrieveUser$0(ActiveDirectoryUnixAuthenticationProvider.java:451) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider$$Lambda/0x00007fd0d6f652f0.apply(Unknown Source) at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2716) - locked com.github.benmanes.caffeine.cache.PSWMS@38c6847c at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache$$Lambda/0x00007fd0d698b490.apply(Unknown Source) at java.base@21.0.4/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1940) - locked java.util.concurrent.ConcurrentHashMap$Node@1f8601f2 at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686) at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669) at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) at PluginClassLoader for caffeine-api//com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:454) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:297) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectoryUnixAuthenticationProvider.retrieveUser(ActiveDirectoryUnixAuthenticationProvider.java:223) at PluginClassLoader for active-directory//hudson.plugins.active_directory.AbstractActiveDirectoryAuthenticationProvider.loadUserByUsername(AbstractActiveDirectoryAuthenticationProvider.java:47) at PluginClassLoader for active-directory//hudson.plugins.active_directory.ActiveDirectorySecurityRealm.loadUserByUsername(ActiveDirectorySecurityRealm.java:941) at hudson.security.AbstractPasswordBasedSecurityRealm.loadUserByUsername2(AbstractPasswordBasedSecurityRealm.java:129) at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.varsetter.impl.UserIdCauseDeterminant.getUserGroups(UserIdCauseDeterminant.java:78) at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.varsetter.impl.UserIdCauseDeterminant.setJenkinsUserBuildVars(UserIdCauseDeterminant.java:47) at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.BuildUser.makeUserBuildVariables(BuildUser.java:72) at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.BuildUser.makeUserBuildVariables(BuildUser.java:90) at PluginClassLoader for build-user-vars-plugin//org.jenkinsci.plugins.builduser.BuildUserVarsEnvironmentContributor.buildEnvironmentFor(BuildUserVarsEnvironmentContributor.java:18) at hudson.model.Run.getEnvironment(Run.java:2432) at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.getEnvironment(WorkflowRun.java:519) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:92) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getProperty(EnvActionImpl.java:108) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsWhitelist.permitsMethod(CpsWhitelist.java:59) at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.ProxyWhitelist.permitsMethod(ProxyWhitelist.java:63) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.GroovyClassLoaderWhitelist.permitsMethod(GroovyClassLoaderWhitelist.java:45) at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:346) at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker$7.call(Checker.java:375) at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:379) at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:355) at PluginClassLoader for script-security//org.kohsuke.groovy.sandbox.impl.Checker.checkedGetProperty(Checker.java:355) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.SandboxInvoker.getProperty(SandboxInvoker.java:29) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.getProperty(LoggingInvoker.java:133) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawGet(PropertyAccessBlock.java:20) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:73) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:65) at java.base@21.0.4/java.lang.invoke.LambdaForm$DMH/0x00007fd0d6859800.invokeVirtual(LambdaForm$DMH) at java.base@21.0.4/java.lang.invoke.LambdaForm$MH/0x00007fd0d6f7c000.invoke(LambdaForm$MH) at java.base@21.0.4/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl(DirectMethodHandleAccessor.java:154) at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83) at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x00007fd0d6d8d900.call(Unknown Source) at java.base@21.0.4/java.util.concurrent.FutureTask.run(FutureTask.java:317) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) at jenkins.util.ErrorLoggingExecutorService$$Lambda/0x00007fd0d6b0e210.run(Unknown Source) at java.base@21.0.4/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base@21.0.4/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50) at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$$Lambda/0x00007fd0d6d8db20.run(Unknown Source) at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596) at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@36b0e380
Thread-1682 holding the lock
"Thread-1682" Id=5386 Group=main RUNNABLE (in native) at java.base@21.0.4/sun.nio.ch.Net.poll(Native Method) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:191) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309) at java.base@21.0.4/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) at java.base@21.0.4/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) at java.base@21.0.4/java.net.Socket$SocketInputStream.read(Socket.java:1099) at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) at java.base@21.0.4/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) at java.base@21.0.4/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) at java.naming@21.0.4/com.sun.jndi.ldap.Connection.run(Connection.java:899) at java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596) at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583) Number of locked synchronizers = 2 - java.util.concurrent.locks.ReentrantLock$NonfairSync@208160a4 - java.util.concurrent.locks.ReentrantLock$NonfairSync@685da118
all Jobs running at the time the lockup happens (rougly when one of the jobs described above finished) are in state BLOCKED as their are waiting for locked java.util.concurrent.ConcurrentHashMap$Node@1f8601f2 that was previously blocked by the CPS Thread.
Why is this an Issue:
The ConcurrentMap that is getting blocked in the CPS execution seems to be used by ALL jobs running on an instance.
The result of the deadlock isd the following:
- All jobs on this Jenkins instance stop evaluating their pipeline (e.g. running sh calls finish but then the pipeline is stuck)
- once a blocked job times out, jenkins starts to notice that something is wrong
- jenkins will try to stop all blocked jobs for roughly 15min before terminating them
- jenkins will continue to accept and execute new jobs
This process blocks all builds on the jenkins for roughly 30+ minutes
I hope this can help you identifying the issue.
pagoel can you provide more details that describe any ideas you might have that would indicate why this plugin was slowing Pipeline execution? Was there something specific in your configuration that might not be used in other configurations?