Loading...

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: core
Labels:
Environment:
Jenkins 2.277.4
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10)

Similar Issues:

Show
Released As:
2.297, 2.289.2

After upgrading a controller from 2.264.x to 2.277.4, as well as upgrading all its plugins, the first page I've opened was /jenkins/configure (to check if there was any obvious tables-to-divs regressions there).

I've noticed an error in the Docker Pipeline section, next to a credentials selector.

The stack from jenkins.log:

2021-06-02 08:57:17.369+0000 [id=570]	WARNING	o.e.j.s.h.ContextHandler$Context#log: Error while serving http://.../jenkins/descriptorByName/org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryEndpoint/fillCredentialsIdItems
java.lang.LinkageError: loader (instance of  hudson/ClassicPluginStrategy$AntClassLoader2): attempted  duplicate class definition for name: "com/cloudbees/plugins/credentials/CredentialsMatchers"
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
	at jenkins.util.AntClassLoader.defineClassFromData(AntClassLoader.java:1155)
	at hudson.ClassicPluginStrategy$AntClassLoader2.defineClassFromData(ClassicPluginStrategy.java:718)
	at jenkins.util.AntClassLoader.getClassFromStream(AntClassLoader.java:1326)
	at jenkins.util.AntClassLoader.findClassInComponents(AntClassLoader.java:1377)
	at jenkins.util.AntClassLoader.findClass(AntClassLoader.java:1342)
	at jenkins.ClassLoaderReflectionToolkit._findClass(ClassLoaderReflectionToolkit.java:107)
	at hudson.ClassicPluginStrategy$DependencyClassLoader.findClass(ClassicPluginStrategy.java:637)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at jenkins.util.AntClassLoader.findBaseClass(AntClassLoader.java:1406)
	at jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1085)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at jenkins.authentication.tokens.api.AuthenticationTokenSource.matcher(AuthenticationTokenSource.java:89)
	at jenkins.authentication.tokens.api.AuthenticationTokens.matcher(AuthenticationTokens.java:87)
	at jenkins.authentication.tokens.api.AuthenticationTokens.matcher(AuthenticationTokens.java:72)
	at org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryEndpoint$DescriptorImpl.doFillCredentialsIdItems(DockerRegistryEndpoint.java:375)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
Caused: java.lang.reflect.InvocationTargetException
<snip>

When reloading the page, same error, with a truncated stacktrace (the classloader remembers that loading this class has already failed).

2021-06-02 09:04:58.621+0000 [id=571]	WARNING	o.e.j.s.h.ContextHandler$Context#log: Error while serving http://.../jenkins/descriptorByName/org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryEndpoint/fillCredentialsIdItems
java.lang.LinkageError: loader (instance of  hudson/ClassicPluginStrategy$AntClassLoader2): attempted  duplicate class definition for name: "com/cloudbees/plugins/credentials/CredentialsMatchers"
	at jenkins.authentication.tokens.api.AuthenticationTokenSource.matcher(AuthenticationTokenSource.java:89)
	at jenkins.authentication.tokens.api.AuthenticationTokens.matcher(AuthenticationTokens.java:87)
	at jenkins.authentication.tokens.api.AuthenticationTokens.matcher(AuthenticationTokens.java:72)
	at org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryEndpoint$DescriptorImpl.doFillCredentialsIdItems(DockerRegistryEndpoint.java:375)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
Caused: java.lang.reflect.InvocationTargetException
<snip>

So, it looks like there was several threads leading to loading the CredentialsMatchers class at the same time. Indeed, that's probably because from /jenkins/configure there can be several concurrent requests for filling some credentials selectors:

... [02/Jun/2021:10:57:17 +0200] "POST /jenkins/descriptorByName/hudson.plugins.jira.JiraSite/fillCredentialsIdItems HTTP/1.1" 200 684 "http://.../jenkins/configure" "Mozilla/5.0 ..."
... [02/Jun/2021:10:57:17 +0200] "POST /jenkins/descriptorByName/org.jenkinsci.plugins.docker.commons.credentials.DockerRegistryEndpoint/fillCredentialsIdItems HTTP/1.1" 500 5431 "http://.../jenkins/configure" "Mozilla/5.0 ..."

I have already (stupidly) restarted this Jenkins controller (which has resolved the issue) without keeping a heap dump, so I don't have access to all the info we could need to understand this bug.
But anyway, walking up the stacktrace, I think we can (kind of) understand how this case might happen.
First, there is jenkins.util.AntClassLoader.loadClass(AntClassLoader.java:1085) => this is a synchronized method, but that alone would not prevent an other thread following the same path via a different AntClassLoader instance (which, I assume, is what happens here).
Then, well, we reach some parent classloader, and from what I can tell there is no other layer of locking/synchronization involved.

This frame is maybe where a lock is missing:
jenkins.ClassLoaderReflectionToolkit._findClass(ClassLoaderReflectionToolkit.java:107)
Is it intended that AntClassLoader.findClass(name) is called from outside the synchronized block there? It's a recent change from PR-5110. I don't see any review comment on this specific chunk, but still, it's not obvious (to me) why it would be okay to avoid synchronization for an AntClassLoader. Ping timja and jglick , from PR-5110, regarding this point.

And then, there is the calling frame which looks a bit funny too:
hudson.ClassicPluginStrategy$DependencyClassLoader.findClass(ClassicPluginStrategy.java:637)

                        Class<?> c = ClassLoaderReflectionToolkit._findLoadedClass(pw.classLoader, name);
                        if (c!=null)    return c;
                        return ClassLoaderReflectionToolkit._findClass(pw.classLoader, name);

Both _findLoadedClass and _findClass are lock-holding (well, not really for the later, see previous point), but still, since the lock is released between the two methods, we can have two threads which both go through the first method (each in turn), and then the later (each in turn). I think that second thread would then lead to a duplicate class definition exception (once in findClass, I don't see what would stop it from reaching the same point, but maybe I have missed something). Maybe the ClassLoaderReflectionToolkit could expose a method which would implement this logic under a single synchronized section?

is duplicated by

JENKINS-64864 LinkageError loader attempted duplicate class definition

Resolved

JENKINS-65510 attempted duplicate class definition for name: "com/cloudbees/plugins/credentials/CredentialsMatcher$CQL"

Resolved

JENKINS-65114 Project loading fails with 3.0.1

Closed

relates to

JENKINS-46618 [JDK9] Illegal reflective access from jenkins core

Closed

JENKINS-23784 Allow parallel class loading

In Progress

JENKINS-65261 Warnings trend chart does not appear; duplicate classes in echarts-api.jar and echarts-build-trends-2.0.0.jar in ECharts API Plugin

Resolved

links to

jenkins #5550

(1 relates to, 1 links to)

Details

Description

Attachments

Issue Links

Activity

People

Dates