markewaite, we have a customer running into this with git-client 3.0.0 and git-plugin 3.12.0. The way I understand your analysis here is, an older jgit is leaking from some other plugin so git-(client-)plugin built against jgit5 is invoking signatures that was not present in the older version that is leaking in - so the fix is to get all those plugins to versions using more or less same (major) version of jgit. I tend to agree this might be the case here, as my Jenkins has loaded 2 older versions of jgit:

$ ls -l /proc/9972/fd/ | grep jgit
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 333 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.lfs-5.5.1.201910021850-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 334 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.http.apache-5.5.1.201910021850-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 337 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.http.server-5.5.1.201910021850-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 338 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit-5.5.1.201910021850-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 548 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit-4.5.4.201711221230-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 549 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit.http.server-4.5.4.201711221230-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 553 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit.http.apache-4.5.4.201711221230-r.jar
lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 764 -> /var/lib/jenkins/plugins/gitlab-plugin/WEB-INF/lib/org.eclipse.jgit-3.5.2.201411120430-r.jar

However, I was under assumption that Jenkins hierarchical classloading is here to prevent dependencies to leak from plugin to plugin (so in this case, git-plugin would get the version it requires and so will the other plugins). What makes this situation special?

Thanks for any pointers!

olivergondza sorry for the confusion caused by not having the same major version number on git client plugin and git plugin releases.

Git client 3.x bundles JGit 5.x and is intended to be used by git plugin 4.x and later. Git client 2.x bundles JGit 4.x and is intended to be used by git plugin 3.x. In the case for your customer, the preferred change is probably to switch them either from git client plugin 3.0.0 to git client plugin 2.9.0 or from git plugin 3.12.0 to git plugin 4.2.2.

You're correct that the JGit project made API breaking changes between JGit 4.x and JGit 5.x. Those API breaking changes required that we "lock-step" release the git plugin 4.x release and the git client plugin 3.x release.

I'm not expert enough with the Jenkins class loader or with Java class loading to know if it is possible to have git client plugin 3.x loaded in memory with its bundled copy of JGit 5.x and still have git plugin 3.x cause the loading of JGit 4.x into memory. The git plugin depends on git client and does not declare a dependency on JGit. The git plugin relies on the git client plugin's commitment to always provide a JGit implementation.

I think in your case that you want to either downgrade from git client plugin 3.0.0 to git client plugin 2.9.0 or upgrade from git plugin 3.12.0 to git plugin 4.2.2. In terms of functionality change, the downgrade from git client plugin 3.0.0 to git client plugin 2.9.0 is smaller than the change from git plugin 3.12.0 to git plugin 4.2.2.

Thanks, Mark. Provided both git-plugin and git-client-plugin depend on jgit passing its objects between the codebases, I doubt any amount of classloader isolation is going to save us. I have not realized it is this 2 plugins that are clashing. Thanks for your recommendations.

Hi Mark Waite

I am having a similar problem in that when i try an SCM build from a git repo i get the error below:

Jenkins v2.204.1

git version 2.23.1

java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref; at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117) at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:114) at jenkins.plugins.git.GitSCMFileSystem$3.invoke(GitSCMFileSystem.java:193) at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:29) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:80) at jenkins.plugins.git.GitSCMFileSystem.invoke(GitSCMFileSystem.java:189) at jenkins.plugins.git.GitSCMFileSystem.<init>(GitSCMFileSystem.java:114) at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:353) at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:197) at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:173) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:115) at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:69) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:309) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:427) Finished: FAILURE

This doesn't seem to match the other errors submitted exactly but pretty close. Is this again a problem between one plugin or another. Thanks Steve

Yes ciscokid, that is the message that happens when a plugin (like git plugin 3.x) attempts to access a method getRef that no longer exists in the JGit implementation that is bundled with git client plugin 3.x. Git plugin 4.x must be used with git client plugin 3.x. Git plugin 3.x is not intended to be used with git client plugin 3.x. Upgrading the called plugin (git client) from 2 to 3 without upgrading the callers (git plugin from 3 to 4, git server plugin, GitHub branch source, Gitea, Bitbucket, etc.) can lead to that error.