-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
Jenkins 2.361.4
GitLab Branch Source Version 642.v9ed86b_b_54384
We recently moved from Bitbucket to GitLab (both on-premise installations) and we encounter a huge delay when processing GitLabPushSCMEvent and GitLabMergeRequestSCMEvent events.
The problem seems to be related to one of the repositories which has over 5500 branches and additionally relates to 10 multibranch pipeline projects (10 different Jenkinsfiles inside the repo) in Jenkins.
When an event is processed, Jenkins seems to iterate over all related projects, collecting all branches through the GitLab branch source plugin using REST.
While our setup for sure is far from optimal, processing the event still takes a lot longer than I would expect. I can see up to 10 parallel event handler threads "hanging" in socketRead() for over 4 hours.
class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#3] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:41:10 MEZ 2022 / SCMEvent [#5] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:41:10 MEZ 2022 / SCMEvent [#6] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:46 MEZ 2022 / SCMEvent [#4] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:46 MEZ 2022 / SCMEvent [#7] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:54 MEZ 2022 / SCMEvent [#1] class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:54 MEZ 2022 / SCMEvent [#2] class io.jenkins.plugins.gitlabbranchsource.GitLabPushSCMEvent Mo. Dez. 05 14:40:41 MEZ 2022 / SCMEvent [#8] class io.jenkins.plugins.gitlabbranchsource.GitLabPushSCMEvent Mo. Dez. 05 14:40:41 MEZ 2022 / SCMEvent [#9]
class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10] java.base@11.0.16/java.net.SocketInputStream.socketRead0(Native Method) java.base@11.0.16/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:168) java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:140) java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478) java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472) java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) java.base@11.0.16/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454) java.base@11.0.16/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065) java.base@11.0.16/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) java.base@11.0.16/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) java.base@11.0.16/java.io.BufferedInputStream.read(BufferedInputStream.java:351) java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:788) java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:723) java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615) java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) java.base@11.0.16/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527) java.base@11.0.16/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334) org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:380) org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:268) org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662) org.glassfish.jersey.client.JerseyInvocation$$Lambda$850/0x00000008413b8840.call(Unknown Source) org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) org.glassfish.jersey.client.JerseyInvocation$$Lambda$851/0x00000008413b8c40.call(Unknown Source) org.glassfish.jersey.internal.Errors.process(Errors.java:292) org.glassfish.jersey.internal.Errors.process(Errors.java:274) org.glassfish.jersey.internal.Errors.process(Errors.java:205) org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661) org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413) org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313) org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:416) org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:404) org.gitlab4j.api.AbstractApi.get(AbstractApi.java:214) org.gitlab4j.api.Pager.page(Pager.java:327) org.gitlab4j.api.Pager.next(Pager.java:241) org.gitlab4j.api.Pager.all(Pager.java:357) org.gitlab4j.api.RepositoryApi.getBranches(RepositoryApi.java:51) io.jenkins.plugins.gitlabbranchsource.GitLabSCMSource.retrieve(GitLabSCMSource.java:319) jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) jenkins.scm.api.SCMSource.fetch(SCMSource.java:327) jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1587) jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1191) jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246) jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229) jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:546) jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69) java.base@11.0.16/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
(this thread, started by the merge event at 14:39:51 MEZ, is still being processed at 18:30:00 MEZ)
In the GitLab log I can see that requesting all branches/tags takes about 1:30 minutes
Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=1" for xxx.xxx.xxxx.xxxx at 2022-12-05 18:12:55 +0100 .... Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=58" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100 ... Started GET "/api/v4/projects/169/repository/tags?per_page=96&page=1" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100 ... Started GET "/api/v4/projects/169/repository/branches/master" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:23 +0100
So I would assume that processing all 10 projects should take about 15 Minutes.
Is this a bug or something to expect when using such a setup?
Is there anything I could do to increase processing speed, e.g. increasing number of threads for event processing?
(side note: when using Bitbucket (Bitbucket Branch Source Plugin), we did not encounter such a problem, but I do understand that the implementaion for obtaining the branches might be very different)