Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70212

Merge/push event processing takes hours for multi branch projects

    XMLWordPrintable

Details

    Description

      We recently moved from Bitbucket to GitLab (both on-premise installations) and we encounter a huge delay when processing GitLabPushSCMEvent and GitLabMergeRequestSCMEvent events.

      The problem seems to be related to one of the repositories which has over 5500 branches and additionally relates to 10 multibranch pipeline projects (10 different Jenkinsfiles inside the repo) in Jenkins.

      When an event is processed, Jenkins seems to iterate over all related projects, collecting all branches through the GitLab branch source plugin using REST.

      While our setup for sure is far from optimal, processing the event still takes a lot longer than I would expect. I can see up to 10 parallel event handler threads "hanging" in  socketRead() for over 4 hours.

      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#3]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:41:10 MEZ 2022 / SCMEvent [#5]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:41:10 MEZ 2022 / SCMEvent [#6]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:46 MEZ 2022 / SCMEvent [#4]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:46 MEZ 2022 / SCMEvent [#7]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:54 MEZ 2022 / SCMEvent [#1]
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:54 MEZ 2022 / SCMEvent [#2]
      class io.jenkins.plugins.gitlabbranchsource.GitLabPushSCMEvent Mo. Dez. 05 14:40:41 MEZ 2022 / SCMEvent [#8]
      class io.jenkins.plugins.gitlabbranchsource.GitLabPushSCMEvent Mo. Dez. 05 14:40:41 MEZ 2022 / SCMEvent [#9] 
      
      class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10]
      
      java.base@11.0.16/java.net.SocketInputStream.socketRead0(Native Method)
      java.base@11.0.16/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
      java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:168)
      java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:140)
      java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
      java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
      java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
      java.base@11.0.16/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
      java.base@11.0.16/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
      java.base@11.0.16/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
      java.base@11.0.16/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
      java.base@11.0.16/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
      java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:788)
      java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:723)
      java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
      java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
      java.base@11.0.16/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
      java.base@11.0.16/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
      org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:380)
      org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:268)
      org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
      org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
      org.glassfish.jersey.client.JerseyInvocation$$Lambda$850/0x00000008413b8840.call(Unknown Source)
      org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
      org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
      org.glassfish.jersey.client.JerseyInvocation$$Lambda$851/0x00000008413b8c40.call(Unknown Source)
      org.glassfish.jersey.internal.Errors.process(Errors.java:292)
      org.glassfish.jersey.internal.Errors.process(Errors.java:274)
      org.glassfish.jersey.internal.Errors.process(Errors.java:205)
      org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
      org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
      org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
      org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413)
      org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313)
      org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:416)
      org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:404)
      org.gitlab4j.api.AbstractApi.get(AbstractApi.java:214)
      org.gitlab4j.api.Pager.page(Pager.java:327)
      org.gitlab4j.api.Pager.next(Pager.java:241)
      org.gitlab4j.api.Pager.all(Pager.java:357)
      org.gitlab4j.api.RepositoryApi.getBranches(RepositoryApi.java:51)
      io.jenkins.plugins.gitlabbranchsource.GitLabSCMSource.retrieve(GitLabSCMSource.java:319)
      jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
      jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
      jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1587)
      jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1191)
      jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
      jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
      jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:546)
      jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
      java.base@11.0.16/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
      java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      java.base@11.0.16/java.lang.Thread.run(Thread.java:829) 

      (this thread, started by the merge event at 14:39:51 MEZ, is still being processed at 18:30:00 MEZ)

      In the GitLab log I can see that requesting all branches/tags takes about 1:30 minutes

       

      Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=1" for xxx.xxx.xxxx.xxxx at 2022-12-05 18:12:55 +0100
      ....
      Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=58" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100
      ...
      Started GET "/api/v4/projects/169/repository/tags?per_page=96&page=1" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100
      ...
      Started GET "/api/v4/projects/169/repository/branches/master" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:23 +0100

      So I would assume that processing all 10 projects should take about 15 Minutes.

       

      Is this a bug or something to expect when using such a setup?

      Is there anything I could do to increase processing speed, e.g. increasing number of threads for event processing?

       

      (side note: when using Bitbucket (Bitbucket Branch Source Plugin), we did not encounter such a problem, but I do understand that the implementaion for obtaining the branches might be very different)

       

      Attachments

        Activity

          gordin Christoph Vogtländer created issue -
          gordin Christoph Vogtländer made changes -
          Field Original Value New Value
          Description We recently moved from Bitbucket to GitLab (both on-premise installations) and we encounter a huge delay when processing GitLabPushSCMEvent and GitLabMergeRequestSCMEvent events.

          The problem seems to be related to one of the repositories which has over 5500 branches and additionally relates to 10 multibranch pipeline projects (10 different Jenkinsfiles inside the repo) in Jenkins.

          When an event is processed, Jenkins seems to iterate over all related projects, collecting all branches through the GitLab branch source plugin using REST.

          While our setup for sure is far from optimal, processing the event still takes a lot longer than I would expect. I can see up to 10 parallel event handler threads "hanging" in  socketRead() for over 4 hours.

           
          {code:java}
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10]

          java.base@11.0.16/java.net.SocketInputStream.socketRead0(Native Method)
          java.base@11.0.16/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
          java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:168)
          java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:140)
          java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
          java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
          java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
          java.base@11.0.16/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
          java.base@11.0.16/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
          java.base@11.0.16/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
          java.base@11.0.16/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
          java.base@11.0.16/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
          java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:788)
          java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:723)
          java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
          java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
          java.base@11.0.16/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
          java.base@11.0.16/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
          org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:380)
          org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:268)
          org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
          org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
          org.glassfish.jersey.client.JerseyInvocation$$Lambda$850/0x00000008413b8840.call(Unknown Source)
          org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
          org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
          org.glassfish.jersey.client.JerseyInvocation$$Lambda$851/0x00000008413b8c40.call(Unknown Source)
          org.glassfish.jersey.internal.Errors.process(Errors.java:292)
          org.glassfish.jersey.internal.Errors.process(Errors.java:274)
          org.glassfish.jersey.internal.Errors.process(Errors.java:205)
          org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
          org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
          org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
          org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413)
          org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313)
          org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:416)
          org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:404)
          org.gitlab4j.api.AbstractApi.get(AbstractApi.java:214)
          org.gitlab4j.api.Pager.page(Pager.java:327)
          org.gitlab4j.api.Pager.next(Pager.java:241)
          org.gitlab4j.api.Pager.all(Pager.java:357)
          org.gitlab4j.api.RepositoryApi.getBranches(RepositoryApi.java:51)
          io.jenkins.plugins.gitlabbranchsource.GitLabSCMSource.retrieve(GitLabSCMSource.java:319)
          jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
          jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
          jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1587)
          jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1191)
          jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
          jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
          jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:546)
          jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
          java.base@11.0.16/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          java.base@11.0.16/java.lang.Thread.run(Thread.java:829) {code}
          (this thread, started by the merge event at 14:39:51 MEZ, is still being processed at 18:30:00 MEZ)

          In the GitLab log I can see that requesting all branches/tags takes about 1:30 minutes

           
          {code:java}
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=1" for xxx.xxx.xxxx.xxxx at 2022-12-05 18:12:55 +0100
          ....
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=58" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100
          ...
          Started GET "/api/v4/projects/169/repository/tags?per_page=96&page=1" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100
          ...
          Started GET "/api/v4/projects/169/repository/branches/master" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:23 +0100{code}
          So I would assume that processing all 10 projects should take about 15 Minutes.

           

          Is this a bug or something to expect when using such a setup?

          Is there anything I could do to increase processing speed, e.g. increasing number of threads for event processing?

           

          (side note: when using Bitbucket (Bitbucket Branch Source Plugin), we did not encounter such a problem, but I do understand that the implementaion for obtaining the branches might be very different)

           
          We recently moved from Bitbucket to GitLab (both on-premise installations) and we encounter a huge delay when processing GitLabPushSCMEvent and GitLabMergeRequestSCMEvent events.

          The problem seems to be related to one of the repositories which has over 5500 branches and additionally relates to 10 multibranch pipeline projects (10 different Jenkinsfiles inside the repo) in Jenkins.

          When an event is processed, Jenkins seems to iterate over all related projects, collecting all branches through the GitLab branch source plugin using REST.

          While our setup for sure is far from optimal, processing the event still takes a lot longer than I would expect. I can see up to 10 parallel event handler threads "hanging" in  socketRead() for over 4 hours.
          {code:java}
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#3]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:41:10 MEZ 2022 / SCMEvent [#5]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:41:10 MEZ 2022 / SCMEvent [#6]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:46 MEZ 2022 / SCMEvent [#4]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:46 MEZ 2022 / SCMEvent [#7]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:54 MEZ 2022 / SCMEvent [#1]
          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:47:54 MEZ 2022 / SCMEvent [#2]
          class io.jenkins.plugins.gitlabbranchsource.GitLabPushSCMEvent Mo. Dez. 05 14:40:41 MEZ 2022 / SCMEvent [#8]
          class io.jenkins.plugins.gitlabbranchsource.GitLabPushSCMEvent Mo. Dez. 05 14:40:41 MEZ 2022 / SCMEvent [#9] {code}
          {code:java}

          class io.jenkins.plugins.gitlabbranchsource.GitLabMergeRequestSCMEvent Mo. Dez. 05 14:39:51 MEZ 2022 / SCMEvent [#10]

          java.base@11.0.16/java.net.SocketInputStream.socketRead0(Native Method)
          java.base@11.0.16/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
          java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:168)
          java.base@11.0.16/java.net.SocketInputStream.read(SocketInputStream.java:140)
          java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
          java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
          java.base@11.0.16/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
          java.base@11.0.16/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1454)
          java.base@11.0.16/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1065)
          java.base@11.0.16/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
          java.base@11.0.16/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
          java.base@11.0.16/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
          java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:788)
          java.base@11.0.16/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:723)
          java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
          java.base@11.0.16/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
          java.base@11.0.16/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
          java.base@11.0.16/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
          org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:380)
          org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:268)
          org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
          org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
          org.glassfish.jersey.client.JerseyInvocation$$Lambda$850/0x00000008413b8840.call(Unknown Source)
          org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
          org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
          org.glassfish.jersey.client.JerseyInvocation$$Lambda$851/0x00000008413b8c40.call(Unknown Source)
          org.glassfish.jersey.internal.Errors.process(Errors.java:292)
          org.glassfish.jersey.internal.Errors.process(Errors.java:274)
          org.glassfish.jersey.internal.Errors.process(Errors.java:205)
          org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
          org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
          org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
          org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413)
          org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313)
          org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:416)
          org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:404)
          org.gitlab4j.api.AbstractApi.get(AbstractApi.java:214)
          org.gitlab4j.api.Pager.page(Pager.java:327)
          org.gitlab4j.api.Pager.next(Pager.java:241)
          org.gitlab4j.api.Pager.all(Pager.java:357)
          org.gitlab4j.api.RepositoryApi.getBranches(RepositoryApi.java:51)
          io.jenkins.plugins.gitlabbranchsource.GitLabSCMSource.retrieve(GitLabSCMSource.java:319)
          jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
          jenkins.scm.api.SCMSource.fetch(SCMSource.java:327)
          jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1587)
          jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1191)
          jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)
          jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)
          jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:546)
          jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)
          java.base@11.0.16/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          java.base@11.0.16/java.lang.Thread.run(Thread.java:829) {code}
          (this thread, started by the merge event at 14:39:51 MEZ, is still being processed at 18:30:00 MEZ)

          In the GitLab log I can see that requesting all branches/tags takes about 1:30 minutes

           
          {code:java}
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=1" for xxx.xxx.xxxx.xxxx at 2022-12-05 18:12:55 +0100
          ....
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=58" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100
          ...
          Started GET "/api/v4/projects/169/repository/tags?per_page=96&page=1" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:11 +0100
          ...
          Started GET "/api/v4/projects/169/repository/branches/master" for xxx.xxx.xxx.xxx at 2022-12-05 18:14:23 +0100{code}
          So I would assume that processing all 10 projects should take about 15 Minutes.

           

          Is this a bug or something to expect when using such a setup?

          Is there anything I could do to increase processing speed, e.g. increasing number of threads for event processing?

           

          (side note: when using Bitbucket (Bitbucket Branch Source Plugin), we did not encounter such a problem, but I do understand that the implementaion for obtaining the branches might be very different)

           

          Another thing I just noticed which seems very weird: it seems that all event threads are executing the exact same API call at the same time. For testing I startet a second Jenkins instance that is then handling the same events. Even this new instance is then using the exact same request at the exact same time. This seems wrong to me:

          ...
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:03 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:04 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:04 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:04 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:04 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:06 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:06 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:07 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:07 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:07 +0100
          Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:07 +0100
          ...
          

          Is there any explanation for this behaviour?

          gordin Christoph Vogtländer added a comment - Another thing I just noticed which seems very weird: it seems that all event threads are executing the exact same API call at the same time. For testing I startet a second Jenkins instance that is then handling the same events. Even this new instance is then using the exact same request at the exact same time. This seems wrong to me: ... Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:03 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:04 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:04 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:04 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:04 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=25" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:05 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:06 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:06 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:07 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:07 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for yyy.yyy.yyy.yyy at 2022-12-06 22:25:07 +0100 Started GET "/api/v4/projects/169/repository/branches?per_page=96&page=26" for xxx.xxx.xxx.xxx at 2022-12-06 22:25:07 +0100 ... Is there any explanation for this behaviour?

          People

            baymac Parichay Barpanda
            gordin Christoph Vogtländer
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: