• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • jenkins 2.153
      bitbucket branch source plugin 2.2.15

      Branch scan is very slow when using Bitbucket source in multibranch pipeline jobs.

      For example: repo with ~20 branches,  when using Git source it took 6 seconds, when using Bitbucket source it took up to 10 minutes. Looks like it is Bitbucket API related issue. 

      According to Jenkins server log Issue begun  at 05 dec 2018. 03:00 AM (GMT+3)

      I tested it on different geographically separated environments. 

          [JENKINS-55071] Slow branch scan when using Bitbucket source

          Igor Tretyak added a comment -

          I've checked, but there is nothing else. Just a scan report after some time. 

          Igor Tretyak added a comment - I've checked, but there is nothing else. Just a scan report after some time. 

          David Kostal added a comment - - edited

          I am experiencing the same issue for cca 2-3 months already. The branch scan is fast if jenkins is down for longer time and then started, but gets slow over time. We use https access to Bitbucket with same username for all projects (and all projects' scans get slow at the same time). There is not notification in the jenkins log about API limits, just the final report like 'branch indexing action completed: SUCCESS in 13 min'

          In the extreme, project with 130 branches and 20 PRs can get as slow as 1 hour for scanning. Then jenkins restart and it "only" takes 18 minutes. When it runs as expected it needs cca 2 mins.

          David Kostal added a comment - - edited I am experiencing the same issue for cca 2-3 months already. The branch scan is fast if jenkins is down for longer time and then started, but gets slow over time. We use https access to Bitbucket with same username for all projects (and all projects' scans get slow at the same time). There is not notification in the jenkins log about API limits, just the final report like 'branch indexing action completed: SUCCESS in 13 min' In the extreme, project with 130 branches and 20 PRs can get as slow as 1 hour for scanning. Then jenkins restart and it "only" takes 18 minutes. When it runs as expected it needs cca 2 mins.

          Here's a "Me Too". It starts off quick, but then we get to stuff like this:

          branch indexing action completed: SUCCESS in 19 min

          Any suggestion on where to look for the bottleneck would be greatly appreciated.

          Slobodan Mišković added a comment - Here's a "Me Too". It starts off quick, but then we get to stuff like this: branch indexing action completed: SUCCESS in 19 min Any suggestion on where to look for the bottleneck would be greatly appreciated.

          David Kostal added a comment -

          at in my case this was really related to Bitbucket API throttling. However there was no report in Jenkins log. After switch of the biggest and most used repo to dedicated BB user, it's fast again.

          David Kostal added a comment - at in my case this was really related to Bitbucket API throttling. However there was no report in Jenkins log. After switch of the biggest and most used repo to dedicated BB user, it's fast again.

          Not sure I'll bring anything to the topic but at least I can share my "solution" for the same problem.

          In my case, we have projects with crazy amount of branches backed by mercurial which results in 3+ minutes delays.

          In practice, we need Jenkins build kick in, only for a very small subset of branches so at some point I've ended up building my own version of bitbucket-branch-source-plugin.

          Note, that I have "zero" experience in Java dev world.

          So, with below change I got rid of that nasty delay before every build and in the end it was all I needed.
          https://github.com/lukaswoj/bitbucket-branch-source-plugin/commit/549899cf8d8b0229c37a3a155b588f36b98f2c78

          This is how I build it, launched from repo root dir

          docker run -it -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:latest mvn -DskipTests package

          You should end up with HPI file generated under "target" subdirectory.

          HPI file can be installed under Upload Plugin section on "/pluginManager/advanced" page of your Jenkins installation.

           

          Hope this will be useful for someone.

          Łukasz Wojciechowski added a comment - Not sure I'll bring anything to the topic but at least I can share my "solution" for the same problem. In my case, we have projects with crazy amount of branches backed by mercurial which results in 3+ minutes delays. In practice, we need Jenkins build kick in, only for a very small subset of branches so at some point I've ended up building my own version of bitbucket-branch-source-plugin. Note, that I have "zero" experience in Java dev world. So, with below change I got rid of that nasty delay before every build and in the end it was all I needed. https://github.com/lukaswoj/bitbucket-branch-source-plugin/commit/549899cf8d8b0229c37a3a155b588f36b98f2c78 This is how I build it, launched from repo root dir docker run -it -v "$(pwd)" :/usr/src/mymaven -w /usr/src/mymaven maven:latest mvn -DskipTests package You should end up with HPI file generated under "target" subdirectory. HPI file can be installed under  Upload Plugin section on "/pluginManager/advanced" page of your Jenkins installation.   Hope this will be useful for someone.

          I can confirm this issue with Jenkins 2.172 and Bitbucket Branche Plugin 2.4.4

          However, to see the logs you need to add a new logger in the Jenkins interface. Go to "Configure Jenkins" > "System logs" and add a new logger. See my attached files.

           

          Mohamed HAIDARA added a comment - I can confirm this issue with Jenkins 2.172 and Bitbucket Branche Plugin 2.4.4 However, to see the logs you need to add a new logger in the Jenkins interface. Go to "Configure Jenkins" > "System logs" and add a new logger. See my attached files.  

          if you go into Jenkins global settings, you can enable cache for bitbucket cloud endpoint.

          You should increase your periodic scans to 8 hours or more and rely on events from Bitbucket cloud.

          lukaswoj you can achieve the same effect by adding branch filters in the behavior sections. No need to hardcode it.

          Joseph Petersen (old) added a comment - if you go into Jenkins global settings, you can enable cache for bitbucket cloud endpoint. You should increase your periodic scans to 8 hours or more and rely on events from Bitbucket cloud. lukaswoj you can achieve the same effect by adding branch filters in the behavior sections. No need to hardcode it.

          casz - can't agree with you. I still use what you referred, to limit Jenkins actions to only selected set of branches. But even then, Jenkins "blindly" asks Bitbucket API about all branches for given repository. Even worse, I believe it does only 10 rows per request (while it could ask for 100).

          Check my comment again, please.

          I am about places like this one: https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java#L454

          There is no code that modifies API calls for list of branches. In my case it takes 3-4 minutes between clicking on Build now and first Pipeline line being executed. This is why I needed a patch.

          Thanks!

          Łukasz Wojciechowski added a comment - casz - can't agree with you. I still use what you referred, to limit Jenkins actions to only selected set of branches. But even then, Jenkins "blindly" asks Bitbucket API about all branches for given repository. Even worse, I believe it does only 10 rows per request (while it could ask for 100). Check my comment again, please. I am about places like this one: https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/client/BitbucketCloudApiClient.java#L454 There is no code that modifies API calls for list of branches. In my case it takes 3-4 minutes between clicking on Build now and first Pipeline line being executed. This is why I needed a patch. Thanks!

          A potential fix has just been released (kindly provided by scddev) - could you all please verify if the issue is now resolved?

          Günter Grodotzki added a comment - A potential fix has just been released (kindly provided by scddev ) - could you all please verify if the issue is now resolved?

          Nikolas Falco added a comment -

          Is this bug still present in the latest plugin version?

          Nikolas Falco added a comment - Is this bug still present in the latest plugin version?

            nfalco Nikolas Falco
            fragpit Igor Tretyak
            Votes:
            2 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: