Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62479

Rate limit reached for Bitbucket server

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have a repository with ~120+ opened PRs and >2000 branches at own Bitbucket server (not a cloud instance) which is scanned for Jenkins files.

      The plugin hits the rate limit for Bitbucket every time it scans the repository for Jenkinsfiles. For cloud instances a fix was implemented and merged in scope of JENKINS-42458

      We need the same fix for on-premise instances.

      java.io.IOException: Communication error for url: Jenkinsfile status code: 429
      	at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:493)
      	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1478)
      	at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99)
      	at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
      	at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
      	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrievePullRequests(BitbucketSCMSource.java:675)
      	at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:593)
      	at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
      	at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
      	at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
      	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
      	at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
      	at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:428)
      

        Attachments

          Issue Links

            Activity

            gnome Dzmitry Moisa created issue -
            gnome Dzmitry Moisa made changes -
            Field Original Value New Value
            Priority Critical [ 2 ] Major [ 3 ]
            gnome Dzmitry Moisa made changes -
            Environment Jenkins 2.32.3
            Bitbucket-branch-source-plugin 2.1.0 (working version 1.9)
            Jenkins 2.222.1
            Bitbucket-branch-source-plugin 2.7.0
            Hide
            amuniz Antonio Muñiz added a comment -

            Latest release is 2.8.0, did you try it?

            Show
            amuniz Antonio Muñiz added a comment - Latest release is 2.8.0, did you try it?
            Hide
            amuniz Antonio Muñiz added a comment -

            Rate limit support added in 2.2.4.

            Show
            amuniz Antonio Muñiz added a comment - Rate limit support added in 2.2.4.
            amuniz Antonio Muñiz made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            amuniz Antonio Muñiz made changes -
            Resolution Fixed [ 1 ]
            Status Fixed but Unreleased [ 10203 ] Reopened [ 4 ]
            Hide
            amuniz Antonio Muñiz added a comment -

            Reopening, rate limit support is implemented only for Bitbucket Cloud, not Server.

            Show
            amuniz Antonio Muñiz added a comment - Reopening, rate limit support is implemented only for Bitbucket Cloud, not Server.
            amuniz Antonio Muñiz made changes -
            Assignee Antonio Muñiz [ amuniz ]
            gnome Dzmitry Moisa made changes -
            Description We have a team repository with about 300 repositories which is scanned for jenkins files.

            We updated Bitbucket-branch-source-plugin to 2.1.0 (working version 1.9) yesterday and now we hit the rate limit every time.

            This means that repositories that are last in the arbitrarily sorted list is never added. We think that this plugin should know about the rate limit and throttle it self. It should always handle newly updated repositories first.

            The best solution would be if Atlasian created a webhook in the team resource for repository creation. But we need a workaround in the mean time.

            {code:java}
            [Fri Mar 03 05:29:24 UTC 2017] Finished organization scan. Scan took 8 min 23 sec
            FATAL: Failed to recompute children of {Removed Job name}
            com.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 429: Unknown Status Code.
            Rate limit for this resource has been exceeded
            at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRequest(BitbucketCloudApiClient.java:568)
            at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.getRepository(BitbucketCloudApiClient.java:232)
            at com.cloudbees.jenkins.plugins.bitbucket.client.BitbucketCloudApiClient.isPrivate(BitbucketCloudApiClient.java:405)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrievePullRequests(BitbucketSCMSource.java:334)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:325)
            at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:300)
            at jenkins.scm.api.SCMSource.fetch(SCMSource.java:254)
            at jenkins.branch.MultiBranchProjectFactory$BySCMSourceCriteria.recognizes(MultiBranchProjectFactory.java:263)
            at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.recognizes(OrganizationFolder.java:1266)
            at jenkins.branch.OrganizationFolder$SCMSourceObserverImpl$1.complete(OrganizationFolder.java:1281)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.add(BitbucketSCMNavigator.java:212)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator.visitSources(BitbucketSCMNavigator.java:187)
            at jenkins.branch.OrganizationFolder.computeChildren(OrganizationFolder.java:399)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:219)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:154)
            at jenkins.branch.OrganizationFolder$OrganizationScan.run(OrganizationFolder.java:850)
            at hudson.model.ResourceController.execute(ResourceController.java:98)
            at hudson.model.Executor.run(Executor.java:404)
            Finished: FAILURE
            {code}
            We have a repository with ~120+ opened PRs and >2000 branches at own Bitbucket server (not a cloud instance) which is scanned for Jenkins files.

            The plugin hits the rate limit for Bitbucket every time it scans the repository for Jenkinsfiles. For cloud instances a fix was implemented and merged in scope of JENKINS-42458

            We need the same fix for in-premise instances.
            {code:java}
            java.io.IOException: Communication error for url: Jenkinsfile status code: 429
            at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:493)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1478)
            at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99)
            at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
            at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrievePullRequests(BitbucketSCMSource.java:675)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:593)
            at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
            at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:428)
            {code}
            Hide
            gnome Dzmitry Moisa added a comment -

            Antonio Muñiz yes, I've updated the description and exception stack with the actual information.

            Show
            gnome Dzmitry Moisa added a comment - Antonio Muñiz yes, I've updated the description and exception stack with the actual information.
            gnome Dzmitry Moisa made changes -
            Description We have a repository with ~120+ opened PRs and >2000 branches at own Bitbucket server (not a cloud instance) which is scanned for Jenkins files.

            The plugin hits the rate limit for Bitbucket every time it scans the repository for Jenkinsfiles. For cloud instances a fix was implemented and merged in scope of JENKINS-42458

            We need the same fix for in-premise instances.
            {code:java}
            java.io.IOException: Communication error for url: Jenkinsfile status code: 429
            at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:493)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1478)
            at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99)
            at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
            at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrievePullRequests(BitbucketSCMSource.java:675)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:593)
            at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
            at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:428)
            {code}
            We have a repository with ~120+ opened PRs and >2000 branches at own Bitbucket server (not a cloud instance) which is scanned for Jenkins files.

            The plugin hits the rate limit for Bitbucket every time it scans the repository for Jenkinsfiles. For cloud instances a fix was implemented and merged in scope of JENKINS-42458

            We need the same fix for on-premise instances.
            {code:java}
            java.io.IOException: Communication error for url: Jenkinsfile status code: 429
            at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:493)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1478)
            at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99)
            at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
            at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrievePullRequests(BitbucketSCMSource.java:675)
            at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:593)
            at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
            at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
            at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:634)
            at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:277)
            at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:164)
            at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1025)
            at hudson.model.ResourceController.execute(ResourceController.java:97)
            at hudson.model.Executor.run(Executor.java:428)
            {code}
            Show
            amuniz Antonio Muñiz added a comment - BTW: https://github.com/jenkinsci/bitbucket-branch-source-plugin/issues/288
            Hide
            smirky Bogomil Vasilev added a comment -

            I can confirm we hit this every time with up-to-date plugins and Jenkins:

            ERROR: [Thu Jul 30 10:08:02 CEST 2020] Could not fetch branches from source fb32e1e7-96a7-4001-8fd0-fcc97610aea6
            java.io.IOException: Communication error for url: devops/jenkins/Jenkinsfile status code: 429
             at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:519)
             at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1477)
             at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99)
             at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
             at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
             at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieveBranches(BitbucketSCMSource.java:747)
             at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:589)
             at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
             at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
             at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641)
             at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
             at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
             at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
             at hudson.model.ResourceController.execute(ResourceController.java:97)
             at hudson.model.Executor.run(Executor.java:428)
            [Thu Jul 30 10:08:02 CEST 2020] Finished branch indexing. Indexing took 1.7 sec
            FATAL: Failed to recompute children of ***********
            java.io.IOException: Communication error for url: devops/jenkins/Jenkinsfile status code: 429
             at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:519)
             at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1477)
             at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99)
             at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74)
             at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342)
             at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieveBranches(BitbucketSCMSource.java:747)
             at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:589)
             at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373)
             at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283)
             at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641)
             at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278)
             at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165)
             at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032)
             at hudson.model.ResourceController.execute(ResourceController.java:97)
             at hudson.model.Executor.run(Executor.java:428)
            Finished: FAILURE
            

            Could we please get a fix, as it is definitely out of the question to preserve a certain count of branches without a solid reason?

            Show
            smirky Bogomil Vasilev added a comment - I can confirm we hit this every time with up-to-date plugins and Jenkins: ERROR: [Thu Jul 30 10:08:02 CEST 2020] Could not fetch branches from source fb32e1e7-96a7-4001-8fd0-fcc97610aea6 java.io.IOException: Communication error for url: devops/jenkins/Jenkinsfile status code: 429 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:519) at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1477) at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99) at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74) at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342) at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieveBranches(BitbucketSCMSource.java:747) at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:589) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165) at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428) [Thu Jul 30 10:08:02 CEST 2020] Finished branch indexing. Indexing took 1.7 sec FATAL: Failed to recompute children of *********** java.io.IOException: Communication error for url: devops/jenkins/Jenkinsfile status code: 429 at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.checkPathExists(BitbucketServerAPIClient.java:519) at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource$BitbucketProbeFactory$1.exists(BitbucketSCMSource.java:1477) at jenkins.scm.api.SCMSourceCriteria$Probe.stat(SCMSourceCriteria.java:99) at org.jenkinsci.plugins.workflow.multibranch.WorkflowBranchProjectFactory$1.isHead(WorkflowBranchProjectFactory.java:74) at jenkins.scm.api.trait.SCMSourceRequest.process(SCMSourceRequest.java:342) at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieveBranches(BitbucketSCMSource.java:747) at com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMSource.retrieve(BitbucketSCMSource.java:589) at jenkins.scm.api.SCMSource._retrieve(SCMSource.java:373) at jenkins.scm.api.SCMSource.fetch(SCMSource.java:283) at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:641) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:278) at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:165) at jenkins.branch.MultiBranchProject$BranchIndexing.run(MultiBranchProject.java:1032) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428) Finished: FAILURE Could we please get a fix, as it is definitely out of the question to preserve a certain count of branches without a solid reason?
            Hide
            shankar128 Shankar Ramasamy added a comment -

            I have added a fix on this PR.
            When ratelimit on Bitbucket server is encountered for any requests, the same API call is retried after a short wait time.

            Show
            shankar128 Shankar Ramasamy added a comment - I have added a fix on this PR. When ratelimit on Bitbucket server is encountered for any requests, the same API call is retried after a short wait time.
            kon Kalle Niemitalo made changes -
            Link This issue relates to JENKINS-64418 [ JENKINS-64418 ]
            kon Kalle Niemitalo made changes -
            Link This issue is related to JENKINS-64418 [ JENKINS-64418 ]
            kon Kalle Niemitalo made changes -
            Link This issue relates to JENKINS-64418 [ JENKINS-64418 ]
            Hide
            kon Kalle Niemitalo added a comment -

            From Improving instance stability with rate limiting | Bitbucket Data Center and Server 7.16 | Atlassian Documentation, it looks like rate limiting requires a Bitbucket Data Center license, so the error won't occur if the server has only a Bitbucket Server license.

            Show
            kon Kalle Niemitalo added a comment - From Improving instance stability with rate limiting | Bitbucket Data Center and Server 7.16 | Atlassian Documentation , it looks like rate limiting requires a Bitbucket Data Center license, so the error won't occur if the server has only a Bitbucket Server license.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              gnome Dzmitry Moisa
              Votes:
              5 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated: