Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47891

Bitbucket branch source plugin does not understand the new Bitbucket Server webhooks

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • bitbucket-branch-source-plugin
    • None
    • bitbucket-branch-source 2.2.4
      Bitbucket Server 5.5.0

      The latest Bitbucket Server version, 5.5, has built-in support for webhooks (as opposed to the plugin previously providing this functionality).

      For some reason, the built-in webhook does not send the same event keys as before, specifically it sets `X-Event-Key: repo:refs_changed` instead of `X-Event-Key: repo:push`, which the plugin expects.

      See https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html - apparently there are other changes to the payloads as well compared to Bitbucket Cloud.

      The latest comments on JENKINS-28877 touch on this issue as well.

      Example error message in the Bitbucket log:

      java.lang.Exception: X-Event-Key HTTP header invalid: repo:refs_changed 
   at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) 
   at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:99) 
   at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) 
   at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) 
   at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) 
   at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) 
   at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) 
   at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) 
   at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) 
   at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) 
   at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) 
   at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) 
   at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) 
   at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) 
   at org.kohsuke.stapler.Stapler.service(Stapler.java:238) 
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) 
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) 
   at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
   at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
   at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
   at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
   at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
   at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232) 
   at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209) 
   at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) 
   at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113) 
   at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
   at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
   at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.process(BitbucketSCMSourcePushHookReceiver.java:70) 
   at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:59) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) 
   at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) 
   at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) 
   at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
   at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) 
   at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
   at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
   at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
   at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) 
   at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) 
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
   at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) 
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
   at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) 
   at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) 
   at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) 
   at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) 
   at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) 
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) 
   at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) 
   at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) 
   at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) 
   at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
   at org.eclipse.jetty.server.Server.handle(Server.java:564) 
   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) 
   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) 
   at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) 
   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) 
   at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) 
   at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) 
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
   at java.lang.Thread.run(Thread.java:748)

       

        1. native_webhooks.png
          native_webhooks.png
          43 kB

          [JENKINS-47891] Bitbucket branch source plugin does not understand the new Bitbucket Server webhooks

          Tom Wieczorek added a comment -

          I'm currently working on native webhook support for Bitbucket Server here: https://github.com/jenkinsci/bitbucket-branch-source-plugin/compare/master...yieldlab:native-webhooks

          Not yet complete, hence no open Pull Request, but it's partially working already. Maybe this helps already.

          Tom Wieczorek added a comment - I'm currently working on native webhook support for Bitbucket Server here: https://github.com/jenkinsci/bitbucket-branch-source-plugin/compare/master...yieldlab:native-webhooks Not yet complete, hence no open Pull Request, but it's partially working already. Maybe this helps already.

          Joseph Petersen (old) added a comment - - edited

          twz123 would actually be great if you would open a PR cause I definitely have some input  the changes look good! Few minor tweaks I would do.

          For now please stop misusing jelly enums. You can do this a lot better with a field and descriptor with doFill.

          See here for a good example use: 

          https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/java/org/jenkinsci/plugins/jvctb/config/ViolationConfig.java#L120-L127

          https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/resources/org/jenkinsci/plugins/jvctb/config/ViolationConfig/config.jelly#L5-L7

           

          Joseph Petersen (old) added a comment - - edited twz123 would actually be great if you would open a PR cause I definitely have some input  the changes look good! Few minor tweaks I would do. For now please stop misusing jelly enums. You can do this a lot better with a field and descriptor with doFill. See here for a good example use:  https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/java/org/jenkinsci/plugins/jvctb/config/ViolationConfig.java#L120-L127 https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/resources/org/jenkinsci/plugins/jvctb/config/ViolationConfig/config.jelly#L5-L7  

          Tom Wieczorek added a comment -

          Pull request #138 opened. Looking forward to get your feedback casz

          Tom Wieczorek added a comment - Pull request #138 opened. Looking forward to get your feedback casz

          Andy Airey added a comment -

          FYI, my company uses Bitbucket Server. With several teams asking about webhook support for Jenkins and speculation running wild, our Engineering Tools manager sought an official answer from Atlassian. Here is his report:

          I've been in contact with Atlassian Premier Support and the lead developers of Bitbucket, who reviewed JENKINS-47891 at my request. They confirm that there are differences in the web hook behaviors between Server and Cloud. This is because although the two products share a name (Bitbucket) and general purpose (remote Git repository management), they serve different customer needs and their code bases are distinct. They said that making Bitbucket Server and Bitbucket Cloud to be exactly the same is not currently planned. From their perspective, Bitbucket Server web hooks are working as designed and customizing the functionality to meet the specific needs of individual third party applications is not on the roadmap.

          Andy Airey added a comment - FYI, my company uses Bitbucket Server. With several teams asking about webhook support for Jenkins and speculation running wild, our Engineering Tools manager sought an official answer from Atlassian. Here is his report: I've been in contact with Atlassian Premier Support and the lead developers of Bitbucket, who reviewed JENKINS-47891 at my request. They confirm that there are differences in the web hook behaviors between Server and Cloud. This is because although the two products share a name (Bitbucket) and general purpose (remote Git repository management), they serve different customer needs and their code bases are distinct . They said that making Bitbucket Server and Bitbucket Cloud to be exactly the same is not currently planned. From their perspective, Bitbucket Server web hooks are working as designed and customizing the functionality to meet the specific needs of individual third party applications is not on the roadmap.

          Jim Klimov added a comment - - edited

          The PR has now been completed, merged and made its way into the new release of the plugin.

          This worked nicely for our Multi-Branch Pipelines generated via Bitbucket team folder "jobs" that are the one item we configure "manually" to get our repos and branches discovered to spawn the pipeline jobs. A rescan of the organization "job" in Jenkins (after setting the global config to use the "Native" webhook implementation in the connector for our BB server) made configurations in all repos that had a Jenkinsfile in at least one branch, but no configurations in repos that had no such branches (corollary: to initially introduce pipeline support in codebase of a component, one has to rescan that organization or MBP job).

          One caveat I hit in our local setup (constrained by a web of corporate IT policies) was that our lab's Jenkins server uses a self-signed certificate which the centrally managed corporate Bitbucket Server instance reasonably does not trust, so it can not HTTP POST to. Our other consumers (developers' browsers and internal build farm) have no issues with trusting this certificate.

          I had to change the (apache httpd) reverse proxy handling to let it through as plaintext:

          RedirectMatch 302 ^/?((?!neo.*$|bitbucket-scmsource-hook$|bitbucket-scmsource-hook/.*$).*)$ https://jenkins.domain.com/$1
ProxyPass         /bitbucket-scmsource-hook  http://jenkins.domain.com:8080/bitbucket-scmsource-hook/ nocanon connectiontimeout=3600 timeout=3600 retry=0

          ...and had to change the Jenkins global config to announce the server URL as the "http" (non-"https") version. UPDATE: Added RFE JENKINS-55423 for a better solution to this sort of problem.

          Thanks to everybody involved in making this feature!

          Jim Klimov added a comment - - edited The PR has now been completed, merged and made its way into the new release of the plugin. This worked nicely for our Multi-Branch Pipelines generated via Bitbucket team folder "jobs" that are the one item we configure "manually" to get our repos and branches discovered to spawn the pipeline jobs. A rescan of the organization "job" in Jenkins (after setting the global config to use the "Native" webhook implementation in the connector for our BB server) made configurations in all repos that had a Jenkinsfile in at least one branch, but no configurations in repos that had no such branches (corollary: to initially introduce pipeline support in codebase of a component, one has to rescan that organization or MBP job). One caveat I hit in our local setup (constrained by a web of corporate IT policies) was that our lab's Jenkins server uses a self-signed certificate which the centrally managed corporate Bitbucket Server instance reasonably does not trust, so it can not HTTP POST to. Our other consumers (developers' browsers and internal build farm) have no issues with trusting this certificate. I had to change the (apache httpd) reverse proxy handling to let it through as plaintext: RedirectMatch 302 ^/?((?!neo.*$|bitbucket-scmsource-hook$|bitbucket-scmsource-hook/.*$).*)$ https: //jenkins.domain.com/$1 ProxyPass /bitbucket-scmsource-hook http: //jenkins.domain.com:8080/bitbucket-scmsource-hook/ nocanon connectiontimeout=3600 timeout=3600 retry=0 ...and had to change the Jenkins global config to announce the server URL as the "http" (non-"https") version. UPDATE: Added RFE JENKINS-55423 for a better solution to this sort of problem. Thanks to everybody involved in making this feature!

          Peter Leibiger added a comment -

          I can't get native hooks to be auto registered, is there something special needed compared to the plugin?

          Plugin has always worked fine but does not (yet) support Bitbucket Server 6.0.0

          Peter Leibiger added a comment - I can't get native hooks to be auto registered, is there something special needed compared to the plugin? Plugin has always worked fine but does not (yet) support Bitbucket Server 6.0.0

          Peter Leibiger added a comment -

          After adding a logger in Jenkins for com.cloudbees.jenkins.plugins.bitbucket I figured out that the configured user did not have Admin access for those repositories/projects in Bitbucket. This was not required with the plugin and is not easy to figure out. It would be nice to have some warning message in the scan log for this case. 

          Peter Leibiger added a comment - After adding a logger in Jenkins for com.cloudbees.jenkins.plugins.bitbucket I figured out that the configured user did not have Admin access for those repositories/projects in Bitbucket. This was not required with the plugin and is not easy to figure out. It would be nice to have some warning message in the scan log for this case. 

          Torsten Kleiber added a comment -

          I'm on bitbucket-branch-source 2.4.1 and Bitbucket Server 5.16. Native Webhooks does not work for Pull requests here:

          Torsten Kleiber added a comment - I'm on bitbucket-branch-source 2.4.1 and Bitbucket Server 5.16. Native Webhooks does not work for Pull requests here:

          Kalle Niemitalo added a comment -

          tkleiber wrote:

          Native Webhooks does not work for Pull requests here:

          In your screenshot, the webhook URI ends with "/bitbucket-hook/". That path seems to be handled by Bitbucket plugin and Bitbucket Push and Pull Request plugin, which do not support Bitbucket Server native webhooks. Bitbucket Branch Source plugin instead uses a URI like "https://jenkins.example/bitbucket-scmsource-hook/notify?server_url=http://bitbucket.example:7990", where the server_url parameter must match what you have in "Bitbucket Endpoints" in the Jenkins configuration.

          Kalle Niemitalo added a comment - tkleiber wrote: Native Webhooks does not work for Pull requests here: In your screenshot, the webhook URI ends with "/bitbucket-hook/". That path seems to be handled by Bitbucket plugin and Bitbucket Push and Pull Request plugin , which do not support Bitbucket Server native webhooks. Bitbucket Branch Source plugin instead uses a URI like "https://jenkins.example/bitbucket-scmsource-hook/notify?server_url= http://bitbucket.example:7990 ", where the server_url parameter must match what you have in "Bitbucket Endpoints" in the Jenkins configuration.

          Torsten Kleiber added a comment -

          Thank you, that was the missing piece, not it works!

          Torsten Kleiber added a comment - Thank you, that was the missing piece, not it works!

            Unassigned Unassigned
            estyrke Emil Styrke
            Votes:
            38 Vote for this issue
            Watchers:
            55 Start watching this issue

              Created:
              Updated:
              Resolved: