Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47891

Bitbucket branch source plugin does not understand the new Bitbucket Server webhooks

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The latest Bitbucket Server version, 5.5, has built-in support for webhooks (as opposed to the plugin previously providing this functionality).

      For some reason, the built-in webhook does not send the same event keys as before, specifically it sets `X-Event-Key: repo:refs_changed` instead of `X-Event-Key: repo:push`, which the plugin expects.

      See https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html - apparently there are other changes to the payloads as well compared to Bitbucket Cloud.

      The latest comments on JENKINS-28877 touch on this issue as well.

      Example error message in the Bitbucket log:

      java.lang.Exception: X-Event-Key HTTP header invalid: repo:refs_changed 
         at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) 
         at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:99) 
         at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) 
         at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) 
         at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) 
         at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) 
         at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) 
         at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) 
         at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) 
         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) 
         at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) 
         at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) 
         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) 
         at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) 
         at org.kohsuke.stapler.Stapler.service(Stapler.java:238) 
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) 
         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) 
         at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
         at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
         at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
         at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
         at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
         at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232) 
         at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209) 
         at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) 
         at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113) 
         at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) 
         at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
         at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.process(BitbucketSCMSourcePushHookReceiver.java:70) 
         at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:59) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) 
         at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) 
         at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) 
         at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) 
         at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) 
         at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
         at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
         at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
         at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) 
         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) 
         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) 
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) 
         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) 
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) 
         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) 
         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) 
         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) 
         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) 
         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) 
         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) 
         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) 
         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) 
         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) 
         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
         at org.eclipse.jetty.server.Server.handle(Server.java:564) 
         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) 
         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) 
         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) 
         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) 
         at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) 
         at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) 
         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
         at java.lang.Thread.run(Thread.java:748)
      

       

        Attachments

          Issue Links

            Activity

            estyrke Emil Styrke created issue -
            kon Kalle Niemitalo made changes -
            Field Original Value New Value
            Link This issue blocks JENKINS-46946 [ JENKINS-46946 ]
            Hide
            kon Kalle Niemitalo added a comment -

            It seems this JENKINS-47891 is about Jenkins receiving notifications from Bitbucket Server built-in webhooks, and JENKINS-46946 is about Jenkins configuring Bitbucket Server to send those notifications.

            Show
            kon Kalle Niemitalo added a comment - It seems this JENKINS-47891 is about Jenkins receiving notifications from Bitbucket Server built-in webhooks, and JENKINS-46946 is about Jenkins configuring Bitbucket Server to send those notifications.
            Hide
            paderka Jan Padera added a comment -

            Is there any workaround?

            Show
            paderka Jan Padera added a comment - Is there any workaround?
            Hide
            estyrke Emil Styrke added a comment -

            Jan Padera Yes, disable the built-in webhooks and use the add-in bitbucket-webhooks. Its configuration page in the repo settings is called "Post webhooks".

            Show
            estyrke Emil Styrke added a comment - Jan Padera Yes, disable the built-in webhooks and use the add-in bitbucket-webhooks. Its configuration page in the repo settings is called "Post webhooks".
            Hide
            paderka Jan Padera added a comment -

            Emil Styrke Thank you for your reply, but what is the right URL for that webhook if

            <JENKINS_URL>/bitbucket-scmsource-hook/notify

            doesn't work?

            Show
            paderka Jan Padera added a comment - Emil Styrke Thank you for your reply, but what is the right URL for that webhook if <JENKINS_URL>/bitbucket-scmsource-hook/notify doesn't work?
            Hide
            estyrke Emil Styrke added a comment -

            That is what I use, it should be correct. Possibly, you have a broken version of the Bitbucket branch source plugin in Jenkins. I seem to recall that the webhook triggering was broken in one of the recent versions. I'm running 2.2.5 here.

            Show
            estyrke Emil Styrke added a comment - That is what I use, it should be correct. Possibly, you have a broken version of the Bitbucket branch source plugin in Jenkins. I seem to recall that the webhook triggering was broken in one of the recent versions. I'm running 2.2.5 here.
            Hide
            paderka Jan Padera added a comment - - edited

            I'm using Bitbucket branch source plugin version 2.2.7 and Jenkins 2.92

            Show
            paderka Jan Padera added a comment - - edited I'm using Bitbucket branch source plugin version 2.2.7 and Jenkins 2.92
            Hide
            namraclik Martin Hegarty added a comment -

            I'm also using '<JENKINS_URL>/bitbucket-scmsource-hook/notify/' successfully with:
            Jenkins 2.85 &  cloudbees-bitbucket-branch-source 2.2.7

            Show
            namraclik Martin Hegarty added a comment - I'm also using '<JENKINS_URL>/bitbucket-scmsource-hook/notify/' successfully with: Jenkins 2.85 &  cloudbees-bitbucket-branch-source 2.2.7
            Hide
            paderka Jan Padera added a comment -

            Even with 2.2.5 version I got exception below.

            java.lang.Exception: X-Event-Key HTTP header not found
            	at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83)
            	at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:94)
            
            Show
            paderka Jan Padera added a comment - Even with 2.2.5 version I got exception below. java.lang.Exception: X-Event-Key HTTP header not found at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:94)
            Show
            namraclik Martin Hegarty added a comment - Are you using this Bitbucket add-on - https://marketplace.atlassian.com/plugins/nl.topicus.bitbucket.bitbucket-webhooks/server/overview   ?  
            Hide
            paderka Jan Padera added a comment -
            Show
            paderka Jan Padera added a comment - Ah, I've used https://marketplace.atlassian.com/plugins/com.atlassian.stash.plugin.stash-web-post-receive-hooks-plugin/server/overview  Thanks for your advice. I'm going to try it!
            Hide
            dsudduth Derek Sudduth added a comment -

            Jan Padera I'm having the exact same issue and I exclusively use the plugin Martin Hegarty mentioned.  A bug must have been introduced.  I've had this working for the better part of 8 months and it stopped with some of the latest Jenkins/Bitbucket plugin updates.

            Show
            dsudduth Derek Sudduth added a comment - Jan Padera I'm having the exact same issue and I exclusively use the plugin Martin Hegarty mentioned.  A bug must have been introduced.  I've had this working for the better part of 8 months and it stopped with some of the latest Jenkins/Bitbucket plugin updates.
            Hide
            tallandtree Mariska Tallandtree added a comment - - edited

            this really is becoming a nightmare. I'm trying to get this to work with jenkins 2.107.1 and Bitbucket 5.8.0 which has webhooks built-in. And now I should install another webhook plugin to get it to work in combination with jenkins and the latest Bitbucket branch source plugin ...

            But it seems to work with Eernies plugin. Hopefully it stays free.

            Show
            tallandtree Mariska Tallandtree added a comment - - edited this really is becoming a nightmare. I'm trying to get this to work with jenkins 2.107.1 and Bitbucket 5.8.0 which has webhooks built-in. And now I should install another webhook plugin to get it to work in combination with jenkins and the latest Bitbucket branch source plugin ... But it seems to work with Eernies plugin. Hopefully it stays free.
            Hide
            aairey Andy Airey added a comment - - edited

            Still no feedback on https://github.com/jenkinsci/bitbucket-branch-source-plugin/compare/master...kwin:bbserver-use-native-webhooks ?

            We unfortunately do not have the liberty to install third-party plugins on Bitbucket Server, so it would be really great to use (and manage) the native webhooks.

            Show
            aairey Andy Airey added a comment - - edited Still no feedback on https://github.com/jenkinsci/bitbucket-branch-source-plugin/compare/master...kwin:bbserver-use-native-webhooks  ? We unfortunately do not have the liberty to install third-party plugins on Bitbucket Server, so it would be really great to use (and manage) the native webhooks.
            Hide
            zack Zack Snyder added a comment - - edited

            Any update here?
            I get the error, even using the plugin: https://marketplace.atlassian.com/plugins/nl.topicus.bitbucket.bitbucket-webhooks/server/overview
            I use bitbucket 5.9.1 and jenkins 2.107.2

            java.lang.Exception: X-Event-Key HTTP header not found
            	at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83)
            	at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:94)
            	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
            	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
            	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
            	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
            	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
            	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
            	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
            	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
            	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
            	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
            	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
            	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
            	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
            	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
            	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
            	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
            	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
            	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
            	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
            	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
            	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
            	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
            	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
            	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
            	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
            	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
            	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
            	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
            	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
            	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
            	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
            	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
            	at org.eclipse.jetty.server.Server.handle(Server.java:564)
            	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
            	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
            	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
            	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
            	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
            	at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
            	at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
            	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            
            Show
            zack Zack Snyder added a comment - - edited Any update here? I get the error, even using the plugin: https://marketplace.atlassian.com/plugins/nl.topicus.bitbucket.bitbucket-webhooks/server/overview I use bitbucket 5.9.1 and jenkins 2.107.2 java.lang.Exception: X-Event-Key HTTP header not found at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:94) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
            Hide
            maennel Manuel Jeckelmann added a comment -

            Hey, unfortunately, I'm not a Jenkins plugin developer nor a Java developer, so I really don't have the necessary tools at hand. But I'll try to contribute to resolving this issue anyways. 

            Currently, I believe supported hook types are handled here: https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/HookEventType.java

            Event types seem unchanged and thus correct for Bitbucket Cloud (see https://confluence.atlassian.com/bitbucket/event-payloads-740262817.html). However, they seem to have changed for Bitbucket Server.

            From https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html I conclude the following event type names (for Bitbucket Server):

            • ``repo:push`` becomes ``repo:refs_changed``
            • ``pullrequest:created`` becomes ``pr:opened``
            • ``pullrequest:updated`` becomes ``pr:opened`` (assuming it corresponds to Atlassian's understanding of 'Modifying' a PR)
            • ``pullrequest:fulfilled`` becomes ``pr:merged``
            • ``pullrequest:rejected`` becomes ``pr:declined``

            If I interpret the code correctly, one can now change behaviour in https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/PullRequestHookProcessor.java in order to handle Bitbucket Cloud and Bitbuket Server events differently. 

             

            Unfortunately, it looks like the POST request body has changed considerably as well, so I guess existing Interfaces (mainly the two ending on *Event.java in https://github.com/jenkinsci/bitbucket-branch-source-plugin/tree/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api) need to be adapted. 

            Anyone knows, whether Atlassian is transiting towards the format they're using for Bitbucket Server? It looks cleaner to me. 

            Hope this helps in some way.

            PS: I have that same issue and I believe that it would make things simpler and cleaner if this plugin would take over changes that were applied to Bitbucket Server. 

            Show
            maennel Manuel Jeckelmann added a comment - Hey, unfortunately, I'm not a Jenkins plugin developer nor a Java developer, so I really don't have the necessary tools at hand. But I'll try to contribute to resolving this issue anyways.  Currently, I believe supported hook types are handled here:  https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/HookEventType.java Event types seem unchanged and thus correct for Bitbucket Cloud (see https://confluence.atlassian.com/bitbucket/event-payloads-740262817.html ). However, they seem to have changed for Bitbucket Server. From  https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html  I conclude the following event type names (for Bitbucket Server): ``repo:push`` becomes ``repo:refs_changed`` ``pullrequest:created`` becomes ``pr:opened`` ``pullrequest:updated`` becomes ``pr:opened`` (assuming it corresponds to Atlassian's understanding of 'Modifying' a PR) ``pullrequest:fulfilled`` becomes ``pr:merged`` ``pullrequest:rejected`` becomes ``pr:declined`` If I interpret the code correctly, one can now change behaviour in  https://github.com/jenkinsci/bitbucket-branch-source-plugin/blob/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/hooks/PullRequestHookProcessor.java  in order to handle Bitbucket Cloud and Bitbuket Server events differently.    Unfortunately, it looks like the POST request body has changed considerably as well, so I guess existing Interfaces (mainly the two ending on *Event.java in https://github.com/jenkinsci/bitbucket-branch-source-plugin/tree/master/src/main/java/com/cloudbees/jenkins/plugins/bitbucket/api ) need to be adapted.  Anyone knows, whether Atlassian is transiting towards the format they're using for Bitbucket Server? It looks cleaner to me.  Hope this helps in some way. PS: I have that same issue and I believe that it would make things simpler and cleaner if this plugin would take over changes that were applied to Bitbucket Server. 
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2714 (Web Link)" [ 20805 ]
            Hide
            amuniz Antonio Muñiz added a comment - - edited

            Ok, let me try to put a bit of context here:

            Why are we using a "custom" BB Server plugin for webhooks?

            Because when bb-branch-source development started there were no built-in webhooks support in BB Server. However BB Cloud already had it. So the most sensible solution by that time was to use a plugin in BB Server to somehow "mimic" webhook messages from BB Cloud. That way the same code could handle webhooks from both BB Server and BB Cloud.

            The "official" plugin, as many of you already said, is: https://marketplace.atlassian.com/plugins/nl.topicus.bitbucket.bitbucket-webhooks/server/overview - which is OSS and the code is on GitHub: https://github.com/Eernie/bitbucket-webhooks-plugin

            BB Server comes with built-in webhooks now

            Yes. And that's causing confusion, sorry for that. I'll try to write fresh and clear docs on this in the short term.
            In the long term we have to decide what to do about this. Options are:

            1. Support both webhook types (plugin generated and built-in)
            2. Support built-in webhooks only
            3. Support plugin webhook only

            Each options has pros and cons:

            1. Supporting both has significant maintenance overhead
            2. Supporting built-in only will probably break most of existent setups, and we would lose any control over the evolution of the webhooks source (the current plugin is OSS, actively maintained and accepting contributions).
            3. Supporting plugin webhooks only, it seems to be producing confusion and installing a plugin is not always well-seen (specially on rigid BB Server installations) when there is a built-in alternative.

            IMO, until BB Server webhooks are "compatible" with BB Cloud webhooks payload we should stick to the "custom" plugin approach.

            Show
            amuniz Antonio Muñiz added a comment - - edited Ok, let me try to put a bit of context here: Why are we using a "custom" BB Server plugin for webhooks? Because when bb-branch-source development started there were no built-in webhooks support in BB Server. However BB Cloud already had it. So the most sensible solution by that time was to use a plugin in BB Server to somehow "mimic" webhook messages from BB Cloud. That way the same code could handle webhooks from both BB Server and BB Cloud. The "official" plugin, as many of you already said, is: https://marketplace.atlassian.com/plugins/nl.topicus.bitbucket.bitbucket-webhooks/server/overview - which is OSS and the code is on GitHub: https://github.com/Eernie/bitbucket-webhooks-plugin BB Server comes with built-in webhooks now Yes. And that's causing confusion, sorry for that. I'll try to write fresh and clear docs on this in the short term. In the long term we have to decide what to do about this. Options are: 1. Support both webhook types (plugin generated and built-in) 2. Support built-in webhooks only 3. Support plugin webhook only Each options has pros and cons: 1. Supporting both has significant maintenance overhead 2. Supporting built-in only will probably break most of existent setups, and we would lose any control over the evolution of the webhooks source (the current plugin is OSS, actively maintained and accepting contributions). 3. Supporting plugin webhooks only, it seems to be producing confusion and installing a plugin is not always well-seen (specially on rigid BB Server installations) when there is a built-in alternative. IMO, until BB Server webhooks are "compatible" with BB Cloud webhooks payload we should stick to the "custom" plugin approach.
            Hide
            aairey Andy Airey added a comment -

            IMO, until BB Server webhooks are "compatible" with BB Cloud webhooks payload we should stick to the "custom" plugin approach.

            This seems a very naive approach. Who says Atlassian will make them "compatible"? Or can you point us to some information that they will do the above at some point?
            BB Server webhooks is newer code than the BB cloud webhooks, so they could've just sticked to the more used one.
            If anything, BB cloud would move to the same code as BB server uses - but that would still be an assumption/wild guess.

            IMHO it's better to support the official webhooks provided by Atlassian.
            This will make the most sense to all end-users and foster greater adoption of Jenkins and the bitbucket-branch-source plugin in general.

            In this way you keep support the "custom plugin" method too, as you say that uses the same payload as BB cloud.
            As you said here:

            So the most sensible solution by that time was to use a plugin in BB Server to somehow "mimic" webhook messages from BB Cloud

            Show
            aairey Andy Airey added a comment - IMO, until BB Server webhooks are "compatible" with BB Cloud webhooks payload we should stick to the "custom" plugin approach. This seems a very naive approach. Who says Atlassian will make them "compatible"? Or can you point us to some information that they will do the above at some point? BB Server webhooks is newer code than the BB cloud webhooks, so they could've just sticked to the more used one. If anything, BB cloud would move to the same code as BB server uses - but that would still be an assumption/wild guess. IMHO it's better to support the official webhooks provided by Atlassian. This will make the most sense to all end-users and foster greater adoption of Jenkins and the bitbucket-branch-source plugin in general. In this way you keep support the "custom plugin" method too, as you say that uses the same payload as BB cloud. As you said here: So the most sensible solution by that time was to use a plugin in BB Server to somehow "mimic" webhook messages from BB Cloud
            Hide
            twz123 Tom Wieczorek added a comment -

            I'm currently working on native webhook support for Bitbucket Server here: https://github.com/jenkinsci/bitbucket-branch-source-plugin/compare/master...yieldlab:native-webhooks

            Not yet complete, hence no open Pull Request, but it's partially working already. Maybe this helps already.

            Show
            twz123 Tom Wieczorek added a comment - I'm currently working on native webhook support for Bitbucket Server here: https://github.com/jenkinsci/bitbucket-branch-source-plugin/compare/master...yieldlab:native-webhooks Not yet complete, hence no open Pull Request, but it's partially working already. Maybe this helps already.
            Hide
            casz Joseph Petersen (old) added a comment - - edited

            Tom Wieczorek would actually be great if you would open a PR cause I definitely have some input  the changes look good! Few minor tweaks I would do.

            For now please stop misusing jelly enums. You can do this a lot better with a field and descriptor with doFill.

            See here for a good example use: 

            https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/java/org/jenkinsci/plugins/jvctb/config/ViolationConfig.java#L120-L127

            https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/resources/org/jenkinsci/plugins/jvctb/config/ViolationConfig/config.jelly#L5-L7

             

            Show
            casz Joseph Petersen (old) added a comment - - edited Tom Wieczorek would actually be great if you would open a PR cause I definitely have some input  the changes look good! Few minor tweaks I would do. For now please stop misusing jelly enums. You can do this a lot better with a field and descriptor with doFill. See here for a good example use:  https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/java/org/jenkinsci/plugins/jvctb/config/ViolationConfig.java#L120-L127 https://github.com/jenkinsci/violation-comments-to-stash-plugin/blob/06424100d8fe25c1dd15c1ff7476ac2d735694f1/src/main/resources/org/jenkinsci/plugins/jvctb/config/ViolationConfig/config.jelly#L5-L7  
            Hide
            twz123 Tom Wieczorek added a comment -

            Pull request #138 opened. Looking forward to get your feedback Joseph Petersen (old)

            Show
            twz123 Tom Wieczorek added a comment - Pull request #138 opened. Looking forward to get your feedback Joseph Petersen (old)
            twz123 Tom Wieczorek made changes -
            Remote Link This issue links to "PR #138 (Web Link)" [ 21302 ]
            Hide
            aairey Andy Airey added a comment -

            FYI, my company uses Bitbucket Server. With several teams asking about webhook support for Jenkins and speculation running wild, our Engineering Tools manager sought an official answer from Atlassian. Here is his report:

            I've been in contact with Atlassian Premier Support and the lead developers of Bitbucket, who reviewed JENKINS-47891 at my request. They confirm that there are differences in the web hook behaviors between Server and Cloud. This is because although the two products share a name (Bitbucket) and general purpose (remote Git repository management), they serve different customer needs and their code bases are distinct. They said that making Bitbucket Server and Bitbucket Cloud to be exactly the same is not currently planned. From their perspective, Bitbucket Server web hooks are working as designed and customizing the functionality to meet the specific needs of individual third party applications is not on the roadmap.

            Show
            aairey Andy Airey added a comment - FYI, my company uses Bitbucket Server. With several teams asking about webhook support for Jenkins and speculation running wild, our Engineering Tools manager sought an official answer from Atlassian. Here is his report: I've been in contact with Atlassian Premier Support and the lead developers of Bitbucket, who reviewed JENKINS-47891 at my request. They confirm that there are differences in the web hook behaviors between Server and Cloud. This is because although the two products share a name (Bitbucket) and general purpose (remote Git repository management), they serve different customer needs and their code bases are distinct . They said that making Bitbucket Server and Bitbucket Cloud to be exactly the same is not currently planned. From their perspective, Bitbucket Server web hooks are working as designed and customizing the functionality to meet the specific needs of individual third party applications is not on the roadmap.
            cleclerc Cyrille Le Clerc made changes -
            Description The latest Bitbucket Server version, 5.5, has built-in support for webhooks (as opposed to the plugin previously providing this functionality).

            For some reason, the built-in webhook does not send the same event keys as before, specifically it sets `X-Event-Key: repo:refs_changed` instead of `X-Event-Key: repo:push`, which the plugin expects.

            See [https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html] - apparently there are other changes to the payloads as well compared to Bitbucket Cloud.

            The latest comments on JENKINS-28877 touch on this issue as well.

            Example error message in the Bitbucket log:

            {{java.lang.Exception: X-Event-Key HTTP header invalid: repo:refs_changed at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:99) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.process(BitbucketSCMSourcePushHookReceiver.java:70) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:59) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)}}

             
            The latest Bitbucket Server version, 5.5, has built-in support for webhooks (as opposed to the plugin previously providing this functionality).

            For some reason, the built-in webhook does not send the same event keys as before, specifically it sets `X-Event-Key: repo:refs_changed` instead of `X-Event-Key: repo:push`, which the plugin expects.

            See [https://confluence.atlassian.com/bitbucketserver/event-payload-938025882.html] - apparently there are other changes to the payloads as well compared to Bitbucket Cloud.

            The latest comments on JENKINS-28877 touch on this issue as well.

            Example error message in the Bitbucket log:

            {noformat}
            java.lang.Exception: X-Event-Key HTTP header invalid: repo:refs_changed
               at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83)
               at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.doNotify(BitbucketSCMSourcePushHookReceiver.java:99)
               at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
               at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
               at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
               at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
               at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
               at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
               at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
               at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
               at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
               at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
               at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
               at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
               at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
               at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
               at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
               at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:225)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
               at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
               at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
               at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
               at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
               at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232)
               at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)
               at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
               at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)
               at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
               at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
               at com.cloudbees.jenkins.plugins.bitbucket.hooks.BitbucketSCMSourcePushHookReceiver.process(BitbucketSCMSourcePushHookReceiver.java:70)
               at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:59)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
               at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
               at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
               at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
               at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
               at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
               at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
               at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
               at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
               at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
               at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
               at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
               at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
               at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
               at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
               at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
               at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
               at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
               at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
               at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
               at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
               at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
               at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
               at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
               at org.eclipse.jetty.server.Server.handle(Server.java:564)
               at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
               at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
               at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
               at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
               at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
               at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
               at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
               at java.lang.Thread.run(Thread.java:748)
            {noformat}

             
            cleclerc Cyrille Le Clerc made changes -
            Summary Bitbucket branch source plugin does not understand the new Bitbucket webhooks Bitbucket branch source plugin does not understand the new Bitbucket Server webhooks
            twz123 Tom Wieczorek made changes -
            Link This issue duplicates JENKINS-46946 [ JENKINS-46946 ]
            twz123 Tom Wieczorek made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            jimklimov Jim Klimov added a comment - - edited

            The PR has now been completed, merged and made its way into the new release of the plugin.

            This worked nicely for our Multi-Branch Pipelines generated via Bitbucket team folder "jobs" that are the one item we configure "manually" to get our repos and branches discovered to spawn the pipeline jobs. A rescan of the organization "job" in Jenkins (after setting the global config to use the "Native" webhook implementation in the connector for our BB server) made configurations in all repos that had a Jenkinsfile in at least one branch, but no configurations in repos that had no such branches (corollary: to initially introduce pipeline support in codebase of a component, one has to rescan that organization or MBP job).

            One caveat I hit in our local setup (constrained by a web of corporate IT policies) was that our lab's Jenkins server uses a self-signed certificate which the centrally managed corporate Bitbucket Server instance reasonably does not trust, so it can not HTTP POST to. Our other consumers (developers' browsers and internal build farm) have no issues with trusting this certificate.

            I had to change the (apache httpd) reverse proxy handling to let it through as plaintext:

            RedirectMatch 302 ^/?((?!neo.*$|bitbucket-scmsource-hook$|bitbucket-scmsource-hook/.*$).*)$ https://jenkins.domain.com/$1
            ProxyPass         /bitbucket-scmsource-hook  http://jenkins.domain.com:8080/bitbucket-scmsource-hook/ nocanon connectiontimeout=3600 timeout=3600 retry=0
            

            ...and had to change the Jenkins global config to announce the server URL as the "http" (non-"https") version. UPDATE: Added RFE JENKINS-55423 for a better solution to this sort of problem.

            Thanks to everybody involved in making this feature!

            Show
            jimklimov Jim Klimov added a comment - - edited The PR has now been completed, merged and made its way into the new release of the plugin. This worked nicely for our Multi-Branch Pipelines generated via Bitbucket team folder "jobs" that are the one item we configure "manually" to get our repos and branches discovered to spawn the pipeline jobs. A rescan of the organization "job" in Jenkins (after setting the global config to use the "Native" webhook implementation in the connector for our BB server) made configurations in all repos that had a Jenkinsfile in at least one branch, but no configurations in repos that had no such branches (corollary: to initially introduce pipeline support in codebase of a component, one has to rescan that organization or MBP job). One caveat I hit in our local setup (constrained by a web of corporate IT policies) was that our lab's Jenkins server uses a self-signed certificate which the centrally managed corporate Bitbucket Server instance reasonably does not trust, so it can not HTTP POST to. Our other consumers (developers' browsers and internal build farm) have no issues with trusting this certificate. I had to change the (apache httpd) reverse proxy handling to let it through as plaintext: RedirectMatch 302 ^/?((?!neo.*$|bitbucket-scmsource-hook$|bitbucket-scmsource-hook/.*$).*)$ https: //jenkins.domain.com/$1 ProxyPass /bitbucket-scmsource-hook http: //jenkins.domain.com:8080/bitbucket-scmsource-hook/ nocanon connectiontimeout=3600 timeout=3600 retry=0 ...and had to change the Jenkins global config to announce the server URL as the "http" (non-"https") version. UPDATE: Added RFE JENKINS-55423 for a better solution to this sort of problem. Thanks to everybody involved in making this feature!
            jimklimov Jim Klimov made changes -
            Link This issue relates to JENKINS-55423 [ JENKINS-55423 ]
            Hide
            pleibiger Peter Leibiger added a comment -

            I can't get native hooks to be auto registered, is there something special needed compared to the plugin?

            Plugin has always worked fine but does not (yet) support Bitbucket Server 6.0.0

            Show
            pleibiger Peter Leibiger added a comment - I can't get native hooks to be auto registered, is there something special needed compared to the plugin? Plugin has always worked fine but does not (yet) support Bitbucket Server 6.0.0
            Hide
            pleibiger Peter Leibiger added a comment -

            After adding a logger in Jenkins for com.cloudbees.jenkins.plugins.bitbucket I figured out that the configured user did not have Admin access for those repositories/projects in Bitbucket. This was not required with the plugin and is not easy to figure out. It would be nice to have some warning message in the scan log for this case. 

            Show
            pleibiger Peter Leibiger added a comment - After adding a logger in Jenkins for com.cloudbees.jenkins.plugins.bitbucket I figured out that the configured user did not have Admin access for those repositories/projects in Bitbucket. This was not required with the plugin and is not easy to figure out. It would be nice to have some warning message in the scan log for this case. 
            tkleiber Torsten Kleiber made changes -
            Attachment native_webhooks.png [ 46126 ]
            Hide
            tkleiber Torsten Kleiber added a comment -

            I'm on bitbucket-branch-source 2.4.1 and Bitbucket Server 5.16. Native Webhooks does not work for Pull requests here:

            Show
            tkleiber Torsten Kleiber added a comment - I'm on bitbucket-branch-source 2.4.1 and Bitbucket Server 5.16. Native Webhooks does not work for Pull requests here:
            Hide
            kon Kalle Niemitalo added a comment -

            Torsten Kleiber wrote:

            Native Webhooks does not work for Pull requests here:

            In your screenshot, the webhook URI ends with "/bitbucket-hook/". That path seems to be handled by Bitbucket plugin and Bitbucket Push and Pull Request plugin, which do not support Bitbucket Server native webhooks. Bitbucket Branch Source plugin instead uses a URI like "https://jenkins.example/bitbucket-scmsource-hook/notify?server_url=http://bitbucket.example:7990", where the server_url parameter must match what you have in "Bitbucket Endpoints" in the Jenkins configuration.

            Show
            kon Kalle Niemitalo added a comment - Torsten Kleiber wrote: Native Webhooks does not work for Pull requests here: In your screenshot, the webhook URI ends with "/bitbucket-hook/". That path seems to be handled by Bitbucket plugin and Bitbucket Push and Pull Request plugin , which do not support Bitbucket Server native webhooks. Bitbucket Branch Source plugin instead uses a URI like "https://jenkins.example/bitbucket-scmsource-hook/notify?server_url= http://bitbucket.example:7990 ", where the server_url parameter must match what you have in "Bitbucket Endpoints" in the Jenkins configuration.
            Hide
            tkleiber Torsten Kleiber added a comment -

            Thank you, that was the missing piece, not it works!

            Show
            tkleiber Torsten Kleiber added a comment - Thank you, that was the missing piece, not it works!

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              estyrke Emil Styrke
              Votes:
              38 Vote for this issue
              Watchers:
              55 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: