Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50154

Webhook delivery failed as X-Hub-Signature did not match calculated

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • github-plugin
    • None
    • Jenkins 2.89.4, Github Plugin 1.29.0

      Our users were reporting that several PR's across a number of different repositories were not being built by Jenkins. Our Webhooks are setup manually and we use Organization Folders as documented here and Freestyle/Pipeline Jobs as documented here

      We noticed that Github was showing a number of failures in both our organization and in individual repositories with messages like so:

       

      <html>
       <head>
       <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
       <title>Error 400 Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</title>
       </head>
       <body><h2>HTTP ERROR 400</h2>
       <p>Problem accessing /github-webhook/. Reason:
       <pre> Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/>
       </body>
      </html>
      

      This appeared to be limited to three repositories at first but it soon spread to other repositories. It's also intermittent and also tied to specific pull requests (i.e once a PR has had a failed webhook, subsequent updates to that PR will also fail).

      We had upgraded from Jenkins 2.64.3 and Github Plugin 1.27.0 and migrated to new instances in AWS over the weekend and had wondered if this may have made the issue worse - we've seen similar failures before now but less frequently.

      So far we have tried:

      • Rolling back to Github Plugin 1.28.1 which completely broke webhooks
      • Clearing the Github Plugin Cache (setting to 0, and back to a positive integer).
      • Changing the secret used for the webhook on Github and Jenkins
      • Changing the hook content type to `application/json`
      • Deleting and re-creating the hook

      The only workaround we have right now has been to:

      • Give our bot user Admin permissions on the affected repository
      • Enable 'Manage Hooks' and re-create hooks on all repositories
      • The created hook on the repository is, so far, delivering with 100% success

          [JENKINS-50154] Webhook delivery failed as X-Hub-Signature did not match calculated

          I've created a PR with possible solution to this https://github.com/jenkinsci/github-plugin/pull/242

          Reviewers are appreciated 

          Andrey Babushkin added a comment - I've created a PR with possible solution to this https://github.com/jenkinsci/github-plugin/pull/242 Reviewers are appreciated 

          Edward Nys added a comment -

          It looks like since this PR was merged and released as version 1.33 there are quite a few people with triggers failing.

          <title>Error 400 Provided signature [c0180c723bbe3ddb266be493f5c9c1977896ea15] did not match to calculated</title>
          </head>
          

          Edward Nys added a comment - It looks like since this PR was merged and released as version 1.33 there are quite a few people with triggers failing. <title>Error 400 Provided signature [c0180c723bbe3ddb266be493f5c9c1977896ea15] did not match to calculated</title> </head>

          enys could you please provide an example JSON payload for me to investigate this?

          Andrey Babushkin added a comment - enys  could you please provide an example JSON payload for me to investigate this?

          Edward Nys added a comment - - edited

          Sure, will prepare an annonimised payload.

           

          I've more or less drilled it down to 2 types.

           A commit containing unicode char, appearing in the payload as  : 

          "message": "Ë << unicode",

           

          Or merging a PR with multi line description :

          "Message\r\n\r\n message continued"

          Edward Nys added a comment - - edited Sure, will prepare an annonimised payload.   I've more or less drilled it down to 2 types.  A commit containing unicode char, appearing in the payload as  :  "message" : "Ë << unicode" ,   Or merging a PR with multi line description : "Message\r\n\r\n message continued"

          Edward Nys added a comment -

          I can confirm that above payloads cause the issue.

          Edward Nys added a comment - I can confirm that above payloads cause the issue.

          vinutha added a comment -

          this issue exits in git plugin version 1.33.0

          vinutha added a comment - this issue exits in git plugin version  1.33.0

          enys, vinutha_garimella thanks for your reports. I've created a PR to revert this changes I've made https://github.com/jenkinsci/github-plugin/pull/246
          I apologize to everyone who suffered from this change - looks like the bug is not in the plugin.
          I've tried to setup webhooks on Jenkins instance behind corporate firewall using this instruction https://www.jenkins.io/blog/2019/01/07/webhook-firewalls/, but I've taken pysmee (https://github.com/Akrog/pysmee) instead of official smee.io client written in JavaScript. If the bug exists, it is in pysmee - official client passes webhook payload correctly.

          Andrey Babushkin added a comment - enys , vinutha_garimella  thanks for your reports. I've created a PR to revert this changes I've made https://github.com/jenkinsci/github-plugin/pull/246 I apologize to everyone who suffered from this change - looks like the bug is not in the plugin. I've tried to setup webhooks on Jenkins instance behind corporate firewall using this instruction https://www.jenkins.io/blog/2019/01/07/webhook-firewalls/ , but I've taken pysmee ( https://github.com/Akrog/pysmee ) instead of official smee.io client written in JavaScript. If the bug exists, it is in pysmee - official client passes webhook payload correctly.

          Edward Nys added a comment -

          oxygenxo Thanks for all the information, and trying to fix a bug to start with!

          The only thing that would have been useful is an entry in the plugin changelog for 1.33 release.

           

          Thanks,

          Edward Nys added a comment - oxygenxo  Thanks for all the information, and trying to fix a bug to start with! The only thing that would have been useful is an entry in the plugin changelog for 1.33 release.   Thanks,

          Niklas Hambuechen added a comment - - edited

          I am still suffering from this problem.

          It occurs when I force-push a commit that has the message:

          mycommit*

          Here the thing that breaks it seems to be the star in the commit message. In the Github hook JSON body this is:

          ...
            "commits": [
          ...
                "message": "mycommit*",

          If I use a + instead, of the *** the issue does not occur and Jenkins accepts the webhook.

          The star seems to be forbidden anywhere in the message, e.g. on a new line as part of a Markdown-style list.

          Equally breaking are commit messages like

          a*

          or even just

          *

          as the commit message which results in JSON being

          "message": "*",

          Niklas Hambuechen added a comment - - edited I am still suffering from this problem. It occurs when I force-push a commit that has the message: mycommit* Here the thing that breaks it seems to be the star in the commit message. In the Github hook JSON body this is: ...   "commits" : [ ...       "message" : "mycommit*" , If I use a + instead, of the *** the issue does not occur and Jenkins accepts the webhook. The star seems to be forbidden anywhere in the message, e.g. on a new line as part of a Markdown-style list. Equally breaking are commit messages like a* or even just * as the commit message which results in JSON being "message" : "*" ,

          I think I found a solution for my case:

          In the "Manage webhook" settings on Github, I had Content-type as application/x-www-form-urlencoded. Changing it to application/json fixes the issue with the stars in commit messages.

          I'm struggling to find documentation that tells me which "Content type" I should be using for the Github plugin.

          On https://support.cloudbees.com/hc/en-us/articles/224543927-GitHub-Integration-Webhooks#manualmode it says

          Content type should be set up as application/json (application/x-www-form-urlencoded for PR events in Freestyle Jobs)

          So this suggests that either is OK, which is evidently not the case if you use stars in your commit messages.

          Could a plugin developer clarify?

          Niklas Hambuechen added a comment - I think I found a solution for my case: In the "Manage webhook" settings on Github, I had Content-type as application/x-www-form-urlencoded . Changing it to application/json fixes the issue with the stars in commit messages. I'm struggling to find documentation that tells me which "Content type" I should be using for the Github plugin. On https://support.cloudbees.com/hc/en-us/articles/224543927-GitHub-Integration-Webhooks#manualmode it says Content type should be set up as application/json ( application/x-www-form-urlencoded for PR events in Freestyle Jobs) So this suggests that either is OK, which is evidently not the case if you use stars in your commit messages. Could a plugin developer clarify?

            lanwen Kirill Merkushev
            dave_tucker Dave Tucker
            Votes:
            5 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: