-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.89.4, Github Plugin 1.29.0
Our users were reporting that several PR's across a number of different repositories were not being built by Jenkins. Our Webhooks are setup manually and we use Organization Folders as documented here and Freestyle/Pipeline Jobs as documented here
We noticed that Github was showing a number of failures in both our organization and in individual repositories with messages like so:
<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 400 Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</title> </head> <body><h2>HTTP ERROR 400</h2> <p>Problem accessing /github-webhook/. Reason: <pre> Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/> </body> </html>
This appeared to be limited to three repositories at first but it soon spread to other repositories. It's also intermittent and also tied to specific pull requests (i.e once a PR has had a failed webhook, subsequent updates to that PR will also fail).
We had upgraded from Jenkins 2.64.3 and Github Plugin 1.27.0 and migrated to new instances in AWS over the weekend and had wondered if this may have made the issue worse - we've seen similar failures before now but less frequently.
So far we have tried:
- Rolling back to Github Plugin 1.28.1 which completely broke webhooks
- Clearing the Github Plugin Cache (setting to 0, and back to a positive integer).
- Changing the secret used for the webhook on Github and Jenkins
- Changing the hook content type to `application/json`
- Deleting and re-creating the hook
The only workaround we have right now has been to:
- Give our bot user Admin permissions on the affected repository
- Enable 'Manage Hooks' and re-create hooks on all repositories
- The created hook on the repository is, so far, delivering with 100% success
[JENKINS-50154] Webhook delivery failed as X-Hub-Signature did not match calculated
| Description |
Original:
Our users were reporting that several PR's across a number of different repositories were not being built by Jenkins. Our Webhooks are setup manually and we use Organization Folders as documented [here|https://support.cloudbees.com/hc/en-us/articles/115003015711-GitHub-Webhook-Organization-Folder] and Freestyle/Pipeline Jobs as documented [here|https://support.cloudbees.com/hc/en-us/articles/115003015691-GitHub-Webhook-Non-Multibranch-Jobs] We noticed that Github was showing a number of failures in both our organization and in individual repositories with messages like so: {code:java} <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 400 Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</title> </head> <body><h2>HTTP ERROR 400</h2> <p>Problem accessing /github-webhook/. Reason: <pre> Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/> </body> </html> {code} This appeared to be limited to three repositories at first but it soon spread to other repositories. It's also intermittent and also tied to specific pull requests (i.e once a PR has had a failed webhook, subsequent updates to that PR will also fail). We had upgraded from Jenkins 2.64.3 and Github Plugin 1.27.0 and migrated to new instances in AWS over the weekend and had wondered if this may have made the issue worse - we've seen similar failures before now but less frequently. So far we have tried: * Rolling back to Github Plugin 1.28.1 which completely broke webhooks * Clearing the Github Plugin Cache (setting to 0, and back to a positive integer). * Changing the secret used for the webhook on Github and Jenkins * Changing the hook content type to `application/json` * Deleting and re-creating the hook The only workaround we have right now has been to: * Give our bot user Admin permissions on the affected respoitory * Enable 'Manage Hooks' and re-create hooks on all repositories * The created hook on the repository is, so far, delivering with 100% success |
New:
Our users were reporting that several PR's across a number of different repositories were not being built by Jenkins. Our Webhooks are setup manually and we use Organization Folders as documented [here|https://support.cloudbees.com/hc/en-us/articles/115003015711-GitHub-Webhook-Organization-Folder] and Freestyle/Pipeline Jobs as documented [here|https://support.cloudbees.com/hc/en-us/articles/115003015691-GitHub-Webhook-Non-Multibranch-Jobs] We noticed that Github was showing a number of failures in both our organization and in individual repositories with messages like so: {code:java} <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 400 Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</title> </head> <body><h2>HTTP ERROR 400</h2> <p>Problem accessing /github-webhook/. Reason: <pre> Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/> </body> </html> {code} This appeared to be limited to three repositories at first but it soon spread to other repositories. It's also intermittent and also tied to specific pull requests (i.e once a PR has had a failed webhook, subsequent updates to that PR will also fail). We had upgraded from Jenkins 2.64.3 and Github Plugin 1.27.0 and migrated to new instances in AWS over the weekend and had wondered if this may have made the issue worse - we've seen similar failures before now but less frequently. So far we have tried: * Rolling back to Github Plugin 1.28.1 which completely broke webhooks * Clearing the Github Plugin Cache (setting to 0, and back to a positive integer). * Changing the secret used for the webhook on Github and Jenkins * Changing the hook content type to `application/json` * Deleting and re-creating the hook The only workaround we have right now has been to: * Give our bot user Admin permissions on the affected repository * Enable 'Manage Hooks' and re-create hooks on all repositories * The created hook on the repository is, so far, delivering with 100% success |
Yes we see lots of instances of:
Mar 14, 2018 3:12:58 PM FINEST org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor shouldProvideValidSignatureTrying to verify sign from header sha1=8e290f17632932bed4138eb632dd2e761f529bcfMar 14, 2018 3:12:58 PM FINEST org.jenkinsci.plugins.github.webhook.GHWebhookSignature matchesSignature: calculated=deff2fe4d0173b1f2c39d086fa6cb6ee4f85b1fd provided=8e290f17632932bed4138eb632dd2e761f529bcf
| Attachment | New: Screenshot 2018-08-14 12.32.55.png [ 43744 ] |
We are seeing the same problem, but it wasn't tied to specific PR/branch, completely random and then just disappear.
I'm also seeing this issue this morning but only for a specific repo (1 out of 20+). Looking at the webhook history, this was working fine and then this same repo starting showing failed webhooks as of 2018-08-09 14:36:05.
Note that we do not have repo-level hooks, just a single org-level hook. All other builds are working fine!
Josh McCullough
added a comment - - edited I'm also seeing this issue this morning but only for a specific repo (1 out of 20+). Looking at the webhook history, this was working fine and then this same repo starting showing failed webhooks as of 2018-08-09 14:36:05.
Note that we do not have repo-level hooks, just a single org-level hook. All other builds are working fine! Josh McCullough
added a comment - BUMP! This is getting worse for us. We have more failed webhooks than succeeded now. We recently moved to multi-branch config, and now most of our webhook requests from GitHub are showing the "Provided signature [...] did not match to calculated" message.
Josh McCullough
added a comment - BUMP! This is getting worse for us. We have more failed webhooks than succeeded now. We recently moved to multi-branch config, and now most of our webhook requests from GitHub are showing the "Provided signature [...] did not match to calculated" message. In my case I was getting this randomly – for some requests, not others – but it seems to have been the result of choosing "application/x-www-form-urlencoded" as the Content Type on the GitHub side. Once I switched this to "application/json", the webhook payloads started consistently delivering successfully.
All webhooks fail with 400 from Jenkins with a message like this.
<tr><th>URI:</th><td>/github-webhook/</td></tr> <tr><th>STATUS:</th><td>400</td></tr> <tr><th>MESSAGE:</th><td>Provided signature [2ea5940025571f3314b15e7d1283a3c258f70912] did not match to calculated</td></tr> <tr><th>SERVLET:</th><td>Stapler</td></tr>
Dave, do you have any log messages with the category org.jenkinsci.plugins.github.webhook?