-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.89.4, Github Plugin 1.29.0
Our users were reporting that several PR's across a number of different repositories were not being built by Jenkins. Our Webhooks are setup manually and we use Organization Folders as documented here and Freestyle/Pipeline Jobs as documented here
We noticed that Github was showing a number of failures in both our organization and in individual repositories with messages like so:
<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 400 Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</title> </head> <body><h2>HTTP ERROR 400</h2> <p>Problem accessing /github-webhook/. Reason: <pre> Provided signature [139e89df4530d9827a6a8f32c1ee28564fff178f] did not match to calculated</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.z-SNAPSHOT</a><hr/> </body> </html>
This appeared to be limited to three repositories at first but it soon spread to other repositories. It's also intermittent and also tied to specific pull requests (i.e once a PR has had a failed webhook, subsequent updates to that PR will also fail).
We had upgraded from Jenkins 2.64.3 and Github Plugin 1.27.0 and migrated to new instances in AWS over the weekend and had wondered if this may have made the issue worse - we've seen similar failures before now but less frequently.
So far we have tried:
- Rolling back to Github Plugin 1.28.1 which completely broke webhooks
- Clearing the Github Plugin Cache (setting to 0, and back to a positive integer).
- Changing the secret used for the webhook on Github and Jenkins
- Changing the hook content type to `application/json`
- Deleting and re-creating the hook
The only workaround we have right now has been to:
- Give our bot user Admin permissions on the affected repository
- Enable 'Manage Hooks' and re-create hooks on all repositories
- The created hook on the repository is, so far, delivering with 100% success
