GitHub support got back to me with the following finding: each application installation has its own token. Seems like my Jenkins instance was using the same token for all the installations. Further diving into the issue, the `Owner` field description for GitHub App credentials entry reads as this:
The organisation or user that this app is to be used for. Only required if this app is installed to multiple organisations.
This is not the owner of the App, it is an org that has this app installed! The name of the field is confusing. Also I am still confused as my Jenkins instance was successfully able to access private repos with the "wrong" token, it's just failing to manage webhooks.
However, that means for each org I need to have its own credentials entry, even though it is the same App ID and key? I'm not sure this is expected/desired behavior. I want to be able to create the App and add its ID and token to my Jenkins instance, and then I want my Jenkins users to be able to install this App to their orgs and when they create jobs they should be able to use GitHub App credentials entry corresponding to this app. Current behavior requires me to be adding credentials entry separately for each org.
I haven't looked into the code, but it should be possible to dynamically inject the `Owner` field before requesting the token based on the current SCM configuration in the job.