Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49332

Jenkins unable to manage webhooks of Github organization

      Under GitHub Pull Request Builder configuration Auto-manage webhooks is enabled.

      In the GitHub Pull Request Builder added user credentials. User belongs to owners' User's credential tested and are working.

      Webhooks management however does not work. For every private repository there's an error message displayed in Jenkins: 

      There is no credentials with admin access to manage hooks on GitHubRepositoryName[host=github.com,username=ORG_NAME,repository=REPO_NAME]

      From the error message it looks like plugin is trying to use organisation name as a user name, so authentication fails.

          [JENKINS-49332] Jenkins unable to manage webhooks of Github organization

          Hi guys we are struggling with this issue since a few weeks.

          What makes it worse is that we disabled the Auto-manage Webhooks as we don't need it but registered a webhook on organisation level instead. Still randomly some webhooks don't trigger a pipeline (maybe 98/100 PRs trigger our Jenkins pipelines, 2/100 do nothing except writing an error message in the logs). In that case we get the same error message as mentioned in the description.

          These are our settings in Jenkins:

          We use the Jenkins Github Organisation feature. In our organisation we have ~80 repos that are scanned by the plugin. The issue appears totally randomly throughout different repos. We create very simple PRs without descriptions or any fancy stuff and even after thorough inspection I could not find any difference in the PRs that result in an error and those that trigger a pipeline. The issue is always solved by closing the PR and creating a new one immediately afterwards, which is exact same as the one that failed. That new PR triggers the pipeline as expected.

          Maybe a hint is, that > 90% of the failing webhooks belong to PRs of a single user, who is also admin of our GitHub organisation (but not Jenkins admin). Again this user also creates the majority of our PRs so if you see it like this, it might be statistically possible that this is coincidence.

          Neidhart Orlich added a comment - Hi guys we are struggling with this issue since a few weeks. What makes it worse is that we disabled the Auto-manage Webhooks as we don't need it but registered a webhook on organisation level instead. Still randomly some webhooks don't trigger a pipeline (maybe 98/100 PRs trigger our Jenkins pipelines, 2/100 do nothing except writing an error message in the logs). In that case we get the same error message as mentioned in the description. These are our settings in Jenkins: We use the Jenkins Github Organisation feature. In our organisation we have ~80 repos that are scanned by the plugin. The issue appears totally randomly throughout different repos. We create very simple PRs without descriptions or any fancy stuff and even after thorough inspection I could not find any difference in the PRs that result in an error and those that trigger a pipeline. The issue is always solved by closing the PR and creating a new one immediately afterwards, which is exact same as the one that failed. That new PR triggers the pipeline as expected. Maybe a hint is, that > 90% of the failing webhooks belong to PRs of a single user, who is also admin of our GitHub organisation (but not Jenkins admin). Again this user also creates the majority of our PRs so if you see it like this, it might be statistically possible that this is coincidence.

          neidhart your issue is that you do not have a 'Secret Text' credentials with a personal access token. Try my instructions above (https://issues.jenkins-ci.org/browse/JENKINS-49332?focusedCommentId=340820&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-340820) with a 'Secret Text' credential and see if that makes a difference.

          Phillip Verheyden added a comment - neidhart your issue is that you do not have a 'Secret Text' credentials with a personal access token. Try my instructions above ( https://issues.jenkins-ci.org/browse/JENKINS-49332?focusedCommentId=340820&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-340820 ) with a 'Secret Text' credential and see if that makes a difference.

          phillipuniverse thank you I was going to try that workaround - but this does not explain why it works in 98% of the cases but randomly in 2% not so the issue remains.

          Neidhart Orlich added a comment - phillipuniverse thank you I was going to try that workaround - but this does not explain why it works in 98% of the cases but randomly in 2% not so the issue remains.

          phillipuniverse I tried your workaround and indeed the error messages in the Jenkins logs went away but the issue remained the same: 2% of our PRs do not trigger the CI.

          I disabled the workaround again because it was not helping and I am still sure it's related:
          Everytime when a PR does not trigger the CI I see the message
          Everytime when I check the logs after a PR that actually triggered the CI, the message is not there

          Neidhart Orlich added a comment - phillipuniverse I tried your workaround and indeed the error messages in the Jenkins logs went away but the issue remained the same: 2% of our PRs do not trigger the CI. I disabled the workaround again because it was not helping and I am still sure it's related: Everytime when a PR does not trigger the CI I see the message Everytime when I check the logs after a PR that actually triggered the CI, the message is  not there

          Mark Stosberg added a comment -

          neidhart You have a lot of repos to scan. Maybe periodically you hit your quota for the Github API. Do you only experience on problems on days with lot of pushes to Github?

          Mark Stosberg added a comment - neidhart You have a lot of repos to scan. Maybe periodically you hit your quota for the Github API. Do you only experience on problems on days with lot of pushes to Github?

          Tarjei Huse added a comment -

          Any news here?

          I have tried very hard to find a place in Jenkins to set the username for the secret key, but have failed and thus Jenkins reports my ORGNAME as the username when connecting to GitHub.

          Is there a way to configure this in Jenkins I'm not seeing? In the comments above I see some references to "Projects->GitHub Organization->Owner under your Jenkins GitHub organisation" but I cannot find any such link in my Jenkins install.

          Regards,

          Tarjei Huse added a comment - Any news here? I have tried very hard to find a place in Jenkins to set the username for the secret key, but have failed and thus Jenkins reports my ORGNAME as the username when connecting to GitHub. Is there a way to configure this in Jenkins I'm not seeing? In the comments above I see some references to "Projects->GitHub Organization->Owner under your Jenkins GitHub organisation" but I cannot find any such link in my Jenkins install. Regards,

          magnayn added a comment -

          I don't know if this is of help, and it's a bit embarrasing that the most common usecase ("I have a github org, I want to build all the projects, and have webhooks added so this is efficient/fast") is so terrible UX-wise.

          But:

          In the project configuration, top level (github organization), you need to set up the organisation name (so it finds the right projects).

          You need some credentials to access github. It only lists "username/password" items on the list.

          However : for github if you're using a personal access token, the username is irrelevant. It really should offer 'secret' types in this list.

          I got ours to work by (re)creating a username/password credentials as username = ORG_NAME, password = personal_access_token.

          This might be of help.

          magnayn added a comment - I don't know if this is of help, and it's a bit embarrasing that the most common usecase ("I have a github org, I want to build all the projects, and have webhooks added so this is efficient/fast") is so terrible UX-wise. But: In the project configuration, top level (github organization), you need to set up the organisation name (so it finds the right projects). You need some credentials to access github. It only lists "username/password" items on the list. However : for github if you're using a personal access token, the username is  irrelevant . It really should offer 'secret' types in this list. I got ours to work by (re)creating a username/password credentials as username = ORG_NAME, password = personal_access_token. This might be of help.

          Saul Cruz added a comment -

          This is not only happening on GitHub organizations items, I have a free-style job that started failing with this same error. 

          Saul Cruz added a comment - This is not only happening on GitHub organizations items, I have a free-style job that started failing with this same error. 

          A possible solution in the org.jenkinsci.plugins.github.webhook.WebhookManager plugin would be to look up the username given the secret text via the API: https://api.github.com/user?  Or, if its an enterprise github, the API for current_user_url from https://<MY ENTERPRISE API URL>/? Use the login from that response as the username in the connection:

          host=github.com,username=<MY ORG>,repository=<MY REPO>

           

          I think this will work for all use cases, including private orgs, public, and enterprise github.

          Michael Merrill added a comment - A possible solution in the org.jenkinsci.plugins.github.webhook.WebhookManager plugin would be to look up the username given the secret text via the API: https://api.github.com/user?   Or, if its an enterprise github, the API for current_user_url from  https://<MY ENTERPRISE API URL>/ ? Use the login from that response as the username in the connection: host=github.com,username=<MY ORG>,repository=<MY REPO>   I think this will work for all use cases, including private orgs, public, and enterprise github.

          In my case I didn't have an entry in jenkins configuration, under GitHub -> GitHub Servers
          It was not obvious from the error and the pipeline configuration that you needed to have the github repo configured both in global jenkins configuration and in the pipeline job itself

          Carlos Sanchez added a comment - In my case I didn't have an entry in jenkins configuration, under GitHub -> GitHub Servers It was not obvious from the error and the pipeline configuration that you needed to have the github repo configured both in global jenkins configuration and in the pipeline job itself

            lanwen Kirill Merkushev
            dmytro_kovalov Dmytro Kovalov
            Votes:
            27 Vote for this issue
            Watchers:
            46 Start watching this issue

              Created:
              Updated: