Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46366

REGRESSION: Github branch source does not consistently set up webhook

    XMLWordPrintable

    Details

    • Similar Issues:
    • Sprint:
      Pipeline - December

      Description

      Summary: 

      Creating a pipeline via Github, either org folder or a single MBP, does not correctly setup webhooks. 

      To reproduce: 

      • Create a github token with correct scope allowing for admin of webhooks
      • Create a MBP with the github project type, using this credential
      • Note that there is no webhook created on the github project. 
      • You will see a message in the log like: 
        INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])

       

      This also happens if you set it up via github org folders. R. Tyler Croy swears on his grave that it used to work (he used it on jenkins.io) so this is possibly a bad regression. 

       

      ---------

       

      See the attached screenshot, I haven't yet been able to figure out why sometimes, but mostly not, a webhook is created at an Org level.

      *Most* of the time this doesn't work, but sometimes it does. MUCH CONFUSION.

      Failed logs:

      Aug 23, 2017 5:45:23 AM org.jenkinsci.plugins.github.webhook.WebhookManager$1 run              
      INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])   
      

      Successful logs:

      INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])                          
      Aug 23, 2017 5:41:04 AM org.jenkinsci.plugins.github_branch_source.GitHubOrgWebHook register              
      INFO: A webhook was registered for the organization https://github.com/CodeValet                                                                                                                  
      Aug 23, 2017 5:41:05 AM org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber onEvent                                                                                             
      INFO: PING webhook received from org <https://api.github.com/orgs/CodeValet>!   
      

      Related-ish to JENKINS-46365

        Attachments

          Issue Links

            Activity

            rtyler R. Tyler Croy created issue -
            michaelneale Michael Neale made changes -
            Field Original Value New Value
            Summary GitHub Org Folder does not consistently set up an org-level webhook Github branch source does not consistently set up webhook
            michaelneale Michael Neale made changes -
            Description See the attached screenshot, I haven't yet been able to figure out why sometimes, but mostly not, a webhook is created at an Org level.

            **Most** of the time this doesn't work, but sometimes it does. MUCH CONFUSION.

            Failed logs:
            {code}
            Aug 23, 2017 5:45:23 AM org.jenkinsci.plugins.github.webhook.WebhookManager$1 run
            INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])
            {code}

            Successful logs:
            {code}
            INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])
            Aug 23, 2017 5:41:04 AM org.jenkinsci.plugins.github_branch_source.GitHubOrgWebHook register
            INFO: A webhook was registered for the organization https://github.com/CodeValet
            Aug 23, 2017 5:41:05 AM org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber onEvent
            INFO: PING webhook received from org <https://api.github.com/orgs/CodeValet&gt;!
            {code}

            Related-ish to JENKINS-46365

            !https://github.com/rtyler/codevalet/raw/master/assets/monkey-128.png!
            Summary: 

            Creating a pipeline via Github, either org folder or a single MBP, does not correctly setup webhooks. 

            To reproduce: 
             * Create a github token with correct scope allowing for admin of webhooks
             * Create a MBP with the github project type, using this credential
             * Note that there is no webhook created on the github project. 
             * You will see a message in the log like: 
            {code:java}
            INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY]){code}

             

            This also happens if you set it up via github org folders. [~rtyler] swears on his grave that it used to work (he used it on jenkins.io) so this is possibly a bad regression. 

             

            ---------

             

            See the attached screenshot, I haven't yet been able to figure out why sometimes, but mostly not, a webhook is created at an Org level.

            **Most** of the time this doesn't work, but sometimes it does. MUCH CONFUSION.

            Failed logs:
            {code:java}
            Aug 23, 2017 5:45:23 AM org.jenkinsci.plugins.github.webhook.WebhookManager$1 run
            INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])
            {code}
            Successful logs:
            {code:java}
            INFO: GitHub webhooks activated for job CodeValet with [] (events: [REPOSITORY])
            Aug 23, 2017 5:41:04 AM org.jenkinsci.plugins.github_branch_source.GitHubOrgWebHook register
            INFO: A webhook was registered for the organization https://github.com/CodeValet
            Aug 23, 2017 5:41:05 AM org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber onEvent
            INFO: PING webhook received from org <https://api.github.com/orgs/CodeValet&gt;!
            {code}
            Related-ish to JENKINS-46365

            !https://github.com/rtyler/codevalet/raw/master/assets/monkey-128.png!
            michaelneale Michael Neale made changes -
            Priority Minor [ 4 ] Critical [ 2 ]
            michaelneale Michael Neale made changes -
            Summary Github branch source does not consistently set up webhook REGRESSION: Github branch source does not consistently set up webhook
            michaelneale Michael Neale made changes -
            Link This issue blocks JENKINS-46365 [ JENKINS-46365 ]
            Hide
            rtyler R. Tyler Croy added a comment -

            Michael Neale "swears it used to work", bah! In my screenshot I'm showing that it does actually work once in a blue moon, but for the life of me I cannot figure out what series of events led to that working

            Show
            rtyler R. Tyler Croy added a comment - Michael Neale "swears it used to work", bah! In my screenshot I'm showing that it does actually work once in a blue moon, but for the life of me I cannot figure out what series of events led to that working
            stephenconnolly Stephen Connolly made changes -
            Link This issue duplicates JENKINS-33228 [ JENKINS-33228 ]
            stephenconnolly Stephen Connolly made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            Hide
            stephenconnolly Stephen Connolly added a comment -

            I am willing to bet that this is a side-effect of the split "responsibility" for registering hooks resulting from the duplication of config between the github plugin and the github branch source plugin

            Show
            stephenconnolly Stephen Connolly added a comment - I am willing to bet that this is a side-effect of the split "responsibility" for registering hooks resulting from the duplication of config between the github plugin and the github branch source plugin
            Hide
            rtyler R. Tyler Croy added a comment -

            Stephen Connolly, I don't understand how this is a duplicate of what you referenced. This isn't with GitHub Enterprise or anything like that set up, this is with GitHub.com

            Show
            rtyler R. Tyler Croy added a comment - Stephen Connolly , I don't understand how this is a duplicate of what you referenced. This isn't with GitHub Enterprise or anything like that set up, this is with GitHub.com
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Org Folder hook level registration is currently managed by the GHBS plugin and requires global config for enterprise servers only (but the org folder must be using credentials that have required permissions.

            Multibranch project hook level registration is managed by the GitHub plugin and requires global config for both github.com and github enterprise servers in the "GitHub Servers" section, and you have to enable Manage Hooks and provide credentials there with the ability to manage the hooks.

            Part of the fix for JENKINS-33228 will be to consolidate down to a single list with more consistent credentials selection policy.

            You are finding hooks failing to be set up because of the general multiplication of configuration... it does work if you know the magic config and setup credentials correctly... unsure if it can currently work for codevalet's use cases, though they should be an important part of the fix for JENKINS-33228

            Show
            stephenconnolly Stephen Connolly added a comment - Org Folder hook level registration is currently managed by the GHBS plugin and requires global config for enterprise servers only (but the org folder must be using credentials that have required permissions. Multibranch project hook level registration is managed by the GitHub plugin and requires global config for both github.com and github enterprise servers in the "GitHub Servers" section, and you have to enable Manage Hooks and provide credentials there with the ability to manage the hooks. Part of the fix for JENKINS-33228 will be to consolidate down to a single list with more consistent credentials selection policy. You are finding hooks failing to be set up because of the general multiplication of configuration... it does work if you know the magic config and setup credentials correctly... unsure if it can currently work for codevalet's use cases, though they should be an important part of the fix for JENKINS-33228
            Hide
            michaelneale Michael Neale added a comment -

            Stephen Connolly it didnt' even work for the most basic case running on local host (nothing to do with code valet, blue ocean or org folders, there was no webhook created). 

            I can't see how this is related to other fix - but there are commits on it - will there be a future release where webhooks actually work they way they used to? 

            Show
            michaelneale Michael Neale added a comment - Stephen Connolly it didnt' even work for the most basic case running on local host (nothing to do with code valet, blue ocean or org folders, there was no webhook created).  I can't see how this is related to other fix - but there are commits on it - will there be a future release where webhooks actually work they way they used to? 
            michaelneale Michael Neale made changes -
            Resolution Duplicate [ 3 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Michael Neale so you know there was a bug where a useless webhook was created if the jenkins url is http://localhost that would have been fixed a while back, can you confirm that you have the JenkinsLocationConfiguration with a non-default URL?

            Do you have a reproducible test scenario that you can describe?

            The consolidation should actually result in webhook management being comprehensible by mere mortals, but I argue that it is functioning as (badly) designed given the current dichotomy of configuration and the need to understand exactly how to configure for the split... and JENKINS-33228 when implemented should fix all the woes... but that does not mean it will work like before, rather it will work better than before! 

            Show
            stephenconnolly Stephen Connolly added a comment - Michael Neale  so you know there was a bug where a useless webhook was created if the jenkins url is http://localhost  that would have been fixed a while back, can you confirm that you have the JenkinsLocationConfiguration with a non-default URL? Do you have a reproducible test scenario that you can describe? The consolidation should actually result in webhook management being comprehensible by mere mortals, but I argue that it is functioning as (badly) designed given the current dichotomy of configuration and the need to understand exactly how to configure for the split... and JENKINS-33228 when implemented should fix all the woes... but that does not mean it will work like before, rather it will work better than before! 
            Hide
            michaelneale Michael Neale added a comment -

            Ah Stephen Connolly right - if it is smart about localhost - that is what I was seeing. 

             

            OK, sounds like the consolidation should be good, the case I was thinking was non org folders but single MBP - that is still valid right? 

            Show
            michaelneale Michael Neale added a comment - Ah Stephen Connolly right - if it is smart about localhost - that is what I was seeing.    OK, sounds like the consolidation should be good, the case I was thinking was non org folders but single MBP - that is still valid right? 
            michaelneale Michael Neale made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Yep consolidation should fix the confusion, and some edge cases. Otherwise this is working as very badly designed and the fix of the design is JENKINS-33228

            Show
            stephenconnolly Stephen Connolly added a comment - Yep consolidation should fix the confusion, and some edge cases. Otherwise this is working as very badly designed and the fix of the design is JENKINS-33228
            Hide
            michaelneale Michael Neale added a comment -

            thanks Stephen Connolly - so what is happening with JENKINS-33228 - is anyone on it? are you planning to touch it? 

            Show
            michaelneale Michael Neale added a comment - thanks Stephen Connolly - so what is happening with JENKINS-33228 - is anyone on it? are you planning to touch it? 
            Hide
            rtyler R. Tyler Croy added a comment -

            So Stephen Connolly, if I'm understanding the problem here correctly, it's that by default nothing is set up here:

            If that's the case, which I totally believe is realistic, then this is the default configuration with Blue Ocean and a freshly minted Jenkins instance.

            Which, Michael Neale, is such a laughably broken experience for new users with GitHub and Blue Ocean IMHO.

            Show
            rtyler R. Tyler Croy added a comment - So Stephen Connolly , if I'm understanding the problem here correctly, it's that by default nothing is set up here: If that's the case, which I totally believe is realistic, then this is the default configuration with Blue Ocean and a freshly minted Jenkins instance. Which, Michael Neale , is such a laughably broken experience for new users with GitHub and Blue Ocean IMHO.
            Hide
            michaelneale Michael Neale added a comment - - edited

            R. Tyler Croy right - if https://issues.jenkins-ci.org/browse/JENKINS-33228 is done - I was under the impression the defaults would be "more correct" - am I wrong? Is there something specific blue ocean could do to make the default better here? (blue ocean as of 1.3 does away with github org folders as its default mechanism, but I assume the problem remains). 

             

            (I think code monkey is more likely to be bit by this than anyone else, but yeah, this is confusing). I might re-open this to keep one eye on it from this angle, in case there is anything we can improve. 

            Show
            michaelneale Michael Neale added a comment - - edited R. Tyler Croy right - if https://issues.jenkins-ci.org/browse/JENKINS-33228  is done - I was under the impression the defaults would be "more correct" - am I wrong? Is there something specific blue ocean could do to make the default better here? (blue ocean as of 1.3 does away with github org folders as its default mechanism, but I assume the problem remains).    (I think code monkey is more likely to be bit by this than anyone else, but yeah, this is confusing). I might re-open this to keep one eye on it from this angle, in case there is anything we can improve. 
            michaelneale Michael Neale made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            michaelneale Michael Neale made changes -
            Component/s blueocean-plugin [ 21481 ]
            michaelneale Michael Neale made changes -
            Sprint Blue Ocean 1.3 - beta 1 [ 386 ]
            michaelneale Michael Neale made changes -
            Rank Ranked higher
            michaelneale Michael Neale made changes -
            Sprint Blue Ocean 1.3 - beta 1 [ 386 ]
            michaelneale Michael Neale made changes -
            Sprint Blue Ocean 1.3 - candidates [ 326 ]
            michaelneale Michael Neale made changes -
            Rank Ranked higher
            Hide
            michaelneale Michael Neale added a comment -

            James Dumay the blue ocean angle might be bad defaults here BTW... 

            Show
            michaelneale Michael Neale added a comment - James Dumay the blue ocean angle might be bad defaults here BTW... 
            jamesdumay James Dumay made changes -
            Sprint Blue Ocean 1.4 - candidates [ 326 ] Blue Ocean 1.4 - beta 2 [ 416 ]
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Rank Ranked higher
            jamesdumay James Dumay made changes -
            Rank Ranked lower
            Hide
            stevenfoster Steven Foster added a comment -

            Just commenting my notes about this issue which I had to deal with recently.

            There are a few different problems interacting I think, most of them are covered already. The GitHub server does need to be configured under the GitHub plugin as you've seen. The second problem was a bit more hidden. From what I found, a branch needs to have been built AND have a checkout as part of the pipeline (my tests often had no checkouts so this took awhile to pin down) because of how the GitHub plugin determines repo names. This means that you need to press "re-register webhooks" on the global configuration, or re-save the MBP after a build in order to register the webhook. My workaround was to write a plugin that added a GitHubRepositoryNameContributor which checked the MBP configuration rather than the SCM of a run to get the name of the repo. It also uses an ItemListener to kick off the webhook registration, so it's not a nice solution.

            The symptom of this name resolution problem was the empty list in the log like in the example
            INFO: GitHub webhooks activated for job CodeValet with []
            although I don't know about the interaction with GitHub org folders.

            It's been a little while since I've looked at this so hopefully that's accurate info, assuming I'm not missing something important.

            Show
            stevenfoster Steven Foster added a comment - Just commenting my notes about this issue which I had to deal with recently. There are a few different problems interacting I think, most of them are covered already. The GitHub server does need to be configured under the GitHub plugin as you've seen. The second problem was a bit more hidden. From what I found, a branch needs to have been built AND have a checkout as part of the pipeline (my tests often had no checkouts so this took awhile to pin down) because of how the GitHub plugin determines repo names. This means that you need to press "re-register webhooks" on the global configuration, or re-save the MBP after a build in order to register the webhook. My workaround was to write a plugin that added a GitHubRepositoryNameContributor which checked the MBP configuration rather than the SCM of a run to get the name of the repo. It also uses an ItemListener to kick off the webhook registration, so it's not a nice solution. The symptom of this name resolution problem was the empty list in the log like in the example INFO: GitHub webhooks activated for job CodeValet with [] although I don't know about the interaction with GitHub org folders. It's been a little while since I've looked at this so hopefully that's accurate info, assuming I'm not missing something important.
            jamesdumay James Dumay made changes -
            Sprint Blue Ocean 1.4 - beta 3 [ 416 ] Blue Ocean 1.4 - beta 4 [ 441 ]
            jamesdumay James Dumay made changes -
            Sprint Blue Ocean 1.4 - beta 4 [ 441 ] Pipeline - December [ 446 ]
            jamesdumay James Dumay made changes -
            Rank Ranked lower
            michaelneale Michael Neale made changes -
            Assignee rsandell [ rsandell ]
            michaelneale Michael Neale made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            jamesdumay James Dumay made changes -
            Component/s blueocean-plugin [ 21481 ]
            Hide
            michaelneale Michael Neale added a comment -

            Steven Foster rsandell mind if we close this in favour of https://issues.jenkins-ci.org/browse/JENKINS-48035 as a duplicate? 
             

            Show
            michaelneale Michael Neale added a comment - Steven Foster rsandell mind if we close this in favour of https://issues.jenkins-ci.org/browse/JENKINS-48035  as a duplicate?   
            michaelneale Michael Neale made changes -
            Link This issue duplicates JENKINS-48035 [ JENKINS-48035 ]
            michaelneale Michael Neale made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

              People

              Assignee:
              rsandell rsandell
              Reporter:
              rtyler R. Tyler Croy
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: