Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72047

scm-filter-jervis gives up after 1 API request to GitHub which can lead to missed webhooks

XMLWordPrintable

    • 2.0-66.vc21d0c1d936d

      Bug description

      Webhooks get received by Jenkins but do not create jobs or start builds. This only happens sometimes.

      Other info

      I noticed clock drift on GitHub servers but it wasn't a factor.

      I verified GitHub API servers have about a 12 second clock drift currently compared to time.gov.

      We've been having several webhooks issues and I'm suspicious about the clock differences (I haven't nailed down a specific bug in code, yet).

      For example, GitHub will send a webhook at 22:07:04 and Jenkins will process the hook payload with signature verification at 22:07:03. No builds trigger for this clock difference and the log is missing from the multibranch pipeline event log.

      However, if I close and re-open the pull request to trigger another webhook its timestamps are in chronological order and succeed. Is it possible there's a clock drift bug in code? I'm still struggling to track it down with traces.

      Custom loggers

      I installed the support-core plugin and created a custom logger named "GitHub webhooks debugging".

      I have logging enabled for the following classes currently (level ALL):

      com.cloudbees.jenkins.GitHubWebHook
      org.jenkinsci.plugins.github.webhook.WebhookManager
      org.jenkinsci.plugins.github.admin.GitHubHookRegisterProblemMonitor
      org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber
      org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber
      org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler
      org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler
      org.jenkinsci.plugins.github.webhook.GHWebhookSignature
      org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor
      org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty
      org.jenkinsci.plugins.github_branch_source.GitHubRepositoryEventSubscriber
      org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber
      org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber
      org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject
      jenkins.branch.buildstrategies.basic.TagBuildStrategyImpl
      jenkins.branch.buildstrategies.basic.ChangeRequestBuildStrategyImpl
      jenkins.scm.api.SCMHeadEvent
      jenkins.branch.MultiBranchProject
      

      I'm able to trace webhook events from GitHub to Jenkins and inside of Jenkins: pull request event, payload received, signature verification succeeded.

      However, the trail stops at signature verification and there's no multibranch pipeline event log. If I retry it goes through all of the above and an event shows up in multibranch pipeline event log with a build being started.

      Sample job

      See attachment sample-job.xml

      Jenkins war and plugin versions

      See dependencies.gradle and the companion comment "How to reproduce" in the comments section of this issue.
       

        1. dependencies.gradle
          14 kB
          Sam Gleske
        2. sample-job.xml
          6 kB
          Sam Gleske

            sag47 Sam Gleske
            sag47 Sam Gleske
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: