Bug description

Webhooks get received by Jenkins but do not create jobs or start builds. This only happens sometimes.

Other info

I noticed clock drift on GitHub servers but it wasn't a factor.

I verified GitHub API servers have about a 12 second clock drift currently compared to time.gov.

We've been having several webhooks issues and I'm suspicious about the clock differences (I haven't nailed down a specific bug in code, yet).

For example, GitHub will send a webhook at 22:07:04 and Jenkins will process the hook payload with signature verification at 22:07:03. No builds trigger for this clock difference and the log is missing from the multibranch pipeline event log.

However, if I close and re-open the pull request to trigger another webhook its timestamps are in chronological order and succeed. Is it possible there's a clock drift bug in code? I'm still struggling to track it down with traces.

Custom loggers

I installed the support-core plugin and created a custom logger named "GitHub webhooks debugging".

I have logging enabled for the following classes currently (level ALL):

com.cloudbees.jenkins.GitHubWebHook
org.jenkinsci.plugins.github.webhook.WebhookManager
org.jenkinsci.plugins.github.admin.GitHubHookRegisterProblemMonitor
org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber
org.jenkinsci.plugins.github.webhook.subscriber.PingGHEventSubscriber
org.jenkinsci.plugins.github.webhook.GHEventHeader$PayloadHandler
org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler
org.jenkinsci.plugins.github.webhook.GHWebhookSignature
org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor
org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty
org.jenkinsci.plugins.github_branch_source.GitHubRepositoryEventSubscriber
org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber
org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber
org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject
jenkins.branch.buildstrategies.basic.TagBuildStrategyImpl
jenkins.branch.buildstrategies.basic.ChangeRequestBuildStrategyImpl
jenkins.scm.api.SCMHeadEvent
jenkins.branch.MultiBranchProject

I'm able to trace webhook events from GitHub to Jenkins and inside of Jenkins: pull request event, payload received, signature verification succeeded.

However, the trail stops at signature verification and there's no multibranch pipeline event log. If I retry it goes through all of the above and an event shows up in multibranch pipeline event log with a build being started.

Sample job

See attachment sample-job.xml

Jenkins war and plugin versions

See dependencies.gradle and the companion comment "How to reproduce" in the comments section of this issue.