Example

curl -sI https://api.github.com/meta | grep -F date:; date

date: Thu, 21 Sep 2023 00:04:33 GMT
Wed Sep 20 20:04:53 EDT 2023

The second timestamp matches time.gov

It's worth noting that some GitHub server timestamps are wildly different and do not appear to be synchonized on GitHub's end (while some do appear to be synchronized). I filed a support ticket for this with GitHub already but the dropped builds are a bug concern which is why I filed this ticket.

Clock drift between GitHub and NIST was up to 20 seconds last night when I was investigating this.

Updated my logger in description

I have verified this bug; I'll post details soon

I have attached sample-job.xml (which is a config.xml) that reproduces the bug so you know what configurations are at play.

github-branch-source@1704.ias_vd5a_2b_29c6cdc.1 plugin is a custom fork of 1703.vd5a_2b_29c6cdc with patch applied from https://github.com/jenkinsci/github-branch-source-plugin/pull/653

I will also attach a list of plugins I'm using.

The temporary workaround

Before I dive into details I found a temporary workaround. GitHub clocks being out of sync required delaying between payload processing and triggering multibranch pipeline builds. This was achieved via the following system property.

-Dorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.eventDelaySeconds=22

I had to restart Jenkins. I would like to change this property (specifically the static method getEventDelaySeconds()) to return the property or fall back to static value so that it can be changed without restart to runtime.

Why does it work?

GitHub servers were out of sync. Jenkins processed multibranch events BEFORE GitHub sent webhook payloads. This triggered a bug (I've yet to find in source but now I have an idea).

By forcing a delay the Jenkins controller system clock has a chance to catch up to the payload event so that multibranch pipeline events are processed AFTER the hook payload timestamp.

Sample logs

I've narrowed down the issue to branch matchers. This gives me a specific avenue of source code to review

Failed webhook trace log

[Fri Sep 22 14:41:34 GMT 2023] Received Push event for tag 1.0.146 in repository **REDACTED ORG**/**REDACTED REPO** CREATED event from **REDACTED IP ADDRESS** ⇒ https://jenkins-webhooks.REDACTED.net/github-webhook/ with timestamp Fri Sep 22 14:41:29 GMT 2023

This means:

• GitHub sent webhook.
• Jenkins received webhook.
• Jenkins processed payload and successfully verified the signature to return 200 status to GitHub.
• Jenkins multibranch pipeline processed the event
• Nothing happened. No jobs created or builds started.

Successful trace log after replay webhook

[Fri Sep 22 16:20:03 GMT 2023] Received Push event for tag 1.0.146 in repository **REDACTED ORG**/**REDACTED REPO** CREATED event from **REDACTED IP ADDRESS** ⇒ https://jenkins-webhooks.REDACTED.net/github-webhook/ with timestamp Fri Sep 22 16:19:58 GMT 2023
Found match against Reporting-Platform/fantastic-signals-sso (new branch 1.0.146)

[Fri Sep 22 16:20:05 GMT 2023] Finished processing Push event for tag 1.0.146 in repository **REDACTED ORG**/**REDACTED REPO** CREATED event from **REDACTED IP ADDRESS** ⇒ https://jenkins-webhooks.REDACTED.net/github-webhook/ with timestamp Fri Sep 22 16:19:58 GMT 2023, processed in 1746ms. Matched 1.

This means:

• GitHub sent webhook.
• Jenkins received webhook.
• Jenkins processed payload and successfully verified the signature to return 200 status to GitHub.
• Jenkins central multibranch pipeline processed the event. Found a match.
• Jenkins central multibranch pipeline processed the event and notified the multibranch pipeline job.
• Jenkins multibranch pipeline job successfully processed the event against branch matchers and created a Jenkins job for a GitHub tag.
• Jenkins automatically started a build for the GitHub tag.