-
Bug
-
Resolution: Fixed
-
Major
-
None
-
core:1.642.4.2
github:1.17.1
github-api:1.75
github-branch-source:1.6
branch-api:1.7
When committing multiple changes successively to the same branch and repo, some builds are never triggered.
How to reproduce this (Assuming you have a GitHub Organization folder with WebHooks already setup):
- Commit multiple changes successively with a minimum interval (few seconds). Usually 5 are enough.
- You should see that a build is trigger containing the changes until a certain revision/sha, but not the last one.
I have tried many times to do this and I believe this happen when changes are pushed while the Branch Indexing is running. Hence it seems easier to hit that issue when you have lots of branches on your repository.
In my example, I have the following commit in that order:
1. fb1de9187286a1c815dcda7bf143c18f4953a9e0 (included in the build)
2. 933fd9547927aa545f48dfab0021ea69458ea6f7 (included in the build)
3. 06cc4d2cc70a4a27b2004de84d50dbdc3b602bd4 (never built)
4. ca5fbd5854b629beb6e407c0ed6356e222bcf638 (never built)
Branch Indexing logs when the commit is detected shows:
Changes detected in feature/feature1 (7e45d8ffaafb3c16567f82769ab6c2400d98068a → fb1de9187286a1c815dcda7bf143c18f4953a9e0)
Scheduled build for branch: feature/feature1
Next 3 branch indexing cannot detect the changes and show:
No changes detected in feature/feature1 (still at fb1de9187286a1c815dcda7bf143c18f4953a9e0)
This is where the revision check fails: https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L312
I attached relevant logs (jenkins.log, branch-source.log, github-webhook.log and folder-queue-executor.log).
- is duplicated by
-
-
- Resolved
-
-
-
- Resolved
-
- is related to
-
JENKINS-34600 Improve the performance of scheduling a build
-
- Closed
-
- links to
[JENKINS-34727] WebHook events are not always successfully triggering Jenkins pipeline
That would also require GitHubWebHook.Listener to be extended with a new method taking a branch name and commit hash, with DefaultPushGHEventSubscriber extracting that information from its json.
I needed that as a next step here JENKINS-34600. I need to know the branch or pull request involved in the PUSH_EVENT.
At any rate, the first order of business is to verify whether GitHub or Jenkins is at fault here.
Root cause found. GitHub is not reliably returning the right commit SHA through the API if you request that information just after the push is done.
It seems to be some kind of cache at Github side delaying the information to be available in the API. As github-branch-source is using the API to retrieve the last commit in a branch just after the webhook is received, that commit is not visible (sometimes, depending on timing) thus not built.
Worth to note that this behavior is only reproducible when pushing commits by HTTPS to Github. It seems that pushing by SSH is triggering the webhooks after the commit is available in the API (which seems the right behavior).
There is another level of cache at client side. PR updated disabling the client side cache also.
Code changed in jenkins
User: Antonio
Path:
src/main/java/org/jenkinsci/plugins/github_branch_source/Connector.java
src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubWebhookListenerImpl.java
src/main/java/org/jenkinsci/plugins/github_branch_source/PullRequestGHEventSubscriber.java
http://jenkins-ci.org/commit/github-branch-source-plugin/096eb3867c90fbfcd4066720918ed8e0f0b5da91
Log:
JENKINS-34727 Delaying branch indexing (#54)
JENKINS-34727Delaying branch indexingJENKINS-34727Disabling GH API client cacheJENKINS-34727Reduce indexing delay to 5 seconds
Solving the trust issues is easy, as GH webhooks can be signed with a shared secret. So you just need to enforce that a secret is used, Jenkins already runs verification IIRC.
I suspect the GitHub API for /repos/:owner/:repo/branches is returning a stale commit.sha due to some server-side caching layer. If that is true, and GitHub refuses to fix the bug, our only option would be to create an SCMSourceOwner2 with an additional method like void onSCMSourceUpdated(@NonNull SCMSource source, @NonNull SCMRevision revision) permitting GitHubWebhookListenerImpl to send a targeted event which MultiBranchProject would interpret specially (not inside computeChildren). That would also require GitHubWebHook.Listener to be extended with a new method taking a branch name and commit hash, with DefaultPushGHEventSubscriber extracting that information from its json.
Would be desirable anyway for performance reasons. On the other hand, it would be a security vulnerability if not done properly: we do not trust webhooks, as they are delivered without Jenkins authentication, so scheduling a build at a particular commit hash that might have come from an attacker could permit someone to build an inappropriate commit. (It would have to be in the origin repository—but perhaps from an unprotected branch.) The only defense against this is to use secured webhooks, which GitHubWebHook would have to be made to support, and MainLogic.applyOrg would have to also be made to set a secure token when registering the webhook.