Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
None
-
-
github-branch-source-2.8.2, github-branch-source-2.9.0
Description
github-branch-source-plugin v2.7.1 allows to configure Jenkins as a GitHub App, which
After upgrading to github-branch-source-plugin v2.7.1, I followed the GitHub app authentication guide to connect Jenkins to GitHub.
I can successfully clone our repositories using these new credentials most of the time, but I can also observe intermittent authentication failures.
I have some builds that manage to successfully execute `checkout scm` using the new credentials 10 times in a row, but the 11th time, it fails with:
using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) ... 1 more
(I don't have more details)
Then, subsequent builds fail with a slighly better error:
using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: remote: Repository not found. fatal: Authentication failed for 'https://github.com/my-org/my-repo/' at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) ... 1 more
Usually, this lasts until we destroy the Jenkins worker used for this build and create a brand new one.
There are no changes, either on Jenkins or on GitHub between that 10th build (which was successful) and the subsequent builds (which are failing). It was easily reproducible last week (it was failing after ~5 or 10 builds). I don't have any suspicious logs about these failures anymore.
Attachments
Issue Links
- is duplicated by
-
JENKINS-64911 Github Branch Source: rate limit sleep interrupted
-
- Closed
-
- links to
TL;DR: Fixed on our MacOS node with git credential-osxkeychain erase host=github.com protocol=https (then enter again)
We had the same problem on our MacOS node. The problem started after changing the credentials used to acces private Github repositories. The symptoms:
We then enabled GIT_TRACE=1 and GIT_CURL_VERBOSE=1 environment variables in my jobs. We did it via a Git wrapper in /usr/local/bin/git, but it could work also by setting the env variable directly in the job or in the Jenkins global config.
We quickly found out that Git was using the keystore from MacOS with the command with this line:
...
run-command.c:663 trace: run_command: 'git credential-osxkeychain get'
...
And a few lines below, we could see that Git was using the wrong user:
...
Server auth using Basic with user '<wrong-user>'
...
We then removed the credential from the MacOS key store with the following command:
git credential-osxkeychain erase host=github.com protocol=https
After that, jobs running on the MacOS node we working properly again.