Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
None
-
-
github-branch-source-2.8.2, github-branch-source-2.9.0
Description
github-branch-source-plugin v2.7.1 allows to configure Jenkins as a GitHub App, which
After upgrading to github-branch-source-plugin v2.7.1, I followed the GitHub app authentication guide to connect Jenkins to GitHub.
I can successfully clone our repositories using these new credentials most of the time, but I can also observe intermittent authentication failures.
I have some builds that manage to successfully execute `checkout scm` using the new credentials 10 times in a row, but the 11th time, it fails with:
using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) ... 1 more
(I don't have more details)
Then, subsequent builds fail with a slighly better error:
using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10 ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: remote: Repository not found. fatal: Authentication failed for 'https://github.com/my-org/my-repo/' at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) ... 1 more
Usually, this lasts until we destroy the Jenkins worker used for this build and create a brand new one.
There are no changes, either on Jenkins or on GitHub between that 10th build (which was successful) and the subsequent builds (which are failing). It was easily reproducible last week (it was failing after ~5 or 10 builds). I don't have any suspicious logs about these failures anymore.
Attachments
Issue Links
- is duplicated by
-
-
- Closed
-
- links to
Activity
I'm a collegue of Hans Koster who added some comments earlier. We are facing this issue on a regular basis. I've put a lot of effort in testing and analyzing it and have come to the following conclusion: The branch-source plugin is managing tokens properly through GIT_ASKPASS, but GIT_ASKPASS is NOT used if the token is still in a git credential helper like 'cache'. That's the one we use with a timeout of 1800 seconds. This timeout is respected, as long as the credential/token in the cache it is not used! In that case, it is stored again by the git client so the timeout is reset (see https://github.com/git/git/blob/master/http.c#L1637). So when the credential is reused within the timeout period over and over again, the credential will remain in the git credential cache until the token is expired in GitHub. In that case, the cache will be cleared by the git client after the usage of the cached token failed:
... returned status code 128: stdout: stderr: remote: Invalid username or password. ...
For now I see no other solution then disabling the git credential cache helper.
I used a small Jenkins pipeline job to check if there is an entry in the git credential cache:
pipeline {
agent any
options {
timeout(time: 1, unit: 'HOURS')
timestamps()
}
triggers \{ cron('0 */1 * * *') }
stages {
stage('Get Cache') {
steps {
script {
sh 'git config --list'
writeFile(file: 'cred.txt', text: '''protocol=https\nhost=github.com\n''')
for(i=0;i<12;i++) {
sh 'cat cred.txt | git credential fill || exit 0'
sleep time: 295, unit: 'SECONDS'
}
}
}
}
}
}
Hugo Hoitink
added a comment - - edited I'm a collegue of Hans Koster who added some comments earlier. We are facing this issue on a regular basis. I've put a lot of effort in testing and analyzing it and have come to the following conclusion: The branch-source plugin is managing tokens properly through GIT_ASKPASS , but GIT_ASKPASS is NOT used if the token is still in a git credential helper like 'cache'. That's the one we use with a timeout of 1800 seconds. This timeout is respected, as long as the credential/token in the cache it is not used! In that case, it is stored again by the git client so the timeout is reset (see https://github.com/git/git/blob/master/http.c#L1637 ). So when the credential is reused within the timeout period over and over again, the credential will remain in the git credential cache until the token is expired in GitHub. In that case, the cache will be cleared by the git client after the usage of the cached token failed:
...
returned status code 128:
stdout:
stderr: remote: Invalid username or password.
...
For now I see no other solution then disabling the git credential cache helper.
I used a small Jenkins pipeline job to check if there is an entry in the git credential cache:
pipeline {
agent any
options {
timeout(time: 1, unit: 'HOURS')
timestamps()
}
triggers \{ cron('0 */1 * * *') }
stages {
stage('Get Cache') {
steps {
script {
sh 'git config --list'
writeFile(file: 'cred.txt', text: '''protocol=https\nhost=github.com\n''')
for(i=0;i<12;i++) {
sh 'cat cred.txt | git credential fill || exit 0'
sleep time: 295, unit: 'SECONDS'
}
}
}
}
}
}
Hugo Hoitink
added a comment - Maybe it it possible to directly manage the token-credentials in the credential helper by initiating git credential approve when generating a new token instead of using GIT_ASKPASS which will only be involved when there's nothing in the cache ...
Hugo Hoitink
added a comment - Maybe it it possible to directly manage the token-credentials in the credential helper by initiating git credential approve when generating a new token instead of using GIT_ASKPASS which will only be involved when there's nothing in the cache ... I came across: https://serverfault.com/a/824596/144361
Ideally you could pass HTTPS credentials to Git by just setting some environment variable like GIT_AUTH=git:abcd1234, but AFAIK it is not so straightforward.
Tejaswi Battuwar
added a comment - We at Jenkins (2.313) are still getting the repo not found error only for the private repositories. The plugin 'GitHub branch source' is in its latest upgraded version (2.11.3).
currently we are bypassing this error by adding Jgit as the client instead of default. Any other solution?
Tejaswi Battuwar
added a comment - We at Jenkins (2.313) are still getting the repo not found error only for the private repositories. The plugin 'GitHub branch source' is in its latest upgraded version (2.11.3).
currently we are bypassing this error by adding Jgit as the client instead of default. Any other solution? 
TL;DR: Fixed on our MacOS node with git credential-osxkeychain erase host=github.com protocol=https (then enter again)
We had the same problem on our MacOS node. The problem started after changing the credentials used to acces private Github repositories. The symptoms:
We then enabled GIT_TRACE=1 and GIT_CURL_VERBOSE=1 environment variables in my jobs. We did it via a Git wrapper in /usr/local/bin/git, but it could work also by setting the env variable directly in the job or in the Jenkins global config.
We quickly found out that Git was using the keystore from MacOS with the command with this line:
...
run-command.c:663 trace: run_command: 'git credential-osxkeychain get'
...
And a few lines below, we could see that Git was using the wrong user:
...
Server auth using Basic with user '<wrong-user>'
...
We then removed the credential from the MacOS key store with the following command:
git credential-osxkeychain erase host=github.com protocol=https
After that, jobs running on the MacOS node we working properly again.