[JENKINS-62249] Unreliable authentication when using the GitHub App credentials - Jenkins Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Component/s: github-branch-source-plugin
Labels:
None

Similar Issues:

Show
Released As:
github-branch-source-2.8.2, github-branch-source-2.9.0

Description

github-branch-source-plugin v2.7.1 allows to configure Jenkins as a GitHub App, which

After upgrading to github-branch-source-plugin v2.7.1, I followed the GitHub app authentication guide to connect Jenkins to GitHub.

I can successfully clone our repositories using these new credentials most of the time, but I can also observe intermittent authentication failures.

I have some builds that manage to successfully execute `checkout scm` using the new credentials 10 times in a row, but the 11th time, it fails with:

using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization
 > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: 
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
	... 1 more

(I don't have more details)

Then, subsequent builds fail with a slighly better error:

using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization
 > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Repository not found.
fatal: Authentication failed for 'https://github.com/my-org/my-repo/'

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
	... 1 more

Usually, this lasts until we destroy the Jenkins worker used for this build and create a brand new one.

There are no changes, either on Jenkins or on GitHub between that 10th build (which was successful) and the subsequent builds (which are failing). It was easily reproducible last week (it was failing after ~5 or 10 builds). I don't have any suspicious logs about these failures anymore.

Attachments

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Attachments

github-notfound-1.jpg
1.41 MB
2020-09-10 10:05
github-notfound-2.png
16 kB
2020-09-10 10:06
github-notfound-3.jpg
1.49 MB
2020-09-10 10:06

Issue Links

is duplicated by

JENKINS-64911 Github Branch Source: rate limit sleep interrupted

Closed

links to

PR 326

Activity

Ascending order - Click to sort in descending order

38 older comments

Hide

Permalink

Liam Newman added a comment - 2021-04-14 05:44

Jt
The output you've provided supports the idea this is synchronization issue on the GitHub cloud.

But I could still use the output as described in this comment: https://issues.jenkins.io/browse/JENKINS-62249?focusedCommentId=402723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-402723

That would give a clearer picture of what is going on.

Show

Liam Newman added a comment - 2021-04-14 05:44 Jt The output you've provided supports the idea this is synchronization issue on the GitHub cloud. But I could still use the output as described in this comment: https://issues.jenkins.io/browse/JENKINS-62249?focusedCommentId=402723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-402723 That would give a clearer picture of what is going on.

Hide

Permalink

Dev added a comment - 2021-04-19 15:24

TL;DR: Fixed on our MacOS node with git credential-osxkeychain erase host=github.com protocol=https (then enter again)

We had the same problem on our MacOS node. The problem started after changing the credentials used to acces private Github repositories. The symptoms:

Restart Jenkins
Jobs using a private repo are working correctly
Wait about 2 days
Jobs using a private repo are still working correctly
Suddenly, jobs using a private repo on the MacOS node start giving the stderr: remote: Repository not found error. After that, no other jobs on the node can access private repositories.

We then enabled GIT_TRACE=1 and GIT_CURL_VERBOSE=1 environment variables in my jobs. We did it via a Git wrapper in /usr/local/bin/git, but it could work also by setting the env variable directly in the job or in the Jenkins global config.

We quickly found out that Git was using the keystore from MacOS with the command with this line:

...
run-command.c:663 trace: run_command: 'git credential-osxkeychain get'
...

And a few lines below, we could see that Git was using the wrong user:
...
Server auth using Basic with user '<wrong-user>'
...

We then removed the credential from the MacOS key store with the following command:

git credential-osxkeychain erase host=github.com protocol=https

After that, jobs running on the MacOS node we working properly again.

Show

Dev added a comment - 2021-04-19 15:24 TL;DR: Fixed on our MacOS node with git credential-osxkeychain erase host=github.com protocol=https (then enter again) We had the same problem on our MacOS node. The problem started after changing the credentials used to acces private Github repositories. The symptoms: Restart Jenkins Jobs using a private repo are working correctly Wait about 2 days Jobs using a private repo are still working correctly Suddenly, jobs using a private repo on the MacOS node start giving the stderr: remote: Repository not found error. After that, no other jobs on the node can access private repositories. We then enabled GIT_TRACE=1 and GIT_CURL_VERBOSE=1 environment variables in my jobs. We did it via a Git wrapper in /usr/local/bin/git , but it could work also by setting the env variable directly in the job or in the Jenkins global config. We quickly found out that Git was using the keystore from MacOS with the command with this line: ... run-command.c:663 trace: run_command: 'git credential-osxkeychain get' ... And a few lines below, we could see that Git was using the wrong user: ... Server auth using Basic with user '<wrong-user>' ... We then removed the credential from the MacOS key store with the following command: git credential-osxkeychain erase host=github.com protocol=https After that, jobs running on the MacOS node we working properly again.

Hide

Permalink

Hugo Hoitink added a comment - 2021-05-06 11:37 - edited

I'm a collegue of Hans Koster who added some comments earlier. We are facing this issue on a regular basis. I've put a lot of effort in testing and analyzing it and have come to the following conclusion: The branch-source plugin is managing tokens properly through GIT_ASKPASS, but GIT_ASKPASS is NOT used if the token is still in a git credential helper like 'cache'. That's the one we use with a timeout of 1800 seconds. This timeout is respected, as long as the credential/token in the cache it is not used! In that case, it is stored again by the git client so the timeout is reset (see https://github.com/git/git/blob/master/http.c#L1637). So when the credential is reused within the timeout period over and over again, the credential will remain in the git credential cache until the token is expired in GitHub. In that case, the cache will be cleared by the git client after the usage of the cached token failed:

...
returned status code 128:
stdout: 
stderr: remote: Invalid username or password.
...

For now I see no other solution then disabling the git credential cache helper.

I used a small Jenkins pipeline job to check if there is an entry in the git credential cache:

pipeline {
  agent any

  options {
    timeout(time: 1, unit: 'HOURS')
    timestamps()
  }

  triggers \{ cron('0 */1 * * *') }

  stages {
    stage('Get Cache') {
      steps {
        script {
          sh 'git config --list'
          writeFile(file: 'cred.txt', text: '''protocol=https\nhost=github.com\n''')
          for(i=0;i<12;i++) {
            sh 'cat cred.txt | git credential fill || exit 0'
            sleep time: 295, unit: 'SECONDS'
          }
        }
      }
    }
  }
}

Show

Hugo Hoitink added a comment - 2021-05-06 11:37 - edited I'm a collegue of Hans Koster who added some comments earlier. We are facing this issue on a regular basis. I've put a lot of effort in testing and analyzing it and have come to the following conclusion: The branch-source plugin is managing tokens properly through GIT_ASKPASS , but GIT_ASKPASS is NOT used if the token is still in a git credential helper like 'cache'. That's the one we use with a timeout of 1800 seconds. This timeout is respected, as long as the credential/token in the cache it is not used! In that case, it is stored again by the git client so the timeout is reset (see https://github.com/git/git/blob/master/http.c#L1637 ). So when the credential is reused within the timeout period over and over again, the credential will remain in the git credential cache until the token is expired in GitHub. In that case, the cache will be cleared by the git client after the usage of the cached token failed: ... returned status code 128: stdout: stderr: remote: Invalid username or password. ... For now I see no other solution then disabling the git credential cache helper. I used a small Jenkins pipeline job to check if there is an entry in the git credential cache: pipeline { agent any options { timeout(time: 1, unit: 'HOURS') timestamps() } triggers \{ cron('0 */1 * * *') } stages { stage('Get Cache') { steps { script { sh 'git config --list' writeFile(file: 'cred.txt', text: '''protocol=https\nhost=github.com\n''') for(i=0;i<12;i++) { sh 'cat cred.txt | git credential fill || exit 0' sleep time: 295, unit: 'SECONDS' } } } } } }

Hide

Permalink

Hugo Hoitink added a comment - 2021-05-07 17:51

Maybe it it possible to directly manage the token-credentials in the credential helper by initiating git credential approve when generating a new token instead of using GIT_ASKPASS which will only be involved when there's nothing in the cache ...

Show

Hugo Hoitink added a comment - 2021-05-07 17:51 Maybe it it possible to directly manage the token-credentials in the credential helper by initiating git credential approve when generating a new token instead of using GIT_ASKPASS which will only be involved when there's nothing in the cache ...

Hide

Permalink

Jesse Glick added a comment - 2021-05-07 18:42

I came across: https://serverfault.com/a/824596/144361

Ideally you could pass HTTPS credentials to Git by just setting some environment variable like GIT_AUTH=git:abcd1234, but AFAIK it is not so straightforward.

Show

Jesse Glick added a comment - 2021-05-07 18:42 I came across: https://serverfault.com/a/824596/144361 Ideally you could pass HTTPS credentials to Git by just setting some environment variable like GIT_AUTH=git:abcd1234 , but AFAIK it is not so straightforward.

Jenkins

Unreliable authentication when using the GitHub App credentials

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates