Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62249

Unreliable authentication when using the GitHub App credentials

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      github-branch-source-2.8.2, github-branch-source-2.9.0

      Description

      github-branch-source-plugin v2.7.1 allows to configure Jenkins as a GitHub App, which

      After upgrading to github-branch-source-plugin v2.7.1, I followed the GitHub app authentication guide to connect Jenkins to GitHub.

      I can successfully clone our repositories using these new credentials most of the time, but I can also observe intermittent authentication failures.

      I have some builds that manage to successfully execute `checkout scm` using the new credentials 10 times in a row, but the 11th time, it fails with:

      using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization
       > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10
      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: 
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
      	... 1 more
      

      (I don't have more details)

      Then, subsequent builds fail with a slighly better error:

      using GIT_ASKPASS to set credentials Jenkins as a GitHub App for the my-org organization
       > git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/* # timeout=10
      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from https://github.com/my-org/my-repo
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:909)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1131)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1167)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress -- https://github.com/my-org/my-repo +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: remote: Repository not found.
      fatal: Authentication failed for 'https://github.com/my-org/my-repo/'
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:81)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:569)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
      	... 1 more
      

      Usually, this lasts until we destroy the Jenkins worker used for this build and create a brand new one.

      There are no changes, either on Jenkins or on GitHub between that 10th build (which was successful) and the subsequent builds (which are failing). It was easily reproducible last week (it was failing after ~5 or 10 builds). I don't have any suspicious logs about these failures anymore.

        Attachments

          Issue Links

            Activity

            Hide
            bitwiseman Liam Newman added a comment -

            Jt
            The output you've provided supports the idea this is synchronization issue on the GitHub cloud.

            But I could still use the output as described in this comment: https://issues.jenkins.io/browse/JENKINS-62249?focusedCommentId=402723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-402723

            That would give a clearer picture of what is going on.

            Show
            bitwiseman Liam Newman added a comment - Jt The output you've provided supports the idea this is synchronization issue on the GitHub cloud. But I could still use the output as described in this comment: https://issues.jenkins.io/browse/JENKINS-62249?focusedCommentId=402723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-402723 That would give a clearer picture of what is going on.
            Hide
            wazo_dev Dev added a comment -

            TL;DR: Fixed on our MacOS node with git credential-osxkeychain erase host=github.com protocol=https (then enter again)

            We had the same problem on our MacOS node. The problem started after changing the credentials used to acces private Github repositories. The symptoms:

            • Restart Jenkins
            • Jobs using a private repo are working correctly
            • Wait about 2 days
            • Jobs using a private repo are still working correctly
            • Suddenly, jobs using a private repo on the MacOS node start giving the stderr: remote: Repository not found error. After that, no other jobs on the node can access private repositories.

            We then enabled GIT_TRACE=1 and GIT_CURL_VERBOSE=1 environment variables in my jobs. We did it via a Git wrapper in /usr/local/bin/git, but it could work also by setting the env variable directly in the job or in the Jenkins global config.

            We quickly found out that Git was using the keystore from MacOS with the command with this line:

            ...
            run-command.c:663 trace: run_command: 'git credential-osxkeychain get'
            ...

            And a few lines below, we could see that Git was using the wrong user:
            ...
            Server auth using Basic with user '<wrong-user>'
            ...

            We then removed the credential from the MacOS key store with the following command:

            git credential-osxkeychain erase host=github.com protocol=https

            After that, jobs running on the MacOS node we working properly again.

            Show
            wazo_dev Dev added a comment - TL;DR: Fixed on our MacOS node with git credential-osxkeychain erase host=github.com protocol=https (then enter again) We had the same problem on our MacOS node. The problem started after changing the credentials used to acces private Github repositories. The symptoms: Restart Jenkins Jobs using a private repo are working correctly Wait about 2 days Jobs using a private repo are still working correctly Suddenly, jobs using a private repo on the MacOS node start giving the stderr: remote: Repository not found error. After that, no other jobs on the node can access private repositories. We then enabled GIT_TRACE=1 and GIT_CURL_VERBOSE=1 environment variables in my jobs. We did it via a Git wrapper in /usr/local/bin/git , but it could work also by setting the env variable directly in the job or in the Jenkins global config. We quickly found out that Git was using the keystore from MacOS with the command with this line: ... run-command.c:663 trace: run_command: 'git credential-osxkeychain get' ... And a few lines below, we could see that Git was using the wrong user: ... Server auth using Basic with user '<wrong-user>' ... We then removed the credential from the MacOS key store with the following command: git credential-osxkeychain erase host=github.com protocol=https After that, jobs running on the MacOS node we working properly again.
            Hide
            hoitih Hugo Hoitink added a comment - - edited

            I'm a collegue of Hans Koster who added some comments earlier. We are facing this issue on a regular basis. I've put a lot of effort in testing and analyzing it and have come to the following conclusion: The branch-source plugin is managing tokens properly through GIT_ASKPASS, but GIT_ASKPASS is NOT used if the token is still in a git credential helper like 'cache'. That's the one we use with a timeout of 1800 seconds. This timeout is respected, as long as the credential/token in the cache it is not used! In that case, it is stored again by the git client so the timeout is reset (see https://github.com/git/git/blob/master/http.c#L1637). So when the credential is reused within the timeout period over and over again, the credential will remain in the git credential cache until the token is expired in GitHub. In that case, the cache will be cleared by the git client after the usage of the cached token failed:

            ...
            returned status code 128:
            stdout: 
            stderr: remote: Invalid username or password.
            ...
            

            For now I see no other solution then disabling the git credential cache helper.

            I used a small Jenkins pipeline job to check if there is an entry in the git credential cache:

            pipeline {
              agent any
            
              options {
                timeout(time: 1, unit: 'HOURS')
                timestamps()
              }
            
              triggers \{ cron('0 */1 * * *') }
            
              stages {
                stage('Get Cache') {
                  steps {
                    script {
                      sh 'git config --list'
                      writeFile(file: 'cred.txt', text: '''protocol=https\nhost=github.com\n''')
                      for(i=0;i<12;i++) {
                        sh 'cat cred.txt | git credential fill || exit 0'
                        sleep time: 295, unit: 'SECONDS'
                      }
                    }
                  }
                }
              }
            }
            Show
            hoitih Hugo Hoitink added a comment - - edited I'm a collegue of Hans Koster who added some comments earlier. We are facing this issue on a regular basis. I've put a lot of effort in testing and analyzing it and have come to the following conclusion: The branch-source plugin is managing tokens properly through GIT_ASKPASS , but GIT_ASKPASS is NOT used if the token is still in a git credential helper like 'cache'. That's the one we use with a timeout of 1800 seconds. This timeout is respected, as long as the credential/token in the cache it is not used! In that case, it is stored again by the git client so the timeout is reset (see https://github.com/git/git/blob/master/http.c#L1637 ). So when the credential is reused within the timeout period over and over again, the credential will remain in the git credential cache until the token is expired in GitHub. In that case, the cache will be cleared by the git client after the usage of the cached token failed: ... returned status code 128: stdout: stderr: remote: Invalid username or password. ... For now I see no other solution then disabling the git credential cache helper. I used a small Jenkins pipeline job to check if there is an entry in the git credential cache: pipeline { agent any options { timeout(time: 1, unit: 'HOURS') timestamps() } triggers \{ cron('0 */1 * * *') } stages { stage('Get Cache') { steps { script { sh 'git config --list' writeFile(file: 'cred.txt', text: '''protocol=https\nhost=github.com\n''') for(i=0;i<12;i++) { sh 'cat cred.txt | git credential fill || exit 0' sleep time: 295, unit: 'SECONDS' } } } } } }
            Hide
            hoitih Hugo Hoitink added a comment -

            Maybe it it possible to directly manage the token-credentials in the credential helper by initiating git credential approve when generating a new token instead of using GIT_ASKPASS which will only be involved when there's nothing in the cache ...

            Show
            hoitih Hugo Hoitink added a comment - Maybe it it possible to directly manage the token-credentials in the credential helper by initiating git credential approve when generating a new token instead of using GIT_ASKPASS which will only be involved when there's nothing in the cache ...
            Hide
            jglick Jesse Glick added a comment -

            I came across: https://serverfault.com/a/824596/144361

            Ideally you could pass HTTPS credentials to Git by just setting some environment variable like GIT_AUTH=git:abcd1234, but AFAIK it is not so straightforward.

            Show
            jglick Jesse Glick added a comment - I came across: https://serverfault.com/a/824596/144361 Ideally you could pass HTTPS credentials to Git by just setting some environment variable like GIT_AUTH=git:abcd1234 , but AFAIK it is not so straightforward.

              People

              Assignee:
              bitwiseman Liam Newman
              Reporter:
              multani Jonathan Ballet
              Votes:
              13 Vote for this issue
              Watchers:
              27 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: