Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63909

git credentials fail during check out, after fetching all branches

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Trivial
    • Resolution: Not A Defect
    • Component/s: git-plugin
    • Labels:
    • Environment:
      Jenkins casc 1.44
      github:1.30.0
      running in docker with jenkins/jenkins:2.249.2-lts-alpine
    • Similar Issues:

      Description

      A multibranch pipeline fails to checkout a submodule repo from an on-prem github using user/password credentials.

      The pipeline succeeds first getting all branches from the repo (using the username/password) but then next fail to pull that specific branch.

      Tried with a username/ssh key and git:// scheme, tried also with a https://username:password@ scheme. Each has the same failure.

      It's as if the username/password is forgotten after its first time use to fetch branch information.

      The error goes like this (sensitive parts are replaced):

      Branch indexing
       > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to https://github.private.com/Dep/project.git
       > git config remote.origin.url https://github.private.com/Dep/project.git # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
       > git --version # timeout=10
       > git --version # 'git version 2.26.2'
       > git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials Access github repo
       > git fetch --tags --force --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
      Seen branch in repository origin/Dep/add_submodule
      ... cut some lines here ...
      Seen 5 remote branches
      Obtained Jenkinsfile from 463f3d60c604e489e847e20f880942f4cdf68b0b
      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] Start of Pipeline
      [Pipeline] library
      Loading library jenkins-pipeline-test@Dep/add_submodule
      Attempting to resolve Dep/add_submodule from remote references...
       > git --version # timeout=10
       > git --version # 'git version 2.26.2'
       > git ls-remote -- https://github.private.com/Dep/project # timeout=10
      ERROR: Checkout failed
      hudson.plugins.git.GitException: Command "git ls-remote -- https://github.private.com/Dep/project" returned status code 128:
      stdout: 
      stderr: remote: Invalid username or password.
      fatal: Authentication failed for 'https://github.private.com/Dep/project/'
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2450)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2051)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1951)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1942)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.getRemoteReferences(CliGitAPIImpl.java:3381)
       at jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:838)
       at jenkins.scm.api.SCMSource.fetch(SCMSource.java:636)
       at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$retrieve$0(SCMSourceRetriever.java:92)
       at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:107)
       at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:92)
       at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
       at org.jenkinsci.plugins.workflow.libs.LibraryStep$Execution.run(LibraryStep.java:205)
       at org.jenkinsci.plugins.workflow.libs.LibraryStep$Execution.run(LibraryStep.java:154)
       at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
       at hudson.security.ACL.impersonate(ACL.java:367)
       at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      [Pipeline] End of Pipeline
      ERROR: Maximum checkout retry attempts reached, aborting
      Finished: FAILURE
      

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment - - edited

          Is the submodule defined to use the same protocol and same credentials as the parent repository?

          Authenticated submodule checkout requires that the submodule must be accessed using the same protocol and same credentials as the parent repository. If the original repository is fetched with https (as it appears in your case), then the submodule reference in that branch of the original repository must refer to the submodule using https protocol, not using ssh protocol.

          Is the checkout step defined to use the parent credentials ( parentCredentials: true ) when performing the submodule checkout?

          In my acceptance test job, I see the following definition of the checkout step:

          def branch='JENKINS-38860'
          
          node('git-1.8+') {
            stage('Checkout') {
              checkout([$class: 'GitSCM',
                        userRemoteConfigs: [[url: 'git@github.com:MarkEWaite/jenkins-bugs-private.git',
                                             credentialsId: 'MarkEWaite-github-rsa-private-key',
                                             name: 'bugs-private-origin',
                                             refspec: "+refs/heads/${branch}:refs/remotes/bugs-private-origin/${branch}",
                                            ]],
                        branches: [[name: branch]],
                        extensions: [
                                     [$class: 'SubmoduleOption',
                                       disableSubmodules: false,
                                       parentCredentials: true,
                                       recursiveSubmodules: true,
                                       reference: '/var/lib/git/mwaite/bugs/jenkins-bugs-private.git',
                                       trackingSubmodules: false],
                                     ],
                        gitTool: 'Default', /* Submodule authentication not supported in JGit */
                       ])
            }
          }
          

          The command line git version 2.26.2 has full support for submodules. While investigating to see if there is an issue here, I ran an acceptance test job on agents with Git 2.26.2 (FreeBSD and OpenBSD in my test cases). Those jobs were all successful at submodule checkout.

          If it helps, you're welcome to refer to the public version of my acceptance test job for details that I know are working in my acceptance test environment. I have a similar job that includes credentials and uses https to access a private GitHub repository.

          Show
          markewaite Mark Waite added a comment - - edited Is the submodule defined to use the same protocol and same credentials as the parent repository? Authenticated submodule checkout requires that the submodule must be accessed using the same protocol and same credentials as the parent repository. If the original repository is fetched with https (as it appears in your case), then the submodule reference in that branch of the original repository must refer to the submodule using https protocol, not using ssh protocol. Is the checkout step defined to use the parent credentials ( parentCredentials: true ) when performing the submodule checkout? In my acceptance test job, I see the following definition of the checkout step: def branch='JENKINS-38860' node('git-1.8+') { stage('Checkout') { checkout([$class: 'GitSCM', userRemoteConfigs: [[url: 'git@github.com:MarkEWaite/jenkins-bugs-private.git', credentialsId: 'MarkEWaite-github-rsa-private-key', name: 'bugs-private-origin', refspec: "+refs/heads/${branch}:refs/remotes/bugs-private-origin/${branch}", ]], branches: [[name: branch]], extensions: [ [$class: 'SubmoduleOption', disableSubmodules: false, parentCredentials: true, recursiveSubmodules: true, reference: '/var/lib/git/mwaite/bugs/jenkins-bugs-private.git', trackingSubmodules: false], ], gitTool: 'Default', /* Submodule authentication not supported in JGit */ ]) } } The command line git version 2.26.2 has full support for submodules. While investigating to see if there is an issue here, I ran an acceptance test job on agents with Git 2.26.2 (FreeBSD and OpenBSD in my test cases). Those jobs were all successful at submodule checkout. If it helps, you're welcome to refer to the public version of my acceptance test job for details that I know are working in my acceptance test environment. I have a similar job that includes credentials and uses https to access a private GitHub repository.
          Hide
          jenkins_in_be F M added a comment -

          Thanks for looking into it.Any idea where to look at next (your test script works for me; see below)?

          Note that the branch I try to checkout/test is the only one with submodules in that repo yet.

          • I updated the Docker FROM jenkins/jenkins:2.260-alpine and and also downgraded with an older version (but got the same issues).
          • I made sure to use the git:// scheme (the way submodules are referenced in .gitmodules) and used my own SSH credentials (but got the same issue).

          I used your test script pipeline with some ls -al to verify to get the expected files. This passes!

          I see in this console logging multiple times:

          using GIT_SSH to set credentials Access github repo - me

          But I don't see this when using the multibranch pipeline at the time it fails I think that's the cause. For completeness I added the test script's output (the actual checkout of git submodule update --init --recursive submodule is removed):

          Started by user admin
          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] Start of Pipeline
          [Pipeline] node
          Running on Jenkins in /var/jenkins_home/workspace/simple
          [Pipeline] {
          [Pipeline] stage
          [Pipeline] { (Checkout)
          [Pipeline] checkout
          The recommended git tool is: git
          using credential 1d634ef3-2d37-4386-a736-4008b2d118db
           > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
           > git config remote.checkout_test.url git@github.private.com:Dep/project.git # timeout=10
          Fetching upstream changes from git@github.private.com:Dep/project.git
           > git --version # timeout=10
           > git --version # 'git version 2.26.2'
          using GIT_SSH to set credentials Access  github repo - me
           > git fetch --tags --force --progress -- git@github.private.com:Dep/project.git +refs/heads/dep/add_submodule:refs/remotes/checkout_test/dep/add_submodule # timeout=10
           > git rev-parse refs/remotes/checkout_test/dep/add_submodule^{commit} # timeout=10
           > git rev-parse dep/add_submodule^{commit} # timeout=10
          Checking out Revision 463f3d60c604e489e847e20f880942f4cdf68b0b (refs/remotes/checkout_test/dep/add_submodule)
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 463f3d60c604e489e847e20f880942f4cdf68b0b # timeout=10
          Commit message: "test directory is gone"
           > git rev-list --no-walk 463f3d60c604e489e847e20f880942f4cdf68b0b # timeout=10
           > git remote # timeout=10
           > git submodule init # timeout=10
           > git submodule sync # timeout=10
           > git config --get remote.bugs-private-origin.url # timeout=10
           > git submodule init # timeout=10
           > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
           > git config --get submodule.submodule.url # timeout=10
           > git remote # timeout=10
           > git config --get remote.bugs-private-origin.url # timeout=10
           > git config -f .gitmodules --get submodule.submodule.path # timeout=10
          using GIT_SSH to set credentials Access  github repo - me
           > git submodule update --init --recursive submodule # timeout=10
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] sh
          + ls -al
          total 124
          ... lines cut here ...
          [Pipeline] sh
          + ls -al submodule
          total 104
          ... lines cut here ...
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESSREST API
          
          

           

           

           

          Show
          jenkins_in_be F M added a comment - Thanks for looking into it.Any idea where to look at next (your test script works for me; see below)? Note that the branch I try to checkout/test is the only one with submodules in that repo yet. I updated the Docker FROM jenkins/jenkins:2.260-alpine and and also downgraded with an older version (but got the same issues). I made sure to use the git:// scheme (the way submodules are referenced in .gitmodules) and used my own SSH credentials (but got the same issue). I used your test script pipeline with some ls -al to verify to get the expected files. This passes! I see in this console logging multiple times: using GIT_SSH to set credentials Access github repo - me But I don't see this when using the multibranch pipeline at the time it fails I think that's the cause. For completeness I added the test script's output (the actual checkout of git submodule update --init --recursive submodule is removed): Started by user admin Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in / var /jenkins_home/workspace/simple [Pipeline] { [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] checkout The recommended git tool is: git using credential 1d634ef3-2d37-4386-a736-4008b2d118db > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.checkout_test.url git@github. private .com:Dep/project.git # timeout=10 Fetching upstream changes from git@github. private .com:Dep/project.git > git --version # timeout=10 > git --version # 'git version 2.26.2' using GIT_SSH to set credentials Access github repo - me > git fetch --tags --force --progress -- git@github. private .com:Dep/project.git +refs/heads/dep/add_submodule:refs/remotes/checkout_test/dep/add_submodule # timeout=10 > git rev-parse refs/remotes/checkout_test/dep/add_submodule^{commit} # timeout=10 > git rev-parse dep/add_submodule^{commit} # timeout=10 Checking out Revision 463f3d60c604e489e847e20f880942f4cdf68b0b (refs/remotes/checkout_test/dep/add_submodule) > git config core.sparsecheckout # timeout=10 > git checkout -f 463f3d60c604e489e847e20f880942f4cdf68b0b # timeout=10 Commit message: "test directory is gone" > git rev-list --no-walk 463f3d60c604e489e847e20f880942f4cdf68b0b # timeout=10 > git remote # timeout=10 > git submodule init # timeout=10 > git submodule sync # timeout=10 > git config --get remote.bugs- private -origin.url # timeout=10 > git submodule init # timeout=10 > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10 > git config --get submodule.submodule.url # timeout=10 > git remote # timeout=10 > git config --get remote.bugs- private -origin.url # timeout=10 > git config -f .gitmodules --get submodule.submodule.path # timeout=10 using GIT_SSH to set credentials Access github repo - me > git submodule update --init --recursive submodule # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] sh + ls -al total 124 ... lines cut here ... [Pipeline] sh + ls -al submodule total 104 ... lines cut here ... [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESSREST API      
          Hide
          markewaite Mark Waite added a comment -

          The logfile reports that you are using git@github.private.com:Dep/project.git as the repository URL. That uses ssh protocol and requires an ssh private key to authenticate. If you retrieve the initial repository with ssh, then the submodule must be defined to use ssh as well. One way to check that you're using consistent protocol for parent repository and for the submodules is:

          $ git remote -v
          origin  git@github.com:MarkEWaite/parent-private.git (fetch)
          origin  git@github.com:MarkEWaite/parent-private.git (push)
          
          $ git submodule foreach git remote -v
          Entering 'test-submodules/JENKINS-43022-325'
          origin  git@github.com:MarkEWaite/submodule-private.git (fetch)
          origin  git@github.com:MarkEWaite/submodule-private.git (push)
          

          In that output, you see that the submodule uses git@github.com just as the parent repository does.

          Show
          markewaite Mark Waite added a comment - The logfile reports that you are using git@github.private.com:Dep/project.git as the repository URL. That uses ssh protocol and requires an ssh private key to authenticate. If you retrieve the initial repository with ssh, then the submodule must be defined to use ssh as well. One way to check that you're using consistent protocol for parent repository and for the submodules is: $ git remote -v origin git@github.com:MarkEWaite/parent-private.git (fetch) origin git@github.com:MarkEWaite/parent-private.git (push) $ git submodule foreach git remote -v Entering 'test-submodules/JENKINS-43022-325' origin git@github.com:MarkEWaite/submodule-private.git (fetch) origin git@github.com:MarkEWaite/submodule-private.git (push) In that output, you see that the submodule uses git@github.com just as the parent repository does.
          Hide
          jenkins_in_be F M added a comment -

          I'm sorry, I found the mistake to be on my side. The multibranch is handling the credentials correctly.

          Show
          jenkins_in_be F M added a comment - I'm sorry, I found the mistake to be on my side. The multibranch is handling the credentials correctly.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jenkins_in_be F M
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: