Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60440

Invalid git username/password on Jenkins agent when using Vault Username-Password Credential with '@' in username

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • VM host: Windows 10 1909 running Docker Desktop Community 19.03.5
      Jenkins master: Docker image jenkins/jenkins:latest
      Jenkins agent: Docker image openjdk:8-stretch + Swarm-Client 3.17 plugin
      Hashicorp Vault: Docker image vault:latest

      My Jenkins master is running Debian 9 with Jenkins 2.208, Git-plugin 4.0.0 and Hashicorp-vault-plugin 3.0.0. My Jenkins agent is running Debian 9 with Swarm-client plugin 3.17. My master is set to 0 executors so that all jobs run on the agent. I have my Bitbucket credentials saved in Jenkins 3 times - "Username with password", "Vault Username-Password Credential" with K/V engine 1 and "Vault Username-Password Credential" with K/V engine 2.

      I have a test freestyle job that does nothing except fetch a Git repository from https://bitbucket.org/... If set the job's Git credentials to use the "Username with password" credentials then the agent successfully fetches the repository. If I use either of the "Vault Username-Password Credential" credentials then the agent fails on the command "git fetch --tags --progress ..." with "remote: Invalid username or password"

      In a pipeline job with script from SCM, the master is able to fetch the repository with all 3 credential types but the agent can only fetch when using "Username with password" credentials - it is unable to fetch with "Vault Username-Password Credential" credentials.

      Bitbucket usernames are email addresses so they contain "@" special character. In Vault I have tried URL encoding the username to "user%40domain.com" but this causes the master to also fail with invalid username.

          [JENKINS-60440] Invalid git username/password on Jenkins agent when using Vault Username-Password Credential with '@' in username

          Mark Waite added a comment - - edited

          Submitter notes that an @ sign embedded in the username will cause authentication failures in the git client plugin. Also an issue for the google code repositories since their user names include an @ sign as well.

          I was not aware of Bitbucket Cloud supporting a username which includes an @ character. My Bitbucket Cloud account username (used to perform the clone) does not contain an embedded @ character.

          I assume the use of an embedded @ character in the username is used on Bitbucket Server and Bitbucket Data Center. I use markewaite as my Bitbucket Cloud username. Bitbucket Cloud knows my google e-mail address and has connected my google e-mail address to my Bitbucket Cloud account.

          Can you define a username in Bitbucket server that does not include the @ character in the username?

          Are you able to define an app password in Bitbucket Cloud, store that app password in Hashicorp Vault, and use that app password as part of a Vault username / password credential?

          Mark Waite added a comment - - edited Submitter notes that an @ sign embedded in the username will cause authentication failures in the git client plugin. Also an issue for the google code repositories since their user names include an @ sign as well. I was not aware of Bitbucket Cloud supporting a username which includes an @ character. My Bitbucket Cloud account username (used to perform the clone) does not contain an embedded @ character. I assume the use of an embedded @ character in the username is used on Bitbucket Server and Bitbucket Data Center. I use markewaite as my Bitbucket Cloud username . Bitbucket Cloud knows my google e-mail address and has connected my google e-mail address to my Bitbucket Cloud account. Can you define a username in Bitbucket server that does not include the @ character in the username? Are you able to define an app password in Bitbucket Cloud, store that app password in Hashicorp Vault, and use that app password as part of a Vault username / password credential?

          Gordon Li added a comment -

          Bitbucket.org no longer allows logging in with a username - they only allow email address or Google/Microsoft logins.

          I don't think @ character is the problem because the Git plugin works fine if my Bitbucket credentials are stored in Jenkins as "Username with password". Also note that it's only the agent server that fails to use a Vault username/password to checkout a Git repository. The master server is correctly setting the Git credentials and checking out the retrieve the pipeline's jenkinsfile.

          Gordon Li added a comment - Bitbucket.org no longer allows logging in with a username - they only allow email address or Google/Microsoft logins. I don't think @ character is the problem because the Git plugin works fine if my Bitbucket credentials are stored in Jenkins as "Username with password". Also note that it's only the agent server that fails to use a Vault username/password to checkout a Git repository. The master server is correctly setting the Git credentials and checking out the retrieve the pipeline's jenkinsfile.

          Mark Waite added a comment - - edited

          I agree that they don't allow login with a simple username that is not an e-mail address. As far as I can tell, they do seem to allow clone with a simple username even when I login with my e-mail address. My question was attempting to find an alternative that will allow you to operate in your environment without requiring a change from the git client plugin.

          I don't think that there are major differences between the use of the username / password credential on the master and the use of the username / password credential on the agent. However, there must be enough of a difference to be creating the issue you're seeing. If the failing pipeline checkout operation inside the Jenkinsfile is intentionally executed on the master, does it fail in the same way, or does it succeed?

          Mark Waite added a comment - - edited I agree that they don't allow login with a simple username that is not an e-mail address. As far as I can tell, they do seem to allow clone with a simple username even when I login with my e-mail address. My question was attempting to find an alternative that will allow you to operate in your environment without requiring a change from the git client plugin. I don't think that there are major differences between the use of the username / password credential on the master and the use of the username / password credential on the agent. However, there must be enough of a difference to be creating the issue you're seeing. If the failing pipeline checkout operation inside the Jenkinsfile is intentionally executed on the master, does it fail in the same way, or does it succeed?

          Gordon Li added a comment -

          The pipeline from SCM job checks out the repo on the master first in order to get the jenkinsfile, then the agent checks out the repo again in order to execute the jenkinsfile. The master is always about to check out while the slave is unable to check out using Vault username/password.

          On further investigation, I suspect that on the agent, GIT_ASKPASS doesn't get configured correctly when using Vault. If I use a pipeline script defined in the job rather than from SCM, I can set my Git credentials to environment variables GITUSER and GITPASS and execute the following steps to manually configure GIT_ASKPASS before checking out a repository.

          sh "git config credential.helper '!f() { sleep 1; echo \"username=${GITUSER}\"; echo \"password=${GITPASS}\"; }; f'"
          
          git url: 'https://bitbucket.org/<username>/<repository>.git'
          

          This runs successfully on a Linux agent. I haven't figured out how to replicate the git config credential.helper function on Windows so my Windows workaround instead is

          git url: "https://${env.GITUSER}:${env.GITPASS}@bitbucket.org/<username>/<repository>.git"
          

           

          Gordon Li added a comment - The pipeline from SCM job checks out the repo on the master first in order to get the jenkinsfile, then the agent checks out the repo again in order to execute the jenkinsfile. The master is always about to check out while the slave is unable to check out using Vault username/password. On further investigation, I suspect that on the agent, GIT_ASKPASS doesn't get configured correctly when using Vault. If I use a pipeline script defined in the job rather than from SCM, I can set my Git credentials to environment variables GITUSER and GITPASS and execute the following steps to manually configure GIT_ASKPASS before checking out a repository. sh "git config credential.helper '!f() { sleep 1; echo \" username=${GITUSER}\ "; echo \" password=${GITPASS}\ "; }; f' " git url: 'https: //bitbucket.org/<username>/<repository>.git' This runs successfully on a Linux agent. I haven't figured out how to replicate the git config credential.helper function on Windows so my Windows workaround instead is git url: "https: //${env.GITUSER}:${env.GITPASS}@bitbucket.org/<username>/<repository>.git"  

          Josip Gracin added a comment -

          Hi! Just confirming that I have the same issue with GitHub.

          Josip Gracin added a comment - Hi! Just confirming that I have the same issue with GitHub.

          Jibin Babu added a comment -

          can anyone say what should be the value of "username Key" and "Password key" when we use Vault username password credential?. I have my Gitlab username and password stored in the Vault. I need it to authenticate on my pipeline job where I fetch files from SCM.

          Jibin Babu added a comment - can anyone say what should be the value of "username Key" and "Password key" when we use Vault username password credential?. I have my Gitlab username and password stored in the Vault. I need it to authenticate on my pipeline job where I fetch files from SCM.

            Unassigned Unassigned
            elgordo Gordon Li
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: