Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63146

Pipeline Clone fails on zOS with SSH Key-pair with Passphrase

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: git-client-plugin
    • Labels:
    • Environment:
      Jenkins:2.222.3
      git-client: 3.3.1
      git-plugin: 4.3.0
      z/OS R2.4
      git 2.14
    • Similar Issues:
    • Released As:
      3.4.0

      Description

      When attempting to clone a repository using an SSH key-pair in a pipeline, the clone fails with
      Permission denied (publickey).
      if the key has a passphrase. The authentication succeeds if there is no passphrase. Various encodings have been attempted to get past this, including utf8, ISO8859-1, and IBM-1047.

      We are using an SSH Agent as a workaround for the time being, but that requires manually modifying the git configuration in new workspaces.

        Attachments

          Activity

          rsbeckerca Randall Becker created issue -
          markewaite Mark Waite made changes -
          Field Original Value New Value
          Assignee Mark Waite [ markewaite ]
          Hide
          markewaite Mark Waite added a comment -

          Thanks for the report and good luck with your investigation. I don't have access to z/OS so can only offer to code review proposed changes. Refer to the CredentialsTest in git client plugin if you'd like an automated test environment that can check passphrase protected private keys.

          Show
          markewaite Mark Waite added a comment - Thanks for the report and good luck with your investigation. I don't have access to z/OS so can only offer to code review proposed changes. Refer to the CredentialsTest in git client plugin if you'd like an automated test environment that can check passphrase protected private keys.
          Hide
          rsbeckerca Randall Becker added a comment -

          Thanks. Any pointers on where the passphrase is passed in? I'm guessing this is an encoding issue. The credentials comes from a controller on x86 Gentoo Docker image of Jenkins, going to a z/OS agent. I'm not at all sure what happens with all the if (isZos()) { code branches. Sadly, this is not my native platform either, but I do have a machine on which I can try to debug the agent (with difficulty).

          Show
          rsbeckerca Randall Becker added a comment - Thanks. Any pointers on where the passphrase is passed in? I'm guessing this is an encoding issue. The credentials comes from a controller on x86 Gentoo Docker image of Jenkins, going to a z/OS agent. I'm not at all sure what happens with all the if (isZos()) { code branches. Sadly, this is not my native platform either, but I do have a machine on which I can try to debug the agent (with difficulty).
          Show
          markewaite Mark Waite added a comment - Search for the 'passphrase' in git-client-plugin. https://github.com/jenkinsci/git-client-plugin/blob/eeec334af0b6447f3db9fb88d55728911a092d73/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L1962
          Hide
          rsbeckerca Randall Becker added a comment -

          I am not sure about the line at 2167:

          try (PrintWriter w = new PrintWriter(passphraseFile, "UTF-8")) {
             w.println(Secret.toString(sshUser.getPassphrase()));

          }

          On the zOS machine, I think the passphrase file needs to be encoded in IBM-1047. I will verify with my contacts at IBM on this.

          Show
          rsbeckerca Randall Becker added a comment - I am not sure about the line at 2167: try (PrintWriter w = new PrintWriter(passphraseFile, "UTF-8")) {    w.println(Secret.toString(sshUser.getPassphrase())); } On the zOS machine, I think the passphrase file needs to be encoded in IBM-1047. I will verify with my contacts at IBM on this.
          Hide
          rsbeckerca Randall Becker added a comment -

          So far, it looks like the IBM team is requesting a parameter to set the passphrase encoding rather than hard coding "UTF-8". Would that be possible?

          Show
          rsbeckerca Randall Becker added a comment - So far, it looks like the IBM team is requesting a parameter to set the passphrase encoding rather than hard coding "UTF-8". Would that be possible?
          Hide
          markewaite Mark Waite added a comment -

          Is it the same encoding used for the process on the agent or does it need to be independent of the encoding used for the agent?

          I could envision an agent specific environment variable defined for the zOS agent and not defined on other agents. If that variable is defined, then it could be used when encoding those temporary files that transmit credentials information to command line git.

          If your repositories are not huge, you might consider enabling JGit and using JGit instead of command line git. It does not use temporary files for credentials so it should not rely on file encoding.

          Show
          markewaite Mark Waite added a comment - Is it the same encoding used for the process on the agent or does it need to be independent of the encoding used for the agent? I could envision an agent specific environment variable defined for the zOS agent and not defined on other agents. If that variable is defined, then it could be used when encoding those temporary files that transmit credentials information to command line git. If your repositories are not huge, you might consider enabling JGit and using JGit instead of command line git. It does not use temporary files for credentials so it should not rely on file encoding.
          Hide
          rsbeckerca Randall Becker added a comment -

          The situation is that original passphrase is coming from a UTF8 controller going to an IBM-1047 agent. So yes, this should be something on the agent config. Maybe something like -Dibm.passphrase.encoding=IBM-1047. I already tried JGit - it does not seem to like being on the platform at all. I could not make it work on R2.3 at all - either standalone or inside Jenkins.

          Show
          rsbeckerca Randall Becker added a comment - The situation is that original passphrase is coming from a UTF8 controller going to an IBM-1047 agent. So yes, this should be something on the agent config. Maybe something like -Dibm.passphrase.encoding=IBM-1047. I already tried JGit - it does not seem to like being on the platform at all. I could not make it work on R2.3 at all - either standalone or inside Jenkins.
          Hide
          markewaite Mark Waite added a comment -

          I'd prefer a name like GitSCM.credentials.file.encoding since there may be other platforms that want to change the encoding of the files.

          Can the encoding be used for all the git credentials files (username/password, private key, and passphrase files)?

          Show
          markewaite Mark Waite added a comment - I'd prefer a name like GitSCM.credentials.file.encoding since there may be other platforms that want to change the encoding of the files. Can the encoding be used for all the git credentials files (username/password, private key, and passphrase files)?
          Hide
          rsbeckerca Randall Becker added a comment -

          That name sounds ok. The thing is, it looks like the private key is going across fine, it's just the passphrase that is not working because it's UTF-8. As above, when there is no passphrase, GitSCM authenticates fine and does the clone correctly. It seems to be just the passphrase. Can you MacGyver a version for me with a candidate and I will try it?

          Show
          rsbeckerca Randall Becker added a comment - That name sounds ok. The thing is, it looks like the private key is going across fine, it's just the passphrase that is not working because it's UTF-8. As above, when there is no passphrase, GitSCM authenticates fine and does the clone correctly. It seems to be just the passphrase. Can you MacGyver a version for me with a candidate and I will try it?
          Hide
          markewaite Mark Waite added a comment -

          Not until after work today (6 or 8 hours from now). Git plugin maintenance is a personal time thing.

          Show
          markewaite Mark Waite added a comment - Not until after work today (6 or 8 hours from now). Git plugin maintenance is a personal time thing.
          Hide
          rsbeckerca Randall Becker added a comment -

          At your convenience

          Show
          rsbeckerca Randall Becker added a comment - At your convenience
          markewaite Mark Waite made changes -
          Component/s git-plugin [ 15543 ]
          Hide
          rsbeckerca Randall Becker added a comment -

          I had confirmation of what's going on. On z/OS, files in the USS space (used by Jenkins) there is a tag attribute of each file that describes the encoding of the file as these can be a mixed bag. The default encoding on the box is assumed to be IBM-1047, but may not necessarily be so. The pipeline code seems to be interpreted correctly because the default encoding of the agent was set to ISO8859-1, so the shell steps understand that and pass the contents appropriately. The passphrase file, being hard-coded to UTF-8 and untagged is interpreted by git as IBM-1047, which it is not, so the authentication fails. So looks like this is just the passphrase file, not all credentials. I put in an interception script to change the tag of the passphrase file to ISO8859-1 and that got the authentication to work - a hack not a generic solution. So I think we're on the right track by allowing the passphrase file to be encoded as something other than UTF-8.

          Show
          rsbeckerca Randall Becker added a comment - I had confirmation of what's going on. On z/OS, files in the USS space (used by Jenkins) there is a tag attribute of each file that describes the encoding of the file as these can be a mixed bag. The default encoding on the box is assumed to be IBM-1047, but may not necessarily be so. The pipeline code seems to be interpreted correctly because the default encoding of the agent was set to ISO8859-1, so the shell steps understand that and pass the contents appropriately. The passphrase file, being hard-coded to UTF-8 and untagged is interpreted by git as IBM-1047, which it is not, so the authentication fails. So looks like this is just the passphrase file, not all credentials. I put in an interception script to change the tag of the passphrase file to ISO8859-1 and that got the authentication to work - a hack not a generic solution. So I think we're on the right track by allowing the passphrase file to be encoded as something other than UTF-8.
          markewaite Mark Waite made changes -
          Remote Link This issue links to "PR-585 - set character set based on environment variable (Web Link)" [ 25350 ]
          markewaite Mark Waite made changes -
          Remote Link This issue links to "PR-584 - set passphrase character set based on ibm.system.encoding property (Web Link)" [ 25351 ]
          Hide
          markewaite Mark Waite added a comment - - edited

          I've submitted two pull requests that try to provide a character set adjustment for the zOS environment without risking breaking other use cases.

          The incremental build with the ibm encoding property is [^git-client-3.3.2-PR-584-ibm.encoding.property.hpi]

          The incremental build with the GitSCM.credentials encoding environment variable is [^git-client-3.3.2-PR-585-credentials-file.encoding.hpi]

          Show
          markewaite Mark Waite added a comment - - edited I've submitted two pull requests that try to provide a character set adjustment for the zOS environment without risking breaking other use cases. The incremental build with the ibm encoding property is [^git-client-3.3.2-PR-584-ibm.encoding.property.hpi] The incremental build with the GitSCM.credentials encoding environment variable is [^git-client-3.3.2-PR-585-credentials-file.encoding.hpi]
          markewaite Mark Waite made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          markewaite Mark Waite made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          markewaite Mark Waite made changes -
          Assignee Mark Waite [ markewaite ]
          markewaite Mark Waite made changes -
          Attachment git-client-3.3.2-PR-584-ibm.encoding.property.hpi [ 51999 ]
          markewaite Mark Waite made changes -
          Attachment git-client-3.3.2-PR-585-credentials-file.encoding.hpi [ 52000 ]
          rsbeckerca Randall Becker made changes -
          Comment [ I have to completely rebuild my entire Jenkins agent structure. The dependencies of this fix appears to have caused all agent configurations to be lost. This is going to take a lot of time - as in days. I don't know what happened or how to recover. ]
          Hide
          rsbeckerca Randall Becker added a comment -

          Environment variables cannot have this particular format, as in GitSCM.credentials.file.encoding, at least not on z/OS. I can use -DGitSCM.credentials.file.encoding or a different name, but this combination doesn't work. Can you change this to System.getProperty("GitSCM.credentials.file.encoding") instead of System.getenv("GitSCM.credentials.file.encoding"), please?

          Show
          rsbeckerca Randall Becker added a comment - Environment variables cannot have this particular format, as in GitSCM.credentials.file.encoding, at least not on z/OS. I can use -DGitSCM.credentials.file.encoding or a different name, but this combination doesn't work. Can you change this to System.getProperty("GitSCM.credentials.file.encoding") instead of System.getenv("GitSCM.credentials.file.encoding"), please?
          Hide
          markewaite Mark Waite added a comment - - edited

          Sure, it can use a property instead of an environment variable. New build will be available on the CI job for that PR.

          In order to make the property match with other properties that control the command line git implementation, the name of the property is now:

          org.jenkinsci.plugins.gitclient.CliGitAPIImpl.credentials.file.encoding

          Show
          markewaite Mark Waite added a comment - - edited Sure, it can use a property instead of an environment variable. New build will be available on the CI job for that PR. In order to make the property match with other properties that control the command line git implementation, the name of the property is now: org.jenkinsci.plugins.gitclient.CliGitAPIImpl.credentials.file.encoding
          Hide
          rsbeckerca Randall Becker added a comment -

          CI did not build

          Show
          rsbeckerca Randall Becker added a comment - CI did not build
          Hide
          rsbeckerca Randall Becker added a comment -

          It does not look like the build failure was related to the PR; rather, it looks like the CI job has an environment issue.

          Show
          rsbeckerca Randall Becker added a comment - It does not look like the build failure was related to the PR; rather, it looks like the CI job has an environment issue.
          markewaite Mark Waite made changes -
          Attachment git-client-3.3.2-PR-585-credentials-file.encoding.hpi [ 52000 ]
          markewaite Mark Waite made changes -
          Attachment git-client-3.3.2-PR-584-ibm.encoding.property.hpi [ 51999 ]
          Hide
          rsbeckerca Randall Becker added a comment -

          Currently under test on my z/OS system

          Show
          rsbeckerca Randall Becker added a comment - Currently under test on my z/OS system
          Hide
          rsbeckerca Randall Becker added a comment - - edited

          Looks good. I ran two different jobs - one on a new workspace, one on an existing workspace with no updates. The expected messages came out correctly (repository name changed to protect the guilty):

          > git --version # timeout=10
          using GIT_SSH to set credentials
          Using passphrase charset 'IBM1047'
          > git fetch --tags --progress – git@Bitbucket.org:REPO-SLUG.git +refs/heads/:refs/remotes/origin/ # timeout=10
          Checking out Revision ce210a4618766170a0dd9cc6f2aee4a8563d4070 (refs/remotes/origin/development)

          and we had no authentication issues.

          As a side nit - the plugin does not recognize the repository format - probably also an encoding issue but that would be a separate case.

          My sincere thanks.

          Show
          rsbeckerca Randall Becker added a comment - - edited Looks good. I ran two different jobs - one on a new workspace, one on an existing workspace with no updates. The expected messages came out correctly (repository name changed to protect the guilty): > git --version # timeout=10 using GIT_SSH to set credentials Using passphrase charset 'IBM1047' > git fetch --tags --progress – git@Bitbucket.org:REPO-SLUG.git +refs/heads/ :refs/remotes/origin/ # timeout=10 Checking out Revision ce210a4618766170a0dd9cc6f2aee4a8563d4070 (refs/remotes/origin/development) and we had no authentication issues. As a side nit - the plugin does not recognize the repository format - probably also an encoding issue but that would be a separate case. My sincere thanks.
          markewaite Mark Waite made changes -
          Assignee Mark Waite [ markewaite ]
          Hide
          markewaite Mark Waite added a comment -

          Randall Becker I'm unassigning this issue from me now. I won't merge the proposed pull request until you can confirm that the entire use case is working. Keep this issue updated as you make further progress. Thanks!

          Show
          markewaite Mark Waite added a comment - Randall Becker I'm unassigning this issue from me now. I won't merge the proposed pull request until you can confirm that the entire use case is working. Keep this issue updated as you make further progress. Thanks!
          Hide
          rsbeckerca Randall Becker added a comment -

          Mark Waite We have been using the fix for a week with no issues. I can confirm that the SSH key-pair is correctly authenticating with this option. The repository format recognition is a completely separate issue and unrelated. So the entire use case is working.

          Show
          rsbeckerca Randall Becker added a comment - Mark Waite We have been using the fix for a week with no issues. I can confirm that the SSH key-pair is correctly authenticating with this option. The repository format recognition is a completely separate issue and unrelated. So the entire use case is working.
          Hide
          markewaite Mark Waite added a comment -

          Thanks. Then all I need to do is add the documentation that describes the property. That's a good excuse to document the other properties provided by the plugin as well.

          Show
          markewaite Mark Waite added a comment - Thanks. Then all I need to do is add the documentation that describes the property. That's a good excuse to document the other properties provided by the plugin as well.
          markewaite Mark Waite made changes -
          Assignee Mark Waite [ markewaite ]
          Hide
          markewaite Mark Waite added a comment -

          Randall Becker I've added documentation for the other properties. Does the layout look reasonable to you? If so, then I'll submit a pull request for that documentation, then once it is merged, I'll update the docs in the PR for this property.

          Show
          markewaite Mark Waite added a comment - Randall Becker I've added documentation for the other properties . Does the layout look reasonable to you? If so, then I'll submit a pull request for that documentation, then once it is merged, I'll update the docs in the PR for this property.
          Hide
          rsbeckerca Randall Becker added a comment -

          I like the format. Actually learned something about tags that I was annoying - I want my tags being updated silently as long as they are verified. Side issue. Go for PR

          Show
          rsbeckerca Randall Becker added a comment - I like the format. Actually learned something about tags that I was annoying - I want my tags being updated silently as long as they are verified. Side issue. Go for PR
          markewaite Mark Waite made changes -
          Resolution Fixed [ 1 ]
          Status In Review [ 10005 ] Fixed but Unreleased [ 10203 ]
          Hide
          markewaite Mark Waite added a comment -

          Released in git client plugin 3.4.0 with the properties documented in plugin properties

          Show
          markewaite Mark Waite added a comment - Released in git client plugin 3.4.0 with the properties documented in plugin properties
          markewaite Mark Waite made changes -
          Released As 3.4.0
          Status Fixed but Unreleased [ 10203 ] Closed [ 6 ]

            People

            Assignee:
            markewaite Mark Waite
            Reporter:
            rsbeckerca Randall Becker
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: