Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64913

CLI git access over ssh fails with SELinux Enforcing

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • git-client-plugin
    • None
    • Jenkins LTS 2.263.x
      git and branch related packages updated 2021-02-22 to whatever is current AND applicable to 2.263.x baseline
      CentOS 7 with SELinux enabled and enforcing

    Description

      While setting up a Jenkins controller instance (and a worker using SSH Build Agent on same machine in different account) I've had some hurdles due to SELinux enabled and desired by machine's owners, so the majority of internet lore about disabling it wherever you see it did not apply Out of context here is probably allowing HTTP port usage for the Jenkins service, which was relatively easy.

      The big problem is using git checkouts over ssh, using the SSH Credentials (name + private key + passphrase) saved in Jenkins Credential store. Configurations that healthily worked for me elsewhere, using the checkout([$class...]) magic (job needing to fetch from several repos and platforms), failed here with "Permission denied" and/or with "Check if you are allowed access" sort of messages.

      I hope to post methodology details below, but in short it was a big hunt in the terminal with loops to ln the temporary files from the build workspace into a safe haven, and to detail the command lines and environment variables of ssh and git processes spawned during a failed job run. Eventually I could grab the files that Jenkins generates and reproduce the failure from command-line - just using the same ssh wrapping script, ssh-askpass script, password and private key files that Jenkins makes, and failing to just connect the ssh client to Git platform with no git involved.

      One big discovery was that the internet lore is right Running setenforce 0 (as root) allows Jenkins git+ssh access to work as expected, as well as the manually-made reproduction for same sort of SSH access in shell. But that is not what the security-conscious customer wants.

      As soon as we restore setenforce 1, ssh client fails again with "Permission denied". This was strace'd to be actually a local filesystem error in the openat() syscall, not a message from remote Git server about some key it won't trust (and actually does trust in other client conditions). It was also traced that while the Jenkins generated wrapper script calls ssh -i /path/to/temporary.key and tries to access that key, fails, and does not look into /.ssh/ directory (which is correct with that CLI option), placing a copy of the same key file (without a passphrase for simplicity) into /.ssh/ and not using the wrapper script worked to allow access. So it is the SELinux shims over filesystems...

      A bit more googling, and I found that SE-cured SSH is aware of filesystem labels for the key files it uses - those few blogs that do not detail how to kill selinux, suggest to restorecon or explicitly chcon the ~/.ssh directory and its contents. This led me to relabeling the directory used by the Jenkins agent (and as a file owner, the unprivileged linux account of the build agent can do this):

          chcon -R --type=ssh_home_t ~jenkins-worker/jenkins/workspace
      

      This worked, to an extent that the job which starts from scratch with deleteDir() before checkouts, creates new ...@tmp subdirs with the label and launches SSH client that can read the key file.

      It however fails further with launching the ssh-askpass helper (wrapper script) that would provide the passphrase; possibly SSH rejects it due to same labels that do not befit an executable that SE-cured SSH would be okay with running.

      A short-term workaround at this point was to save into Jenkins Credentials a copy of the key without a passphrase.

      Probably a proper solution would be to detect at run-time whether SELinux is a concern on the current system, and label just the newly made temporary private key file (hopefully that would suffice? or dedicate it a subdir like ~/.ssh/ is used in real life...) someplace around https://github.com/jenkinsci/git-client-plugin/blob/master/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2072

      On a side note, I also tried the SSH Agent (not that for the Build agents, but the one which launches ssh-agent and uses ssh-add to enroll known keys into it). This works well regardless of SELinux enforcement, and sh steps calling git program explicitly can use the key provided from Credentials this way. Even programs in terminal to which I export the same envvar values as I see in the job console (with job just sleeping inside the sshagent clause) can use this key. Only the pretty integrated sshagent{checkout} does not work, the git and ssh processes launched do not have these values in their /proc/PID/environ... They also do not have e.g. GIT_TRACE that I set in stage's environment clause, but they do have envvars made from build parameters - so I guess the context for inheritable env of a child process branches off sometime between start of job and start of stage, or the inherited environment for git launched by checkout() step is too sanitized...

      I did not check jGit, never used it (knowingly at least) TBH, but I did not quickly find indications in the source that it would be messing with temporary files to pass key data - so maybe it is immune to the problem.

      Attachments

        Activity

          markewaite I did not configure SELinux explicitly - so this is actually news to me but I guess it is there. The machine is an Antsle box with a custom Gentoo. Jenkins is running in a root Docker, not inside a separate VM. Yes to the jenkins:lts image. I think the issue is that the Docker image only has limited access to SELinux so that may be a problem. What I'm wondering is whether we can do a mod to the git plugin to disable the git /usr/bin/chcon functionality from the pipeline in question with an option to make this go away.

          rsbeckerca Randall Becker added a comment - markewaite I did not configure SELinux explicitly - so this is actually news to me but I guess it is there. The machine is an Antsle box with a custom Gentoo. Jenkins is running in a root Docker, not inside a separate VM. Yes to the jenkins:lts image. I think the issue is that the Docker image only has limited access to SELinux so that may be a problem. What I'm wondering is whether we can do a mod to the git plugin to disable the git /usr/bin/chcon functionality from the pipeline in question with an option to make this go away.
          markewaite Mark Waite added a comment -

          Pull requests to the git client plugin and git plugin are welcomed, though your pull request would need to be reviewed by jimklimov in his environment to confirm that it didn't do him harm. I'm willing to review the pull request but don't have the capacity or the equipment to test it interactively.

          markewaite Mark Waite added a comment - Pull requests to the git client plugin and git plugin are welcomed, though your pull request would need to be reviewed by jimklimov in his environment to confirm that it didn't do him harm. I'm willing to review the pull request but don't have the capacity or the equipment to test it interactively.

          I can't say it will be any time soon. My $DAYJOB has me pretty crushed for the next two weeks (at least). Some summer . but I will do what I can. I think I'd need some guidance as to where the option needs to go as this is magic code done by the controller rather than the agent. My own block is:

          {{ checkout([$class: 'GitSCM',}}
            branches: [[name: '*/production'],[name: '*/20*-Sprint-*']],
            extensions: [
             [$class: 'CleanBeforeCheckout'],
             [$class: 'CloneOption', timeout: 120, shallow: false],
             [$class: 'CheckoutOption', timeout: 120],
             userRemoteConfigs: [[credentialsId: yeah-this]])

          It seems like this might need to be a 'CheckoutOption', SELinux: false (default true) or something along those lines.

          rsbeckerca Randall Becker added a comment - I can't say it will be any time soon. My $DAYJOB has me pretty crushed for the next two weeks (at least). Some summer . but I will do what I can. I think I'd need some guidance as to where the option needs to go as this is magic code done by the controller rather than the agent. My own block is: {{ checkout([$class: 'GitSCM',}}   branches: [ [name: '*/production'] , [name: '*/20*-Sprint-*'] ],   extensions: [    [$class: 'CleanBeforeCheckout'] ,    [$class: 'CloneOption', timeout: 120, shallow: false] ,    [$class: 'CheckoutOption', timeout: 120] ,    userRemoteConfigs: [ [credentialsId: yeah-this] ]) It seems like this might need to be a 'CheckoutOption', SELinux: false (default true) or something along those lines.
          jimklimov Jim Klimov added a comment - - edited

          Just in case, remembering the context: what is the problem to solve here? Noise in logs due to SELinux problems avoidance?

          Reminder for first cause: in some but later apparently not all setups, SELinux forbids Jenkins/agent to read the temporary ssh key-related files from the ssh client program; this was semi-fixed by tweaking the file context for one toxic system I had access to, and only for password-less keys – trusting the script to inject passphrase to private key is a separate can of worms, and we anyway store both the key and its pass in credential store (and would instantiate both on the agent for the ssh operation), so there is moderate threat in just storing it open there.

          That noise is minimal since first "complaint", assuming all my PRs on the matter were merged. It should just log running the chcon (one line) but only burst into details if there are checkout errors that may be related to constraints on that worker.

          UPDATE: Checked, the noise reduction PR was https://github.com/jenkinsci/git-client-plugin/pull/693 and merged April 20; your Git Client version 3.7.1 was released April 4, so...

          Regarding an option to manage it - be my guest PRs linked above (IIRC 673, 690 and 693) should point you to the area of git-client-plugin code where this happens. Note that this is not just a checkout or just a clone option - it applies to all operations that happen over ssh (probably including implicit ones, without the schema in URI - just `git@server...` patterns), and only on (SE)Linux systems. That is, no-op for http(s) or Windows/BSD/Solaris/... and older Linuxes that were not tainted by harsh security methods

          jimklimov Jim Klimov added a comment - - edited Just in case, remembering the context: what is the problem to solve here? Noise in logs due to SELinux problems avoidance? Reminder for first cause: in some but later apparently not all setups, SELinux forbids Jenkins/agent to read the temporary ssh key-related files from the ssh client program; this was semi-fixed by tweaking the file context for one toxic system I had access to, and only for password-less keys – trusting the script to inject passphrase to private key is a separate can of worms, and we anyway store both the key and its pass in credential store (and would instantiate both on the agent for the ssh operation), so there is moderate threat in just storing it open there. That noise is minimal since first "complaint", assuming all my PRs on the matter were merged. It should just log running the chcon (one line) but only burst into details if there are checkout errors that may be related to constraints on that worker. UPDATE: Checked, the noise reduction PR was https://github.com/jenkinsci/git-client-plugin/pull/693 and merged April 20; your Git Client version 3.7.1 was released April 4, so... Regarding an option to manage it - be my guest PRs linked above (IIRC 673, 690 and 693) should point you to the area of git-client-plugin code where this happens. Note that this is not just a checkout or just a clone option - it applies to all operations that happen over ssh (probably including implicit ones, without the schema in URI - just `git@server...` patterns), and only on (SE)Linux systems. That is, no-op for http(s) or Windows/BSD/Solaris/... and older Linuxes that were not tainted by harsh security methods
          markewaite Mark Waite added a comment -

          Amazon Linux 2022 enables selinux by default. It is still in preview, but seems a likely candidate to be fully supported by Amazon before the end of life of Amazon Linux 2 in 2023. Mark Waite needs to explore this further and identify what changes are needed in order to support Amazon Linux 2022.

          markewaite Mark Waite added a comment - Amazon Linux 2022 enables selinux by default. It is still in preview, but seems a likely candidate to be fully supported by Amazon before the end of life of Amazon Linux 2 in 2023. Mark Waite needs to explore this further and identify what changes are needed in order to support Amazon Linux 2022.

          People

            jimklimov Jim Klimov
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated: