Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64913

CLI git access over ssh fails with SELinux Enforcing

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-client-plugin
    • None
    • Jenkins LTS 2.263.x
      git and branch related packages updated 2021-02-22 to whatever is current AND applicable to 2.263.x baseline
      CentOS 7 with SELinux enabled and enforcing

      While setting up a Jenkins controller instance (and a worker using SSH Build Agent on same machine in different account) I've had some hurdles due to SELinux enabled and desired by machine's owners, so the majority of internet lore about disabling it wherever you see it did not apply Out of context here is probably allowing HTTP port usage for the Jenkins service, which was relatively easy.

      The big problem is using git checkouts over ssh, using the SSH Credentials (name + private key + passphrase) saved in Jenkins Credential store. Configurations that healthily worked for me elsewhere, using the checkout([$class...]) magic (job needing to fetch from several repos and platforms), failed here with "Permission denied" and/or with "Check if you are allowed access" sort of messages.

      I hope to post methodology details below, but in short it was a big hunt in the terminal with loops to ln the temporary files from the build workspace into a safe haven, and to detail the command lines and environment variables of ssh and git processes spawned during a failed job run. Eventually I could grab the files that Jenkins generates and reproduce the failure from command-line - just using the same ssh wrapping script, ssh-askpass script, password and private key files that Jenkins makes, and failing to just connect the ssh client to Git platform with no git involved.

      One big discovery was that the internet lore is right Running setenforce 0 (as root) allows Jenkins git+ssh access to work as expected, as well as the manually-made reproduction for same sort of SSH access in shell. But that is not what the security-conscious customer wants.

      As soon as we restore setenforce 1, ssh client fails again with "Permission denied". This was strace'd to be actually a local filesystem error in the openat() syscall, not a message from remote Git server about some key it won't trust (and actually does trust in other client conditions). It was also traced that while the Jenkins generated wrapper script calls ssh -i /path/to/temporary.key and tries to access that key, fails, and does not look into /.ssh/ directory (which is correct with that CLI option), placing a copy of the same key file (without a passphrase for simplicity) into /.ssh/ and not using the wrapper script worked to allow access. So it is the SELinux shims over filesystems...

      A bit more googling, and I found that SE-cured SSH is aware of filesystem labels for the key files it uses - those few blogs that do not detail how to kill selinux, suggest to restorecon or explicitly chcon the ~/.ssh directory and its contents. This led me to relabeling the directory used by the Jenkins agent (and as a file owner, the unprivileged linux account of the build agent can do this):

          chcon -R --type=ssh_home_t ~jenkins-worker/jenkins/workspace
      

      This worked, to an extent that the job which starts from scratch with deleteDir() before checkouts, creates new ...@tmp subdirs with the label and launches SSH client that can read the key file.

      It however fails further with launching the ssh-askpass helper (wrapper script) that would provide the passphrase; possibly SSH rejects it due to same labels that do not befit an executable that SE-cured SSH would be okay with running.

      A short-term workaround at this point was to save into Jenkins Credentials a copy of the key without a passphrase.

      Probably a proper solution would be to detect at run-time whether SELinux is a concern on the current system, and label just the newly made temporary private key file (hopefully that would suffice? or dedicate it a subdir like ~/.ssh/ is used in real life...) someplace around https://github.com/jenkinsci/git-client-plugin/blob/master/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2072

      On a side note, I also tried the SSH Agent (not that for the Build agents, but the one which launches ssh-agent and uses ssh-add to enroll known keys into it). This works well regardless of SELinux enforcement, and sh steps calling git program explicitly can use the key provided from Credentials this way. Even programs in terminal to which I export the same envvar values as I see in the job console (with job just sleeping inside the sshagent clause) can use this key. Only the pretty integrated sshagent{checkout} does not work, the git and ssh processes launched do not have these values in their /proc/PID/environ... They also do not have e.g. GIT_TRACE that I set in stage's environment clause, but they do have envvars made from build parameters - so I guess the context for inheritable env of a child process branches off sometime between start of job and start of stage, or the inherited environment for git launched by checkout() step is too sanitized...

      I did not check jGit, never used it (knowingly at least) TBH, but I did not quickly find indications in the source that it would be messing with temporary files to pass key data - so maybe it is immune to the problem.

          [JENKINS-64913] CLI git access over ssh fails with SELinux Enforcing

          Jim Klimov added a comment -

          bizdelnick: I understand the annoyance from these logs for users to whose setups they happened to be irrelevant. In testing of the original PR they did not stand out, probably because systems either had SELinux installed and usable (enabled or disabled), or did not have it at all. Your case and one commented above is where it is installed but not usable, whether on OS or filesystem level, and the resulting noise is not relevant.

          On the upside, having the noise and reports like yours (thanks!) allowed to identify scenarios not anticipated originally. In fact the follow-up fix was developed a few weeks ago and merged recently (not sure if there was a plugin release since then, however) so that `chcon` issues are only reported if the subsequent `git` command fails: only then these data are really relevant as the probable cause why git failed.

          As for quoting the `"fix"` as if it is not such... the merge did make git client usable for at least some setups with SELinux active, where it was not usable before. So for that population of users, it was a real fix of a problem. For others it did introduce undesirable noise (and some data points for further development), but so far nobody reported that it would have introduced breakage - causing builds, automated log inspections, etc. to actively fail. So technically it was not a regression, although indeed was something to improve further better.

          Sorry for the inconvenience.

          Jim Klimov added a comment - bizdelnick : I understand the annoyance from these logs for users to whose setups they happened to be irrelevant. In testing of the original PR they did not stand out, probably because systems either had SELinux installed and usable (enabled or disabled), or did not have it at all. Your case and one commented above is where it is installed but not usable, whether on OS or filesystem level, and the resulting noise is not relevant. On the upside, having the noise and reports like yours (thanks!) allowed to identify scenarios not anticipated originally. In fact the follow-up fix was developed a few weeks ago and merged recently (not sure if there was a plugin release since then, however) so that `chcon` issues are only reported if the subsequent `git` command fails: only then these data are really relevant as the probable cause why git failed. As for quoting the `"fix"` as if it is not such... the merge did make git client usable for at least some setups with SELinux active, where it was not usable before. So for that population of users, it was a real fix of a problem. For others it did introduce undesirable noise (and some data points for further development), but so far nobody reported that it would have introduced breakage - causing builds, automated log inspections, etc. to actively fail. So technically it was not a regression, although indeed was something to improve further better. Sorry for the inconvenience.

          Mark Waite added a comment - - edited

          bizdelnick thanks for reporting the message that you're seeing.

          Unfortunately, I'm not seeing that message in any of my test environments. My test environments include:

          • Controller installed on Debian 10 with a deb file, with agents
            • Windows 10
            • Debian Testing
            • Debian 10 on ARM
          • Controller installed on Docker using Jenkins LTS base image with agents
            • Windows 10
            • Debian 9
            • Debian 10
            • CentOS 7
            • CentOS 8
            • OpenBSD 6.9
            • FreeBSD 12.2
            • Ubuntu 18.04
            • Ubuntu 20.04

          Could you describe the environment where you're seeing the noise so that it can be considered for future testing?

          If you'd like to join those who are testing the git plugin before it releases (most typically that is an army of one, me), please send your contact information to mark.earl.waite@gmail.com.

          Mark Waite added a comment - - edited bizdelnick thanks for reporting the message that you're seeing. Unfortunately, I'm not seeing that message in any of my test environments. My test environments include: Controller installed on Debian 10 with a deb file, with agents Windows 10 Debian Testing Debian 10 on ARM Controller installed on Docker using Jenkins LTS base image with agents Windows 10 Debian 9 Debian 10 CentOS 7 CentOS 8 OpenBSD 6.9 FreeBSD 12.2 Ubuntu 18.04 Ubuntu 20.04 Could you describe the environment where you're seeing the noise so that it can be considered for future testing? If you'd like to join those who are testing the git plugin before it releases (most typically that is an army of one, me), please send your contact information to mark.earl.waite@gmail.com.

          I run Jenkins in docker container. Host system is CentOS 7 (SELinux in permissive mode), guest is Alpine (official jenkins/jenkins:lts-alpine image).

          Dmitry Mikhirev added a comment - I run Jenkins in docker container. Host system is CentOS 7 (SELinux in permissive mode), guest is Alpine (official jenkins/jenkins:lts-alpine image).

          I am experiencing the same situation running from a Gentoo Linux/Docker container for the controller, and a Windows 10 Pro VM where the agent is running. I'm on native git 2.20.1, Jenkins 2.289, Git 4.7.1, GitClient 3.7.1.

          Randall Becker added a comment - I am experiencing the same situation running from a Gentoo Linux/Docker container for the controller, and a Windows 10 Pro VM where the agent is running. I'm on native git 2.20.1, Jenkins 2.289, Git 4.7.1, GitClient 3.7.1.

          Mark Waite added a comment -

          I assume rsbeckerca that your Gentoo Linux that is hosting the container for the controller also has SELinux configured. Can you confirm that?

          I assume you are running one of the centos Docker images like jenkins/jenkins:lts-centos so that the operating system utilities inside the Docker image are aware of SELinux. Can you confirm that?

          I don't plan to spend time developing or testing the git plugin with SELinux. I'm willing to consider pull requests from others, but will have to rely on community validation of the pull requests because I won't test them myself.

          Mark Waite added a comment - I assume rsbeckerca that your Gentoo Linux that is hosting the container for the controller also has SELinux configured. Can you confirm that? I assume you are running one of the centos Docker images like jenkins/jenkins:lts-centos so that the operating system utilities inside the Docker image are aware of SELinux. Can you confirm that? I don't plan to spend time developing or testing the git plugin with SELinux. I'm willing to consider pull requests from others, but will have to rely on community validation of the pull requests because I won't test them myself.

          markewaite I did not configure SELinux explicitly - so this is actually news to me but I guess it is there. The machine is an Antsle box with a custom Gentoo. Jenkins is running in a root Docker, not inside a separate VM. Yes to the jenkins:lts image. I think the issue is that the Docker image only has limited access to SELinux so that may be a problem. What I'm wondering is whether we can do a mod to the git plugin to disable the git /usr/bin/chcon functionality from the pipeline in question with an option to make this go away.

          Randall Becker added a comment - markewaite I did not configure SELinux explicitly - so this is actually news to me but I guess it is there. The machine is an Antsle box with a custom Gentoo. Jenkins is running in a root Docker, not inside a separate VM. Yes to the jenkins:lts image. I think the issue is that the Docker image only has limited access to SELinux so that may be a problem. What I'm wondering is whether we can do a mod to the git plugin to disable the git /usr/bin/chcon functionality from the pipeline in question with an option to make this go away.

          Mark Waite added a comment -

          Pull requests to the git client plugin and git plugin are welcomed, though your pull request would need to be reviewed by jimklimov in his environment to confirm that it didn't do him harm. I'm willing to review the pull request but don't have the capacity or the equipment to test it interactively.

          Mark Waite added a comment - Pull requests to the git client plugin and git plugin are welcomed, though your pull request would need to be reviewed by jimklimov in his environment to confirm that it didn't do him harm. I'm willing to review the pull request but don't have the capacity or the equipment to test it interactively.

          I can't say it will be any time soon. My $DAYJOB has me pretty crushed for the next two weeks (at least). Some summer . but I will do what I can. I think I'd need some guidance as to where the option needs to go as this is magic code done by the controller rather than the agent. My own block is:

          {{ checkout([$class: 'GitSCM',}}
            branches: [[name: '*/production'],[name: '*/20*-Sprint-*']],
            extensions: [
             [$class: 'CleanBeforeCheckout'],
             [$class: 'CloneOption', timeout: 120, shallow: false],
             [$class: 'CheckoutOption', timeout: 120],
             userRemoteConfigs: [[credentialsId: yeah-this]])

          It seems like this might need to be a 'CheckoutOption', SELinux: false (default true) or something along those lines.

          Randall Becker added a comment - I can't say it will be any time soon. My $DAYJOB has me pretty crushed for the next two weeks (at least). Some summer . but I will do what I can. I think I'd need some guidance as to where the option needs to go as this is magic code done by the controller rather than the agent. My own block is: {{ checkout([$class: 'GitSCM',}}   branches: [ [name: '*/production'] , [name: '*/20*-Sprint-*'] ],   extensions: [    [$class: 'CleanBeforeCheckout'] ,    [$class: 'CloneOption', timeout: 120, shallow: false] ,    [$class: 'CheckoutOption', timeout: 120] ,    userRemoteConfigs: [ [credentialsId: yeah-this] ]) It seems like this might need to be a 'CheckoutOption', SELinux: false (default true) or something along those lines.

          Jim Klimov added a comment - - edited

          Just in case, remembering the context: what is the problem to solve here? Noise in logs due to SELinux problems avoidance?

          Reminder for first cause: in some but later apparently not all setups, SELinux forbids Jenkins/agent to read the temporary ssh key-related files from the ssh client program; this was semi-fixed by tweaking the file context for one toxic system I had access to, and only for password-less keys – trusting the script to inject passphrase to private key is a separate can of worms, and we anyway store both the key and its pass in credential store (and would instantiate both on the agent for the ssh operation), so there is moderate threat in just storing it open there.

          That noise is minimal since first "complaint", assuming all my PRs on the matter were merged. It should just log running the chcon (one line) but only burst into details if there are checkout errors that may be related to constraints on that worker.

          UPDATE: Checked, the noise reduction PR was https://github.com/jenkinsci/git-client-plugin/pull/693 and merged April 20; your Git Client version 3.7.1 was released April 4, so...

          Regarding an option to manage it - be my guest PRs linked above (IIRC 673, 690 and 693) should point you to the area of git-client-plugin code where this happens. Note that this is not just a checkout or just a clone option - it applies to all operations that happen over ssh (probably including implicit ones, without the schema in URI - just `git@server...` patterns), and only on (SE)Linux systems. That is, no-op for http(s) or Windows/BSD/Solaris/... and older Linuxes that were not tainted by harsh security methods

          Jim Klimov added a comment - - edited Just in case, remembering the context: what is the problem to solve here? Noise in logs due to SELinux problems avoidance? Reminder for first cause: in some but later apparently not all setups, SELinux forbids Jenkins/agent to read the temporary ssh key-related files from the ssh client program; this was semi-fixed by tweaking the file context for one toxic system I had access to, and only for password-less keys – trusting the script to inject passphrase to private key is a separate can of worms, and we anyway store both the key and its pass in credential store (and would instantiate both on the agent for the ssh operation), so there is moderate threat in just storing it open there. That noise is minimal since first "complaint", assuming all my PRs on the matter were merged. It should just log running the chcon (one line) but only burst into details if there are checkout errors that may be related to constraints on that worker. UPDATE: Checked, the noise reduction PR was https://github.com/jenkinsci/git-client-plugin/pull/693 and merged April 20; your Git Client version 3.7.1 was released April 4, so... Regarding an option to manage it - be my guest PRs linked above (IIRC 673, 690 and 693) should point you to the area of git-client-plugin code where this happens. Note that this is not just a checkout or just a clone option - it applies to all operations that happen over ssh (probably including implicit ones, without the schema in URI - just `git@server...` patterns), and only on (SE)Linux systems. That is, no-op for http(s) or Windows/BSD/Solaris/... and older Linuxes that were not tainted by harsh security methods

          Mark Waite added a comment -

          Amazon Linux 2022 enables selinux by default. It is still in preview, but seems a likely candidate to be fully supported by Amazon before the end of life of Amazon Linux 2 in 2023. Mark Waite needs to explore this further and identify what changes are needed in order to support Amazon Linux 2022.

          Mark Waite added a comment - Amazon Linux 2022 enables selinux by default. It is still in preview, but seems a likely candidate to be fully supported by Amazon before the end of life of Amazon Linux 2 in 2023. Mark Waite needs to explore this further and identify what changes are needed in order to support Amazon Linux 2022.

            jimklimov Jim Klimov
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: