Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64913

CLI git access over ssh fails with SELinux Enforcing

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-client-plugin
    • None
    • Jenkins LTS 2.263.x
      git and branch related packages updated 2021-02-22 to whatever is current AND applicable to 2.263.x baseline
      CentOS 7 with SELinux enabled and enforcing

      While setting up a Jenkins controller instance (and a worker using SSH Build Agent on same machine in different account) I've had some hurdles due to SELinux enabled and desired by machine's owners, so the majority of internet lore about disabling it wherever you see it did not apply Out of context here is probably allowing HTTP port usage for the Jenkins service, which was relatively easy.

      The big problem is using git checkouts over ssh, using the SSH Credentials (name + private key + passphrase) saved in Jenkins Credential store. Configurations that healthily worked for me elsewhere, using the checkout([$class...]) magic (job needing to fetch from several repos and platforms), failed here with "Permission denied" and/or with "Check if you are allowed access" sort of messages.

      I hope to post methodology details below, but in short it was a big hunt in the terminal with loops to ln the temporary files from the build workspace into a safe haven, and to detail the command lines and environment variables of ssh and git processes spawned during a failed job run. Eventually I could grab the files that Jenkins generates and reproduce the failure from command-line - just using the same ssh wrapping script, ssh-askpass script, password and private key files that Jenkins makes, and failing to just connect the ssh client to Git platform with no git involved.

      One big discovery was that the internet lore is right Running setenforce 0 (as root) allows Jenkins git+ssh access to work as expected, as well as the manually-made reproduction for same sort of SSH access in shell. But that is not what the security-conscious customer wants.

      As soon as we restore setenforce 1, ssh client fails again with "Permission denied". This was strace'd to be actually a local filesystem error in the openat() syscall, not a message from remote Git server about some key it won't trust (and actually does trust in other client conditions). It was also traced that while the Jenkins generated wrapper script calls ssh -i /path/to/temporary.key and tries to access that key, fails, and does not look into /.ssh/ directory (which is correct with that CLI option), placing a copy of the same key file (without a passphrase for simplicity) into /.ssh/ and not using the wrapper script worked to allow access. So it is the SELinux shims over filesystems...

      A bit more googling, and I found that SE-cured SSH is aware of filesystem labels for the key files it uses - those few blogs that do not detail how to kill selinux, suggest to restorecon or explicitly chcon the ~/.ssh directory and its contents. This led me to relabeling the directory used by the Jenkins agent (and as a file owner, the unprivileged linux account of the build agent can do this):

          chcon -R --type=ssh_home_t ~jenkins-worker/jenkins/workspace
      

      This worked, to an extent that the job which starts from scratch with deleteDir() before checkouts, creates new ...@tmp subdirs with the label and launches SSH client that can read the key file.

      It however fails further with launching the ssh-askpass helper (wrapper script) that would provide the passphrase; possibly SSH rejects it due to same labels that do not befit an executable that SE-cured SSH would be okay with running.

      A short-term workaround at this point was to save into Jenkins Credentials a copy of the key without a passphrase.

      Probably a proper solution would be to detect at run-time whether SELinux is a concern on the current system, and label just the newly made temporary private key file (hopefully that would suffice? or dedicate it a subdir like ~/.ssh/ is used in real life...) someplace around https://github.com/jenkinsci/git-client-plugin/blob/master/src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java#L2072

      On a side note, I also tried the SSH Agent (not that for the Build agents, but the one which launches ssh-agent and uses ssh-add to enroll known keys into it). This works well regardless of SELinux enforcement, and sh steps calling git program explicitly can use the key provided from Credentials this way. Even programs in terminal to which I export the same envvar values as I see in the job console (with job just sleeping inside the sshagent clause) can use this key. Only the pretty integrated sshagent{checkout} does not work, the git and ssh processes launched do not have these values in their /proc/PID/environ... They also do not have e.g. GIT_TRACE that I set in stage's environment clause, but they do have envvars made from build parameters - so I guess the context for inheritable env of a child process branches off sometime between start of job and start of stage, or the inherited environment for git launched by checkout() step is too sanitized...

      I did not check jGit, never used it (knowingly at least) TBH, but I did not quickly find indications in the source that it would be messing with temporary files to pass key data - so maybe it is immune to the problem.

          [JENKINS-64913] CLI git access over ssh fails with SELinux Enforcing

          Jim Klimov added a comment -

          Reproduction snippets:

          • Cloning in shell, using (passless) key file:
          :; git init; GIT_SSH_COMMAND="(strace -f) ssh -vvv -i /home/jenkins-worker/jenkins/workspace/jobname/subrepo@tmp/jenkins-gitclient-ssh123123123.key -l git" git fetch --tags --force --progress -- git@reposerver.local:org/repo/git '+refs/heads/*:refs/remotes/origin/*'
          • Catching temp files made and discarded by git-client:
            :; mkdir /home/jenkins-worker/jenkins/workspace/.haven
            :; cd /home/jenkins-worker/jenkins/workspace/.haven
            :; (while : ; do find ../jobname/subrepo@tmp/* -type f -exec ln '{}' ./ \; ; done)
          • Catching process info for git-related processes launched during a job:
            :; ( F=/dev/shm/pidlist; echo "" > "$F"; while : ; do while ! (ps -ef | grep [g]it) ; do : ; done ; \
                 date; ps -ef | grep [g]it | awk '{print $2}' ; while read P ; do \
                     grep -w "$P" "$F" >/dev/null && continue ; \
                     echo "=== `date`: $P :" ; find "/proc/$P/" -ls ; \
                     echo "===== ENV:" ; tr '\0' '\n' < "/proc/$P/environ" | sort -n ; \
                     echo "===== CMD:" ; tr '\0' '\n' < "/proc/$P/cmdline" ; \
                     echo "$P" >> "$F" ; \
                 done; done )

          Jim Klimov added a comment - Reproduction snippets: Cloning in shell, using (passless) key file: :; git init; GIT_SSH_COMMAND= "(strace -f) ssh -vvv -i /home/jenkins-worker/jenkins/workspace/jobname/subrepo@tmp/jenkins-gitclient-ssh123123123.key -l git" git fetch --tags --force --progress -- git@reposerver.local:org/repo/git '+refs/heads/*:refs/remotes/origin/*' Catching temp files made and discarded by git-client: :; mkdir /home/jenkins-worker/jenkins/workspace/.haven :; cd /home/jenkins-worker/jenkins/workspace/.haven :; ( while : ; do find ../jobname/subrepo@tmp/* -type f -exec ln '{}' ./ \; ; done) Catching process info for git-related processes launched during a job: :; ( F=/dev/shm/pidlist; echo "" > " $F"; while : ; do while ! (ps -ef | grep [g]it) ; do : ; done ; \ date; ps -ef | grep [g]it | awk '{print $2}' ; while read P ; do \ grep -w "$P" "$F" >/dev/ null && continue ; \ echo "=== `date`: $P :" ; find "/proc/$P/" -ls ; \ echo "===== ENV:" ; tr '\0' '\n' < "/proc/$P/environ" | sort -n ; \ echo "===== CMD:" ; tr '\0' '\n' < "/proc/$P/cmdline" ; \ echo "$P" >> "$F" ; \ done; done )

          Jim Klimov added a comment -

          Jim Klimov added a comment - Experimenting in https://github.com/jenkinsci/git-client-plugin/pull/673

          Jim Klimov added a comment -

          As detailed in the PR above (merged recently), currently the issue is resolved for unencrypted private keys - these files can be labeled so ssh client called from git agrees to read them on SELinux-enabled builds. There is still a problem for keys which need a passphrase, because of permissions needed for executing helper scripts (posing as ssh-askpass compatible) that appear in non-system paths (temporary workspace dirs).

          This particular issue can be tackled later by just pre-unencrypting key files and likely simplifying git-client-plugin code.

          The more general issue about arbitrary execution from non-system paths and data access can be addressed by defining a SELinux profile template for jenkins agent which can be pre-installed on the worker host (if all goes well, eventually upstreamed to be part of general SELinux baseline; their IRC channel was quite helpful for the research and experiments during the investigation), and constraining the agent setups to lay out file data in patterns that match such definition. The latter bit can be done by documentation for admins: if you have to run on such hosts, follow such rules...

          Jim Klimov added a comment - As detailed in the PR above (merged recently), currently the issue is resolved for unencrypted private keys - these files can be labeled so ssh client called from git agrees to read them on SELinux-enabled builds. There is still a problem for keys which need a passphrase, because of permissions needed for executing helper scripts (posing as ssh-askpass compatible) that appear in non-system paths (temporary workspace dirs). This particular issue can be tackled later by just pre-unencrypting key files and likely simplifying git-client-plugin code. The more general issue about arbitrary execution from non-system paths and data access can be addressed by defining a SELinux profile template for jenkins agent which can be pre-installed on the worker host (if all goes well, eventually upstreamed to be part of general SELinux baseline; their IRC channel was quite helpful for the research and experiments during the investigation), and constraining the agent setups to lay out file data in patterns that match such definition. The latter bit can be done by documentation for admins: if you have to run on such hosts, follow such rules...

          Jim Klimov added a comment -

          I am not sure how to better track this issue's state: it is not likely I'll get to cleanly implement the pre-unencryption of key files anytime soon, so likely someone else would eventually progress on that to complete this ticket at face value.

          So at the moment it is not in progress, and not "assigned" to me anymore, takers welcome

          Jim Klimov added a comment - I am not sure how to better track this issue's state: it is not likely I'll get to cleanly implement the pre-unencryption of key files anytime soon, so likely someone else would eventually progress on that to complete this ticket at face value. So at the moment it is not in progress, and not "assigned" to me anymore, takers welcome

          Jim Klimov added a comment -

          As raised in the initial PR's comments, that change added a lot of noise for Linux systems where tools for SELinux are present but it is not active per se (and `chcon` fails to relabel files and git-client-plugin complains into the build log). Opened https://github.com/jenkinsci/git-client-plugin/pull/690 to check better if we should try relabeling in fact.

          Jim Klimov added a comment - As raised in the initial PR's comments, that change added a lot of noise for Linux systems where tools for SELinux are present but it is not active per se (and `chcon` fails to relabel files and git-client-plugin complains into the build log). Opened https://github.com/jenkinsci/git-client-plugin/pull/690 to check better if we should try relabeling in fact.

          Mark Waite added a comment -

          Released as an initial attempt in git client plugin 3.7.0. As noted by jimklimov, not a complete solution, but steps towards a solution

          Mark Waite added a comment - Released as an initial attempt in git client plugin 3.7.0. As noted by jimklimov , not a complete solution, but steps towards a solution

          Marc Brugger added a comment -

          Hi 

          I'm running Jenkins inside OpenShift 4.6.
          I recently updated to git-client-plugin v4.7.0.

          When running pipelines, the changes on the SELinux context result in warnings, since container in OpenShift run unprivileged and with an arbitrary user where the SELinux changes are not allowed.
          The build is running, but generates a lot of warning noises.

          Is there way the fixSELinuxLabel can be skipped for OpenShift environments?

          Running in Durability level: MAX_SURVIVABILITY
          Loading library jenkins-shared-library@master
          Attempting to resolve master from remote references...
           > git --version # timeout=10
           > git --version # 'git version 2.27.0'
          using GIT_SSH to set credentials jenkins-shared-library
          [INFO] Currently running in a labeled security context
           > git /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh1265001173911415896.key
          [WARNING] Failed (1) performing chcon helper command for SELinux: git /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh1265001173911415896.key:
          === STDERR:
          chcon: failed to change context of '/tmp/jenkins-gitclient-ssh1265001173911415896.key' to ‘system_u:object_r:ssh_home_t:s0:c14,c27’: Operation not supported====
          IMPACT: if SELinux is enabled, access to temporary key file may be denied for git+ssh below
           > git ls-remote -h -- ssh://git@stash:22/sdbi/jenkins-shared-library.git # timeout=10
          Found match: refs/heads/master revision 95acfea7b3908a123c24fbad8e661ee1c30f972a
          The recommended git tool is: NONE
          Cloning the remote Git repository
          using credential jenkins-shared-library
          Cloning the remote Git repository
          Using shallow clone with depth 1
          Avoid fetching tags
          Cloning repository ssh://git@stash:22/sdbi/jenkins-shared-library.git
           > git init /var/lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library # timeout=10
          Fetching upstream changes from ssh://git@stash:22/sdbi/jenkins-shared-library.git
           > git --version # timeout=10
           > git --version # 'git version 2.27.0'
          using GIT_SSH to set credentials jenkins-shared-library
          [INFO] Currently running in a labeled security context
           > git /usr/bin/chcon --type=ssh_home_t /var/lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library@tmp/jenkins-gitclient-ssh18053214354803671212.key
          [WARNING] Failed (1) performing chcon helper command for SELinux: git /usr/bin/chcon --type=ssh_home_t /var/lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library@tmp/jenkins-gitclient-ssh18053214354803671212.key:
          === STDERR:
          chcon: failed to change context of '/var/lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library@tmp/jenkins-gitclient-ssh18053214354803671212.key' to ‘system_u:object_r:ssh_home_t:s0:c14,c27’: Permission denied====
          IMPACT: if SELinux is enabled, access to temporary key file may be denied for git+ssh below
           > git fetch --no-tags --force --progress --depth=1 -- ssh://git@stash:22/sdbi/jenkins-shared-library.git +refs/heads/*:refs/remotes/origin/* # timeout=10
           > git config remote.origin.url ssh://git@stash:22/sdbi/jenkins-shared-library.git # timeout=10
           > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
          Avoid second fetch
          Checking out Revision 95acfea7b3908a123c24fbad8e661ee1c30f972a (master)
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 95acfea7b3908a123c24fbad8e661ee1c30f972a # timeout=10
          Commit message: "add failed as valid result"
          [Pipeline] Start of Pipeline
          [Pipeline] echo
          Applying build properties with ...
           - 5 build number(s) to keep
          [Pipeline] properties
          [Pipeline] End of Pipeline
          Finished: SUCCESS
          

           

           

           

           

          Marc Brugger added a comment - Hi  I'm running Jenkins inside OpenShift 4.6. I recently updated to  git-client-plugin  v4.7.0. When running pipelines, the changes on the SELinux context result in warnings, since container in OpenShift run unprivileged and with an arbitrary user where the SELinux changes are not allowed. The build is running, but generates a lot of warning noises. Is there way the fixSELinuxLabel can be skipped for OpenShift environments? Running in Durability level: MAX_SURVIVABILITY Loading library jenkins-shared-library@master Attempting to resolve master from remote references... > git --version # timeout=10 > git --version # 'git version 2.27.0' using GIT_SSH to set credentials jenkins-shared-library [INFO] Currently running in a labeled security context > git /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh1265001173911415896.key [WARNING] Failed (1) performing chcon helper command for SELinux: git /usr/bin/chcon --type=ssh_home_t /tmp/jenkins-gitclient-ssh1265001173911415896.key: === STDERR: chcon: failed to change context of '/tmp/jenkins-gitclient-ssh1265001173911415896.key' to ‘system_u:object_r:ssh_home_t:s0:c14,c27’: Operation not supported==== IMPACT: if SELinux is enabled, access to temporary key file may be denied for git+ssh below > git ls-remote -h -- ssh: //git@stash:22/sdbi/jenkins-shared-library.git # timeout=10 Found match: refs/heads/master revision 95acfea7b3908a123c24fbad8e661ee1c30f972a The recommended git tool is: NONE Cloning the remote Git repository using credential jenkins-shared-library Cloning the remote Git repository Using shallow clone with depth 1 Avoid fetching tags Cloning repository ssh: //git@stash:22/sdbi/jenkins-shared-library.git > git init / var /lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library # timeout=10 Fetching upstream changes from ssh: //git@stash:22/sdbi/jenkins-shared-library.git > git --version # timeout=10 > git --version # 'git version 2.27.0' using GIT_SSH to set credentials jenkins-shared-library [INFO] Currently running in a labeled security context > git /usr/bin/chcon --type=ssh_home_t / var /lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library@tmp/jenkins-gitclient-ssh18053214354803671212.key [WARNING] Failed (1) performing chcon helper command for SELinux: git /usr/bin/chcon --type=ssh_home_t / var /lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library@tmp/jenkins-gitclient-ssh18053214354803671212.key: === STDERR: chcon: failed to change context of '/ var /lib/jenkins/jobs/tst/jobs/asfd/workspace@libs/jenkins-shared-library@tmp/jenkins-gitclient-ssh18053214354803671212.key' to ‘system_u:object_r:ssh_home_t:s0:c14,c27’: Permission denied==== IMPACT: if SELinux is enabled, access to temporary key file may be denied for git+ssh below > git fetch --no-tags --force --progress --depth=1 -- ssh: //git@stash:22/sdbi/jenkins-shared-library.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url ssh: //git@stash:22/sdbi/jenkins-shared-library.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 95acfea7b3908a123c24fbad8e661ee1c30f972a (master) > git config core.sparsecheckout # timeout=10 > git checkout -f 95acfea7b3908a123c24fbad8e661ee1c30f972a # timeout=10 Commit message: "add failed as valid result" [Pipeline] Start of Pipeline [Pipeline] echo Applying build properties with ... - 5 build number(s) to keep [Pipeline] properties [Pipeline] End of Pipeline Finished: SUCCESS        

          Marc Brugger added a comment -

          Perhaps the check in fixSELinuxLabel could also consider the presence of the environment variable KUBERNETES_SERVICE_HOST which indicates, that jenkins is running inside a pod on kubernetes.
          See: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#environment-variables
           

          Marc Brugger added a comment - Perhaps the check in fixSELinuxLabel could also consider the presence of the environment variable KUBERNETES_SERVICE_HOST which indicates, that jenkins is running inside a pod on kubernetes. See: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#environment-variables  

          Jim Klimov added a comment - - edited

          Hi, trying to wrap my head around these news

          So, the worker in OpenShift does run in a system where SELinux is present and enabled, but the worker runs with a user/context not privileged enough to make those changes with chcon? So in this particular case the "kind and generous" system is relaxed and SSH does not reject the temporary key file so you don't want to see the noise, while in generic other cases the system is not so forgiving and people would just wonder why their checkouts failed with same setup they succeed elsewhere (if without a message and without chcon, as before this issue and PR; at least they would not wonder with the message now but have something actionable)?

          While it is possible to pepper the code with exceptional cases for symptoms like presence of this or that envvar, at least temporarily, I think it would become unmanageable in the long run. It may be more worthwhile and generic to find what makes a "kind and generous" system different from the one with SELinux enabled and diligently doing its job to deny a rogue SSH key from being used, and check for that difference (config file, tool output, etc.) in the method – such approach would apply to all other systems using similar technology even if they are not OpenShift and Kubernetes

          Another approach could be to add a configuration toggle for this behavior (Additional behaviors on checkout/clone category), and I'd lean towards being noisy by default and having a way for particular deployments to hush it down deliberately.

          Yet another approach could be to not report a chcon failure, e.g. just cache the message, unless a git operation fails. For example, as a guess, your system might be set up same as any other with SELinux, but the build of ssh in the container is not SELinux-aware so just does not check file labels. This would also reduce log noise for situations where it (and even active chcon failures) is not relevant to the outcome. This currently seems like a reasonable option easiest to implement without additional research and skills I do not readily have (so other solutions would probably need someone else to address).

          And on a side note, seems there is a typo in the logged messages with > git /usr/bin/chcon --type=... (the "git" should not be there)

          Jim Klimov added a comment - - edited Hi, trying to wrap my head around these news So, the worker in OpenShift does run in a system where SELinux is present and enabled, but the worker runs with a user/context not privileged enough to make those changes with chcon? So in this particular case the "kind and generous" system is relaxed and SSH does not reject the temporary key file so you don't want to see the noise, while in generic other cases the system is not so forgiving and people would just wonder why their checkouts failed with same setup they succeed elsewhere (if without a message and without chcon, as before this issue and PR; at least they would not wonder with the message now but have something actionable)? While it is possible to pepper the code with exceptional cases for symptoms like presence of this or that envvar, at least temporarily, I think it would become unmanageable in the long run. It may be more worthwhile and generic to find what makes a "kind and generous" system different from the one with SELinux enabled and diligently doing its job to deny a rogue SSH key from being used, and check for that difference (config file, tool output, etc.) in the method – such approach would apply to all other systems using similar technology even if they are not OpenShift and Kubernetes Another approach could be to add a configuration toggle for this behavior (Additional behaviors on checkout/clone category), and I'd lean towards being noisy by default and having a way for particular deployments to hush it down deliberately. Yet another approach could be to not report a chcon failure, e.g. just cache the message, unless a git operation fails. For example, as a guess, your system might be set up same as any other with SELinux, but the build of ssh in the container is not SELinux-aware so just does not check file labels. This would also reduce log noise for situations where it (and even active chcon failures) is not relevant to the outcome. This currently seems like a reasonable option easiest to implement without additional research and skills I do not readily have (so other solutions would probably need someone else to address). And on a side note, seems there is a typo in the logged messages with > git /usr/bin/chcon --type=... (the "git" should not be there)

          Jim Klimov added a comment -

          I posted a proposal with my understanding of the issue and prospect fix per latter idea (with cached messages, only to log in case of errors) at https://github.com/jenkinsci/git-client-plugin/pull/693 - while this is not yet tested, does the idea seem right? (Oh, and also you can check that branch out, "mvn package" and run the custom plugin build on your Jenkins to see if this solves the issue or adds some others :\ )

          Jim Klimov added a comment - I posted a proposal with my understanding of the issue and prospect fix per latter idea (with cached messages, only to log in case of errors) at https://github.com/jenkinsci/git-client-plugin/pull/693 - while this is not yet tested, does the idea seem right? (Oh, and also you can check that branch out, "mvn package" and run the custom plugin build on your Jenkins to see if this solves the issue or adds some others :\ )

          Marc Brugger added a comment -

          Thank you for your quick response and fix.
          I'll try to build the plugin from the branch and can give you feedback tomorrow.

          Marc Brugger added a comment - Thank you for your quick response and fix. I'll try to build the plugin from the branch and can give you feedback tomorrow.

          Marc Brugger added a comment -

          I could quickly build and test it the branch version.
          This version behaves the same way as 3.6.0 did. All warnings have gone and I could't see any other side effects.

          Thanks again for the quick fix

          Marc Brugger added a comment - I could quickly build and test it the branch version. This version behaves the same way as 3.6.0 did. All warnings have gone and I could't see any other side effects. Thanks again for the quick fix

          Jim Klimov added a comment -

          That updated version seems to (still) work for me on SELinux worker too.

          Jim Klimov added a comment - That updated version seems to (still) work for me on SELinux worker too.

          Dmitry Mikhirev added a comment - - edited

          I see this garbage in build logs on system without SELinux enabled:

          using GIT_SSH to set credentials builder key for slaves
          [INFO] Currently running in a labeled security context
          > git /usr/bin/chcon --type=ssh_home_t /var/jenkins_home/caches/git-f7ddff632098e52e07f4accbfc6a18a0@tmp/jenkins-gitclient-ssh919708132215489744.key
          [WARNING] Failed (1) performing chcon helper command for SELinux: git /usr/bin/chcon --type=ssh_home_t /var/jenkins_home/caches/git-f7ddff632098e52e07f4accbfc6a18a0@tmp/jenkins-gitclient-ssh919708132215489744.key:
          === STDERR:
          chcon: failed to get security context of '/var/jenkins_home/caches/git-f7ddff632098e52e07f4accbfc6a18a0@tmp/jenkins-gitclient-ssh919708132215489744.key': Not supported

          ====
          IMPACT: if SELinux is enabled, access to temporary key file may be denied for git+ssh below

          Why was this "fix" merged and released without testing?

          Dmitry Mikhirev added a comment - - edited I see this garbage in build logs on system without SELinux enabled: using GIT_SSH to set credentials builder key for slaves [INFO] Currently running in a labeled security context > git /usr/bin/chcon --type=ssh_home_t /var/jenkins_home/caches/git-f7ddff632098e52e07f4accbfc6a18a0@tmp/jenkins-gitclient-ssh919708132215489744.key [WARNING] Failed (1) performing chcon helper command for SELinux: git /usr/bin/chcon --type=ssh_home_t /var/jenkins_home/caches/git-f7ddff632098e52e07f4accbfc6a18a0@tmp/jenkins-gitclient-ssh919708132215489744.key: === STDERR: chcon: failed to get security context of '/var/jenkins_home/caches/git-f7ddff632098e52e07f4accbfc6a18a0@tmp/jenkins-gitclient-ssh919708132215489744.key': Not supported ==== IMPACT: if SELinux is enabled, access to temporary key file may be denied for git+ssh below Why was this "fix" merged and released without testing?

          Jim Klimov added a comment -

          bizdelnick: I understand the annoyance from these logs for users to whose setups they happened to be irrelevant. In testing of the original PR they did not stand out, probably because systems either had SELinux installed and usable (enabled or disabled), or did not have it at all. Your case and one commented above is where it is installed but not usable, whether on OS or filesystem level, and the resulting noise is not relevant.

          On the upside, having the noise and reports like yours (thanks!) allowed to identify scenarios not anticipated originally. In fact the follow-up fix was developed a few weeks ago and merged recently (not sure if there was a plugin release since then, however) so that `chcon` issues are only reported if the subsequent `git` command fails: only then these data are really relevant as the probable cause why git failed.

          As for quoting the `"fix"` as if it is not such... the merge did make git client usable for at least some setups with SELinux active, where it was not usable before. So for that population of users, it was a real fix of a problem. For others it did introduce undesirable noise (and some data points for further development), but so far nobody reported that it would have introduced breakage - causing builds, automated log inspections, etc. to actively fail. So technically it was not a regression, although indeed was something to improve further better.

          Sorry for the inconvenience.

          Jim Klimov added a comment - bizdelnick : I understand the annoyance from these logs for users to whose setups they happened to be irrelevant. In testing of the original PR they did not stand out, probably because systems either had SELinux installed and usable (enabled or disabled), or did not have it at all. Your case and one commented above is where it is installed but not usable, whether on OS or filesystem level, and the resulting noise is not relevant. On the upside, having the noise and reports like yours (thanks!) allowed to identify scenarios not anticipated originally. In fact the follow-up fix was developed a few weeks ago and merged recently (not sure if there was a plugin release since then, however) so that `chcon` issues are only reported if the subsequent `git` command fails: only then these data are really relevant as the probable cause why git failed. As for quoting the `"fix"` as if it is not such... the merge did make git client usable for at least some setups with SELinux active, where it was not usable before. So for that population of users, it was a real fix of a problem. For others it did introduce undesirable noise (and some data points for further development), but so far nobody reported that it would have introduced breakage - causing builds, automated log inspections, etc. to actively fail. So technically it was not a regression, although indeed was something to improve further better. Sorry for the inconvenience.

          Mark Waite added a comment - - edited

          bizdelnick thanks for reporting the message that you're seeing.

          Unfortunately, I'm not seeing that message in any of my test environments. My test environments include:

          • Controller installed on Debian 10 with a deb file, with agents
            • Windows 10
            • Debian Testing
            • Debian 10 on ARM
          • Controller installed on Docker using Jenkins LTS base image with agents
            • Windows 10
            • Debian 9
            • Debian 10
            • CentOS 7
            • CentOS 8
            • OpenBSD 6.9
            • FreeBSD 12.2
            • Ubuntu 18.04
            • Ubuntu 20.04

          Could you describe the environment where you're seeing the noise so that it can be considered for future testing?

          If you'd like to join those who are testing the git plugin before it releases (most typically that is an army of one, me), please send your contact information to mark.earl.waite@gmail.com.

          Mark Waite added a comment - - edited bizdelnick thanks for reporting the message that you're seeing. Unfortunately, I'm not seeing that message in any of my test environments. My test environments include: Controller installed on Debian 10 with a deb file, with agents Windows 10 Debian Testing Debian 10 on ARM Controller installed on Docker using Jenkins LTS base image with agents Windows 10 Debian 9 Debian 10 CentOS 7 CentOS 8 OpenBSD 6.9 FreeBSD 12.2 Ubuntu 18.04 Ubuntu 20.04 Could you describe the environment where you're seeing the noise so that it can be considered for future testing? If you'd like to join those who are testing the git plugin before it releases (most typically that is an army of one, me), please send your contact information to mark.earl.waite@gmail.com.

          I run Jenkins in docker container. Host system is CentOS 7 (SELinux in permissive mode), guest is Alpine (official jenkins/jenkins:lts-alpine image).

          Dmitry Mikhirev added a comment - I run Jenkins in docker container. Host system is CentOS 7 (SELinux in permissive mode), guest is Alpine (official jenkins/jenkins:lts-alpine image).

          I am experiencing the same situation running from a Gentoo Linux/Docker container for the controller, and a Windows 10 Pro VM where the agent is running. I'm on native git 2.20.1, Jenkins 2.289, Git 4.7.1, GitClient 3.7.1.

          Randall Becker added a comment - I am experiencing the same situation running from a Gentoo Linux/Docker container for the controller, and a Windows 10 Pro VM where the agent is running. I'm on native git 2.20.1, Jenkins 2.289, Git 4.7.1, GitClient 3.7.1.

          Mark Waite added a comment -

          I assume rsbeckerca that your Gentoo Linux that is hosting the container for the controller also has SELinux configured. Can you confirm that?

          I assume you are running one of the centos Docker images like jenkins/jenkins:lts-centos so that the operating system utilities inside the Docker image are aware of SELinux. Can you confirm that?

          I don't plan to spend time developing or testing the git plugin with SELinux. I'm willing to consider pull requests from others, but will have to rely on community validation of the pull requests because I won't test them myself.

          Mark Waite added a comment - I assume rsbeckerca that your Gentoo Linux that is hosting the container for the controller also has SELinux configured. Can you confirm that? I assume you are running one of the centos Docker images like jenkins/jenkins:lts-centos so that the operating system utilities inside the Docker image are aware of SELinux. Can you confirm that? I don't plan to spend time developing or testing the git plugin with SELinux. I'm willing to consider pull requests from others, but will have to rely on community validation of the pull requests because I won't test them myself.

          markewaite I did not configure SELinux explicitly - so this is actually news to me but I guess it is there. The machine is an Antsle box with a custom Gentoo. Jenkins is running in a root Docker, not inside a separate VM. Yes to the jenkins:lts image. I think the issue is that the Docker image only has limited access to SELinux so that may be a problem. What I'm wondering is whether we can do a mod to the git plugin to disable the git /usr/bin/chcon functionality from the pipeline in question with an option to make this go away.

          Randall Becker added a comment - markewaite I did not configure SELinux explicitly - so this is actually news to me but I guess it is there. The machine is an Antsle box with a custom Gentoo. Jenkins is running in a root Docker, not inside a separate VM. Yes to the jenkins:lts image. I think the issue is that the Docker image only has limited access to SELinux so that may be a problem. What I'm wondering is whether we can do a mod to the git plugin to disable the git /usr/bin/chcon functionality from the pipeline in question with an option to make this go away.

          Mark Waite added a comment -

          Pull requests to the git client plugin and git plugin are welcomed, though your pull request would need to be reviewed by jimklimov in his environment to confirm that it didn't do him harm. I'm willing to review the pull request but don't have the capacity or the equipment to test it interactively.

          Mark Waite added a comment - Pull requests to the git client plugin and git plugin are welcomed, though your pull request would need to be reviewed by jimklimov in his environment to confirm that it didn't do him harm. I'm willing to review the pull request but don't have the capacity or the equipment to test it interactively.

          I can't say it will be any time soon. My $DAYJOB has me pretty crushed for the next two weeks (at least). Some summer . but I will do what I can. I think I'd need some guidance as to where the option needs to go as this is magic code done by the controller rather than the agent. My own block is:

          {{ checkout([$class: 'GitSCM',}}
            branches: [[name: '*/production'],[name: '*/20*-Sprint-*']],
            extensions: [
             [$class: 'CleanBeforeCheckout'],
             [$class: 'CloneOption', timeout: 120, shallow: false],
             [$class: 'CheckoutOption', timeout: 120],
             userRemoteConfigs: [[credentialsId: yeah-this]])

          It seems like this might need to be a 'CheckoutOption', SELinux: false (default true) or something along those lines.

          Randall Becker added a comment - I can't say it will be any time soon. My $DAYJOB has me pretty crushed for the next two weeks (at least). Some summer . but I will do what I can. I think I'd need some guidance as to where the option needs to go as this is magic code done by the controller rather than the agent. My own block is: {{ checkout([$class: 'GitSCM',}}   branches: [ [name: '*/production'] , [name: '*/20*-Sprint-*'] ],   extensions: [    [$class: 'CleanBeforeCheckout'] ,    [$class: 'CloneOption', timeout: 120, shallow: false] ,    [$class: 'CheckoutOption', timeout: 120] ,    userRemoteConfigs: [ [credentialsId: yeah-this] ]) It seems like this might need to be a 'CheckoutOption', SELinux: false (default true) or something along those lines.

          Jim Klimov added a comment - - edited

          Just in case, remembering the context: what is the problem to solve here? Noise in logs due to SELinux problems avoidance?

          Reminder for first cause: in some but later apparently not all setups, SELinux forbids Jenkins/agent to read the temporary ssh key-related files from the ssh client program; this was semi-fixed by tweaking the file context for one toxic system I had access to, and only for password-less keys – trusting the script to inject passphrase to private key is a separate can of worms, and we anyway store both the key and its pass in credential store (and would instantiate both on the agent for the ssh operation), so there is moderate threat in just storing it open there.

          That noise is minimal since first "complaint", assuming all my PRs on the matter were merged. It should just log running the chcon (one line) but only burst into details if there are checkout errors that may be related to constraints on that worker.

          UPDATE: Checked, the noise reduction PR was https://github.com/jenkinsci/git-client-plugin/pull/693 and merged April 20; your Git Client version 3.7.1 was released April 4, so...

          Regarding an option to manage it - be my guest PRs linked above (IIRC 673, 690 and 693) should point you to the area of git-client-plugin code where this happens. Note that this is not just a checkout or just a clone option - it applies to all operations that happen over ssh (probably including implicit ones, without the schema in URI - just `git@server...` patterns), and only on (SE)Linux systems. That is, no-op for http(s) or Windows/BSD/Solaris/... and older Linuxes that were not tainted by harsh security methods

          Jim Klimov added a comment - - edited Just in case, remembering the context: what is the problem to solve here? Noise in logs due to SELinux problems avoidance? Reminder for first cause: in some but later apparently not all setups, SELinux forbids Jenkins/agent to read the temporary ssh key-related files from the ssh client program; this was semi-fixed by tweaking the file context for one toxic system I had access to, and only for password-less keys – trusting the script to inject passphrase to private key is a separate can of worms, and we anyway store both the key and its pass in credential store (and would instantiate both on the agent for the ssh operation), so there is moderate threat in just storing it open there. That noise is minimal since first "complaint", assuming all my PRs on the matter were merged. It should just log running the chcon (one line) but only burst into details if there are checkout errors that may be related to constraints on that worker. UPDATE: Checked, the noise reduction PR was https://github.com/jenkinsci/git-client-plugin/pull/693 and merged April 20; your Git Client version 3.7.1 was released April 4, so... Regarding an option to manage it - be my guest PRs linked above (IIRC 673, 690 and 693) should point you to the area of git-client-plugin code where this happens. Note that this is not just a checkout or just a clone option - it applies to all operations that happen over ssh (probably including implicit ones, without the schema in URI - just `git@server...` patterns), and only on (SE)Linux systems. That is, no-op for http(s) or Windows/BSD/Solaris/... and older Linuxes that were not tainted by harsh security methods

          Mark Waite added a comment -

          Amazon Linux 2022 enables selinux by default. It is still in preview, but seems a likely candidate to be fully supported by Amazon before the end of life of Amazon Linux 2 in 2023. Mark Waite needs to explore this further and identify what changes are needed in order to support Amazon Linux 2022.

          Mark Waite added a comment - Amazon Linux 2022 enables selinux by default. It is still in preview, but seems a likely candidate to be fully supported by Amazon before the end of life of Amazon Linux 2 in 2023. Mark Waite needs to explore this further and identify what changes are needed in order to support Amazon Linux 2022.

            jimklimov Jim Klimov
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: