Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60133

Default HOME folder in containers does not exist

    XMLWordPrintable

Details

    • Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Major
    • Resolution: Fixed
    • kubernetes-plugin
    • None
    • Kubernetes plugin 1.21.1
      Openshift 3.11

    Description

      I upgraded from Kubernetes plugin 1.16 to 1.21 and suddenly many of our Jenkins slaves stopped working. We found that the issue is caused by the default HOME environment variable that the plugin adds to all containers in Jenkins slave pods in Openshift. https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L104

      That variable has a default value of /home/jenkins but if that folder does not exist then the container can fail. 

      Attachments

        Issue Links

          Activity

            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            This is probably a side effect of the change of working directory to /home/jenkins/agent (i.e. JENKINS-58705). If the /home/jenkins does not exist in the image, it will be automatically added during the mounting of the workspace volume but will not be writable.
            The DEFAULT_HOME might need to be changed. Otherwise images need to be adapted for arbitrary users as documented in https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/creating_images/creating-images-guidelines.

            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited This is probably a side effect of the change of working directory to /home/jenkins/agent (i.e. JENKINS-58705 ). If the /home/jenkins does not exist in the image, it will be automatically added during the mounting of the workspace volume but will not be writable. The DEFAULT_HOME might need to be changed. Otherwise images need to be adapted for arbitrary users as documented in https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/creating_images/creating-images-guidelines .
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            pgomez Setting the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent should workaround that problem and mock the behavior that was there before JENKINS-58705.

            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited pgomez Setting the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent should workaround that problem and mock the behavior that was there before JENKINS-58705 .
            pgomez Pablo Gomez added a comment -

            Thanks Allan, but when you are using some random image from DockerHub in the pod (i.e. Maven image), /home/jenkins does not exist and you cannot control which user that image is using

            pgomez Pablo Gomez added a comment - Thanks Allan, but when you are using some random image from DockerHub in the pod (i.e. Maven image), /home/jenkins does not exist and you cannot control which user that image is using
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            pgomez Most of images have a user defined and a HOME directory. As an example the maven image's HOME dir is set to /root and that i what is used when the kubernetes plugin spins up an agent with an additional maven container, like in the example.

            The real challenge with Openshift is that the arbitraty uids are generated dynamically without a $HOME directory / variable. The "trick" that was in place with the kubernetes plugin was to set the HOME directory to a location that for sure exists and is owned by the running user /home/jenkins. Now this has changed and that trick has become a breaking change for that scenario with Openshift. I do agree that this is a bug.

            Not sure what is the best solution here. Change the DEFAULT_HOME to match again the working directory, use another directory (but not sure which one, /tmp/ ?), use an additional emptyDir as HOME ?
            Maybe vlatombe has some ideas ?

            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited pgomez Most of images have a user defined and a HOME directory. As an example the maven image's HOME dir is set to /root and that i what is used when the kubernetes plugin spins up an agent with an additional maven container, like in the example . The real challenge with Openshift is that the arbitraty uids are generated dynamically without a $HOME directory / variable. The "trick" that was in place with the kubernetes plugin was to set the HOME directory to a location that for sure exists and is owned by the running user /home/jenkins . Now this has changed and that trick has become a breaking change for that scenario with Openshift. I do agree that this is a bug. Not sure what is the best solution here. Change the DEFAULT_HOME to match again the working directory, use another directory (but not sure which one, /tmp/ ?), use an additional emptyDir as HOME ? Maybe vlatombe has some ideas ?

            These problems make me think the default should be to do absolutely nothing (essentially, remove https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L291-L296).

            On Openshift, if you want to run an image from DockerHub, you should set up the HOME variable pointing to a volume backed by an empty dir to make sure the random uid has access. This can be done already using envvars and volumes or using the yaml spec.

            But if you have proper images setup following OpenShift recommendations and it behaves properly with a random UID, then the plugin shouldn't do anything.

            vlatombe Vincent Latombe added a comment - These problems make me think the default should be to do absolutely nothing (essentially, remove https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L291-L296 ). On Openshift, if you want to run an image from DockerHub, you should set up the HOME variable pointing to a volume backed by an empty dir to make sure the random uid has access. This can be done already using envvars and volumes or using the yaml spec. But if you have proper images setup following OpenShift recommendations and it behaves properly with a random UID, then the plugin shouldn't do anything.
            pgomez Pablo Gomez added a comment -

            Yes, most of the images have a HOME folder but that folder is not usually /home/jenkins. And if you use different images each one can have different home folders.

            pgomez Pablo Gomez added a comment - Yes, most of the images have a HOME folder but that folder is not usually /home/jenkins. And if you use different images each one can have different home folders.

            People

              vlatombe Vincent Latombe
              pgomez Pablo Gomez
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: