Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60133

Default HOME folder in containers does not exist

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Kubernetes plugin 1.21.1
      Openshift 3.11
    • Similar Issues:

      Description

      I upgraded from Kubernetes plugin 1.16 to 1.21 and suddenly many of our Jenkins slaves stopped working. We found that the issue is caused by the default HOME environment variable that the plugin adds to all containers in Jenkins slave pods in Openshift. https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L104

      That variable has a default value of /home/jenkins but if that folder does not exist then the container can fail. 

        Attachments

          Issue Links

            Activity

            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            This is probably a side effect of the change of working directory to /home/jenkins/agent (i.e. JENKINS-58705). If the /home/jenkins does not exist in the image, it will be automatically added during the mounting of the workspace volume but will not be writable.
            The DEFAULT_HOME might need to be changed. Otherwise images need to be adapted for arbitrary users as documented in https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/creating_images/creating-images-guidelines.

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited This is probably a side effect of the change of working directory to /home/jenkins/agent (i.e. JENKINS-58705 ). If the /home/jenkins does not exist in the image, it will be automatically added during the mounting of the workspace volume but will not be writable. The DEFAULT_HOME might need to be changed. Otherwise images need to be adapted for arbitrary users as documented in https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/creating_images/creating-images-guidelines .
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            Pablo Gomez Setting the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent should workaround that problem and mock the behavior that was there before JENKINS-58705.

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited Pablo Gomez Setting the system property -Dorg.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.defaultHome=/home/jenkins/agent should workaround that problem and mock the behavior that was there before JENKINS-58705 .
            Hide
            pgomez Pablo Gomez added a comment -

            Thanks Allan, but when you are using some random image from DockerHub in the pod (i.e. Maven image), /home/jenkins does not exist and you cannot control which user that image is using

            Show
            pgomez Pablo Gomez added a comment - Thanks Allan, but when you are using some random image from DockerHub in the pod (i.e. Maven image), /home/jenkins does not exist and you cannot control which user that image is using
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            Pablo Gomez Most of images have a user defined and a HOME directory. As an example the maven image's HOME dir is set to /root and that i what is used when the kubernetes plugin spins up an agent with an additional maven container, like in the example.

            The real challenge with Openshift is that the arbitraty uids are generated dynamically without a $HOME directory / variable. The "trick" that was in place with the kubernetes plugin was to set the HOME directory to a location that for sure exists and is owned by the running user /home/jenkins. Now this has changed and that trick has become a breaking change for that scenario with Openshift. I do agree that this is a bug.

            Not sure what is the best solution here. Change the DEFAULT_HOME to match again the working directory, use another directory (but not sure which one, /tmp/ ?), use an additional emptyDir as HOME ?
            Maybe Vincent Latombe has some ideas ?

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited Pablo Gomez Most of images have a user defined and a HOME directory. As an example the maven image's HOME dir is set to /root and that i what is used when the kubernetes plugin spins up an agent with an additional maven container, like in the example . The real challenge with Openshift is that the arbitraty uids are generated dynamically without a $HOME directory / variable. The "trick" that was in place with the kubernetes plugin was to set the HOME directory to a location that for sure exists and is owned by the running user /home/jenkins . Now this has changed and that trick has become a breaking change for that scenario with Openshift. I do agree that this is a bug. Not sure what is the best solution here. Change the DEFAULT_HOME to match again the working directory, use another directory (but not sure which one, /tmp/ ?), use an additional emptyDir as HOME ? Maybe Vincent Latombe has some ideas ?
            Hide
            vlatombe Vincent Latombe added a comment -

            These problems make me think the default should be to do absolutely nothing (essentially, remove https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L291-L296).

            On Openshift, if you want to run an image from DockerHub, you should set up the HOME variable pointing to a volume backed by an empty dir to make sure the random uid has access. This can be done already using envvars and volumes or using the yaml spec.

            But if you have proper images setup following OpenShift recommendations and it behaves properly with a random UID, then the plugin shouldn't do anything.

            Show
            vlatombe Vincent Latombe added a comment - These problems make me think the default should be to do absolutely nothing (essentially, remove https://github.com/jenkinsci/kubernetes-plugin/blob/33737fe0c96615e33b78205fe6e71aaffca2df49/src/main/java/org/csanchez/jenkins/plugins/kubernetes/PodTemplateBuilder.java#L291-L296 ). On Openshift, if you want to run an image from DockerHub, you should set up the HOME variable pointing to a volume backed by an empty dir to make sure the random uid has access. This can be done already using envvars and volumes or using the yaml spec. But if you have proper images setup following OpenShift recommendations and it behaves properly with a random UID, then the plugin shouldn't do anything.
            Hide
            pgomez Pablo Gomez added a comment -

            Yes, most of the images have a HOME folder but that folder is not usually /home/jenkins. And if you use different images each one can have different home folders.

            Show
            pgomez Pablo Gomez added a comment - Yes, most of the images have a HOME folder but that folder is not usually /home/jenkins. And if you use different images each one can have different home folders.

              People

              Assignee:
              vlatombe Vincent Latombe
              Reporter:
              pgomez Pablo Gomez
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: