Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51643

Kubernetes plugin doesn't respect privileged=true in jnlp container

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.89.4
      Kubernetes Plugin Version >= 1.60
    • Similar Issues:

      Description

      I have a library that defines the follow pod template & container templates prior to using the node step to use the container as a slave.

      podTemplate(
        name: "provisioner-${config.version}",
        label: "provisioner-${config.version}",
        cloud: config.cloudName,
        serviceAccount: 'jenkins',
        idleMinutes: 0,
        namespace: config.tenant,
        containers: [
          // This adds the custom provisioner slave container to the pod. Must be first with name 'jnlp'
          containerTemplate(
            name: 'jnlp',
            image: "${config.dockerUrl}/${config.tenant}/${config.provisioningImage}-${config.version}",
            ttyEnabled: false,
            args: '${computer.jnlpmac} ${computer.name}',
            command: '',
            workingDir: '/tmp',
            privileged: true
          )
        ]
      )

      The resulting job fails because the container is not privileged, and thus doesn't have access to utilties that set the corresponding log messages:

      May 31, 2018 6:18:43 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
      
      Created Pod: provisioner-v1.0-4gkgx in namespace redhat-multiarch-qe
      
      May 31, 2018 6:18:43 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
      
      Waiting for Pod to be scheduled (0/100): provisioner-v1.0-4gkgx
      
      May 31, 2018 6:18:49 PM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher logLastLines
      
      Error in provisioning; agent=KubernetesSlave name: provisioner-v1.0-4gkgx, template=PodTemplate{, name='provisioner-v1.0', label='provisioner-v1.0', serviceAccount='jenkins', containers=[ContainerTemplate{name='jnlp', image='172.30.1.1:5000/redhat-multiarch-qe/provisioner-v1.0', alwaysPullImage=true, workingDir='/tmp', command='', args='${computer.jnlpmac} ${computer.name}'}]}. Container jnlp exited with error 127. Logs: failed to link /usr/bin/java -> /etc/alternatives/java: Permission denied
      

      As expected, an easy workaround for this is to go into the Jenkins settings after the job has run for the first time and manually set the pod template "privileged" flag. With this set provisioning completes as expected.

      May 31, 2018 6:40:23 PM INFO hudson.slaves.NodeProvisioner$2 run
      Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)
      May 31, 2018 6:40:23 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
      Created Pod: provisioner-v1.0-85v6m in namespace redhat-multiarch-qe
      May 31, 2018 6:40:23 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
      

       

        Attachments

          Activity

          Hide
          csanchez Carlos Sanchez added a comment -

          I have given it a try and the plugin launches the pod correctly, the problem is that jnlp container does not run as root, you would need USER root in your Dockerfile

          Show
          csanchez Carlos Sanchez added a comment - I have given it a try and the plugin launches the pod correctly, the problem is that jnlp container does not run as root, you would need USER root in your Dockerfile
          Hide
          csanchez Carlos Sanchez added a comment -

          That said , setting the privileged flag for your usecase is not the right approach, you should set the right permissions in your image

          Show
          csanchez Carlos Sanchez added a comment - That said , setting the privileged flag for your usecase is not the right approach, you should set the right permissions in your image
          Hide
          jaypoulz Jeremy Poulin added a comment -

          Carlos Sanchez isn't the containerTemplate supposed to set the permissions of the (jnlp) container being launched, not the pod that contains it?

          I am making the suggested modifications to the container image in questions and will retest upon spinning up a fresh Jenkins master.

          Show
          jaypoulz Jeremy Poulin added a comment - Carlos Sanchez isn't the containerTemplate supposed to set the permissions of the (jnlp) container being launched, not the pod that contains it? I am making the suggested modifications to the container image in questions and will retest upon spinning up a fresh Jenkins master.
          Hide
          jaypoulz Jeremy Poulin added a comment -

          I have updated the underlying image to declare `USER root`. The issue persisted.

          However, when I simplified my container such that it contained only the following three lines,  I no longer encounter the issue. This suggests to me that something about the container configuration causes this issue.

          FROM openshift/jenkins-slave-base-centos7
          USER root
          ENV HOME=/home/jenkins

          I will update this issue and close it once I find the offending line.

          Show
          jaypoulz Jeremy Poulin added a comment - I have updated the underlying image to declare `USER root`. The issue persisted. However, when I simplified my container such that it contained only the following three lines,  I no longer encounter the issue. This suggests to me that something about the container configuration causes this issue. FROM openshift/jenkins-slave-base-centos7 USER root ENV HOME=/home/jenkins I will update this issue and close it once I find the offending line.
          Hide
          jaypoulz Jeremy Poulin added a comment -

          I cannot reproduce the issue with a simplified Dockerfile, so I'm left at a complete loss as to why this is actually occurring. However, I can confirm that the underlying issue is with my Dockerfile, since removing the following line for the original fixes the issue.

          WORKDIR root

          However, adding that line to a simplified Dockerfile does not reintroduce the issue. Therefore I must assume that this line is, in tandem with something else, causing a permissions issue or causing the container to run as something other than root.

          Either way, my initial assumption that the problem was in the Kubernetes plugin (since the error messages seemed to confirm this) cannot be proven in a reproducible manner, so I am closing this issue accordingly.

          Show
          jaypoulz Jeremy Poulin added a comment - I cannot reproduce the issue with a simplified Dockerfile, so I'm left at a complete loss as to why this is actually occurring. However, I can confirm that the underlying issue is with my Dockerfile, since removing the following line for the original fixes the issue. WORKDIR root However, adding that line to a simplified Dockerfile does not reintroduce the issue. Therefore I must assume that this line is, in tandem with something else, causing a permissions issue or causing the container to run as something other than root. Either way, my initial assumption that the problem was in the Kubernetes plugin (since the error messages seemed to confirm this) cannot be proven in a reproducible manner, so I am closing this issue accordingly.

            People

            Assignee:
            csanchez Carlos Sanchez
            Reporter:
            jaypoulz Jeremy Poulin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: