Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52144

unable to provision pods w/ plugin > 1.4.1

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      after upgrading from 1.4.1 to any new version, the latest being 1.8.4 i am unable to provision any pods in kubernetes. NOTHING has changed in my configuration, which means something is different in the plugin.

      in 1.4.1 i did NOT have to specify any working directory as part of the configuration, it is an empty value. for 1.8.4 it seems i am forced to pick a value and it defaults to /home/jenkins (ugh) if nothing is selected. i have explicitly created a /home/jenkins/.jenkins-agent directory and set the value to that, but still no luck.

      here is the pod template (from the logs) that gets created:

      Pod built: Pod(apiVersion=v1, kind=Pod, metadata=ObjectMeta(annotations={}, clusterName=null, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, initializers=null, labels={jenkins=slave, jenkins/maven-8-debian=true}, name=maven-8-debian-s9xnf, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[d71ba3254f49281ef132d9bd8c841dc52386c1b46c919a64e863a57912408399, maven-8-debian-s9xnf], command=[], env=[EnvVar(name=JENKINS_SECRET, value=d71ba3254f49281ef132d9bd8c841dc52386c1b46c919a64e863a57912408399, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_NAME, value=maven-8-debian-s9xnf, valueFrom=null, additionalProperties={}), EnvVar(name=JENKINS_URL, value=http://jenkins-beta.infra.us-east-1.conductor.sh/, valueFrom=null, additionalProperties={}), EnvVar(name=HOME, value=/home/jenkins/.jenkins-agent, valueFrom=null, additionalProperties={})], envFrom=[], image=registry.battery-park.conductor.com/jenkins-slave-8-maven-debian, imagePullPolicy=Always, lifecycle=null, livenessProbe=null, name=jnlp, ports=[], readinessProbe=null, resources=ResourceRequirements(limits={}, requests={}, additionalProperties={}), securityContext=SecurityContext(capabilities=null, privileged=false, readOnlyRootFilesystem=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, additionalProperties={}), stdin=null, stdinOnce=null, terminationMessagePath=null, terminationMessagePolicy=null, tty=true, volumeMounts=[VolumeMount(mountPath=/home/jenkins/.jenkins-agent, name=workspace-volume, readOnly=false, subPath=null, additionalProperties={})], workingDir=/home/jenkins/.jenkins-agent, additionalProperties={})], dnsPolicy=null, hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, restartPolicy=Never, schedulerName=null, securityContext=null, serviceAccount=null, serviceAccountName=null, subdomain=null, terminationGracePeriodSeconds=null, tolerations=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, downwardAPI=null, emptyDir=EmptyDirVolumeSource(medium=, sizeLimit=null, additionalProperties={}), fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=workspace-volume, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), status=null, additionalProperties={})

      and this the resulting error i get back

      Error in provisioning; agent=KubernetesSlave name: maven-8-debian-s9xnf, template=PodTemplate{inheritFrom='', name='maven-8-debian', namespace='', label='maven-8-debian', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], containers=[ContainerTemplate{name='jnlp', image='registry.battery-park.conductor.com/jenkins-slave-8-maven-debian', alwaysPullImage=true, workingDir='/home/jenkins/.jenkins-agent', command='', args='${computer.jnlpmac} ${computer.name}', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@5f5745d0}], yaml=}
      java.lang.IllegalStateException: Containers are terminated with exit codes: {jnlp=127}
      	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:156)
      	at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      i am unable to get any details about the pod b/c it gets deleted before i can get any information about it.

        Attachments

          Activity

          Hide
          csanchez Carlos Sanchez added a comment -

          you need to get the container logs, without that there's not enough info

          You can continuously run something like

          for pod in $(kubectl get pods -o name | grep mypod); do kubectl logs $pod; done

          Show
          csanchez Carlos Sanchez added a comment - you need to get the container logs, without that there's not enough info You can continuously run something like for pod in $(kubectl get pods -o name | grep mypod); do kubectl logs $pod; done
          Hide
          jgangemi Jae Gangemi added a comment -

          this seems to be related to whatever is happening to the working directory / jenkins home when the container is started.

          in my `Dockerfile` i have this:

          USER jenkins
          WORKDIR ${JENKINS_HOME}
          
          ENTRYPOINT ["slave-scripts/jenkins-slave.sh"]
          

          where JENKINS_HOME == /home/jenkins

          and i have found this in the system logs:

          Jun 22 15:45:28 ip-172-22-1-63.ec2.internal containerd[1004]: time="2018-06-22T15:45:28.017677203Z" level=error msg="containerd: start container" error="oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"slave-scripts/jenkins-slave.sh\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\"\n" id=27bc57569f3ce34c9cf120b5fb80d77a44e59b46d8a557190d46835232f6ee28
          Jun 22 15:45:28 ip-172-22-1-63.ec2.internal dockerd[1212]: time="2018-06-22T15:45:28.018189578Z" level=error msg="Create container failed with error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\"\\n\""
          Jun 22 15:45:28 ip-172-22-1-63.ec2.internal dockerd[1212]: time="2018-06-22T15:45:28.036649033Z" level=error msg="Handler for POST /containers/27bc57569f3ce34c9cf120b5fb80d77a44e59b46d8a557190d46835232f6ee28/start returned error: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\"\\n\""
          Jun 22 15:45:28 ip-172-22-1-63.ec2.internal kubelet[1773]: E0622 15:45:28.036968    1773 docker_manager.go:2250] container start failed: RunContainerError: runContainer: Error response from daemon: {"message":"invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"exec: \\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\"\\n\""}
          Jun 22 15:45:28 ip-172-22-1-63.ec2.internal kubelet[1773]: E0622 15:45:28.037029    1773 pod_workers.go:184] Error syncing pod 48cbf1ff-7633-11e8-b3fa-0ef76f0169b6, skipping: failed to "StartContainer" for "jnlp" with RunContainerError: "runContainer: Error response from daemon: {\"message\":\"invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"exec: \\\\\\\\\\\\\\\"slave-scripts/jenkins-slave.sh\\\\\\\\\\\\\\\": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\\\\\"\\\\n\\\"\"}"
          

          which seems to be indicating that it can't find the script in /home/jenkins.

          setting the working directory to something completely different /tmp/jenkins-agent doesn't solve anything either.

          but this seems to be related to something potentially mounting over /home/jenkns when it shouldn't be.

           

           

           

          Show
          jgangemi Jae Gangemi added a comment - this seems to be related to whatever is happening to the working directory / jenkins home when the container is started. in my `Dockerfile` i have this: USER jenkins WORKDIR ${JENKINS_HOME} ENTRYPOINT [ "slave-scripts/jenkins-slave.sh" ] where JENKINS_HOME == /home/jenkins and i have found this in the system logs: Jun 22 15:45:28 ip-172-22-1-63.ec2.internal containerd[1004]: time= "2018-06-22T15:45:28.017677203Z" level=error msg= "containerd: start container" error= "oci runtime error: container_linux.go:247: starting container process caused \" exec: \\\ "slave-scripts/jenkins-slave.sh\\\" : stat slave-scripts/jenkins-slave.sh: no such file or directory\ "\n" id=27bc57569f3ce34c9cf120b5fb80d77a44e59b46d8a557190d46835232f6ee28 Jun 22 15:45:28 ip-172-22-1-63.ec2.internal dockerd[1212]: time= "2018-06-22T15:45:28.018189578Z" level=error msg= "Create container failed with error: invalid header field value \" oci runtime error: container_linux.go:247: starting container process caused \\\ "exec: \\\\\\\" slave-scripts/jenkins-slave.sh\\\\\\\ ": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\" \\n\"" Jun 22 15:45:28 ip-172-22-1-63.ec2.internal dockerd[1212]: time= "2018-06-22T15:45:28.036649033Z" level=error msg= "Handler for POST /containers/27bc57569f3ce34c9cf120b5fb80d77a44e59b46d8a557190d46835232f6ee28/start returned error: invalid header field value \" oci runtime error: container_linux.go:247: starting container process caused \\\ "exec: \\\\\\\" slave-scripts/jenkins-slave.sh\\\\\\\ ": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\" \\n\"" Jun 22 15:45:28 ip-172-22-1-63.ec2.internal kubelet[1773]: E0622 15:45:28.036968    1773 docker_manager.go:2250] container start failed: RunContainerError: runContainer: Error response from daemon: { "message" : "invalid header field value \" oci runtime error: container_linux.go:247: starting container process caused \\\ "exec: \\\\\\\" slave-scripts/jenkins-slave.sh\\\\\\\ ": stat slave-scripts/jenkins-slave.sh: no such file or directory\\\" \\n\""} Jun 22 15:45:28 ip-172-22-1-63.ec2.internal kubelet[1773]: E0622 15:45:28.037029    1773 pod_workers.go:184] Error syncing pod 48cbf1ff-7633-11e8-b3fa-0ef76f0169b6, skipping: failed to "StartContainer" for "jnlp" with RunContainerError: "runContainer: Error response from daemon: {\" message\ ":\" invalid header field value \\\ "oci runtime error: container_linux.go:247: starting container process caused \\\\\\\" exec: \\\\\\\\\\\\\\\ "slave-scripts/jenkins-slave.sh\\\\\\\\\\\\\\\" : stat slave-scripts/jenkins-slave.sh: no such file or directory\\\\\\\ "\\\\n\\\" \ "}" which seems to be indicating that it can't find the script in /home/jenkins . setting the working directory to something completely different /tmp/jenkins-agent doesn't solve anything either. but this seems to be related to something potentially mounting over /home/jenkns when it shouldn't be.      
          Hide
          csanchez Carlos Sanchez added a comment -

          /home/jenkins is mounted as a wolume shared across all containers. There is an option to customize it in the UI, I don't recall if it can also remove it altogether.

          But if you move your startup script somewhere else it should work

          Show
          csanchez Carlos Sanchez added a comment - /home/jenkins is mounted as a wolume shared across all containers. There is an option to customize it in the UI, I don't recall if it can also remove it altogether. But if you move your startup script somewhere else it should work
          Hide
          jgangemi Jae Gangemi added a comment -

          i have other things in the home directory that get used so i can't just up and move everything.

          i used to be able to remove the working directory entirely in 1.4.1 but i can no longer do that in the latest version. i have even tried setting the working dir to a different directory and i still get the same behavior.

          i don't need a shared volume for my use case, i only run a single container, and this change is forcing me to have one.

           

          Show
          jgangemi Jae Gangemi added a comment - i have other things in the home directory that get used so i can't just up and move everything. i used to be able to remove the working directory entirely in 1.4.1 but i can no longer do that in the latest version. i have even tried setting the working dir to a different directory and i still get the same behavior. i don't need a shared volume for my use case, i only run a single container, and this change is forcing me to have one.  
          Hide
          jgangemi Jae Gangemi added a comment -

          any chance of seeing movement on this? JENKINS-50659 references the same issue i am where the behavior changed from 1.4 to 1.5 and it seems to all be related to the working directory between versions.

          i tried the latest version 1.12.2 and it's still broken.

          Show
          jgangemi Jae Gangemi added a comment - any chance of seeing movement on this? JENKINS-50659 references the same issue i am where the behavior changed from 1.4 to 1.5 and it seems to all be related to the working directory between versions. i tried the latest version 1.12.2 and it's still broken.
          Hide
          spugachev Stanislav Pugachov added a comment -

          I'm getting same behaviour but error is "{jnlp=0}"
          Is it related to this ticket or is it a different problem?

          Show
          spugachev Stanislav Pugachov added a comment - I'm getting same behaviour but error is "{jnlp=0}" Is it related to this ticket or is it a different problem?
          Hide
          vlatombe Vincent Latombe added a comment -

          Closing as stale. Please re-open if you have a reproducer with the current version.

          Show
          vlatombe Vincent Latombe added a comment - Closing as stale. Please re-open if you have a reproducer with the current version.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jgangemi Jae Gangemi
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: