[JENKINS-50801] Cannot provision Slave Pod: mountPath: Required value - Jenkins Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Duplicate
Component/s: kubernetes-plugin
Labels:
None
Environment:

Hide
OS: Linux/Docker using jenkins/jenkins:lts
Jenkins Version: 2.107.2
Kubernetes Plugin: 1.5.1
Running Jenkins on Kubernetes. Using offical image from https://hub.docker.com/r/jenkins/jenkins/
Web browser: Chrome Version 65.0.3325.181 (Official Build) (64-bit)

Show
OS: Linux/Docker using jenkins/jenkins:lts Jenkins Version: 2.107.2 Kubernetes Plugin: 1.5.1 Running Jenkins on Kubernetes. Using offical image from https://hub.docker.com/r/jenkins/jenkins/ Web browser: Chrome Version 65.0.3325.181 (Official Build) (64-bit)

Similar Issues:

Show

Description

We are running the standard Jenkins container (lts) on Kubernetes and it is provisioning build slaves on the same cluster. After upgrading to Kubernetes Plugin v1.5.1 (from v1.3.2), Jenkins was no longer able to provision any build slaves and the Jenkins logs showed many errors like:

{{Error in provisioning; agent=KubernetesSlave name: builder-lfc8x, template=PodTemplate{, name='builder', namespace='jenkins', label='builder', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume@68a57ce6, volumes=[org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume@8e436e0, HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], HostPathVolume [mountPath=/usr/bin/docker, hostPath=/usr/bin/docker]], containers=[ContainerTemplate
{name='git-auth', image='us.gcr.io/myproject/git-auth:v1', workingDir='', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@4f8c0bfd}
, ContainerTemplate
{name='jnlp', image='us.gcr.io/myproject/jnlp-linux-node:v1', workingDir='/workspace/', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@26954299}
, ContainerTemplate
{name='builder', image='us.gcr.io/myproject/builder:v8', workingDir='/workspace/', command='cat', args='', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@6d1df705}
]}}}
 io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/jenkins/pods. Message: Pod "builder-lfc8x" is invalid: spec.containers[0].volumeMounts[3].mountPath: Required value. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[3].mountPath, message=Required value, reason=FieldValueRequired, additionalProperties={})], group=null, kind=Pod, name=builder-lfc8x, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "builder-lfc8x" is invalid: spec.containers[0].volumeMounts[3].mountPath: Required value, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
 {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)}}
 {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:411)}}
 {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)}}
 {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)}}
 {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)}}
 {{ at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:756)}}
 {{ at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:334)}}
 {{ at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:105)}}
 {{ at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:288)}}
 {{ at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)}}
 {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
 {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
 {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
 {{ at java.lang.Thread.run(Thread.java:748)}}{{}}

This seems like it is probably related to https://issues.jenkins-ci.org/browse/JENKINS-50525, but since that fix is in 1.5.1, perhaps we are experiencing something slightly different.

We have reverted the Kubernetes plugin back to 1.3.2 while we figure this out.

Attachments

Issue Links

duplicates

JENKINS-50525 Error When Provisioning Slave: mountPath: Required value

Resolved

Activity

Descending order - Click to sort in ascending order

Hide

Permalink

Robert Wehner added a comment - 2018-04-16 15:09

The jobs failing in this way are all pipelines (we don't have any non-pipeline jobs to compare to). Here's one example:

node('builder') { 
stage('Clean Workspace') { 
cleanWs() 
} 
stage('Clone source repo') { 
checkout scm 
} 
stage('Run Tests') { 
container('builder') { 
setGerritReview unsuccessfulMessage: "Tests failed" 
sh 'make test' 
} 
} 
stage('Run Package Build') { 
container('builder') { 
setGerritReview unsuccessfulMessage: "Build failed" 
sh """ 
export BUILD_NUMBER="${env.BUILD_NUMBER}" 
export WORKSPACE="${env.WORKSPACE}" 
make build 
""" 
} 
} 
}

The "builder" node is a PodTemplate defined in the UI. It inherits another UI-defined PodTemplate called "k8s-linux". Here's the cloud section from the config.xml:

<clouds>
 <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin="kubernetes@1.5.1">
 <name>kubernetes</name>
 <defaultsProviderTemplate></defaultsProviderTemplate>
 <templates>
 <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
 <inheritFrom></inheritFrom>
 <name>k8s-linux</name>
 <namespace></namespace>
 <privileged>false</privileged>
 <alwaysPullImage>false</alwaysPullImage>
 <instanceCap>2147483647</instanceCap>
 <slaveConnectTimeout>100</slaveConnectTimeout>
 <idleMinutes>5</idleMinutes>
 <activeDeadlineSeconds>0</activeDeadlineSeconds>
 <label>k8s linux</label>
 <nodeSelector></nodeSelector>
 <nodeUsageMode>NORMAL</nodeUsageMode>
 <customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
 <workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
 <memory>false</memory>
 </workspaceVolume>
 <volumes>
 <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
 <mountPath>/usr/bin/docker</mountPath>
 <hostPath>/usr/bin/docker</hostPath>
 </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
 <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
 <mountPath>/var/run/docker.sock</mountPath>
 <hostPath>/var/run/docker.sock</hostPath>
 </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
 <org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume>
 <mountPath>/var/lib/gob-auth</mountPath>
 <memory>true</memory>
 </org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume>
 </volumes>
 <containers>
 <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
 <name>jnlp</name>
 <image>us.gcr.io/myproject/jnlp-linux-node:v1</image>
 <privileged>false</privileged>
 <alwaysPullImage>false</alwaysPullImage>
 <workingDir>/workspace/</workingDir>
 <command></command>
 <args></args>
 <ttyEnabled>false</ttyEnabled>
 <resourceRequestCpu></resourceRequestCpu>
 <resourceRequestMemory></resourceRequestMemory>
 <resourceLimitCpu></resourceLimitCpu>
 <resourceLimitMemory></resourceLimitMemory>
 <envVars/>
 <ports/>
 <livenessProbe>
 <execArgs></execArgs>
 <timeoutSeconds>0</timeoutSeconds>
 <initialDelaySeconds>0</initialDelaySeconds>
 <failureThreshold>0</failureThreshold>
 <periodSeconds>0</periodSeconds>
 <successThreshold>0</successThreshold>
 </livenessProbe>
 </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
 <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
 <name>git-auth</name>
 <image>us.gcr.io/myproject/git-auth:v1</image>
 <privileged>false</privileged>
 <alwaysPullImage>false</alwaysPullImage>
 <workingDir></workingDir>
 <command></command>
 <args></args>
 <ttyEnabled>false</ttyEnabled>
 <resourceRequestCpu></resourceRequestCpu>
 <resourceRequestMemory></resourceRequestMemory>
 <resourceLimitCpu></resourceLimitCpu>
 <resourceLimitMemory></resourceLimitMemory>
 <envVars/>
 <ports/>
 <livenessProbe>
 <execArgs></execArgs>
 <timeoutSeconds>0</timeoutSeconds>
 <initialDelaySeconds>0</initialDelaySeconds>
 <failureThreshold>0</failureThreshold>
 <periodSeconds>0</periodSeconds>
 <successThreshold>0</successThreshold>
 </livenessProbe>
 </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
 </containers>
 <envVars/>
 <annotations/>
 <imagePullSecrets/>
 </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
 <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
 <inheritFrom>k8s-linux</inheritFrom>
 <name>builder</name>
 <namespace>jenkins</namespace>
 <privileged>false</privileged>
 <alwaysPullImage>false</alwaysPullImage>
 <instanceCap>2147483647</instanceCap>
 <slaveConnectTimeout>100</slaveConnectTimeout>
 <idleMinutes>1</idleMinutes>
 <activeDeadlineSeconds>0</activeDeadlineSeconds>
 <label>builder</label>
 <nodeSelector></nodeSelector>
 <nodeUsageMode>NORMAL</nodeUsageMode>
 <customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
 <workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
 <memory>false</memory>
 </workspaceVolume>
 <volumes/>
 <containers>
 <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
 <name>builder</name>
 <image>us.gcr.io/myproject/builder:v8</image>
 <privileged>false</privileged>
 <alwaysPullImage>false</alwaysPullImage>
 <workingDir>/workspace/</workingDir>
 <command>cat</command>
 <args></args>
 <ttyEnabled>true</ttyEnabled>
 <resourceRequestCpu></resourceRequestCpu>
 <resourceRequestMemory></resourceRequestMemory>
 <resourceLimitCpu></resourceLimitCpu>
 <resourceLimitMemory></resourceLimitMemory>
 <envVars/>
 <ports/>
 <livenessProbe>
 <execArgs></execArgs>
 <timeoutSeconds>0</timeoutSeconds>
 <initialDelaySeconds>0</initialDelaySeconds>
 <failureThreshold>0</failureThreshold>
 <periodSeconds>0</periodSeconds>
 <successThreshold>0</successThreshold>
 </livenessProbe>
 </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
 </containers>
 <envVars/>
 <annotations/>
 <imagePullSecrets/>
 </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
 </templates>
 <serverUrl>https://kubernetes.default</serverUrl>
 <skipTlsVerify>false</skipTlsVerify>
 <namespace>jenkins</namespace>
 <jenkinsUrl>http://jenkins-ui.jenkins.svc.cluster.local:8080</jenkinsUrl>
 <jenkinsTunnel>jenkins-discovery.jenkins.svc.cluster.local:50000</jenkinsTunnel>
 <credentialsId>jenkins</credentialsId>
 <containerCap>10</containerCap>
 <retentionTimeout>5</retentionTimeout>
 <connectTimeout>0</connectTimeout>
 <readTimeout>0</readTimeout>
 <labels class="com.google.common.collect.ImmutableMap">
 <entry>
 <string>jenkins</string>
 <string>slave</string>
 </entry>
 </labels>
 <maxRequestsPerHost>32</maxRequestsPerHost>
 </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>
 </clouds>

Show

Robert Wehner added a comment - 2018-04-16 15:09 The jobs failing in this way are all pipelines (we don't have any non-pipeline jobs to compare to). Here's one example: node( 'builder' ) { stage( 'Clean Workspace' ) { cleanWs() } stage( 'Clone source repo' ) { checkout scm } stage( 'Run Tests' ) { container( 'builder' ) { setGerritReview unsuccessfulMessage: "Tests failed" sh 'make test' } } stage( 'Run Package Build' ) { container( 'builder' ) { setGerritReview unsuccessfulMessage: "Build failed" sh """ export BUILD_NUMBER= "${env.BUILD_NUMBER}" export WORKSPACE= "${env.WORKSPACE}" make build """ } } } The "builder" node is a PodTemplate defined in the UI. It inherits another UI-defined PodTemplate called "k8s-linux". Here's the cloud section from the config.xml: <clouds> <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin= "kubernetes@1.5.1" > <name>kubernetes</name> <defaultsProviderTemplate></defaultsProviderTemplate> <templates> <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <inheritFrom></inheritFrom> <name>k8s-linux</name> <namespace></namespace> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <instanceCap>2147483647</instanceCap> <slaveConnectTimeout>100</slaveConnectTimeout> <idleMinutes>5</idleMinutes> <activeDeadlineSeconds>0</activeDeadlineSeconds> <label>k8s linux</label> <nodeSelector></nodeSelector> <nodeUsageMode>NORMAL</nodeUsageMode> <customWorkspaceVolumeEnabled> false </customWorkspaceVolumeEnabled> <workspaceVolume class= "org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume" > <memory> false </memory> </workspaceVolume> <volumes> <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <mountPath>/usr/bin/docker</mountPath> <hostPath>/usr/bin/docker</hostPath> </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <mountPath>/ var /run/docker.sock</mountPath> <hostPath>/ var /run/docker.sock</hostPath> </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume> <mountPath>/ var /lib/gob-auth</mountPath> <memory> true </memory> </org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume> </volumes> <containers> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>jnlp</name> <image>us.gcr.io/myproject/jnlp-linux-node:v1</image> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <workingDir>/workspace/</workingDir> <command></command> <args></args> <ttyEnabled> false </ttyEnabled> <resourceRequestCpu></resourceRequestCpu> <resourceRequestMemory></resourceRequestMemory> <resourceLimitCpu></resourceLimitCpu> <resourceLimitMemory></resourceLimitMemory> <envVars/> <ports/> <livenessProbe> <execArgs></execArgs> <timeoutSeconds>0</timeoutSeconds> <initialDelaySeconds>0</initialDelaySeconds> <failureThreshold>0</failureThreshold> <periodSeconds>0</periodSeconds> <successThreshold>0</successThreshold> </livenessProbe> </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>git-auth</name> <image>us.gcr.io/myproject/git-auth:v1</image> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <workingDir></workingDir> <command></command> <args></args> <ttyEnabled> false </ttyEnabled> <resourceRequestCpu></resourceRequestCpu> <resourceRequestMemory></resourceRequestMemory> <resourceLimitCpu></resourceLimitCpu> <resourceLimitMemory></resourceLimitMemory> <envVars/> <ports/> <livenessProbe> <execArgs></execArgs> <timeoutSeconds>0</timeoutSeconds> <initialDelaySeconds>0</initialDelaySeconds> <failureThreshold>0</failureThreshold> <periodSeconds>0</periodSeconds> <successThreshold>0</successThreshold> </livenessProbe> </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> </containers> <envVars/> <annotations/> <imagePullSecrets/> </org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <inheritFrom>k8s-linux</inheritFrom> <name>builder</name> <namespace>jenkins</namespace> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <instanceCap>2147483647</instanceCap> <slaveConnectTimeout>100</slaveConnectTimeout> <idleMinutes>1</idleMinutes> <activeDeadlineSeconds>0</activeDeadlineSeconds> <label>builder</label> <nodeSelector></nodeSelector> <nodeUsageMode>NORMAL</nodeUsageMode> <customWorkspaceVolumeEnabled> false </customWorkspaceVolumeEnabled> <workspaceVolume class= "org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume" > <memory> false </memory> </workspaceVolume> <volumes/> <containers> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>builder</name> <image>us.gcr.io/myproject/builder:v8</image> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <workingDir>/workspace/</workingDir> <command>cat</command> <args></args> <ttyEnabled> true </ttyEnabled> <resourceRequestCpu></resourceRequestCpu> <resourceRequestMemory></resourceRequestMemory> <resourceLimitCpu></resourceLimitCpu> <resourceLimitMemory></resourceLimitMemory> <envVars/> <ports/> <livenessProbe> <execArgs></execArgs> <timeoutSeconds>0</timeoutSeconds> <initialDelaySeconds>0</initialDelaySeconds> <failureThreshold>0</failureThreshold> <periodSeconds>0</periodSeconds> <successThreshold>0</successThreshold> </livenessProbe> </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> </containers> <envVars/> <annotations/> <imagePullSecrets/> </org.csanchez.jenkins.plugins.kubernetes.PodTemplate> </templates> <serverUrl>https: //kubernetes. default </serverUrl> <skipTlsVerify> false </skipTlsVerify> <namespace>jenkins</namespace> <jenkinsUrl>http: //jenkins-ui.jenkins.svc.cluster.local:8080</jenkinsUrl> <jenkinsTunnel>jenkins-discovery.jenkins.svc.cluster.local:50000</jenkinsTunnel> <credentialsId>jenkins</credentialsId> <containerCap>10</containerCap> <retentionTimeout>5</retentionTimeout> <connectTimeout>0</connectTimeout> <readTimeout>0</readTimeout> <labels class= "com.google.common.collect.ImmutableMap" > <entry> <string>jenkins</string> <string>slave</string> </entry> </labels> <maxRequestsPerHost>32</maxRequestsPerHost> </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud> </clouds>

Hide

Permalink

Carlos Sanchez added a comment - 2018-04-16 07:21

pipeline or global config? do you have an example?

Show

Carlos Sanchez added a comment - 2018-04-16 07:21 pipeline or global config? do you have an example?

Jenkins

Cannot provision Slave Pod: mountPath: Required value

Details

Description

Attachments

Issue Links

Activity

People

Dates