[JENKINS-41418] Permission denied while accessing workspace

Type: Bug
Resolution: Cannot Reproduce
Priority: Major
Component/s: kubernetes-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

I've added kubernetes cloud in system config and created git project with following Jenkinsfile and trying to run the job and hitting with following permission issues.

podTemplate(label: 'mypod', containers: [
    containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'golang', image: 'golang:1.6.3-alpine', ttyEnabled: true, command: 'cat')
  ],
  volumes: [hostPathVolume(mountPath: "/var/run/docker.sock", hostPath: "/var/run/docker.sock")]) {

    node ('mypod') {
        stage 'Get a Maven project'
        sh """
        cat /etc/resolv.conf
        route
        ip addr show
        """
        git 'https://github.com/jenkinsci/kubernetes-plugin.git'
        container('maven') {
            stage 'Build a Maven project'
            sh 'mvn clean install'
        }

        stage 'Get a Golang project'
        git url: 'https://github.com/hashicorp/terraform.git'
        container('golang') {
            stage 'Build a Go project'
            sh """
            mkdir -p /go/src/github.com/hashicorp
            ln -s `pwd` /go/src/github.com/hashicorp/terraform
            cd /go/src/github.com/hashicorp/terraform && make core-dev
            """
        }

    }
}

Error:

Started by user admin
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to git@github.ibm.com:mkumatag/test-kube-pipeline.git
 > git config remote.origin.url git@github.ibm.com:mkumatag/test-kube-pipeline.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen 1 remote branch
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git@github.ibm.com:mkumatag/test-kube-pipeline.git # timeout=10
Fetching upstream changes from git@github.ibm.com:mkumatag/test-kube-pipeline.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress git@github.ibm.com:mkumatag/test-kube-pipeline.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision 45e459e4645b1dfb2e27bde2b36503fb1d2926d6 (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 45e459e4645b1dfb2e27bde2b36503fb1d2926d6
 > git rev-list 45e459e4645b1dfb2e27bde2b36503fb1d2926d6 # timeout=10
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
kubernetes-634383bd02e4408a9ad879376fa358c7-4e36e7bfa07c9 is offline
Running on kubernetes-634383bd02e4408a9ad879376fa358c7-4e37578f5757a in /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
[Pipeline] {
[Pipeline] stage (Get a Maven project)
Using the 'stage' step without a block argument is deprecated
Entering stage Get a Maven project
Proceeding
[Pipeline] sh
[test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA] Running shell script
+ cat /etc/resolv.conf
+ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      *               255.255.0.0     U     0      0        0 eth0
+ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
14: eth0@if15: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.5/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:5/64 scope link 
       valid_lft forever preferred_lft forever
+ sleep 30
[Pipeline] git
Cloning the remote Git repository
Cloning repository https://github.com/jenkinsci/kubernetes-plugin.git
 > git init /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA # timeout=10
Fetching upstream changes from https://github.com/jenkinsci/kubernetes-plugin.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/jenkinsci/kubernetes-plugin.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/jenkinsci/kubernetes-plugin.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/jenkinsci/kubernetes-plugin.git # timeout=10
Fetching upstream changes from https://github.com/jenkinsci/kubernetes-plugin.git
 > git fetch --tags --progress https://github.com/jenkinsci/kubernetes-plugin.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision a9bb356e021a89fd174472217aa112ea37c0b308 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a9bb356e021a89fd174472217aa112ea37c0b308
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b master a9bb356e021a89fd174472217aa112ea37c0b308
 > git rev-list a9bb356e021a89fd174472217aa112ea37c0b308 # timeout=10
[Pipeline] container
[Pipeline] {
[Pipeline] stage (Build a Maven project)
Using the 'stage' step without a block argument is deprecated
Entering stage Build a Maven project
Proceeding
[Pipeline] sh
[test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA] Running shell script
Executing shell script inside container [maven] of pod [kubernetes-634383bd02e4408a9ad879376fa358c7-4e37578f5757a]
Executing command: sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/pid'; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/script.sh' > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/jenkins-result.txt' 
cd /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/pid'; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/script.sh' > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/jenkins-result.txt' 
exit
/home/jenkins # cd /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQ

LCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
/bin/sh: cd: can't cd to /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
/home/jenkins # sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_mast

er-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/pid

'; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/ho

me/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2

C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/script.sh' > '/home/jenkins/workspace/t

est-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tm

p/durable-ee7e41ae/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/tes

t-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/

durable-ee7e41ae/jenkins-result.txt' 
/bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/pid: Permission denied
/bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/jenkins-log.txt: Permission denied
/bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-ee7e41ae/jenkins-result.txt: Permission denied
/home/jenkins # exit
command terminated with non-zero exit code: Error executing in Docker Container: 1[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE

is related to

JENKINS-47827 Support passing a user/uid into containerTemplate

Resolved

Sergey Kulanov added a comment - 2017-01-26 13:08

please try to copy/paste the exact code from example https://github.com/jenkinsci/kubernetes-plugin#pipeline-support

and paste the result here

Sergey Kulanov added a comment - 2017-01-26 13:08 please try to copy/paste the exact code from example https://github.com/jenkinsci/kubernetes-plugin#pipeline-support and paste the result here

Manjunath Kumatagi added a comment - 2017-01-27 07:02

I copy pasted exact code from above git repo and tried running it,.. Jos is waiting for slave to be up and running to run the job but slave never came up and here what I find in the kubernetes:

[root@<kubernetes-master> kubernetes]# ./cluster/kubectl.sh get pods
NAME                                                        READY     STATUS              RESTARTS   AGE
kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9   0/3       ContainerCreating   0          5m


[root@<kubernetes-master> kubernetes]# ./cluster/kubectl.sh describe pod kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9
Name:		kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9
Namespace:	default
Node:		127.0.0.1/127.0.0.1
Start Time:	Fri, 27 Jan 2017 01:47:41 -0500
Labels:		jenkins=slave
		jenkins/mypod=true
Status:		Pending
IP:		
Controllers:	<none>
Containers:
  maven:
    Container ID:	
    Image:		maven:3.3.9-jdk-8-alpine
    Image ID:		
    Port:		
    Command:
      cat
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /etc/shared-secrets from volume-0 (rw)
      /home/jenkins from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jqq8v (ro)
    Environment Variables from:	<none>
    Environment Variables:
      JENKINS_SECRET:		0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5
      JENKINS_NAME:		kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9
      JENKINS_LOCATION_URL:	http://<jenkins-master>:8080/
      JENKINS_URL:		http://<jenkins-master>:8080
      JENKINS_JNLP_URL:		http://<jenkins-master>:8080/computer/kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9/slave-agent.jnlp
      HOME:			/home/jenkins
  golang:
    Container ID:	
    Image:		golang:1.6.3-alpine
    Image ID:		
    Port:		
    Command:
      cat
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /etc/shared-secrets from volume-0 (rw)
      /home/jenkins from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jqq8v (ro)
    Environment Variables from:	<none>
    Environment Variables:
      JENKINS_SECRET:		0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5
      JENKINS_NAME:		kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9
      JENKINS_LOCATION_URL:	http://<jenkins-master>:8080/
      JENKINS_URL:		http://<jenkins-master>:8080
      JENKINS_JNLP_URL:		http://<jenkins-master>:8080/computer/kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9/slave-agent.jnlp
      HOME:			/home/jenkins
  jnlp:
    Container ID:	
    Image:		jenkinsci/jnlp-slave:alpine
    Image ID:		
    Port:		
    Args:
      0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5
      kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Volume Mounts:
      /etc/shared-secrets from volume-0 (rw)
      /home/jenkins from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jqq8v (ro)
    Environment Variables from:	<none>
    Environment Variables:
      JENKINS_SECRET:		0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5
      JENKINS_NAME:		kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9
      JENKINS_LOCATION_URL:	http://<jenkins-master>:8080/
      JENKINS_URL:		http://<jenkins-master>:8080
      JENKINS_JNLP_URL:		http://<jenkins-master>:8080/computer/kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9/slave-agent.jnlp
      HOME:			/home/jenkins
Conditions:
  Type		Status
  Initialized 	True 
  Ready 	False 
  PodScheduled 	True 
Volumes:
  volume-0:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	shared-secrets
  workspace-volume:
    Type:	EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:	
  default-token-jqq8v:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	default-token-jqq8v
QoS Class:	BestEffort
Node-Selectors:	<none>
Tolerations:	<none>
Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  5m		5m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 to 127.0.0.1
  5m		1m		10	{kubelet 127.0.0.1}			Warning		FailedMount	MountVolume.SetUp failed for volume "kubernetes.io/secret/80020c1e-e45c-11e6-9dc3-005056aa2e0b-volume-0" (spec.Name: "volume-0") pod "80020c1e-e45c-11e6-9dc3-005056aa2e0b" (UID: "80020c1e-e45c-11e6-9dc3-005056aa2e0b") with: secrets "shared-secrets" not found
  3m		1m		2	{kubelet 127.0.0.1}			Warning		FailedMount	Unable to mount volumes for pod "kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9_default(80020c1e-e45c-11e6-9dc3-005056aa2e0b)": timeout expired waiting for volumes to attach/mount for pod "default"/"kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9". list of unattached/unmounted volumes=[volume-0]
  3m		1m		2	{kubelet 127.0.0.1}			Warning		FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9". list of unattached/unmounted volumes=[volume-0]
[root@<kubernetes-master> kubernetes]#

secret volume was the only difference what I had in my earlier try volumes: [secretVolume(secretName: 'shared-secrets', mountPath: '/etc/shared-secrets')].

The steps I'm trying are :

1. export API_HOST_IP="0.0.0.0"
2. Running the kubernetes cluster with the help of "./hack/local-up-cluster.sh"
3. exposed 8080 port on the same machine
4. Configured the kubernetes cloud in the jenkins configuration with IP : http://<kubernetes-master>:8080
5. Created this job and running it.

Please let me know if I'm missing something here? like creating some sort of a secret and put it somewhere!

Manjunath Kumatagi added a comment - 2017-01-27 07:02 I copy pasted exact code from above git repo and tried running it,.. Jos is waiting for slave to be up and running to run the job but slave never came up and here what I find in the kubernetes: [root@<kubernetes-master> kubernetes]# ./cluster/kubectl.sh get pods NAME READY STATUS RESTARTS AGE kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 0/3 ContainerCreating 0 5m [root@<kubernetes-master> kubernetes]# ./cluster/kubectl.sh describe pod kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 Name: kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 Namespace: default Node: 127.0.0.1/127.0.0.1 Start Time: Fri, 27 Jan 2017 01:47:41 -0500 Labels: jenkins=slave jenkins/mypod= true Status: Pending IP: Controllers: <none> Containers: maven: Container ID: Image: maven:3.3.9-jdk-8-alpine Image ID: Port: Command: cat State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Volume Mounts: /etc/shared-secrets from volume-0 (rw) /home/jenkins from workspace-volume (rw) / var /run/secrets/kubernetes.io/serviceaccount from default -token-jqq8v (ro) Environment Variables from: <none> Environment Variables: JENKINS_SECRET: 0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5 JENKINS_NAME: kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 JENKINS_LOCATION_URL: http: //<jenkins-master>:8080/ JENKINS_URL: http: //<jenkins-master>:8080 JENKINS_JNLP_URL: http: //<jenkins-master>:8080/computer/kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9/slave-agent.jnlp HOME: /home/jenkins golang: Container ID: Image: golang:1.6.3-alpine Image ID: Port: Command: cat State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Volume Mounts: /etc/shared-secrets from volume-0 (rw) /home/jenkins from workspace-volume (rw) / var /run/secrets/kubernetes.io/serviceaccount from default -token-jqq8v (ro) Environment Variables from: <none> Environment Variables: JENKINS_SECRET: 0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5 JENKINS_NAME: kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 JENKINS_LOCATION_URL: http: //<jenkins-master>:8080/ JENKINS_URL: http: //<jenkins-master>:8080 JENKINS_JNLP_URL: http: //<jenkins-master>:8080/computer/kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9/slave-agent.jnlp HOME: /home/jenkins jnlp: Container ID: Image: jenkinsci/jnlp-slave:alpine Image ID: Port: Args: 0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5 kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Volume Mounts: /etc/shared-secrets from volume-0 (rw) /home/jenkins from workspace-volume (rw) / var /run/secrets/kubernetes.io/serviceaccount from default -token-jqq8v (ro) Environment Variables from: <none> Environment Variables: JENKINS_SECRET: 0c574b20fd8c968fbc2671d550c454c8c15e9cbb4cb810055c9dae547ac428a5 JENKINS_NAME: kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 JENKINS_LOCATION_URL: http: //<jenkins-master>:8080/ JENKINS_URL: http: //<jenkins-master>:8080 JENKINS_JNLP_URL: http: //<jenkins-master>:8080/computer/kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9/slave-agent.jnlp HOME: /home/jenkins Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: volume-0: Type: Secret (a volume populated by a Secret) SecretName: shared-secrets workspace-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default -token-jqq8v: Type: Secret (a volume populated by a Secret) SecretName: default -token-jqq8v QoS Class : BestEffort Node-Selectors: <none> Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 5m 5m 1 { default -scheduler } Normal Scheduled Successfully assigned kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9 to 127.0.0.1 5m 1m 10 {kubelet 127.0.0.1} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/secret/80020c1e-e45c-11e6-9dc3-005056aa2e0b-volume-0" (spec.Name: "volume-0" ) pod "80020c1e-e45c-11e6-9dc3-005056aa2e0b" (UID: "80020c1e-e45c-11e6-9dc3-005056aa2e0b" ) with: secrets "shared-secrets" not found 3m 1m 2 {kubelet 127.0.0.1} Warning FailedMount Unable to mount volumes for pod "kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9_default(80020c1e-e45c-11e6-9dc3-005056aa2e0b)" : timeout expired waiting for volumes to attach/mount for pod " default " / "kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9" . list of unattached/unmounted volumes=[volume-0] 3m 1m 2 {kubelet 127.0.0.1} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod " default " / "kubernetes-68ce0b016db741cc9bc7f89b961e3bbd-57f30d1779ab9" . list of unattached/unmounted volumes=[volume-0] [root@<kubernetes-master> kubernetes]# secret volume was the only difference what I had in my earlier try volumes: [secretVolume(secretName: 'shared-secrets', mountPath: '/etc/shared-secrets')] . The steps I'm trying are : 1. export API_HOST_IP="0.0.0.0" 2. Running the kubernetes cluster with the help of "./hack/local-up-cluster.sh" 3. exposed 8080 port on the same machine 4. Configured the kubernetes cloud in the jenkins configuration with IP : http://<kubernetes-master>:8080 5. Created this job and running it. Please let me know if I'm missing something here? like creating some sort of a secret and put it somewhere!

Sergey Kulanov added a comment - 2017-01-27 10:06

Actually I have the same issue, but it's not 100% reproducible, `waiting to start slave`, that's why I asked your to check the example. Ok so, please try to define jnlp container explicitly:

          containerTemplate(
              name: 'jnlp',
              image: jnlpSlaveImg,
              args: '${computer.jnlpmac} ${computer.name}'
          )

where jnlpSlaveImg can by upstream jenkinsci/jnlp-slave:2.62-alpine or any other which you customized

Sergey Kulanov added a comment - 2017-01-27 10:06 Actually I have the same issue, but it's not 100% reproducible, `waiting to start slave`, that's why I asked your to check the example. Ok so, please try to define jnlp container explicitly: containerTemplate( name: 'jnlp' , image: jnlpSlaveImg, args: '${computer.jnlpmac} ${computer.name}' ) where jnlpSlaveImg can by upstream jenkinsci/jnlp-slave:2.62-alpine or any other which you customized

Carlos Sanchez added a comment - 2017-01-29 17:14

I have removed the shared-secrets volume from the example, as it won't work out of the bat. You can see the error in your "describe"

Warning		FailedMount	MountVolume.SetUp failed for volume "kubernetes.io/secret/80020c1e-e45c-11e6-9dc3-005056aa2e0b-volume-0" (spec.Name: "volume-0") pod "80020c1e-e45c-11e6-9dc3-005056aa2e0b" (UID: "80020c1e-e45c-11e6-9dc3-005056aa2e0b") with: secrets "shared-secrets" not found

Now, if you describe the pods created in your pipeline, do you get error messages in the events section ?
Also check kubectl logs pods/(your pod name) jnlp for errors

Carlos Sanchez added a comment - 2017-01-29 17:14 I have removed the shared-secrets volume from the example, as it won't work out of the bat. You can see the error in your "describe" Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/secret/80020c1e-e45c-11e6-9dc3-005056aa2e0b-volume-0" (spec.Name: "volume-0" ) pod "80020c1e-e45c-11e6-9dc3-005056aa2e0b" (UID: "80020c1e-e45c-11e6-9dc3-005056aa2e0b" ) with: secrets "shared-secrets" not found Now, if you describe the pods created in your pipeline, do you get error messages in the events section ? Also check kubectl logs pods/(your pod name) jnlp for errors

Manjunath Kumatagi added a comment - 2017-02-01 06:28 - edited

Removed the secret volume and ran the job, hitting with the problem mentioned in the description, "Build a Maven project" stage is failing. Here is the complete build console output:

Started by user admin
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to git@github.ibm.com:mkumatag/test-kube-pipeline.git
 > git config remote.origin.url git@github.ibm.com:mkumatag/test-kube-pipeline.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
Seen branch in repository origin/master
Seen 1 remote branch
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git@github.ibm.com:mkumatag/test-kube-pipeline.git # timeout=10
Fetching upstream changes from git@github.ibm.com:mkumatag/test-kube-pipeline.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress git@github.ibm.com:mkumatag/test-kube-pipeline.git +refs/heads/*:refs/remotes/origin/*
Checking out Revision 916212bda350d9ea81c1419c2501f5177e396a8b (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 916212bda350d9ea81c1419c2501f5177e396a8b
 > git rev-list 916212bda350d9ea81c1419c2501f5177e396a8b # timeout=10
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Running on kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3 in /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
[Pipeline] {
[Pipeline] stage (Get a Maven project)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Get a Maven project
Proceeding
[Pipeline] git
Cloning the remote Git repository
Cloning repository https://github.com/jenkinsci/kubernetes-plugin.git
 > git init /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA # timeout=10
Fetching upstream changes from https://github.com/jenkinsci/kubernetes-plugin.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/jenkinsci/kubernetes-plugin.git +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/jenkinsci/kubernetes-plugin.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/jenkinsci/kubernetes-plugin.git # timeout=10
Fetching upstream changes from https://github.com/jenkinsci/kubernetes-plugin.git
 > git fetch --tags --progress https://github.com/jenkinsci/kubernetes-plugin.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 8078e125c4e2241814ce70a129b8fb7ab4abe81a (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8078e125c4e2241814ce70a129b8fb7ab4abe81a
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b master 8078e125c4e2241814ce70a129b8fb7ab4abe81a
 > git rev-list 8078e125c4e2241814ce70a129b8fb7ab4abe81a # timeout=10
[Pipeline] container
[Pipeline] {
[Pipeline] stage (Build a Maven project)
Using the ‘stage’ step without a block argument is deprecated
Entering stage Build a Maven project
Proceeding
[Pipeline] sh
[test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA] Running shell script
Executing shell script inside container [maven] of pod [kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3]
Executing command: sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid'; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/script.sh' > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-result.txt' 
cd /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid'; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/script.sh' > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-result.txt' 
exit
/home/jenkins # cd /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQ

LCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
/bin/sh: cd: can't cd to /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA
/home/jenkins # sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_mast

er-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid

'; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/ho

me/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2

C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/script.sh' > '/home/jenkins/workspace/t

est-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tm

p/durable-b11ca984/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/tes

t-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/

durable-b11ca984/jenkins-result.txt' 
/bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid: Permission denied
/bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-log.txt: Permission denied
/bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-result.txt: Permission denied
/home/jenkins # exit
command terminated with non-zero exit code: Error executing in Docker Container: 1[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code -2
Finished: FAILURE

I don't see any surprise in jnlp pod log:

[root@kube-master kubernetes]# ./cluster/kubectl.sh logs kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3 jnlp
Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-master:8080]
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-master:4420
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Feb 01, 2017 6:19:05 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
[root@kube-master kubernetes]#

I tried maven logs but couldn't get any logs, I suspect that while getting into maven containers workspace it is getting some permission issues and failing the job. Not really sure with what permissions this shared jenkins workspace with created which is creating the problem here!

Manjunath Kumatagi added a comment - 2017-02-01 06:28 - edited Removed the secret volume and ran the job, hitting with the problem mentioned in the description, "Build a Maven project" stage is failing. Here is the complete build console output: Started by user admin > git rev-parse --is-inside-work-tree # timeout=10 Setting origin to git@github.ibm.com:mkumatag/test-kube-pipeline.git > git config remote.origin.url git@github.ibm.com:mkumatag/test-kube-pipeline.git # timeout=10 Fetching origin... Fetching upstream changes from origin > git --version # timeout=10 using GIT_SSH to set credentials > git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* Seen branch in repository origin/master Seen 1 remote branch > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url git@github.ibm.com:mkumatag/test-kube-pipeline.git # timeout=10 Fetching upstream changes from git@github.ibm.com:mkumatag/test-kube-pipeline.git > git --version # timeout=10 using GIT_SSH to set credentials > git fetch --tags --progress git@github.ibm.com:mkumatag/test-kube-pipeline.git +refs/heads/*:refs/remotes/origin/* Checking out Revision 916212bda350d9ea81c1419c2501f5177e396a8b (master) > git config core.sparsecheckout # timeout=10 > git checkout -f 916212bda350d9ea81c1419c2501f5177e396a8b > git rev-list 916212bda350d9ea81c1419c2501f5177e396a8b # timeout=10 [Pipeline] podTemplate [Pipeline] { [Pipeline] node Running on kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3 in /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA [Pipeline] { [Pipeline] stage (Get a Maven project) Using the ‘stage’ step without a block argument is deprecated Entering stage Get a Maven project Proceeding [Pipeline] git Cloning the remote Git repository Cloning repository https: //github.com/jenkinsci/kubernetes-plugin.git > git init /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA # timeout=10 Fetching upstream changes from https: //github.com/jenkinsci/kubernetes-plugin.git > git --version # timeout=10 > git fetch --tags --progress https: //github.com/jenkinsci/kubernetes-plugin.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https: //github.com/jenkinsci/kubernetes-plugin.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https: //github.com/jenkinsci/kubernetes-plugin.git # timeout=10 Fetching upstream changes from https: //github.com/jenkinsci/kubernetes-plugin.git > git fetch --tags --progress https: //github.com/jenkinsci/kubernetes-plugin.git +refs/heads/*:refs/remotes/origin/* > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 8078e125c4e2241814ce70a129b8fb7ab4abe81a (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 8078e125c4e2241814ce70a129b8fb7ab4abe81a > git branch -a -v --no-abbrev # timeout=10 > git checkout -b master 8078e125c4e2241814ce70a129b8fb7ab4abe81a > git rev-list 8078e125c4e2241814ce70a129b8fb7ab4abe81a # timeout=10 [Pipeline] container [Pipeline] { [Pipeline] stage (Build a Maven project) Using the ‘stage’ step without a block argument is deprecated Entering stage Build a Maven project Proceeding [Pipeline] sh [test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA] Running shell script Executing shell script inside container [maven] of pod [kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3] Executing command: sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid' ; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/script.sh' > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-result.txt' cd /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid' ; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/script.sh' > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-log.txt' 2>&1; echo $? > '/home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-result.txt' exit /home/jenkins # cd /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQ LCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA /bin/sh: cd: can't cd to /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA /home/jenkins # sh -c echo $$ > '/home/jenkins/workspace/test-kube-pipeline_mast er-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid '; jsc=durable-d2db8e2edcfe3ddf1127c78a3c7156e6; JENKINS_SERVER_COOKIE=$jsc ' /ho me/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2 C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/script.sh ' > ' /home/jenkins/workspace/t est-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tm p/durable-b11ca984/jenkins-log.txt ' 2>&1; echo $? > ' /home/jenkins/workspace/tes t-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/ durable-b11ca984/jenkins-result.txt' /bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/pid: Permission denied /bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-log.txt: Permission denied /bin/sh: can't create /home/jenkins/workspace/test-kube-pipeline_master-IUIW3TJVYLQLCZOGWHIM6BOJZ3VXUBKJ2C2L4YNTBE4GL52NRMEA@tmp/durable-b11ca984/jenkins-result.txt: Permission denied /home/jenkins # exit command terminated with non-zero exit code: Error executing in Docker Container: 1[Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline ERROR: script returned exit code -2 Finished: FAILURE I don't see any surprise in jnlp pod log: [root@kube-master kubernetes]# ./cluster/kubectl.sh logs kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3 jnlp Warning: JnlpProtocol3 is disabled by default , use JNLP_PROTOCOL_OPTS to alter the behavior Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main createEngine INFO: Setting up slave: kubernetes-3f499c55e0fb4feea46c66e78e82ded3-70682873cd6d3 Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener <init> INFO: Jenkins agent is running in headless mode. Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http: //jenkins-master:8080] Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins-master:4420 Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Protocol JNLP3-connect is not enabled, skipping Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP2-connect Feb 01, 2017 6:18:44 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Feb 01, 2017 6:19:05 AM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated [root@kube-master kubernetes]# I tried maven logs but couldn't get any logs, I suspect that while getting into maven containers workspace it is getting some permission issues and failing the job. Not really sure with what permissions this shared jenkins workspace with created which is creating the problem here!

Carlos Sanchez added a comment - 2017-02-12 10:58

What version of kubernetes are you using?
The only reason I can think this would fail is if containers were using different permissions, because they are all accessing the same emptyVolume

Can you try exec'ing into the containers and list the permissions in /home/jenkins/workspace and subdirs and possibly trying to write a file in there from the 3 containers to see if it works or fails ?

Carlos Sanchez added a comment - 2017-02-12 10:58 What version of kubernetes are you using? The only reason I can think this would fail is if containers were using different permissions, because they are all accessing the same emptyVolume Can you try exec'ing into the containers and list the permissions in /home/jenkins/workspace and subdirs and possibly trying to write a file in there from the 3 containers to see if it works or fails ?

Manjunath Kumatagi added a comment - 2017-03-06 04:54

After trying with all latest - kubernetes, jenkins plugin I'm not seeing this issue anymore. Will report a new issue or reopen this if I hit this issue again, -Thanks.

Manjunath Kumatagi added a comment - 2017-03-06 04:54 After trying with all latest - kubernetes, jenkins plugin I'm not seeing this issue anymore. Will report a new issue or reopen this if I hit this issue again, -Thanks.

Gytis Ramanauskas added a comment - 2017-04-20 13:47

csanchez this also happens to me when using jenkinsci/jnlp-slave > 2.62

Let's say I add a gradle container:

containerTemplate(name: 'gradle', image: 'gradle:3.4-jdk8-alpine', ttyEnabled: true, command: 'cat', workingDir:'/home/jenkins')

And then try to run gradle build on that container, I will get permission denied. If I understand correctly, this is because in jenkinsci/jnlp-slave image version > 2.62 the jenkins user has and uid of 10000 (in version 2.62 that uid is 1000).

So files are copied to gradle container with permissions for uid 10000. There is no such uid on gradle container (as gradle user has uid of 1000) and therfore it fails.

Debug output:

➜ ~ kubectl -n jenkins exec -it kubernetes-e36a3b0093244aa5a4c6ab7390c0f2dd-1388857a00d2 --container jnlp -- /bin/sh
$ ls -al
total 24
drwxrwxrwx 5 root root 4096 Apr 18 14:10 .
drwxr-xr-x 3 root root 4096 Apr 18 13:56 ..
-rw------- 1 1000 1000 405 Apr 18 14:10 .ash_history
drwxr-xr-x 3 jenkins jenkins 4096 Apr 18 14:09 .jenkins
drwx------ 2 jenkins jenkins 4096 Apr 18 14:09 .ssh
drwxr-xr-x 4 jenkins jenkins 4096 Apr 18 14:10 workspace
$ pwd
/home/jenkins
$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/false
systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/false
systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/false
systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/false
messagebus:x:104:108::/var/run/dbus:/bin/false
jenkins:x:10000:10000:Jenkins user:/home/jenkins:/bin/sh
$
➜ ~ kubectl -n jenkins exec -it kubernetes-e36a3b0093244aa5a4c6ab7390c0f2dd-1388857a00d2 --container gradle -- /bin/sh
/home/jenkins $ ls -al
total 24
drwxrwxrwx 5 root root 4096 Apr 18 14:10 .
drwxr-xr-x 1 root root 4096 Apr 18 14:09 ..
-rw------- 1 gradle gradle 412 Apr 18 14:10 .ash_history
drwxr-xr-x 2 root root 4096 Apr 18 14:09 .jenkins
drwx------ 2 10000 10000 4096 Apr 18 14:09 .ssh
drwxr-xr-x 4 10000 10000 4096 Apr 18 14:10 workspace
/home/jenkins $ cd workspace/
/home/jenkins/workspace $ touch test
touch: test: Permission denied
/home/jenkins/workspace $ whoami
gradle
/home/jenkins/workspace $ cat /etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
postgres:x:70:70::/var/lib/postgresql:/bin/sh
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
gradle:x:1000:1000:Linux User,,,:/home/gradle:/bin/ash
/home/jenkins/workspace $ %

Gytis Ramanauskas added a comment - 2017-04-20 13:47 csanchez this also happens to me when using jenkinsci/jnlp-slave > 2.62 Let's say I add a gradle container: containerTemplate(name: 'gradle' , image: 'gradle:3.4-jdk8-alpine' , ttyEnabled: true , command: 'cat' , workingDir: '/home/jenkins' ) And then try to run gradle build on that container, I will get permission denied. If I understand correctly, this is because in jenkinsci/jnlp-slave image version > 2.62 the jenkins user has and uid of 10000 (in version 2.62 that uid is 1000). So files are copied to gradle container with permissions for uid 10000. There is no such uid on gradle container (as gradle user has uid of 1000) and therfore it fails. Debug output: ➜ ~ kubectl -n jenkins exec -it kubernetes-e36a3b0093244aa5a4c6ab7390c0f2dd-1388857a00d2 --container jnlp -- /bin/sh $ ls -al total 24 drwxrwxrwx 5 root root 4096 Apr 18 14:10 . drwxr-xr-x 3 root root 4096 Apr 18 13:56 .. -rw------- 1 1000 1000 405 Apr 18 14:10 .ash_history drwxr-xr-x 3 jenkins jenkins 4096 Apr 18 14:09 .jenkins drwx------ 2 jenkins jenkins 4096 Apr 18 14:09 .ssh drwxr-xr-x 4 jenkins jenkins 4096 Apr 18 14:10 workspace $ pwd /home/jenkins $ cat /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin sys:x:3:3:sys:/dev:/usr/sbin/nologin sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/usr/sbin/nologin man:x:6:12:man:/ var /cache/man:/usr/sbin/nologin lp:x:7:7:lp:/ var /spool/lpd:/usr/sbin/nologin mail:x:8:8:mail:/ var /mail:/usr/sbin/nologin news:x:9:9:news:/ var /spool/news:/usr/sbin/nologin uucp:x:10:10:uucp:/ var /spool/uucp:/usr/sbin/nologin proxy:x:13:13:proxy:/bin:/usr/sbin/nologin www-data:x:33:33:www-data:/ var /www:/usr/sbin/nologin backup:x:34:34:backup:/ var /backups:/usr/sbin/nologin list:x:38:38:Mailing List Manager:/ var /list:/usr/sbin/nologin irc:x:39:39:ircd:/ var /run/ircd:/usr/sbin/nologin gnats:x:41:41:Gnats Bug-Reporting System (admin):/ var /lib/gnats:/usr/sbin/nologin nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin systemd-timesync:x:100:103:systemd Time Synchronization,,,:/run/systemd:/bin/ false systemd-network:x:101:104:systemd Network Management,,,:/run/systemd/netif:/bin/ false systemd-resolve:x:102:105:systemd Resolver,,,:/run/systemd/resolve:/bin/ false systemd-bus-proxy:x:103:106:systemd Bus Proxy,,,:/run/systemd:/bin/ false messagebus:x:104:108::/ var /run/dbus:/bin/ false jenkins:x:10000:10000:Jenkins user:/home/jenkins:/bin/sh $ ➜ ~ kubectl -n jenkins exec -it kubernetes-e36a3b0093244aa5a4c6ab7390c0f2dd-1388857a00d2 --container gradle -- /bin/sh /home/jenkins $ ls -al total 24 drwxrwxrwx 5 root root 4096 Apr 18 14:10 . drwxr-xr-x 1 root root 4096 Apr 18 14:09 .. -rw------- 1 gradle gradle 412 Apr 18 14:10 .ash_history drwxr-xr-x 2 root root 4096 Apr 18 14:09 .jenkins drwx------ 2 10000 10000 4096 Apr 18 14:09 .ssh drwxr-xr-x 4 10000 10000 4096 Apr 18 14:10 workspace /home/jenkins $ cd workspace/ /home/jenkins/workspace $ touch test touch: test: Permission denied /home/jenkins/workspace $ whoami gradle /home/jenkins/workspace $ cat /etc/passwd root:x:0:0:root:/root:/bin/ash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/ var /adm:/sbin/nologin lp:x:4:7:lp:/ var /spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/ var /spool/mail:/sbin/nologin news:x:9:13:news:/usr/lib/news:/sbin/nologin uucp:x:10:14:uucp:/ var /spool/uucppublic:/sbin/nologin operator :x:11:0: operator :/root:/bin/sh man:x:13:15:man:/usr/man:/sbin/nologin postmaster:x:14:12:postmaster:/ var /spool/mail:/sbin/nologin cron:x:16:16:cron:/ var /spool/cron:/sbin/nologin ftp:x:21:21::/ var /lib/ftp:/sbin/nologin sshd:x:22:22:sshd:/dev/ null :/sbin/nologin at:x:25:25:at:/ var /spool/cron/atjobs:/sbin/nologin squid:x:31:31:Squid:/ var /cache/squid:/sbin/nologin xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin games:x:35:35:games:/usr/games:/sbin/nologin postgres:x:70:70::/ var /lib/postgresql:/bin/sh cyrus:x:85:12::/usr/cyrus:/sbin/nologin vpopmail:x:89:89::/ var /vpopmail:/sbin/nologin ntp:x:123:123:NTP:/ var /empty:/sbin/nologin smmsp:x:209:209:smmsp:/ var /spool/mqueue:/sbin/nologin guest:x:405:100:guest:/dev/ null :/sbin/nologin nobody:x:65534:65534:nobody:/:/sbin/nologin gradle:x:1000:1000:Linux User,,,:/home/gradle:/bin/ash /home/jenkins/workspace $ %

ASHOK MOHANTY added a comment - 2019-07-10 12:49

I am getting same permission issue (but its in-frequent) when using different user with uid:1000. Will it help by changing to uid:10000 !! I have Jenkins - 2.121.3 and Kubernetes Plugin - 1.12.7 .

ASHOK MOHANTY added a comment - 2019-07-10 12:49 I am getting same permission issue (but its in-frequent) when using different user with uid:1000. Will it help by changing to uid:10000 !! I have Jenkins - 2.121.3 and Kubernetes Plugin - 1.12.7 .

Carlos Sanchez added a comment - 2020-03-31 09:22

please do not ask questions in closed issues. Use the jenkins-user mailing list

Carlos Sanchez added a comment - 2020-03-31 09:22 please do not ask questions in closed issues. Use the jenkins-user mailing list

Assignee:: Carlos Sanchez

Reporter:: Manjunath Kumatagi

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: 2017-01-25 10:58

Updated:: 2020-04-01 05:03

Resolved:: 2017-03-06 04:54

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Sergey Kulanov added a comment - 2017-01-26 13:08

Expand comment: Sergey Kulanov added a comment - 2017-01-26 13:08

Collapse comment: Manjunath Kumatagi added a comment - 2017-01-27 07:02

Expand comment: Manjunath Kumatagi added a comment - 2017-01-27 07:02

Collapse comment: Sergey Kulanov added a comment - 2017-01-27 10:06

Expand comment: Sergey Kulanov added a comment - 2017-01-27 10:06

Collapse comment: Carlos Sanchez added a comment - 2017-01-29 17:14

Expand comment: Carlos Sanchez added a comment - 2017-01-29 17:14

Collapse comment: Manjunath Kumatagi added a comment - 2017-02-01 06:28, Edited by Manjunath Kumatagi - 2017-02-01 06:30

Expand comment: Manjunath Kumatagi added a comment - 2017-02-01 06:28, Edited by Manjunath Kumatagi - 2017-02-01 06:30

Collapse comment: Carlos Sanchez added a comment - 2017-02-12 10:58

Expand comment: Carlos Sanchez added a comment - 2017-02-12 10:58

Collapse comment: Manjunath Kumatagi added a comment - 2017-03-06 04:54

Expand comment: Manjunath Kumatagi added a comment - 2017-03-06 04:54

Collapse comment: Gytis Ramanauskas added a comment - 2017-04-20 13:47

Expand comment: Gytis Ramanauskas added a comment - 2017-04-20 13:47

Collapse comment: ASHOK MOHANTY added a comment - 2019-07-10 12:49

Expand comment: ASHOK MOHANTY added a comment - 2019-07-10 12:49

Collapse comment: Carlos Sanchez added a comment - 2020-03-31 09:22

Expand comment: Carlos Sanchez added a comment - 2020-03-31 09:22

People

Dates