how are you authenticating to openshift? using a service account, user/password, certs,...?

A user/password.
And even with this error we can still logging in to Openshift with this credentials.

Maybe iocanel or jstrachan have an idea

The jenkins pod needs to use a service account and that service account needs to be allowed to list, create, delete pods.

For example the following command `oc adm policy who-can create pods` needs to return a list that contains the service account assigned to jenkins.

This might explain things better than me:
https://docs.openshift.com/enterprise/3.0/admin_guide/manage_authorization_policy.html

My comment above applies to setups where jenkins is deployed as pod. From the description above I understand that this is not the case and Jenkins lives outside of openshift, right?

If this is the case, you may be hitting a bug in the openshift client, which was fixed recently: https://github.com/fabric8io/kubernetes-client/commit/97990ab51fe8d8bc307b7494098f0ebe3a3422d6

So, we need to update to a more recent version.

iocanel You're right. I've an external Jenkins installation.
Great news if this bug was fixed!
So csanchez, the next version of Kubernetes-plugin when is expected to be released?

I do not know if it is related to this but I am getting an error like the one described in here: https://github.com/fabric8io/kubernetes-client/issues/558

I am using Jenkins 2.19.3 and kubernetes-plugin 0.9

If the pod template I use does NOT have volumes, everything goes fine. But if in the same template, I add an NFS volume is when I get this error:

SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-template-12849497fadef4, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@4687fe6a
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.49.160.125:8443/api/v1/namespaces/jenkins-slaves/pods. Message: Forbidden! User admin/10-49-160-125:8443 doesn't have permission..

agustinbs:

• Which is the default user-id and group-id for the NFS?
• Which is the user-id and group-id of your jenkins user?
• Which directories do you have in the NFS?

I think that this problem is not related with the plugin. Anyway it would be better if you open a new issue for your problem and we will try to help you

agustinbs You need to make sure that the service account you are using has permission to use volumes.

Thank you for your comments, I am bound to make some mistakes since I do not have that much experience. Nevertheless, it is when using NFS volume or Hosted volume (aprox. name, cannot check it right now) that I get the error. With no volume or Empty dir volume, it works fine, the plugin does start-up a pod in Openshift. I am using a Vagrant all-in-one version of Openshift Origin as kubernetes host, and therefore I just use a system-admin login for all purposes, so I would not expect permission issues. I am replicating the setup described in here https://blog.openshift.com/jenkins-slaves-in-openshift-using-an-external-jenkins-environment/ where two ways are used to have jenkins slaves, one is Swarm plugin (preallocated slave pods selfdiscover the master) the other kubernetes-plugin (the master actively launches slaves on kubernetes, that would be within openshift in this case). For swarm plugin I found that v2.2 does have regression in terms of honoring the NO_PROXY env. var, which is important on my setup, so it works fine if I stay in 2.0. For kubernetes-plugin I am finding the problem already stated, it launches slaves OK only if they have no volumes or empty dir volumes. I am wildguessing it might have to do with some bug in kubernetes-client which is used for the plugin?, Since it is a jenkins for experimenting, I am using an 'admin' account in jenkins too. The pod slaves preinstantiated (the ones that connect to master jenkins via swarm) do also have the same NFS volume defined (via openshift deployment config in this case) and they work just fine. NFS directory has an nfsnobody owner and group, that volumes is accessed ok for the preallocated pods launched via openshift, and anyway the other option via 'host volume' does not work either (same error). NFS directories, Openshift origin and master jenkins all are located in the same machine. Master jenkins cohabits with Openshift Origin installation (so master jenkins is not a pod within Openshift, the slaves are meant to be deployed in Openshift). The whole point of the setup is to have all slaves share a persistent volume to keep there maven's repository, so that when maven 'downloads half of internet' upon a build, is kept there and all those jars do not disappear and have to be downloaded again for a next build. Slaves preallocated with swarm do fine on this, the ones with kubernetes-plugin... seemingly cannot have a persistent volume for now...Sorry for the lengthy comment! Your suggestions are very welcome. Should this be a separate issue?

k8s client is upgraded in https://github.com/jenkinsci/kubernetes-plugin/pull/101

Should be fixed in 0.10

Suddenly I'm experiencing the same issue again in the 0.10 version of Kubernetes Plugin:

Dec 29, 2016 11:58:55 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://myopenshift.url/api/v1/namespaces/jenkins/pods?labelSelector=jenkins%3Dslave. Message: Unauthorized
.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:315)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:268)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:237)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:506)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:62)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:651)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:513)
	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

iocanel Could be another Kubernetes (Openshift) Client issue?

I missed to re-open the issue. More info in this comment.

Any idea to how can we fix this issue? csanchez or iocanel??

I've used the term "suddenly" because after the plugin update everything had working well but without any reason (or the same one than causes this error in the 0.9 plugin version) this happened again. I could not find any pattern to reproduce this error.
The output that you asked is this:

$ oc adm policy who-can list pods -n jenkins
Namespace: jenkins
Verb:      list
Resource:  pods

Users:  myUser1
        myUser2
        system:admin
        system:serviceaccount:jenkins:deployer
        system:serviceaccount:management-infra:management-admin
        system:serviceaccount:openshift-infra:build-controller
        system:serviceaccount:openshift-infra:daemonset-controller
        system:serviceaccount:openshift-infra:deployment-controller
        system:serviceaccount:openshift-infra:deploymentconfig-controller
        system:serviceaccount:openshift-infra:endpoint-controller
        system:serviceaccount:openshift-infra:gc-controller
        system:serviceaccount:openshift-infra:hpa-controller
        system:serviceaccount:openshift-infra:job-controller
        system:serviceaccount:openshift-infra:namespace-controller
        system:serviceaccount:openshift-infra:pet-set-controller
        system:serviceaccount:openshift-infra:pv-attach-detach-controller
        system:serviceaccount:openshift-infra:pv-binder-controller
        system:serviceaccount:openshift-infra:pv-recycler-controller
        system:serviceaccount:openshift-infra:replicaset-controller
        system:serviceaccount:openshift-infra:replication-controller

Groups: GL_APP_OSHIFT_Admins
        system:cluster-admins
        system:cluster-readers
        system:masters
        system:nodes

Restarting the jenkins isntance it still repair this error.

Thank you for helping me with this

So I am assuming that you are either connection with myUser1, myUser2 or admin, right?

Could it be possible that each time jenkins starts, it authenticates but after 24 hours, the token expires and you start getting these errors until, you restart again?
If this is the case, then we need to make sure in the kubernetes-client that we handle properlly token expiration.

Yes, I'm using one to those myUser1 user.

It seems that could be a token expiration problem because (I can't tell you surely) this error happened once per day.

Is there any update of this? It is still happening even in the latest (0.11) version.

Thanks!

 have the same issue with 0.11. I have tracked it to the following in my case.

org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect() method is caching the client connection. If the credentials were not for a long lived service account , the token would expire after some time and you would get an authorization failure

 Below is snippet I modified so it would reset in one hour. 

private static final long CONNECTION_RESET = 60 * 60 * 1000;

private long savedTime = 0;

public KubernetesClient connect()
throws UnrecoverableKeyException, NoSuchAlgorithmException,
KeyStoreException, IOException, CertificateEncodingException {

LOGGER.log(Level.FINE, "Building connection to Kubernetes host "
+ name + " URL " + serverUrl);
final long curTime = System.currentTimeMillis();
if ((client == null)

}
}
}
return client;

I've fixed it setting a new credentials on Jenkins as Service Account Token instead of a user.
Then, this is not a defect of this plugin.

Thank you anyway for your help

I have the same issue when upgrading from 0.10 to 0.12.

Steps to reproduce:

• install Jenkins persistent in openshift
  in my case OpenShift Master: v1.5.1+7b451fc Kubernetes Master: v1.5.2+43a9be4 
• simple pipeline job run successfully:

node("maven") {
  echo "hello"
}

• update Kubernetes plugin from 0.10 to 0.12
  following plugins get updated as well:
  • Pipeline: Model API
  • Script Security Plugin
  • Pipeline: Supporting APIs
  • Pipeline: Step API
  • Pipeline: API
  • Pipeline: Groovy
  • Pipeline: Job
  • Pipeline: Declarative Extension Points API
  • Pipeline: Nodes and Processes
• simple pipeline job doesn't run anymore (just hanging)
  warnings appear in the log:

WARNING: Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/api/v1/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:mg-test:jenkins" cannot list all pods in the cluster.

The job runs successfully when downgrading the Kubernetes plugin to 0.10 (and this is the only plugin that gets downgraded).

Fixed in https://github.com/jenkinsci/kubernetes-plugin/pull/189 removing the caching of the kubernetes client