Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39867

Message: Unauthorized from OpenShift API after some time

XMLWordPrintable

      Somehow, after undeterminated time, the kubernetes-plugin from Jenkins can't connect to the Openshift server with this error in the Jenkins log:

      Nov 15, 2016 11:44:13 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Nov 15, 2016 11:44:13 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://openshift.server/api/v1/namespaces/jenkins/pods?labelSelector=jenkins%3Dslave. Message: Unauthorized
.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:310)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:263)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:232)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:416)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:58)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:588)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:463)
	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

      And clicking the "Test Connection" in the plugin interface replies a "Connection successful" but generates the following in the jenkins log:

      Nov 15, 2016 1:01:16 PM WARNING io.fabric8.kubernetes.client.Config tryServiceAccount
Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
Nov 15, 2016 1:01:16 PM WARNING org.apache.http.client.protocol.ResponseProcessCookies processCookies
Invalid cookie header: "Set-Cookie: ssn=MTQ3OTIxMTI3NnxKWjlkclhMOTJhNlU2MFZXdWVhdkdkQTBiVlloT3lMLWZDQkpERE4wdThsVGNkd2h6LVVmTzJvWWw0YUUyWlBqNEJHRlg3cHdIUzJVM2Q3SC1VTFZDY1BMNzVMUXJwOUk5QlJ6blBlQkhnMEZRMFEyZW5pdkxjcEdBbzdwd0dBQ2ZnPT1830DMNv2oQAZm6jBMgQ7RiFXwKY6UAJ7OBXixgk4kPoQ=; Path=/; Expires=Tue, 15 Nov 2016 13:01:16 GMT; Max-Age=3600; HttpOnly; Secure". Invalid 'expires' attribute: Tue, 15 Nov 2016 13:01:16 GMT

      Restarting the Jenkins server this problem has fixed until it happen again after some time.

          [JENKINS-39867] Message: Unauthorized from OpenShift API after some time

          Albert V created issue -
          Carlos Sanchez made changes -
          Summary Original: Message: Unauthorized New: Message: Unauthorized from OpenShift API after some time

          Carlos Sanchez added a comment -

          how are you authenticating to openshift? using a service account, user/password, certs,...?

          Carlos Sanchez added a comment - how are you authenticating to openshift? using a service account, user/password, certs,...?

          Albert V added a comment -

          A user/password.
          And even with this error we can still logging in to Openshift with this credentials.

          Albert V added a comment - A user/password. And even with this error we can still logging in to Openshift with this credentials.

          Carlos Sanchez added a comment -

          Maybe iocanel or jstrachan have an idea

          Carlos Sanchez added a comment - Maybe iocanel or jstrachan have an idea

          Ioannis Canellos added a comment -

          The jenkins pod needs to use a service account and that service account needs to be allowed to list, create, delete pods.

          For example the following command `oc adm policy who-can create pods` needs to return a list that contains the service account assigned to jenkins.

          This might explain things better than me:
          https://docs.openshift.com/enterprise/3.0/admin_guide/manage_authorization_policy.html

          Ioannis Canellos added a comment - The jenkins pod needs to use a service account and that service account needs to be allowed to list, create, delete pods. For example the following command `oc adm policy who-can create pods` needs to return a list that contains the service account assigned to jenkins. This might explain things better than me: https://docs.openshift.com/enterprise/3.0/admin_guide/manage_authorization_policy.html

          Ioannis Canellos added a comment -

          My comment above applies to setups where jenkins is deployed as pod. From the description above I understand that this is not the case and Jenkins lives outside of openshift, right?

          If this is the case, you may be hitting a bug in the openshift client, which was fixed recently: https://github.com/fabric8io/kubernetes-client/commit/97990ab51fe8d8bc307b7494098f0ebe3a3422d6

          So, we need to update to a more recent version.

          Ioannis Canellos added a comment - My comment above applies to setups where jenkins is deployed as pod. From the description above I understand that this is not the case and Jenkins lives outside of openshift, right? If this is the case, you may be hitting a bug in the openshift client, which was fixed recently: https://github.com/fabric8io/kubernetes-client/commit/97990ab51fe8d8bc307b7494098f0ebe3a3422d6 So, we need to update to a more recent version.

          Albert V added a comment -

          iocanel You're right. I've an external Jenkins installation.
          Great news if this bug was fixed!
          So csanchez, the next version of Kubernetes-plugin when is expected to be released?

          Albert V added a comment - iocanel You're right. I've an external Jenkins installation. Great news if this bug was fixed! So csanchez , the next version of Kubernetes-plugin when is expected to be released?

          agustin burguete added a comment -

          I do not know if it is related to this but I am getting an error like the one described in here: https://github.com/fabric8io/kubernetes-client/issues/558

          I am using Jenkins 2.19.3 and kubernetes-plugin 0.9

          If the pod template I use does NOT have volumes, everything goes fine. But if in the same template, I add an NFS volume is when I get this error:

          SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-template-12849497fadef4, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@4687fe6a
          io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.49.160.125:8443/api/v1/namespaces/jenkins-slaves/pods. Message: Forbidden! User admin/10-49-160-125:8443 doesn't have permission..

          agustin burguete added a comment - I do not know if it is related to this but I am getting an error like the one described in here: https://github.com/fabric8io/kubernetes-client/issues/558 I am using Jenkins 2.19.3 and kubernetes-plugin 0.9 If the pod template I use does NOT have volumes, everything goes fine. But if in the same template, I add an NFS volume is when I get this error: SEVERE: Error in provisioning; slave=KubernetesSlave name: jenkins-slave-template-12849497fadef4, template=org.csanchez.jenkins.plugins.kubernetes.PodTemplate@4687fe6a io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.49.160.125:8443/api/v1/namespaces/jenkins-slaves/pods . Message: Forbidden! User admin/10-49-160-125:8443 doesn't have permission..

          Albert V added a comment -

          agustinbs:

          • Which is the default user-id and group-id for the NFS?
          • Which is the user-id and group-id of your jenkins user?
          • Which directories do you have in the NFS?

          I think that this problem is not related with the plugin. Anyway it would be better if you open a new issue for your problem and we will try to help you

          Albert V added a comment - agustinbs : Which is the default user-id and group-id for the NFS? Which is the user-id and group-id of your jenkins user? Which directories do you have in the NFS? I think that this problem is not related with the plugin. Anyway it would be better if you open a new issue for your problem and we will try to help you

            csanchez Carlos Sanchez
            m4x1m0v3r Albert V
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: