[JENKINS-39867] Message: Unauthorized from OpenShift API after some time - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: kubernetes-plugin
Labels:
- authentication
- jenkins
- jenkins2.3
- kubernetes
- openshift
- token
Environment:
External Jenkins:
Kubernetes-plugin v0.9
Jenkins ver. 2.30
----
Slave generator:
openshift v1.3.1
kubernetes v1.3.0+52492b4

Similar Issues:

Show

Description

Somehow, after undeterminated time, the kubernetes-plugin from Jenkins can't connect to the Openshift server with this error in the Jenkins log:

Nov 15, 2016 11:44:13 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Nov 15, 2016 11:44:13 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://openshift.server/api/v1/namespaces/jenkins/pods?labelSelector=jenkins%3Dslave. Message: Unauthorized
.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:310)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:263)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:232)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:416)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:58)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:588)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:463)
	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

And clicking the "Test Connection" in the plugin interface replies a "Connection successful" but generates the following in the jenkins log:

Nov 15, 2016 1:01:16 PM WARNING io.fabric8.kubernetes.client.Config tryServiceAccount
Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
Nov 15, 2016 1:01:16 PM WARNING org.apache.http.client.protocol.ResponseProcessCookies processCookies
Invalid cookie header: "Set-Cookie: ssn=MTQ3OTIxMTI3NnxKWjlkclhMOTJhNlU2MFZXdWVhdkdkQTBiVlloT3lMLWZDQkpERE4wdThsVGNkd2h6LVVmTzJvWWw0YUUyWlBqNEJHRlg3cHdIUzJVM2Q3SC1VTFZDY1BMNzVMUXJwOUk5QlJ6blBlQkhnMEZRMFEyZW5pdkxjcEdBbzdwd0dBQ2ZnPT1830DMNv2oQAZm6jBMgQ7RiFXwKY6UAJ7OBXixgk4kPoQ=; Path=/; Expires=Tue, 15 Nov 2016 13:01:16 GMT; Max-Age=3600; HttpOnly; Secure". Invalid 'expires' attribute: Tue, 15 Nov 2016 13:01:16 GMT

Restarting the Jenkins server this problem has fixed until it happen again after some time.

Attachments

Activity

Ascending order - Click to sort in descending order

View 19 older comments

Albert V added a comment - 2017-03-29 14:51

Is there any update of this? It is still happening even in the latest (0.11) version.

Thanks!

Albert V added a comment - 2017-03-29 14:51 Is there any update of this? It is still happening even in the latest (0.11) version. Thanks!

Lloyd Fernandes added a comment - 2017-05-03 12:12 - edited

have the same issue with 0.11. I have tracked it to the following in my case.

org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect() method is caching the client connection. If the credentials were not for a long lived service account , the token would expire after some time and you would get an authorization failure

Below is snippet I modified so it would reset in one hour.

private static final long CONNECTION_RESET = 60 * 60 * 1000;

private long savedTime = 0;

public KubernetesClient connect()
throws UnrecoverableKeyException, NoSuchAlgorithmException,
KeyStoreException, IOException, CertificateEncodingException {

LOGGER.log(Level.FINE, "Building connection to Kubernetes host "
+ name + " URL " + serverUrl);
final long curTime = System.currentTimeMillis();
if ((client == null)

(curTime > (savedTime + CONNECTION_RESET))) {
client = null;
savedTime = curTime;
synchronized (this) {
if (client == null) {
LOGGER.log(Level.INFO,
"Attempting: Building connection to Kubernetes host "
+ name + " URL " + serverUrl
+ " Namespace " + namespace);
client = new KubernetesFactoryAdapter(serverUrl,
namespace, serverCertificate, credentialsId,
skipTlsVerify, connectTimeout, readTimeout)
.createClient();
LOGGER.log(Level.INFO,
"Success: Building connection to Kubernetes host "
+ name + " URL " + serverUrl
+ " Namespace " + namespace);

}
}
}
return client;

Lloyd Fernandes added a comment - 2017-05-03 12:12 - edited have the same issue with 0.11. I have tracked it to the following in my case. org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect() method is caching the client connection. If the credentials were not for a long lived service account , the token would expire after some time and you would get an authorization failure Below is snippet I modified so it would reset in one hour. private static final long CONNECTION_RESET = 60 * 60 * 1000; private long savedTime = 0; public KubernetesClient connect() throws UnrecoverableKeyException, NoSuchAlgorithmException, KeyStoreException, IOException, CertificateEncodingException { LOGGER.log(Level.FINE, "Building connection to Kubernetes host " + name + " URL " + serverUrl); final long curTime = System.currentTimeMillis(); if ((client == null) (curTime > (savedTime + CONNECTION_RESET))) { client = null; savedTime = curTime; synchronized (this) { if (client == null) { LOGGER.log(Level.INFO, "Attempting: Building connection to Kubernetes host " + name + " URL " + serverUrl + " Namespace " + namespace); client = new KubernetesFactoryAdapter(serverUrl, namespace, serverCertificate, credentialsId, skipTlsVerify, connectTimeout, readTimeout) .createClient(); LOGGER.log(Level.INFO, "Success: Building connection to Kubernetes host " + name + " URL " + serverUrl + " Namespace " + namespace); } } } return client;

Albert V added a comment - 2017-06-22 15:57

I've fixed it setting a new credentials on Jenkins as Service Account Token instead of a user.
Then, this is not a defect of this plugin.

Thank you anyway for your help

Albert V added a comment - 2017-06-22 15:57 I've fixed it setting a new credentials on Jenkins as Service Account Token instead of a user. Then, this is not a defect of this plugin. Thank you anyway for your help

Marc Guillemot added a comment - 2017-08-04 07:59

I have the same issue when upgrading from 0.10 to 0.12.

Steps to reproduce:

install Jenkins persistent in openshift
in my case OpenShift Master: v1.5.1+7b451fc Kubernetes Master: v1.5.2+43a9be4
simple pipeline job run successfully:

node("maven") {
  echo "hello"
}

update Kubernetes plugin from 0.10 to 0.12
following plugins get updated as well:
- Pipeline: Model API
- Script Security Plugin
- Pipeline: Supporting APIs
- Pipeline: Step API
- Pipeline: API
- Pipeline: Groovy
- Pipeline: Job
- Pipeline: Declarative Extension Points API
- Pipeline: Nodes and Processes
simple pipeline job doesn't run anymore (just hanging)
warnings appear in the log:

WARNING: Failed to count the # of live instances on Kubernetes
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1/api/v1/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:mg-test:jenkins" cannot list all pods in the cluster.

The job runs successfully when downgrading the Kubernetes plugin to 0.10 (and this is the only plugin that gets downgraded).

Marc Guillemot added a comment - 2017-08-04 07:59 I have the same issue when upgrading from 0.10 to 0.12. Steps to reproduce: install Jenkins persistent in openshift in my case OpenShift Master: v1.5.1+7b451fc Kubernetes Master: v1.5.2+43a9be4 simple pipeline job run successfully: node( "maven" ) { echo "hello" } update Kubernetes plugin from 0.10 to 0.12 following plugins get updated as well: Pipeline: Model API Script Security Plugin Pipeline: Supporting APIs Pipeline: Step API Pipeline: API Pipeline: Groovy Pipeline: Job Pipeline: Declarative Extension Points API Pipeline: Nodes and Processes simple pipeline job doesn't run anymore (just hanging) warnings appear in the log: WARNING: Failed to count the # of live instances on Kubernetes io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https: //172.30.0.1/api/v1/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:mg-test:jenkins" cannot list all pods in the cluster. The job runs successfully when downgrading the Kubernetes plugin to 0.10 (and this is the only plugin that gets downgraded).

Carlos Sanchez added a comment - 2017-08-07 08:33 - edited

Fixed in https://github.com/jenkinsci/kubernetes-plugin/pull/189 removing the caching of the kubernetes client

Carlos Sanchez added a comment - 2017-08-07 08:33 - edited Fixed in https://github.com/jenkinsci/kubernetes-plugin/pull/189 removing the caching of the kubernetes client

People

Assignee:: Carlos Sanchez

Reporter:: Albert V

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2016-11-18 14:04

Updated:: 2019-07-12 22:15

Resolved:: 2017-08-07 08:33