Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70450

Kuberntes plugin will use the ~/kube/config file if present instead of the provided credential

      We have a multi cloud configuration with multiple pod templates per "cloud".
      On Jenkins masters where there is a ~/.kube/config file in the home directory of the service account running Jenkins, we were seeing multiple spurious warnings and errors in the Jenkins log to the effect of:

      ... Message: Unauthorized! Token may have expired! Please log-in again. Unauthorized.

      Not even the creation of brand new tokens would allow consistent connections.

      To recreate this, set up multiple cloud configurations with different service account tokens for login.  verify that each cloud is working properly first.

      Then from the command line perform a kubectl login to any one of the clusters with the service account token.  If you then attempt to spin up agents in a cluster OTHER THAN the one you manually logged into from the command line, you will get a false positive on an expired token.. .specifically because some of the post authentication steps performed to maintain the build agent pods will use the default context (and credential) from the last kubectl login.

      I have not walked through all of the 3 plugins' source, but this description should make it fairly easy for the authors/contributors to identify and locate the source of the anomaly.

          [JENKINS-70450] Kuberntes plugin will use the ~/kube/config file if present instead of the provided credential

          Kyle Cronin added a comment -

          Moved to kubernetes-plugin component as the client-api is just providing the client library dependency. There is no source code in the client-api plugin.

          Kyle Cronin added a comment - Moved to kubernetes-plugin component as the client-api is just providing the client library dependency. There is no source code in the client-api plugin.

          Toby added a comment - - edited

          I am not sure if this is the same issue or not but we ran into service account problems after upgrading our plugins as well. We are running in two different AWS EKS clusters. The main Jenkins node in the "shared" account spins up worker pods to build our projects in the "dev" account. After upgrading plugins, we were getting intermittent "Unauthorized" errors that killed the build and orphaned the worker pods which had to be killed manually. The specific error was:

          Failed to start websocket connection: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://[our_aws_cluster].us-east-1.eks.amazonaws.com/api/v1/namespaces/jenkins-agents/pods/tdx-24604-2-vn1zm-3s8m3-dj066. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked.

          Sometimes the error was thrown immediately when trying to create the worker pod. Other times it would successfully kick off the first stage of the build but then it would always error out on the next step.

          Downgrading the Kubernetes plugins resolved the errors. The working setup is running these versions:

          kubernetes: 3743.v1fa_4c724c3b_7
          kubernetes-cli: 1.10.3
          kubernetes-client-api: 5.12.2-193.v26a_6078f65a_9
          kubernetes-credentials: 0.9.0

          Toby added a comment - - edited I am not sure if this is the same issue or not but we ran into service account problems after upgrading our plugins as well. We are running in two different AWS EKS clusters. The main Jenkins node in the "shared" account spins up worker pods to build our projects in the "dev" account. After upgrading plugins, we were getting intermittent "Unauthorized" errors that killed the build and orphaned the worker pods which had to be killed manually. The specific error was: Failed to start websocket connection: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://[our_aws_cluster].us-east-1.eks.amazonaws.com/api/v1/namespaces/jenkins-agents/pods/tdx-24604-2-vn1zm-3s8m3-dj066 . Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Sometimes the error was thrown immediately when trying to create the worker pod. Other times it would successfully kick off the first stage of the build but then it would always error out on the next step. Downgrading the Kubernetes plugins resolved the errors. The working setup is running these versions: kubernetes: 3743.v1fa_4c724c3b_7 kubernetes-cli: 1.10.3 kubernetes-client-api: 5.12.2-193.v26a_6078f65a_9 kubernetes-credentials: 0.9.0

          Dominykas added a comment -

          Dominykas added a comment - Related to/duplicate of https://issues.jenkins.io/browse/JENKINS-70416?

          Yes indeed, dupe of JENKINS-70416

          Vincent Latombe added a comment - Yes indeed, dupe of JENKINS-70416

            Unassigned Unassigned
            lfiorino Lou Fiorino
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: