Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49660

Unable to configure plugin using ibm k8s

XMLWordPrintable

      I was able to configure the plugin to use my local k8s cluster however I can't get it working using my ibm k8s instance. The difference seems to be that ibm is using auth tokens and my local instance is using cert+key. See attached

      io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://127.0.0.1:1234/api/v1/namespaces/jenkins/pods. Message:
 Forbidden! User username@company.com doesn't have permission. pods is forbidden: User "system:anonymous" cannot list pods in the namespace "jenki
ns".
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:407)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:327)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:605)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:452)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:405)
    at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
    at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

       

        1. config.cert
          0.4 kB
        2. config.token
          0.5 kB
        3. config.token.fixed
          0.4 kB

          [JENKINS-49660] Unable to configure plugin using ibm k8s

          Steve Szabo added a comment -

          I should mention that using config.token I'm able to list pods using kubectl so the role should have the correct perms:

          sszabo@xxxxxxxxxxx [/home/sszabo] $ kubectl get pods
No resources found.

          Steve Szabo added a comment - I should mention that using config.token I'm able to list pods using kubectl so the role should have the correct perms: sszabo@xxxxxxxxxxx [/home/sszabo] $ kubectl get pods No resources found.

          Carlos Sanchez added a comment -

          if you run the jenkins master in k8s you should just run it using serviceAccount for authentication

          Carlos Sanchez added a comment - if you run the jenkins master in k8s you should just run it using serviceAccount for authentication

          Steve Szabo added a comment -

          I'm not running the master in k8s however I was able to get it working with my local cluster by pointing the service endpoints to the master.

          I don't think this is the issue under ibm as it gives a permission error when I test the connection from jenkins config page.

          How are you authenticating from the master to k8s - you seem to be using the kubectl config.

          ---
kind: Endpoints
apiVersion: v1
metadata:
  name: jenkins-master
  namespace: jenkins
  labels:
    oiqclient: jenkins
subsets:
- addresses:
  - ip: 192.168.0.100
  ports:
  - port: 8080
    name: jenkins-master
- addresses:
  - ip: 192.168.0.100
  ports:
  - port: 50000
    name: jenkins-jnlp

---
kind: Service
apiVersion: v1
metadata:
  name: jenkins-master
  namespace: jenkins
  labels:
    oiqclient: jenkins
spec:
  ports:
  - name: jenkins-master
    protocol: TCP
    port: 8080
    targetPort: 8080
    nodePort: 0
  - name: jenkins-jnlp
    protocol: TCP
    port: 50000
    targetPort: 50000
    nodePort: 0

          Steve Szabo added a comment - I'm not running the master in k8s however I was able to get it working with my local cluster by pointing the service endpoints to the master. I don't think this is the issue under ibm as it gives a permission error when I test the connection from jenkins config page. How are you authenticating from the master to k8s - you seem to be using the kubectl config. --- kind: Endpoints apiVersion: v1 metadata: name: jenkins-master namespace: jenkins labels: oiqclient: jenkins subsets: - addresses: - ip: 192.168.0.100 ports: - port: 8080 name: jenkins-master - addresses: - ip: 192.168.0.100 ports: - port: 50000 name: jenkins-jnlp --- kind: Service apiVersion: v1 metadata: name: jenkins-master namespace: jenkins labels: oiqclient: jenkins spec: ports: - name: jenkins-master protocol: TCP port: 8080 targetPort: 8080 nodePort: 0 - name: jenkins-jnlp protocol: TCP port: 50000 targetPort: 50000 nodePort: 0

          Steve Szabo added a comment -

          Using the token from config.token I'm able to list pods using kubectl and curl:

          sszabo@xxxxxxxxxxx [/home/sszabo/tmp] $ curl -s --cacert $KUBE_CERT -X GET -H "$KUBE_HEADER" $KUBE_MASTER/api/v1/namespaces/$KUBE_NAMESPACE/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/jenkins/pods",
    "resourceVersion": "882144"
  },
  "items": []
}

          Steve Szabo added a comment - Using the token from config.token I'm able to list pods using kubectl and curl: sszabo@xxxxxxxxxxx [/home/sszabo/tmp] $ curl -s --cacert $KUBE_CERT -X GET -H "$KUBE_HEADER" $KUBE_MASTER/api/v1/namespaces/$KUBE_NAMESPACE/pods { "kind" : "PodList" , "apiVersion" : "v1" , "metadata" : { "selfLink" : "/api/v1/namespaces/jenkins/pods" , "resourceVersion" : "882144" }, "items" : [] }

          Steve Szabo added a comment -

          Have you tested the plugin using a token?

          token

          Steve Szabo added a comment - Have you tested the plugin using a token? token

          Steve Szabo added a comment -

          Using the basic token format works:

          users:
- name: username@company.com
  user:
    token: AAA

          This version does not work:

          users:
- name: username@company.com
  user:
    auth-provider:
      name: oidc
      config:
        client-id: bx
        client-secret: bx
        id-token: AAA
        idp-issuer-url: https://iam.ng.bluemix.net/kubernetes
        refresh-token: BBB

          Steve Szabo added a comment - Using the basic token format works: users: - name: username@company.com user: token: AAA This version does not work: users: - name: username@company.com user: auth-provider: name: oidc config: client-id: bx client-secret: bx id-token: AAA idp-issuer-url: https: //iam.ng.bluemix.net/kubernetes refresh-token: BBB

          Carlos Sanchez added a comment -

          typically you would create a ServiceAccount in k8s and use its token for authentication.

          oidc could work if you use refresh-token as token but it has an expiration time

          Carlos Sanchez added a comment - typically you would create a ServiceAccount in k8s and use its token for authentication. oidc could work if you use refresh-token as token but it has an expiration time

          Steve Szabo added a comment -

          I plan to create a service account to manage the namespace from jenkins.

          Thanks for your help on this one - cheers

          Steve Szabo added a comment - I plan to create a service account to manage the namespace from jenkins. Thanks for your help on this one - cheers

            csanchez Carlos Sanchez
            sszabo Steve Szabo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: