Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70436

Kubernetes plugin fails to read kubecfg from yaml

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None
    • jenkins 2.375.2
      kubernetes 3802.vb_b_600831fcb_3

      The Kubernetes plugin fails to read the cluster configuration from the kubecfg yaml file provided as Jenkins secret through JCasC.

      manage/configureClouds -> Kubernetes clud details -> test connection fails with

      {{Error testing connection : com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: special characters are not allowed
      at [Source: (StringReader); line: 1, column: 1]}}

      Jenkins log is full of similar messages, stating that there is a 0x7 in the yaml .... I can't hear any bells though ;-P

      The combination of Jenkins 2.375.1 and Kubernetes plugin 3734.v562b_b_a_627ea_c works fine with the same config file.

          [JENKINS-70436] Kubernetes plugin fails to read kubecfg from yaml

          Georg added a comment -

          I just tried the combination of

          • Jenkins 2.375.2
          • Kubernetes plugin 3734.v562b_b_a_627ea_c

          The same problem occurs. So it might be a problem of Jenkins itself after all.

          Georg added a comment - I just tried the combination of Jenkins 2.375.2 Kubernetes plugin 3734.v562b_b_a_627ea_c The same problem occurs. So it might be a problem of Jenkins itself after all.

          Georg added a comment - - edited

          Update:
          After rolling back my complete set of kubernetes* plugins, it also works with Jenkins 2.375.2. In especially:

          • kubernetes:3734.v562b_b_a_627ea_c
          • kubernetes-credentials:0.9.0
          • kubernetes-client-api:5.12.2-193.v26a_6078f65a_9

          Georg added a comment - - edited Update: After rolling back my complete set of kubernetes* plugins, it also works with Jenkins 2.375.2. In especially: kubernetes:3734.v562b_b_a_627ea_c kubernetes-credentials:0.9.0 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9

          Mauricio added a comment -

          Obviously current jenkins 2.375 is messing up with the kube configurations. We had connection issues from controller to kubernetes-agents because of having both a single kube config file in .kube dir (test) and a single cloud configuration in the ui (prod). Having both configurations in place we had ongoing connection issues saying

          "Unauthorized! Token may have expired! Please log-in again. Unauthorized 401: must authenticate."

          After having removed the test configurartion config file from .kube dir the error disappeared. When reactivating the config again (mv to config.bak and back zu config) the error reappeared. So this hints at a bug where jenkins is messing up with the configs and authentication tokens. 

          Mauricio added a comment - Obviously current jenkins 2.375 is messing up with the kube configurations. We had connection issues from controller to kubernetes-agents because of having both a single kube config file in .kube dir (test) and a single cloud configuration in the ui (prod). Having both configurations in place we had ongoing connection issues saying "Unauthorized! Token may have expired! Please log-in again. Unauthorized 401: must authenticate." After having removed the test configurartion config file from .kube dir the error disappeared. When reactivating the config again (mv to config.bak and back zu config) the error reappeared. So this hints at a bug where jenkins is messing up with the configs and authentication tokens. 

          Probably another occurrence of JENKINS-70416. Check again with latest version.

          Vincent Latombe added a comment - Probably another occurrence of JENKINS-70416 . Check again with latest version.

          Georg added a comment -

          I retried with Jenkins 2.387.2 and with reverted version pinning for the kubernetes plugin (i.e. only specifying kubernetes in plugins.txt).
          The exact same problem occurs :

          Error testing connection : com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: special characters are not allowed

          Georg added a comment - I retried with Jenkins 2.387.2 and with reverted version pinning for the kubernetes plugin (i.e. only specifying kubernetes in plugins.txt). The exact same problem occurs : Error testing connection : com.fasterxml.jackson.dataformat.yaml.JacksonYAMLParseException: special characters are not allowed

          Georg added a comment -

          My situation just got worse:

          • current Jenkins 2.414.3 no longer works with the pinned k8s plugin version 3734.v562b_b_a_627ea_c (breaking change in snake yaml)
          • current Jenkins 2.414.3 with current k8s plugin still shows the same error:
            Nov 21, 2023 11:54:49 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
            
            Error in provisioning; agent=KubernetesSlave name: atgrz-k8s-linux-jnlp-45dvb, template=PodTemplate{id='954105e0-8cc7-429a-971c-03563fe76b78', name='atgrz-k8s-linux-jnlp', label='atgrz-k8s-buildnode-linux generic-linux-build', volumes=[org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim@6080b897, org.csanchez.jenkins.plugins.kubernetes.volumes.ConfigMapVolume@1fc84d4f], containers=[ContainerTemplate{name='jnlp', image='env-docker-release.arti.mycompany.com/env/dev/linux_r7g8:jnlp', alwaysPullImage=true, command='', args='', resourceRequestCpu='1000m', resourceRequestMemory='2000Mi', resourceLimitCpu='4000m', resourceLimitMemory='6000Mi', envVars=[KeyValueEnvVar [getValue()=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/.local/bin, getKey()=PATH]]}], imagePullSecrets=[PodImagePullSecret{name='s19m13-env-docker-arti'}, PodImagePullSecret{name='s19m13-docker-arti'}]}
            unacceptable code point '' (0x7) special characters are not allowed
            in "reader", position 15
            	at org.snakeyaml.engine.v2.scanner.StreamReader.update(StreamReader.java:274)
            	at org.snakeyaml.engine.v2.scanner.StreamReader.ensureEnoughData(StreamReader.java:237)
            	at org.snakeyaml.engine.v2.scanner.StreamReader.ensureEnoughData(StreamReader.java:232)
            	at org.snakeyaml.engine.v2.scanner.StreamReader.peek(StreamReader.java:187)
            	at org.snakeyaml.engine.v2.scanner.ScannerImpl.scanToNextToken(ScannerImpl.java:1114)
            	at org.snakeyaml.engine.v2.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:292)
            	at org.snakeyaml.engine.v2.scanner.ScannerImpl.checkToken(ScannerImpl.java:192)
            	at org.snakeyaml.engine.v2.parser.ParserImpl$ParseImplicitDocumentStart.produce(ParserImpl.java:444)
            	at org.snakeyaml.engine.v2.parser.ParserImpl.lambda$produce$1(ParserImpl.java:232)
            	at java.base/java.util.Optional.ifPresent(Optional.java:183)
            	at org.snakeyaml.engine.v2.parser.ParserImpl.produce(ParserImpl.java:232)
            	at org.snakeyaml.engine.v2.parser.ParserImpl.peekEvent(ParserImpl.java:206)
            	at org.snakeyaml.engine.v2.parser.ParserImpl.checkEvent(ParserImpl.java:198)
            	at org.snakeyaml.engine.v2.composer.Composer.hasNext(Composer.java:110)
            	at org.snakeyaml.engine.v2.api.Load$YamlIterator.hasNext(Load.java:226)
            	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.parseYaml(KubernetesSerialization.java:271)
            	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:252)
            	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:342)
            	at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:327)
            	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:185)
            	at io.fabric8.kubernetes.client.internal.KubeConfigUtils.parseConfigFromString(KubeConfigUtils.java:47)
            	at io.fabric8.kubernetes.client.Config.loadFromKubeconfig(Config.java:706)
            	at io.fabric8.kubernetes.client.Config.fromKubeconfig(Config.java:619)
            	at io.fabric8.kubernetes.client.Config.fromKubeconfig(Config.java:609)
            	at org.jenkinsci.plugins.kubernetes.auth.impl.KubernetesAuthKubeconfig.decorate(KubernetesAuthKubeconfig.java:28)
            	at org.csanchez.jenkins.plugins.kubernetes.KubernetesFactoryAdapter.createClient(KubernetesFactoryAdapter.java:119)
            	at org.csanchez.jenkins.plugins.kubernetes.KubernetesClientProvider.createClient(KubernetesClientProvider.java:59)
            	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect(KubernetesCloud.java:536)
            	at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:118)
            	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
            	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
            	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            	at java.base/java.lang.Thread.run(Thread.java:829)
            

          I fear I am stuck on Jenkins 2.387.3 for the time being - Please help.

          Remark: I doubt that this is duplicate. Sorry for the inconvenience caused by repoening this.

          Georg added a comment - My situation just got worse: current Jenkins 2.414.3 no longer works with the pinned k8s plugin version 3734.v562b_b_a_627ea_c (breaking change in snake yaml) current Jenkins 2.414.3 with current k8s plugin still shows the same error: Nov 21, 2023 11:54:49 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch Error in provisioning; agent=KubernetesSlave name: atgrz-k8s-linux-jnlp-45dvb, template=PodTemplate{id= '954105e0-8cc7-429a-971c-03563fe76b78' , name= 'atgrz-k8s-linux-jnlp' , label= 'atgrz-k8s-buildnode-linux generic -linux-build' , volumes=[org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim@6080b897, org.csanchez.jenkins.plugins.kubernetes.volumes.ConfigMapVolume@1fc84d4f], containers=[ContainerTemplate{name= 'jnlp' , image= 'env-docker-release.arti.mycompany.com/env/dev/linux_r7g8:jnlp' , alwaysPullImage= true , command= '', args=' ', resourceRequestCpu=' 1000m ', resourceRequestMemory=' 2000Mi ', resourceLimitCpu=' 4000m ', resourceLimitMemory=' 6000Mi ', envVars=[KeyValueEnvVar [getValue()=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/.local/bin, getKey()=PATH]]}], imagePullSecrets=[PodImagePullSecret{name=' s19m13-env-docker-arti '}, PodImagePullSecret{name=' s19m13-docker-arti'}]} unacceptable code point '' (0x7) special characters are not allowed in "reader" , position 15 at org.snakeyaml.engine.v2.scanner.StreamReader.update(StreamReader.java:274) at org.snakeyaml.engine.v2.scanner.StreamReader.ensureEnoughData(StreamReader.java:237) at org.snakeyaml.engine.v2.scanner.StreamReader.ensureEnoughData(StreamReader.java:232) at org.snakeyaml.engine.v2.scanner.StreamReader.peek(StreamReader.java:187) at org.snakeyaml.engine.v2.scanner.ScannerImpl.scanToNextToken(ScannerImpl.java:1114) at org.snakeyaml.engine.v2.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:292) at org.snakeyaml.engine.v2.scanner.ScannerImpl.checkToken(ScannerImpl.java:192) at org.snakeyaml.engine.v2.parser.ParserImpl$ParseImplicitDocumentStart.produce(ParserImpl.java:444) at org.snakeyaml.engine.v2.parser.ParserImpl.lambda$produce$1(ParserImpl.java:232) at java.base/java.util.Optional.ifPresent(Optional.java:183) at org.snakeyaml.engine.v2.parser.ParserImpl.produce(ParserImpl.java:232) at org.snakeyaml.engine.v2.parser.ParserImpl.peekEvent(ParserImpl.java:206) at org.snakeyaml.engine.v2.parser.ParserImpl.checkEvent(ParserImpl.java:198) at org.snakeyaml.engine.v2.composer.Composer.hasNext(Composer.java:110) at org.snakeyaml.engine.v2.api.Load$YamlIterator.hasNext(Load.java:226) at io.fabric8.kubernetes.client.utils.KubernetesSerialization.parseYaml(KubernetesSerialization.java:271) at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:252) at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:342) at io.fabric8.kubernetes.client.utils.KubernetesSerialization.unmarshal(KubernetesSerialization.java:327) at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:185) at io.fabric8.kubernetes.client.internal.KubeConfigUtils.parseConfigFromString(KubeConfigUtils.java:47) at io.fabric8.kubernetes.client.Config.loadFromKubeconfig(Config.java:706) at io.fabric8.kubernetes.client.Config.fromKubeconfig(Config.java:619) at io.fabric8.kubernetes.client.Config.fromKubeconfig(Config.java:609) at org.jenkinsci.plugins.kubernetes.auth.impl.KubernetesAuthKubeconfig.decorate(KubernetesAuthKubeconfig.java:28) at org.csanchez.jenkins.plugins.kubernetes.KubernetesFactoryAdapter.createClient(KubernetesFactoryAdapter.java:119) at org.csanchez.jenkins.plugins.kubernetes.KubernetesClientProvider.createClient(KubernetesClientProvider.java:59) at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect(KubernetesCloud.java:536) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:118) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:829) I fear I am stuck on Jenkins 2.387.3 for the time being - Please help. Remark: I doubt that this is duplicate. Sorry for the inconvenience caused by repoening this.

          Can you check that your kubeconfig is valid for kubectl usage? 0x7 is a control character and is not expected to be present.

          Vincent Latombe added a comment - Can you check that your kubeconfig is valid for kubectl usage? 0x7 is a control character and is not expected to be present.

          Georg added a comment -

          Yes of course - I am also very curious where that bell should be coming from:

          • The file itself - mounted as secret to my running jenkins container - looks unsuspicious
          • All kubectl commands from terminal work : e.g. kubectl config current-context, kubectl version, kubectl get pods)
          • It works in Jenkins 2.387.3 with the pinned k8s plugin (e.g. "Test connection" as lined out in the description and in especially spinning up pods)

          Thanks for your prompt reaction!

          Georg added a comment - Yes of course - I am also very curious where that bell should be coming from: The file itself - mounted as secret to my running jenkins container - looks unsuspicious All kubectl commands from terminal work : e.g. kubectl config current-context , kubectl version , kubectl get pods ) It works in Jenkins 2.387.3 with the pinned k8s plugin (e.g. "Test connection" as lined out in the description and in especially spinning up pods) Thanks for your prompt reaction!

          Georg added a comment -

          Update:

          I had a closer look at the Jenkins system log when clicking this "Test Connection":

          • It reports back Connected to Kubernetes v1.24.9 successfully
          • but the log states the all too familiar:
            Nov 21, 2023 2:17:51 PM SEVERE io.fabric8.kubernetes.client.Config loadFromKubeconfig
            
            Failed to parse the kubeconfig.
            unacceptable code point '' (0x7) special characters are not allowed
            in "'reader'", position 15
            	at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:204)
            ...
            

          It seems that this behaviour has been there before - but not causing a fatal error to the plugin.

          Georg added a comment - Update: I had a closer look at the Jenkins system log when clicking this "Test Connection": It reports back Connected to Kubernetes v1.24.9 successfully but the log states the all too familiar: Nov 21, 2023 2:17:51 PM SEVERE io.fabric8.kubernetes.client.Config loadFromKubeconfig Failed to parse the kubeconfig. unacceptable code point '' (0x7) special characters are not allowed in " 'reader' " , position 15 at org.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:204) ... It seems that this behaviour has been there before - but not causing a fatal error to the plugin.

          Georg added a comment -

          The log message refer to position 15; The first line of kubecfg reads apiVersion: v1 which would make the 1st line break the 15th character.
          The base64 blob in the k8s config file defining the kubectl secret mounted into the running jenkins container starts with YXBpVmVyc2lvbjogdjEK - decoding shows that this 1st line break actually is a LF (0x0a).
          Still hunting for the bell

          Georg added a comment - The log message refer to position 15; The first line of kubecfg reads apiVersion: v1 which would make the 1st line break the 15th character. The base64 blob in the k8s config file defining the kubectl secret mounted into the running jenkins container starts with YXBpVmVyc2lvbjogdjEK - decoding shows that this 1st line break actually is a LF (0x0a). Still hunting for the bell

            vlatombe Vincent Latombe
            georg_salentinig Georg
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: