Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70639

Error creating Kubernetes Pods after update

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • kubernetes-client-api-plugin, kubernetes-plugin
    • None
    • JENKINS_VERSION 2.375
      Kubernetes version 1.22 (GCloud)
       Kubernetes Client API Plugin Version 6.4.1-208.vfe09a_9362c2c
       Kubernetes plugin Version 3872.v760b_4a_6c126b_

      When creating a Kubernetes Pod in a pipeline using the following YAML:

      apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    node-role: jenkins-worker
  tolerations:
  - key: "node-role"
    operator: "Equal"
    value: "jenkins-worker"
    effect: "NoExecute"
  containers:
  - name: jnlp
    image: <redacted>
    resources:
        requests:
            cpu: 300m
            memory: 1200Mi
  - name: mongodb
    image:  mongo:4.2.15-bionic
    resources:
        requests:
            cpu: 100m
            memory: 200Mi
    ports:
    - containerPort: 27017
    env:
    - name: MONGO_INITDB_ROOT_USERNAME
      value: "tester"
    - name: MONGO_INITDB_ROOT_PASSWORD
      value: "tester"
  - name: rabbitmq
    image:  rabbitmq:3.9.7-alpine
    resources:
        requests:
            cpu: 100m
            memory: 200Mi
    ports:
    - containerPort: 5672
  - name: dind
    image: docker:20.10.8-dind
    command: ["dockerd", "--host=tcp://0.0.0.0:2375"]
    ports:
    - containerPort: 2375
    - containerPort: 2376
    resources:
        requests:
            cpu: 300m
            memory: 600Mi
    securityContext:
        privileged: true

      the following exception appears:

      hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: name not specified for an operation requiring one.
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:182)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:142)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:93)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.parseFromYaml(PodTemplateUtils.java:611)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.validateYamlContainerNames(PodTemplateUtils.java:640)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.validateYamlContainerNames(PodTemplateUtils.java:630)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution.start(PodTemplateStepExecution.java:145)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
Caused: hudson.remoting.ProxyException: java.lang.RuntimeException: Failed to parse yaml: "
                apiVersion: v1
                kind: Pod
                spec:
                  nodeSelector:
                    node-role: jenkins-worker
                  tolerations:
                  - key: "node-role"
                    operator: "Equal"
                    value: "jenkins-worker"
                    effect: "NoExecute"
                  containers:
                  - name: jnlp
                    image: <redacted>
                    resources:
                        requests:
                            cpu: 300m
                            memory: 1200Mi
                  - name: mongodb
                    image:  mongo:4.2.15-bionic
                    resources:
                        requests:
                            cpu: 100m
                            memory: 200Mi
                    ports:
                    - containerPort: 27017
                    env:
                    - name: MONGO_INITDB_ROOT_USERNAME
                      value: "tester"
                    - name: MONGO_INITDB_ROOT_PASSWORD
                      value: "tester"
                  - name: rabbitmq
                    image:  rabbitmq:3.9.7-alpine
                    resources:
                        requests:
                            cpu: 100m
                            memory: 200Mi
                    ports:
                    - containerPort: 5672
                  - name: dind
                    image: docker:20.10.8-dind
                    command: ["dockerd", "--host=tcp://0.0.0.0:2375"]
                    ports:
                    - containerPort: 2375
                    - containerPort: 2376
                    resources:
                        requests:
                            cpu: 300m
                            memory: 600Mi
                    securityContext:
                        privileged: true
            "
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.parseFromYaml(PodTemplateUtils.java:613)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.validateYamlContainerNames(PodTemplateUtils.java:640)
	at org.csanchez.jenkins.plugins.kubernetes.PodTemplateUtils.validateYamlContainerNames(PodTemplateUtils.java:630)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution.start(PodTemplateStepExecution.java:145)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.KubernetesDeclarativeAgentScript.run(KubernetesDeclarativeAgentScript.groovy:51)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:76)
	at WorkflowScript.run(WorkflowScript:48)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
	at jdk.internal.reflect.GeneratedMethodAccessor373.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

      This worked on a previous plugin version (uncertain which one), and only started appearing after a recent update.

          [JENKINS-70639] Error creating Kubernetes Pods after update

          Peter Simms added a comment -

          Same issue here with new plugin update

          Peter Simms added a comment - Same issue here with new plugin update

          Allan added a comment -

          Same issue here, but with simpler Yaml:

          spec:
            tolerations:
            - key: "nodeType"
              operator: "Equal"
              value: "buildagent"
              effect: "NoExecute"

           

          on GKE 1.21.14 (although I don't think this matters as it's not getting that far)

          Allan added a comment - Same issue here, but with simpler Yaml: spec:   tolerations:   - key: "nodeType"     operator: "Equal"     value: "buildagent"     effect: "NoExecute"   on GKE 1.21.14 (although I don't think this matters as it's not getting that far)

          Joerg Schwaerzler added a comment -

          Same issue for us, even after downgrading the kubernetes plugin back to 3845.va_9823979a_744

          • Jenkins 2.375.3
          • Kubernetes 3872.v760b_4a_6c126b_ & 3845.va_9823979a_744
          • Kubernetes Client API Plugin 6.4.1-208.vfe09a_9362c2c
          • Kubernetes Credentials Plugin 0.10.0

          Joerg Schwaerzler added a comment - Same issue for us, even after downgrading the kubernetes plugin back to 3845.va_9823979a_744 Jenkins 2.375.3 Kubernetes 3872.v760b_4a_6c126b_ & 3845.va_9823979a_744 Kubernetes Client API Plugin 6.4.1-208.vfe09a_9362c2c Kubernetes Credentials Plugin 0.10.0

          Peter Simms added a comment - - edited

          macdrega
          Downgrade the client api plugin to 6.3.1-206.v76d3b_6b_14db_b

          This fixed the creation of pods for me

          Peter Simms added a comment - - edited macdrega Downgrade the client api plugin to 6.3.1-206.v76d3b_6b_14db_b This fixed the creation of pods for me

          Joerg Schwaerzler added a comment -

          Already in progress. Thanks, good to know that this will fix it

          Joerg Schwaerzler added a comment - Already in progress. Thanks, good to know that this will fix it

          Sebastian added a comment -

          How did you downgrade the version? The GUI option for the rollback is grayed out. Are there dependencies that must be downgraded first?

          Sebastian added a comment - How did you downgrade the version? The GUI option for the rollback is grayed out. Are there dependencies that must be downgraded first?

          Joerg Schwaerzler added a comment -

          not sure. I just downloaded the latest version and uploaded it manually.
          While the above error is gone, the JNLP container can now no longer connect to Jenkins. Not sure whether this issue is related to the kubernetes Plugin or some other plugin update we did

          Joerg Schwaerzler added a comment - not sure. I just downloaded the latest version and uploaded it manually. While the above error is gone, the JNLP container can now no longer connect to Jenkins. Not sure whether this issue is related to the kubernetes Plugin or some other plugin update we did

          Morten Birkelund added a comment -

          What am I missing? When trying to downgrade the Kubernetes Client API plugin, it shows that 6.3.1-206.v76d3b_6b_14db_b  is incompatible. 

          kubernetes-credentials:0.10.0

          kubernetes-cli:1.11.0

          kubernetes:3802.vb_b_600831fcb_3

          kubernetes-credentials:0.10.0

          Is there an easy way to see what it is incompatible with?

          Morten Birkelund added a comment - What am I missing? When trying to downgrade the Kubernetes Client API plugin, it shows that 6.3.1-206.v76d3b_6b_14db_b  is incompatible.  kubernetes-credentials:0.10.0 kubernetes-cli:1.11.0 kubernetes:3802.vb_b_600831fcb_3 kubernetes-credentials:0.10.0 Is there an easy way to see what it is incompatible with?

          charly ghislain added a comment - - edited

          i confirm downgrading works around the issue. I used the following values in the helm chart:

          # Locked versions to work around https://issues.jenkins.io/browse/JENKINS-70639
          - kubernetes-client-api:6.3.1-206.v76d3b_6b_14db_b
          - kubernetes:3845.va_9823979a_744

          charly ghislain added a comment - - edited i confirm downgrading works around the issue. I used the following values in the helm chart: # Locked versions to work around https://issues.jenkins.io/browse/JENKINS-70639 - kubernetes-client-api:6.3.1-206.v76d3b_6b_14db_b - kubernetes:3845.va_9823979a_744

          Joerg Schwaerzler added a comment -

          cghislai I can confirm that your combination works for me, too:

          • kubernetes-client-api:6.3.1-206.v76d3b_6b_14db_b
          • kubernetes:3845.va_9823979a_744

          However it did not in the first place. I did the following:

          1. Upgrade both to latest: Got the error described in this ticket
          2. Downgraded the kubernetes plugin to 3845.va_9823979a_744: Still same error
          3. Downgraded the kubernetes-client-api to 6.3.1-206.v76d3b_6b_14db_b: Now the kubernetes nodes could no longer connect to Jenkins
          4. Upgraded kubernetes plugin back to 3872.v760b_4a_6c126b_: No change
          5. Again downgraded the kubernetes plugin to 3845.va_9823979a_744. Back working, finally

          Joerg Schwaerzler added a comment - cghislai I can confirm that your combination works for me, too: kubernetes-client-api:6.3.1-206.v76d3b_6b_14db_b kubernetes:3845.va_9823979a_744 However it did not in the first place. I did the following: 1. Upgrade both to latest: Got the error described in this ticket 2. Downgraded the kubernetes plugin to 3845.va_9823979a_744: Still same error 3. Downgraded the kubernetes-client-api to 6.3.1-206.v76d3b_6b_14db_b: Now the kubernetes nodes could no longer connect to Jenkins 4. Upgraded kubernetes plugin back to 3872.v760b_4a_6c126b_: No change 5. Again downgraded the kubernetes plugin to 3845.va_9823979a_744. Back working, finally

            kylecronin Kyle Cronin
            ngtone Sebastian
            Votes:
            4 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: