Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72662

Connectivity issue with cloud agent post upgrading Jenkins to latest

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Dev

      Hi,

      After upgrade of Jenkins LTS 2.426.3 with helm I could observe that there's a timeout waiting for a Kubernetes Pod with the name 'jenkins-slave-j82rz' to be created in the 'jenkins' namespace. This Pod is part of a Jenkins agent launched using the Kubernetes plugin.

      Kubernetes plugin I am using: 4186.v1d804571d5d4
      EKS Cluster's K8s version: 1.21

      Please help me to fix this connectivity issue with k8s pods. Thanks.

      System log:
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
      Terminated Kubernetes instance for agent jenkins/jenkins-slave-lt9lz
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Disconnected computer jenkins-slave-lt9lz
      Feb 05, 2024 6:46:01 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor
      No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnPodFailed onEvent
      jenkins/jenkins-slave-lt9lz Pod just failed. Removing the corresponding Jenkins agent. Reason: null, Message: null
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-lt9lz
      Feb 05, 2024 6:46:01 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Computer for agent is null: jenkins-slave-lt9lz
      Feb 05, 2024 6:46:01 AM INFO hudson.slaves.AbstractCloudSlave terminate
      FATAL: Computer for agent is null: jenkins-slave-lt9lz
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnContainerTerminated lambda$onEvent$1
      jenkins/jenkins-slave-fvzrd Container jnlp was just terminated, so removing the corresponding Jenkins agent
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-fvzrd
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
      Terminated Kubernetes instance for agent jenkins/jenkins-slave-fvzrd
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Disconnected computer jenkins-slave-fvzrd
      Feb 05, 2024 6:46:01 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor
      No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnPodFailed onEvent
      jenkins/jenkins-slave-fvzrd Pod just failed. Removing the corresponding Jenkins agent. Reason: null, Message: null
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-fvzrd
      Feb 05, 2024 6:46:01 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Computer for agent is null: jenkins-slave-fvzrd
      Feb 05, 2024 6:46:01 AM INFO hudson.slaves.AbstractCloudSlave terminate
      FATAL: Computer for agent is null: jenkins-slave-fvzrd
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnContainerTerminated lambda$onEvent$1
      jenkins/jenkins-slave-b2zf3 Container jnlp was just terminated, so removing the corresponding Jenkins agent
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-b2zf3
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
      Terminated Kubernetes instance for agent jenkins/jenkins-slave-b2zf3
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Disconnected computer jenkins-slave-b2zf3
      Feb 05, 2024 6:46:01 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor
      No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnPodFailed onEvent
      jenkins/jenkins-slave-b2zf3 Pod just failed. Removing the corresponding Jenkins agent. Reason: null, Message: null
      Feb 05, 2024 6:46:01 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-b2zf3
      Feb 05, 2024 6:46:01 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Computer for agent is null: jenkins-slave-b2zf3
      Feb 05, 2024 6:46:01 AM INFO hudson.slaves.AbstractCloudSlave terminate
      FATAL: Computer for agent is null: jenkins-slave-b2zf3
      Feb 05, 2024 6:46:02 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnContainerTerminated lambda$onEvent$1
      jenkins/jenkins-slave-8dlxt Container jnlp was just terminated, so removing the corresponding Jenkins agent
      Feb 05, 2024 6:46:02 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-8dlxt
      Feb 05, 2024 6:46:02 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
      Terminated Kubernetes instance for agent jenkins/jenkins-slave-8dlxt
      Feb 05, 2024 6:46:02 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Disconnected computer jenkins-slave-8dlxt
      Feb 05, 2024 6:46:02 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor
      No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave
      Feb 05, 2024 6:46:02 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnPodFailed onEvent
      jenkins/jenkins-slave-8dlxt Pod just failed. Removing the corresponding Jenkins agent. Reason: null, Message: null
      Feb 05, 2024 6:46:02 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-8dlxt
      Feb 05, 2024 6:46:02 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Computer for agent is null: jenkins-slave-8dlxt
      Feb 05, 2024 6:46:02 AM INFO hudson.slaves.AbstractCloudSlave terminate
      FATAL: Computer for agent is null: jenkins-slave-8dlxt
      Feb 05, 2024 6:46:03 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnContainerTerminated lambda$onEvent$1
      jenkins/jenkins-slave-w6ch8 Container jnlp was just terminated, so removing the corresponding Jenkins agent
      Feb 05, 2024 6:46:03 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-w6ch8
      Feb 05, 2024 6:46:03 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
      Terminated Kubernetes instance for agent jenkins/jenkins-slave-w6ch8
      Feb 05, 2024 6:46:03 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Disconnected computer jenkins-slave-w6ch8
      Feb 05, 2024 6:46:03 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor
      No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave
      Feb 05, 2024 6:46:03 AM INFO org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$TerminateAgentOnPodFailed onEvent
      jenkins/jenkins-slave-w6ch8 Pod just failed. Removing the corresponding Jenkins agent. Reason: null, Message: null
      Feb 05, 2024 6:46:03 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Terminating Kubernetes instance for agent jenkins-slave-w6ch8
      Feb 05, 2024 6:46:03 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
      Computer for agent is null: jenkins-slave-w6ch8
      Feb 05, 2024 6:46:03 AM INFO hudson.slaves.AbstractCloudSlave terminate
      FATAL: Computer for agent is null: jenkins-slave-w6ch8
      Feb 05, 2024 6:46:09 AM INFO hudson.slaves.NodeProvisioner update
      jenkins-slave-mrl3x provisioning successfully completed. We have now 3 computer(s)
      Feb 05, 2024 6:46:09 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
      Error in provisioning; agent=KubernetesSlave name: jenkins-slave-hxkkn, template=PodTemplate{id='520dc4e6-e0df-45fb-b462-589002329332', name='jenkins-slave', slaveConnectTimeout=100, label='jenkins-agent', serviceAccount='default', nodeUsageMode=NORMAL, podRetention='Never', volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate{name='jnlp', image='gcr.io/blackjack-209019/pipeline/dev-build/build-engine-autoscaling/sofy-jenkins-agent:latest', workingDir='/home/jenkins', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceRequestEphemeralStorage='', resourceLimitCpu='', resourceLimitMemory='', resourceLimitEphemeralStorage='', envVars=[KeyValueEnvVar [getValue()=https://jenkins.hclpnpsofy.com/, getKey()=JENKINS_URL]], livenessProbe=ContainerLivenessProbe{execArgs='', timeoutSeconds=0, initialDelaySeconds=0, failureThreshold=0, periodSeconds=0, successThreshold=0}}], imagePullSecrets=[PodImagePullSecret\{name='gcr-json-key'}]} io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[jenkins-slave-hxkkn] in namespace [jenkins]. at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

          [JENKINS-72662] Connectivity issue with cloud agent post upgrading Jenkins to latest

          Jana added a comment -

          Disconnected computer jenkins-slave-lt9lz
          Feb 05, 2024 6:46:01 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor
          No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave

          Jana added a comment - Disconnected computer jenkins-slave-lt9lz Feb 05, 2024 6:46:01 AM INFO org.jenkinsci.plugins.cloudstats.CloudStatistics getIdFor No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave

          Jana added a comment -

          maheshrvm can you check is there any plugin incompatibility issue

          Jana added a comment - maheshrvm can you check is there any plugin incompatibility issue

          Mahesh added a comment -

          Hi Srikanth, Thanks for responding. I don't see any warnings in plugins section in Jenkins with incompatibility. But we can suspect as you mentioned in the first comment its due to  No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave. Do you think this is the cause of issue, you want me to remove this plugin?
           
           

          Mahesh added a comment - Hi Srikanth, Thanks for responding. I don't see any warnings in plugins section in Jenkins with incompatibility. But we can suspect as you mentioned in the first comment its due to  No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave. Do you think this is the cause of issue, you want me to remove this plugin?    

          Mahesh added a comment -

          FYI, I am providing some portion of logs here,

          Feb 05, 2024 6:56:59 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Computer for agent is null: jenkins-slave-pg8vm Feb 05, 2024 6:56:59 AM INFO hudson.slaves.AbstractCloudSlave terminate FATAL: Computer for agent is null: jenkins-slave-pg8vm Feb 05, 2024 6:56:59 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch Error in provisioning; agent=KubernetesSlave name: jenkins-slave-z0k77, template=PodTemplate{id='520dc4e6-e0df-45fb-b462-589002329332', name='jenkins-slave', slaveConnectTimeout=100, label='jenkins-agent', serviceAccount='default', nodeUsageMode=NORMAL, podRetention='Never', volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate{name='jnlp', image='gcr.io/blackjack-209019/pipeline/dev-build/build-engine-autoscaling/sofy-jenkins-agent:latest', workingDir='/home/jenkins', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceRequestEphemeralStorage='', resourceLimitCpu='', resourceLimitMemory='', resourceLimitEphemeralStorage='', envVars=[KeyValueEnvVar [getValue()=https://jenkins.hclpnpsofy.com/, getKey()=JENKINS_URL]], livenessProbe=ContainerLivenessProbe{execArgs='', timeoutSeconds=0, initialDelaySeconds=0, failureThreshold=0, periodSeconds=0, successThreshold=0}}], imagePullSecrets=[PodImagePullSecret\{name='gcr-json-key'}]} io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[jenkins-slave-z0k77] in namespace [jenkins]. at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

          Feb 05, 2024 6:56:59 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminating Kubernetes instance for agent jenkins-slave-z0k77 Feb 05, 2024 6:56:59 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Computer for agent is null: jenkins-slave-z0k77 Feb 05, 2024 6:56:59 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch Error in provisioning; agent=KubernetesSlave name: jenkins-slave-35cmx, template=PodTemplate{id='520dc4e6-e0df-45fb-b462-589002329332', name='jenkins-slave', slaveConnectTimeout=100, label='jenkins-agent', serviceAccount='default', nodeUsageMode=NORMAL, podRetention='Never', volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate{name='jnlp', image='gcr.io/blackjack-209019/pipeline/dev-build/build-engine-autoscaling/sofy-jenkins-agent:latest', workingDir='/home/jenkins', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceRequestEphemeralStorage='', resourceLimitCpu='', resourceLimitMemory='', resourceLimitEphemeralStorage='', envVars=[KeyValueEnvVar [getValue()=https://jenkins.hclpnpsofy.com/, getKey()=JENKINS_URL]], livenessProbe=ContainerLivenessProbe{execArgs='', timeoutSeconds=0, initialDelaySeconds=0, failureThreshold=0, periodSeconds=0, successThreshold=0}}], imagePullSecrets=[PodImagePullSecret\{name='gcr-json-key'}]} io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[jenkins-slave-35cmx] in namespace [jenkins]. at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

          Mahesh added a comment - FYI, I am providing some portion of logs here, Feb 05, 2024 6:56:59 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Computer for agent is null: jenkins-slave-pg8vm Feb 05, 2024 6:56:59 AM INFO hudson.slaves.AbstractCloudSlave terminate FATAL: Computer for agent is null: jenkins-slave-pg8vm Feb 05, 2024 6:56:59 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch Error in provisioning; agent=KubernetesSlave name: jenkins-slave-z0k77, template=PodTemplate{id='520dc4e6-e0df-45fb-b462-589002329332', name='jenkins-slave', slaveConnectTimeout=100, label='jenkins-agent', serviceAccount='default', nodeUsageMode=NORMAL, podRetention='Never', volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock] ], containers=[ContainerTemplate{name='jnlp', image='gcr.io/blackjack-209019/pipeline/dev-build/build-engine-autoscaling/sofy-jenkins-agent:latest', workingDir='/home/jenkins', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceRequestEphemeralStorage='', resourceLimitCpu='', resourceLimitMemory='', resourceLimitEphemeralStorage='', envVars=[KeyValueEnvVar [getValue()= https://jenkins.hclpnpsofy.com/ , getKey()=JENKINS_URL]], livenessProbe=ContainerLivenessProbe{execArgs='', timeoutSeconds=0, initialDelaySeconds=0, failureThreshold=0, periodSeconds=0, successThreshold=0}}], imagePullSecrets= [PodImagePullSecret\{name='gcr-json-key'}] } io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name: [jenkins-slave-z0k77] in namespace [jenkins] . at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Feb 05, 2024 6:56:59 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Terminating Kubernetes instance for agent jenkins-slave-z0k77 Feb 05, 2024 6:56:59 AM SEVERE org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate Computer for agent is null: jenkins-slave-z0k77 Feb 05, 2024 6:56:59 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch Error in provisioning; agent=KubernetesSlave name: jenkins-slave-35cmx, template=PodTemplate{id='520dc4e6-e0df-45fb-b462-589002329332', name='jenkins-slave', slaveConnectTimeout=100, label='jenkins-agent', serviceAccount='default', nodeUsageMode=NORMAL, podRetention='Never', volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock] ], containers=[ContainerTemplate{name='jnlp', image='gcr.io/blackjack-209019/pipeline/dev-build/build-engine-autoscaling/sofy-jenkins-agent:latest', workingDir='/home/jenkins', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceRequestEphemeralStorage='', resourceLimitCpu='', resourceLimitMemory='', resourceLimitEphemeralStorage='', envVars=[KeyValueEnvVar [getValue()= https://jenkins.hclpnpsofy.com/ , getKey()=JENKINS_URL]], livenessProbe=ContainerLivenessProbe{execArgs='', timeoutSeconds=0, initialDelaySeconds=0, failureThreshold=0, periodSeconds=0, successThreshold=0}}], imagePullSecrets= [PodImagePullSecret\{name='gcr-json-key'}] } io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name: [jenkins-slave-35cmx] in namespace [jenkins] . at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97) at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

          Mahesh added a comment -

          As I have roll back to previous version I can't find the recent logs. The above logs were present when I upgraded to latest version. At that time k8s cloud agent not able to provisioned. Please advise.

          Mahesh added a comment - As I have roll back to previous version I can't find the recent logs. The above logs were present when I upgraded to latest version. At that time k8s cloud agent not able to provisioned. Please advise.

          Jana added a comment -

          maheshrvm can you give me some info:
          1. previous jenkins version you used and latest jenkins verison upgraded to.
          2. Kubernetes plugin version
          3. cloud-stats-plugin version

          Jana added a comment - maheshrvm can you give me some info: 1. previous jenkins version you used and latest jenkins verison upgraded to. 2. Kubernetes plugin version 3. cloud-stats-plugin version

          Mahesh added a comment -

          1. Previous Jenkins version: Jenkins-2.414.3 to upgraded to Jenkins-2.426.3 (As I faced the connectivity with cloud agents, rolled back to previous version 2.414.3)
          2. Kubernetes plugin version: Kubernetes pluginVersion4054.v2da_8e2794884
          3. Cloud-stats-plugin version: Cloud Statistics PluginVersion336.v788e4055508b_

          I regularly update plugins whenever the updates available for plugins.

          Mahesh added a comment - 1. Previous Jenkins version: Jenkins-2.414.3 to upgraded to Jenkins-2.426.3 (As I faced the connectivity with cloud agents, rolled back to previous version 2.414.3) 2. Kubernetes plugin version: Kubernetes pluginVersion4054.v2da_8e2794884 3. Cloud-stats-plugin version: Cloud Statistics PluginVersion336.v788e4055508b_ I regularly update plugins whenever the updates available for plugins.

          Mahesh added a comment -

          Could you provide an update please. I'm still not able to find the root cause.

          Mahesh added a comment - Could you provide an update please. I'm still not able to find the root cause.

          Ryker Youd added a comment -

          I am running into a similar issue, but there's nothing in the logs about No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.

          My issue cropped up after upgrading Jenkins to 2.447 and Kubernetes plugin to 4192.vded98e56cc25.

          I've made sure that I am fully up to date on all plugins but that hasn't helped with anything. I had to redeploy a new Jenkins on version 2.443 and Kubernetes 4186.v1d804571d5d4 in order to get any jobs to run again.

          My logs from Jenkins startup until the running into the issue: 

          2024-03-08 17:09:16.092+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file
          2024-03-08 17:09:16.161+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath
          2024-03-08 17:09:16.222+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 11.0.22+7
          2024-03-08 17:09:16.486+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
          2024-03-08 17:09:16.534+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0
          2024-03-08 17:09:16.966+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
          2024-03-08 17:09:17.095+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@522b2631{Jenkins v2.447,/,file:///var/jenkins_cache/war/,AVAILABLE}{/var/jenkins_cache/war}
          2024-03-08 17:09:17.107+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@1ebd319f{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
          2024-03-08 17:09:17.124+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: Started Server@4ef37659{STARTING}[10.0.20,sto=0] @1522ms
          2024-03-08 17:09:17.128+0000 [id=25]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
          2024-03-08 17:09:17.309+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
          2024-03-08 17:09:22.735+0000 [id=43]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          2024-03-08 17:09:26.514+0000 [id=37]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          2024-03-08 17:09:26.531+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
          2024-03-08 17:09:26.537+0000 [id=35]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions2024-03-08 17:09:17.128+0000 [id=25]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled
          2024-03-08 17:09:17.309+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
          2024-03-08 17:09:22.735+0000 [id=43]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          2024-03-08 17:09:26.514+0000 [id=37]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          2024-03-08 17:09:26.531+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
          2024-03-08 17:09:26.537+0000 [id=35]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
          WARNING: An illegal reflective access operation has occurred
          WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/var/jenkins_cache/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
          WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
          WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
          WARNING: All illegal access operations will be denied in a future release
          2024-03-08 17:09:27.151+0000 [id=41]    INFO    h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set
          2024-03-08 17:09:27.427+0000 [id=44]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
          2024-03-08 17:09:27.538+0000 [id=38]    WARNING c.c.h.p.folder.AbstractFolder#loadChildren: could not load /var/jenkins_home/jobs/FOLDER/jobs/JOB1
          java.io.FileNotFoundException: Could not find configuration file /var/jenkins_home/jobs/FOLDER/jobs/JOB1/config.xml
                  at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:367)
                  at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:472)
                  at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:112)
                  at hudson.model.Items.load(Items.java:376)
                  at jenkins.model.Jenkins$13.run(Jenkins.java:3514)
                  at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
                  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
                  at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)
                  at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
                  at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
                  at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
                  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                  at java.base/java.lang.Thread.run(Unknown Source)
          2024-03-08 17:09:27.573+0000 [id=38]    WARNING c.c.h.p.folder.AbstractFolder#loadChildren: could not load /var/jenkins_home/jobs/FOLDER/jobs/JOB2
          java.io.FileNotFoundException: Could not find configuration file /var/jenkins_home/jobs/FOLDER/jobs/JOB2/config.xml
                  at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:367)
                  at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:472)
                  at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:112)
                  at hudson.model.Items.load(Items.java:376)
                  at jenkins.model.Jenkins$13.run(Jenkins.java:3514)
                  at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)
                  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
                  at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)
                  at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
                  at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
                  at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
                  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                  at java.base/java.lang.Thread.run(Unknown Source)
          2024-03-08 17:09:27.915+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.csanchez.jenkins.plugins.kubernetes.PodTemplate#listener: type is abstract but not Describable.
          2024-03-08 17:09:27.982+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
          2024-03-08 17:09:28.423+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.jenkinsci.plugins.github_branch_source.GitHubSCMSource#owner: type is abstract but not Describable.
          2024-03-08 17:09:28.492+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.csanchez.jenkins.plugins.kubernetes.PodTemplate#listener: type is abstract but not Describable.
          2024-03-08 17:09:28.542+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
          2024-03-08 17:09:28.560+0000 [id=57]    INFO    o.c.j.p.k.pod.retention.Reaper#watchCloud: set up watcher on kubernetes
          2024-03-08 17:09:28.903+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.jenkinsci.plugins.github_branch_source.GitHubSCMSource#owner: type is abstract but not Describable.
          2024-03-08 17:09:28.929+0000 [id=35]    INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted
          2024-03-08 17:09:28.930+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
          2024-03-08 17:09:28.930+0000 [id=30]    INFO    o.j.p.l.NodesMirror#createNodeResources: lockable-resources-plugin: configure node resources
          2024-03-08 17:09:28.990+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated
          2024-03-08 17:09:29.095+0000 [id=42]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
          2024-03-08 17:09:29.135+0000 [id=24]    INFO    hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running
          2024-03-08 17:12:53.559+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
          2024-03-08 17:12:53.560+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /job/FOLDER5/widget/ExecutorsWidget/ajax by USER. Returning 403.
          2024-03-08 17:12:53.559+0000 [id=105]   WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
          2024-03-08 17:12:53.561+0000 [id=105]   WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /job/FOLDER5/widget/BuildQueueWidget/ajax by USER. Returning 403.
          2024-03-08 17:12:55.735+0000 [id=71]    WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
          2024-03-08 17:12:55.735+0000 [id=71]    WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /widget/BuildQueueWidget/ajax by USER. Returning 403.
          2024-03-08 17:12:55.736+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
          2024-03-08 17:12:55.736+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /widget/ExecutorsWidget/ajax by USER. Returning 403.
          2024-03-08 17:13:17.501+0000 [id=127]   WARNING o.k.g.e.a.JwtBuilderUtil#createBuilderImpl: You are using an outdated version of the io.jsonwebtoken:jjwt-* suite. v0.12.x or later is recommended.
          2024-03-08 17:13:50.036+0000 [id=188]   INFO    hudson.slaves.NodeProvisioner#update: folder-job-name-40-gdp5j-1h1gv-slcbs provisioning successfully completed. We have now 3 computer(s)
          2024-03-08 17:13:50.136+0000 [id=187]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins-k8s-namespace/folder-job-name-40-gdp5j-1h1gv-slcbs
          2024-03-08 17:14:20.018+0000 [id=249]   INFO    hudson.slaves.NodeProvisioner#update: folder-job-name-360-6bmz5-3v644-1lg93 provisioning successfully completed. We have now 4 computer(s)
          2024-03-08 17:14:20.021+0000 [id=188]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins-k8s-namespace/folder-job-name-360-6bmz5-3v644-1lg93
          2024-03-08 17:15:30.198+0000 [id=187]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: folder-job-name-40-gdp5j-1h1gv-slcbs, template=PodTemplate{id='94c39694-7745-41bc-971a-f9e30326bb70', name='FOLDER_JOB_40-gdp5j-1h1gv', namespace='jenkins-k8s-namespace', slaveConnectTimeout=100, label='FOLDER_JOB_40-gdp5j', serviceAccount='a-valid-service-account', workspaceVolume='org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.PersistentVolumeClaimWorkspaceVolume@19ecd615', containers=[ContainerTemplate{name='containera', image='valid-image', command='sleep', args='99d', envVars=[KeyValueEnvVar [getValue()=valid-value, getKey()=valid-key]]}, ContainerTemplate{name='jnlp', image='valid-jnlp-container-image', workingDir='/home/jenkins-k8s-namespace/agent', command='', args='${computer.jnlpmac} ${computer.name}', resourceRequestCpu='64m', resourceRequestMemory='256Mi', resourceLimitCpu='512m', resourceLimitMemory='1024Mi', envVars=[KeyValueEnvVar [getValue()=http://jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/, getKey()=JENKINS_URL]]}], annotations=[PodAnnotation{key='buildUrl', value='http://jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/job/FOLDER/job/JOB/40/'}, PodAnnotation{key='runUrl', value='job/FOLDER/job/JOB/40/'}], imagePullSecrets=[PodImagePullSecret{name='image-pull-secret'}], unwrapped=true}
          io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[folder-job-name-40-gdp5j-1h1gv-slcbs] in namespace [jenkins-k8s-namespace].
                  at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939)
                  at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921)
                  at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97)
                  at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185)
                  at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
                  at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
                  at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
                  at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
                  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                  at java.base/java.lang.Thread.run(Unknown Source)
          2024-03-08 17:15:30.199+0000 [id=187]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent folder-job-name-40-gdp5j-1h1gv-slcbs
          2024-03-08 17:15:30.228+0000 [id=187]   INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent jenkins-k8s-namespace/folder-job-name-40-gdp5j-1h1gv-slcbs
          2024-03-08 17:15:30.228+0000 [id=187]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer folder-job-name-40-gdp5j-1h1gv-slcbs
          2024-03-08 17:15:30.248+0000 [id=263]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent folder-job-name-39-75mgw-bmxq3-14ggv
          2024-03-08 17:15:30.264+0000 [id=263]   INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent jenkins-k8s-namespace/folder-job-name-39-75mgw-bmxq3-14ggv
          2024-03-08 17:15:30.265+0000 [id=263]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer folder-job-name-39-75mgw-bmxq3-14ggv
          2024-03-08 17:15:40.017+0000 [id=263]   INFO    hudson.slaves.NodeProvisioner#update: folder-job-name-40-gdp5j-1h1gv-c3h9j provisioning successfully completed. We have now 3 computer(s)
          2024-03-08 17:15:40.041+0000 [id=187]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins-k8s-namespace/folder-job-name-40-gdp5j-1h1gv-c3h9j
          2024-03-08 17:16:00.029+0000 [id=188]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: folder-job-name-360-6bmz5-3v644-1lg93, template=PodTemplate{id='f9f39390-970d-40c0-b059-bd7e60b771f5', name='FOLDER_JOB2_360-6bmz5-3v644', namespace='jenkins-k8s-namespace', slaveConnectTimeout=100, label='FOLDER_JOB2_360-6bmz5', serviceAccount='default', workspaceVolume='org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.PersistentVolumeClaimWorkspaceVolume@19ecd615', containers=[ContainerTemplate{name='jnlp', image='valid-jnlp-container-image', workingDir='/home/jenkins-k8s-namespace/agent', command='', args='${computer.jnlpmac} ${computer.name}', resourceRequestCpu='64m', resourceRequestMemory='256Mi', resourceLimitCpu='512m', resourceLimitMemory='1024Mi', envVars=[KeyValueEnvVar [getValue()=http://jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/, getKey()=JENKINS_URL]]}], annotations=[PodAnnotation{key='buildUrl', value='http://jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/job/FOLDER/job/JOB2/360/'}, PodAnnotation{key='runUrl', value='job/FOLDER/job/JOB2/360/'}], imagePullSecrets=[PodImagePullSecret{name='image-pull-secret'}], unwrapped=true}
          io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[folder-job-name-360-6bmz5-3v644-1lg93] in namespace [jenkins-k8s-namespace].
                  at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939)
                  at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921)
                  at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97)
                  at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185)
                  at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
                  at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
                  at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
                  at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
                  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
                  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                  at java.base/java.lang.Thread.run(Unknown Source) 

          Any insight to what the issue is or how to fix it so that I can keep my Jenkins up to date would be much appreciated.

          Ryker Youd added a comment - I am running into a similar issue, but there's nothing in the logs about No support for cloud-stats-plugin by class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave. My issue cropped up after upgrading Jenkins to 2.447 and Kubernetes plugin to 4192.vded98e56cc25. I've made sure that I am fully up to date on all plugins but that hasn't helped with anything. I had to redeploy a new Jenkins on version 2.443 and Kubernetes 4186.v1d804571d5d4 in order to get any jobs to run again. My logs from Jenkins startup until the running into the issue:  2024-03-08 17:09:16.092+0000 [id=1]     INFO    winstone.Logger#logInternal: Beginning extraction from war file 2024-03-08 17:09:16.161+0000 [id=1]     WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath 2024-03-08 17:09:16.222+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 11.0.22+7 2024-03-08 17:09:16.486+0000 [id=1]     INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2024-03-08 17:09:16.534+0000 [id=1]     INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0 2024-03-08 17:09:16.966+0000 [id=1]     INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: / var /jenkins_home found at: EnvVars.masterEnvVars.get( "JENKINS_HOME" ) 2024-03-08 17:09:17.095+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@522b2631{Jenkins v2.447,/,file: /// var /jenkins_cache/war/,AVAILABLE}{/ var /jenkins_cache/war} 2024-03-08 17:09:17.107+0000 [id=1]     INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@1ebd319f{HTTP/1.1, (http/1.1)}{0.0.0.0:8080} 2024-03-08 17:09:17.124+0000 [id=1]     INFO    org.eclipse.jetty.server.Server#doStart: Started Server@4ef37659{STARTING}[10.0.20,sto=0] @1522ms 2024-03-08 17:09:17.128+0000 [id=25]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled 2024-03-08 17:09:17.309+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization 2024-03-08 17:09:22.735+0000 [id=43]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2024-03-08 17:09:26.514+0000 [id=37]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 2024-03-08 17:09:26.531+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins 2024-03-08 17:09:26.537+0000 [id=35]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions2024-03-08 17:09:17.128+0000 [id=25]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled 2024-03-08 17:09:17.309+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization 2024-03-08 17:09:22.735+0000 [id=43]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins 2024-03-08 17:09:26.514+0000 [id=37]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 2024-03-08 17:09:26.531+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins 2024-03-08 17:09:26.537+0000 [id=35]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/ var /jenkins_cache/war/WEB-INF/lib/groovy-all-2.4.21.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang. Class , int ) WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2024-03-08 17:09:27.151+0000 [id=41]    INFO    h.p.b.g.GlobalTimeOutConfiguration#load: global timeout not set 2024-03-08 17:09:27.427+0000 [id=44]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded 2024-03-08 17:09:27.538+0000 [id=38]    WARNING c.c.h.p.folder.AbstractFolder#loadChildren: could not load / var /jenkins_home/jobs/FOLDER/jobs/JOB1 java.io.FileNotFoundException: Could not find configuration file / var /jenkins_home/jobs/FOLDER/jobs/JOB1/config.xml         at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:367)         at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:472)         at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:112)         at hudson.model.Items.load(Items.java:376)         at jenkins.model.Jenkins$13.run(Jenkins.java:3514)         at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)         at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)         at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)         at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)         at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)         at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)         at java.base/java.lang. Thread .run(Unknown Source) 2024-03-08 17:09:27.573+0000 [id=38]    WARNING c.c.h.p.folder.AbstractFolder#loadChildren: could not load / var /jenkins_home/jobs/FOLDER/jobs/JOB2 java.io.FileNotFoundException: Could not find configuration file / var /jenkins_home/jobs/FOLDER/jobs/JOB2/config.xml         at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:367)         at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:472)         at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:112)         at hudson.model.Items.load(Items.java:376)         at jenkins.model.Jenkins$13.run(Jenkins.java:3514)         at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)         at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)         at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)         at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)         at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)         at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)         at java.base/java.lang. Thread .run(Unknown Source) 2024-03-08 17:09:27.915+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.csanchez.jenkins.plugins.kubernetes.PodTemplate#listener: type is abstract but not Describable. 2024-03-08 17:09:27.982+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable. 2024-03-08 17:09:28.423+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.jenkinsci.plugins.github_branch_source.GitHubSCMSource#owner: type is abstract but not Describable. 2024-03-08 17:09:28.492+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.csanchez.jenkins.plugins.kubernetes.PodTemplate#listener: type is abstract but not Describable. 2024-03-08 17:09:28.542+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable. 2024-03-08 17:09:28.560+0000 [id=57]    INFO    o.c.j.p.k.pod.retention.Reaper#watchCloud: set up watcher on kubernetes 2024-03-08 17:09:28.903+0000 [id=37]    WARNING i.j.p.casc.BaseConfigurator#createAttribute: Can't handle class org.jenkinsci.plugins.github_branch_source.GitHubSCMSource#owner: type is abstract but not Describable. 2024-03-08 17:09:28.929+0000 [id=35]    INFO    jenkins.InitReactorRunner$1#onAttained: System config adapted 2024-03-08 17:09:28.930+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 2024-03-08 17:09:28.930+0000 [id=30]    INFO    o.j.p.l.NodesMirror#createNodeResources: lockable-resources-plugin: configure node resources 2024-03-08 17:09:28.990+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Configuration for all jobs updated 2024-03-08 17:09:29.095+0000 [id=42]    INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization 2024-03-08 17:09:29.135+0000 [id=24]    INFO    hudson.lifecycle.Lifecycle#onReady: Jenkins is fully up and running 2024-03-08 17:12:53.559+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https: //www.jenkins.io/redirect/crumb-cannot-be-used- for -script 2024-03-08 17:12:53.560+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /job/FOLDER5/widget/ExecutorsWidget/ajax by USER. Returning 403. 2024-03-08 17:12:53.559+0000 [id=105]   WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https: //www.jenkins.io/redirect/crumb-cannot-be-used- for -script 2024-03-08 17:12:53.561+0000 [id=105]   WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /job/FOLDER5/widget/BuildQueueWidget/ajax by USER. Returning 403. 2024-03-08 17:12:55.735+0000 [id=71]    WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https: //www.jenkins.io/redirect/crumb-cannot-be-used- for -script 2024-03-08 17:12:55.735+0000 [id=71]    WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /widget/BuildQueueWidget/ajax by USER. Returning 403. 2024-03-08 17:12:55.736+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 19347d823a12389d34234. If you are calling this URL with a script, please use the API Token instead. More information: https: //www.jenkins.io/redirect/crumb-cannot-be-used- for -script 2024-03-08 17:12:55.736+0000 [id=84]    WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /widget/ExecutorsWidget/ajax by USER. Returning 403. 2024-03-08 17:13:17.501+0000 [id=127]   WARNING o.k.g.e.a.JwtBuilderUtil#createBuilderImpl: You are using an outdated version of the io.jsonwebtoken:jjwt-* suite. v0.12.x or later is recommended. 2024-03-08 17:13:50.036+0000 [id=188]   INFO    hudson.slaves.NodeProvisioner#update: folder-job-name-40-gdp5j-1h1gv-slcbs provisioning successfully completed. We have now 3 computer(s) 2024-03-08 17:13:50.136+0000 [id=187]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins-k8s-namespace/folder-job-name-40-gdp5j-1h1gv-slcbs 2024-03-08 17:14:20.018+0000 [id=249]   INFO    hudson.slaves.NodeProvisioner#update: folder-job-name-360-6bmz5-3v644-1lg93 provisioning successfully completed. We have now 4 computer(s) 2024-03-08 17:14:20.021+0000 [id=188]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins-k8s-namespace/folder-job-name-360-6bmz5-3v644-1lg93 2024-03-08 17:15:30.198+0000 [id=187]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: folder-job-name-40-gdp5j-1h1gv-slcbs, template=PodTemplate{id= '94c39694-7745-41bc-971a-f9e30326bb70' , name= 'FOLDER_JOB_40-gdp5j-1h1gv' , namespace= 'jenkins-k8s-namespace' , slaveConnectTimeout=100, label= 'FOLDER_JOB_40-gdp5j' , serviceAccount= 'a-valid-service-account' , workspaceVolume= 'org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.PersistentVolumeClaimWorkspaceVolume@19ecd615' , containers=[ContainerTemplate{name= 'containera' , image= 'valid-image' , command= 'sleep' , args= '99d' , envVars=[KeyValueEnvVar [getValue()=valid-value, getKey()=valid-key]]}, ContainerTemplate{name= 'jnlp' , image= 'valid-jnlp-container-image' , workingDir= '/home/jenkins-k8s-namespace/agent' , command= '', args=' ${computer.jnlpmac} ${computer.name} ', resourceRequestCpu=' 64m ', resourceRequestMemory=' 256Mi ', resourceLimitCpu=' 512m ', resourceLimitMemory=' 1024Mi ', envVars=[KeyValueEnvVar [getValue()=http: //jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/, getKey()=JENKINS_URL]]}], annotations=[PodAnnotation{key=' buildUrl ', value=' http://jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/job/FOLDER/job/JOB/40/ '}, PodAnnotation{key=' runUrl ', value=' job/FOLDER/job/JOB/40/ '}], imagePullSecrets=[PodImagePullSecret{name=' image-pull-secret'}], unwrapped= true } io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[folder-job-name-40-gdp5j-1h1gv-slcbs] in namespace [jenkins-k8s-namespace].         at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939)         at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921)         at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97)         at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185)         at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)         at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)         at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)         at java.base/java.util.concurrent.FutureTask.run(Unknown Source)         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)         at java.base/java.lang. Thread .run(Unknown Source) 2024-03-08 17:15:30.199+0000 [id=187]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent folder-job-name-40-gdp5j-1h1gv-slcbs 2024-03-08 17:15:30.228+0000 [id=187]   INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent jenkins-k8s-namespace/folder-job-name-40-gdp5j-1h1gv-slcbs 2024-03-08 17:15:30.228+0000 [id=187]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer folder-job-name-40-gdp5j-1h1gv-slcbs 2024-03-08 17:15:30.248+0000 [id=263]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent folder-job-name-39-75mgw-bmxq3-14ggv 2024-03-08 17:15:30.264+0000 [id=263]   INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent jenkins-k8s-namespace/folder-job-name-39-75mgw-bmxq3-14ggv 2024-03-08 17:15:30.265+0000 [id=263]   INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer folder-job-name-39-75mgw-bmxq3-14ggv 2024-03-08 17:15:40.017+0000 [id=263]   INFO    hudson.slaves.NodeProvisioner#update: folder-job-name-40-gdp5j-1h1gv-c3h9j provisioning successfully completed. We have now 3 computer(s) 2024-03-08 17:15:40.041+0000 [id=187]   INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes jenkins-k8s-namespace/folder-job-name-40-gdp5j-1h1gv-c3h9j 2024-03-08 17:16:00.029+0000 [id=188]   WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: folder-job-name-360-6bmz5-3v644-1lg93, template=PodTemplate{id= 'f9f39390-970d-40c0-b059-bd7e60b771f5' , name= 'FOLDER_JOB2_360-6bmz5-3v644' , namespace= 'jenkins-k8s-namespace' , slaveConnectTimeout=100, label= 'FOLDER_JOB2_360-6bmz5' , serviceAccount= ' default ' , workspaceVolume= 'org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.PersistentVolumeClaimWorkspaceVolume@19ecd615' , containers=[ContainerTemplate{name= 'jnlp' , image= 'valid-jnlp-container-image' , workingDir= '/home/jenkins-k8s-namespace/agent' , command= '', args=' ${computer.jnlpmac} ${computer.name} ', resourceRequestCpu=' 64m ', resourceRequestMemory=' 256Mi ', resourceLimitCpu=' 512m ', resourceLimitMemory=' 1024Mi ', envVars=[KeyValueEnvVar [getValue()=http: //jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/, getKey()=JENKINS_URL]]}], annotations=[PodAnnotation{key=' buildUrl ', value=' http://jenkins.jenkins-k8s-namespace.svc.cluster.local:8080/job/FOLDER/job/JOB2/360/ '}, PodAnnotation{key=' runUrl ', value=' job/FOLDER/job/JOB2/360/ '}], imagePullSecrets=[PodImagePullSecret{name=' image-pull-secret'}], unwrapped= true } io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [100000] milliseconds for [Pod] with name:[folder-job-name-360-6bmz5-3v644-1lg93] in namespace [jenkins-k8s-namespace].         at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilCondition(BaseOperation.java:939)         at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:921)         at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.waitUntilReady(BaseOperation.java:97)         at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:185)         at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)         at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)         at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)         at java.base/java.util.concurrent.FutureTask.run(Unknown Source)         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)         at java.base/java.lang. Thread .run(Unknown Source) Any insight to what the issue is or how to fix it so that I can keep my Jenkins up to date would be much appreciated.

            Unassigned Unassigned
            maheshrvm Mahesh
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: