Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67474

Pipeline is failing due to io.fabric8.kubernetes.client.KubernetesClientException: not ready after n milliseconds

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • kubernetes-plugin
    • None

      Sometimes when pipeline starts or in the middle of pipeline, I receive this error:

      io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS
	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176)
	at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.waitUntilReady(ExecWebSocketListener.java:158)
	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:329)
	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:427)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:344)
	at hudson.Launcher$ProcStarter.start(Launcher.java:508)
	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
	at com.jhh.jenkins.core.HelmManager.packageAndPush(HelmManager.groovy:13)
	at WorkflowScript.run(WorkflowScript:87)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:107)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

      I don't understand what can be the problem. Could you help me please?
      Jenkins is deployed in Azure Kubernetes Service cluster.

          [JENKINS-67474] Pipeline is failing due to io.fabric8.kubernetes.client.KubernetesClientException: not ready after n milliseconds

          Damien Duportal added a comment -

          We see the following behavior:

          • The JNLP container of each pod starts successfully, and is able to initialize a workspace for the pod (the git checkout step works flawlessly)
          • As soon as an `sh''` step is executed (even if there is only one), this error message appears.
          • Increasing the websocket timeout does not change this behavior (it's always reached)
          • The pod templates are always specified through a custom podtemplate.Yaml, with NO explicit inheritance (it seems that the jnlp container is the default from the kubernetes-plugin)
          • All of our pods starts with valid readiness probes

          Damien Duportal added a comment - We see the following behavior: The JNLP container of each pod starts successfully, and is able to initialize a workspace for the pod (the git checkout step works flawlessly) As soon as an `sh''` step is executed (even if there is only one), this error message appears. Increasing the websocket timeout does not change this behavior (it's always reached) The pod templates are always specified through a custom podtemplate.Yaml, with NO explicit inheritance (it seems that the jnlp container is the default from the kubernetes-plugin) All of our pods starts with valid readiness probes

          Damien Duportal added a comment -

          We switched back to a single-container model as per https://support.cloudbees.com/hc/en-us/articles/360054642231-Considerations-for-Kubernetes-Clients-Connections-when-using-Kubernetes-Plugin on infra.ci for now.

          It defeats the whole puprose of Kubernetes pod, but it is the only solution that we can use.

          Damien Duportal added a comment - We switched back to a single-container model as per https://support.cloudbees.com/hc/en-us/articles/360054642231-Considerations-for-Kubernetes-Clients-Connections-when-using-Kubernetes-Plugin on infra.ci for now. It defeats the whole puprose of Kubernetes pod, but it is the only solution that we can use.

          Jesse Glick added a comment -

          The current implementation of the container step is not great. See linked issues.

          Jesse Glick added a comment - The current implementation of the container step is not great. See linked issues.

          Adam Placzek added a comment - - edited

          After recent upgrades of Kubernetes plugins we are observing an increase of the below errors: 

          io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS

          Increasing timeout does not reduce the number of occurrences.
          It's so bad that it makes Jenkins pipelines unreliable and unusable. Is there any fix or at least a detailed investigation coming?

          Adam Placzek added a comment - - edited After recent upgrades of Kubernetes plugins we are observing an increase of the below errors:  io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS Increasing timeout does not reduce the number of occurrences. It's so bad that it makes Jenkins pipelines unreliable and unusable. Is there any fix or at least a detailed investigation coming?

          jianli hujianli added a comment -

          io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS
          at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176)
          at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:322)
          at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84)
          at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:427)
          at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:344)
          at hudson.Launcher$ProcStarter.start(Launcher.java:507)
          at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
          at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
          at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
          at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
          at sun.reflect.GeneratedMethodAccessor1017.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)

           

          I have the same problem here

          Kubernetes plugin：1.31.2

          Kubernetes Client API：5.11.1-179.v12037658df90

          jenkins：2.319.1

           

          jianli hujianli added a comment - io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176) at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:322) at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:427) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:344) at hudson.Launcher$ProcStarter.start(Launcher.java:507) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor1017.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498)   I have the same problem here Kubernetes plugin： 1.31.2 Kubernetes Client API ： 5.11.1-179.v12037658df90 jenkins：2.319.1  

          SparkC added a comment -

          I'm facing same issue across all Jenkins servers. I tried many ways to fix this but nothing worked. 

          SparkC added a comment - I'm facing same issue across all Jenkins servers. I tried many ways to fix this but nothing worked. 

          Terminator added a comment -

          aplaczek which version have you been previosly using?
          When you say "increasing". Does it mean you were also getting these errors previously?

          I am getting the error with 

          • kubernetes plugin: 1.31.3
          • kubernetes client api plugin: 5.11.2-182.v0f1cf4c5904e

          Terminator added a comment - aplaczek  which version have you been previosly using? When you say "increasing". Does it mean you were also getting these errors previously? I am getting the error with  kubernetes plugin: 1.31.3 kubernetes client api plugin: 5.11.2-182.v0f1cf4c5904e

          Terminator added a comment -

          Duplicate / related issue: https://issues.jenkins.io/browse/JENKINS-67664

          Terminator added a comment - Duplicate / related issue:  https://issues.jenkins.io/browse/JENKINS-67664

          Artem Chernenko added a comment - - edited

          I upgraded Jenkins to 2.355 and Kubernetes Plugin to 3651.v908e7db_10d06. And error is gone

           

          UPD: I was wrong. Issue still occurs(

          Artem Chernenko added a comment - - edited I upgraded Jenkins to 2.355 and Kubernetes Plugin to 3651.v908e7db_10d06 . And error is gone   UPD: I was wrong. Issue still occurs(

          Francisco Aguiar added a comment -

          achernenkokaaiot 

          Is your cluster private? As I explained here: https://issues.jenkins.io/browse/JENKINS-67167?focusedCommentId=427982&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-427982

          I only have this problem when the cluster is private. Is that your case?

          Francisco Aguiar added a comment - achernenkokaaiot   Is your cluster private? As I explained here: https://issues.jenkins.io/browse/JENKINS-67167?focusedCommentId=427982&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-427982 I only have this problem when the cluster is private. Is that your case?

            Unassigned Unassigned
            achernenkokaaiot Artem Chernenko
            Votes:
            0 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved: