Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67664

KubernetesClientException: not ready after 5000 MILLISECONDS

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None
    • Prod
    • Blue Ocean - Candidates
    • 3690.va_9ddf6635481

      We have 4 Jenkins servers in 4 AKS Clusters. 

      Af of sudden all Jenkins agent pods started giving Errors as below, Few pods are working and few are giving Errors.  This is Happening 1-2 times out of 4-5 attempts. 

      AKS Version : 1.20.13 

      Jenkins Version, which clusters is having different version. I can reproduce this Error in all versions. 

      AKS-1:

      • kubernetes:1.30.1
      • kubernetes-client-api:5.10.1-171.vaa0774fb8c20
      • kubernetes-credentials:0.8.0

      AKS-2:

      • kubernetes:1.31.3
      • kubernetes-client-api:5.11.2-182.v0f1cf4c5904e
      • kubernetes-credentials:0.9.0

      AKS-3:

      • kubernetes:1.30.1
      • kubernetes-client-api:5.10.1-171.vaa0774fb8c20
      • kubernetes-credentials:0.8.0

      AKS-4:

      • kubernetes:1.31.3
      • workflow-job:1145.v7f2433caa07f
      • workflow-aggregator:2.6
        21:00:49 io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS*21:00:49* at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176)21:00:49 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:322)21:00:49 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84)21:00:49 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:427)21:00:49 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:344)21:00:49 at hudson.Launcher$ProcStarter.start(Launcher.java:507)21:00:49 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)21:00:49 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)21:00:49 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)21:00:49 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)21:00:49 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)21:00:49 at jdk.internal.reflect.GeneratedMethodAccessor546.invoke(Unknown Source)21:00:49 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)21:00:49 at java.base/java.lang.reflect.Method.invoke(Method.java:566)21:00:49 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)21:00:49 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)21:00:49 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)21:00:49 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)21:00:49 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)21:00:49 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)21:00:49 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)21:00:49 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)21:00:49 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)21:00:49 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)21:00:49 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)21:00:49 at WorkflowScript.run(WorkflowScript:63)21:00:49 at __cps.transform__(Native Method)21:00:49 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)21:00:49 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)21:00:49 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)21:00:49 at jdk.internal.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)21:00:49 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)21:00:49 at java.base/java.lang.reflect.Method.invoke(Method.java:566)21:00:49 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)21:00:49 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:107)21:00:49 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)21:00:49 at jdk.internal.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)21:00:49 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)21:00:49 at java.base/java.lang.reflect.Method.invoke(Method.java:566)21:00:49 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)21:00:49 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)21:00:49 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)21:00:49 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)21:00:49 at jdk.internal.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)21:00:49 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)21:00:49 at java.base/java.lang.reflect.Method.invoke(Method.java:566)21:00:49 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)21:00:49 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)21:00:49 at com.cloudbees.groovy.cps.Next.step(Next.java:83)21:00:49 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)21:00:49 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)21:00:49 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)21:00:49 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)21:00:49 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)21:00:49 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)21:00:49 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278)21:00:49 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)21:00:49 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)21:00:49 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)21:00:49 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)21:00:49 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)21:00:49 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)21:00:49 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)21:00:49 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)21:00:49 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)21:00:49 at java.base/java.lang.Thread.run(Thread.java:829)21:00:49 [Bitbucket] Notifying commit build result*21:00:50* [Bitbucket] Build result notified*21:00:50* Finished: FAILURE

        1. evaluateExecResponsiveness.groovy
          3 kB
        2. image-2022-04-06-12-53-43-586.png
          image-2022-04-06-12-53-43-586.png
          19 kB

          [JENKINS-67664] KubernetesClientException: not ready after 5000 MILLISECONDS

          SparkC created issue -

          Adam Placzek added a comment -

          Hi,
          We have exactly the same situation. Also the same exact AKS version 1.20.13

          Failure happens whenever the step is executed in block

          container(Xxx) {
XXX
}

          I think it's  the same as  JENKINS-67167

          Adam Placzek added a comment - Hi, We have exactly the same situation. Also the same exact AKS version 1.20.13 Failure happens whenever the step is executed in block container(Xxx) { XXX } I think it's  the same as  JENKINS-67167

          SparkC added a comment -

          Do anyone know how to resolve this. Full Error Log:
          This is Happening in random places in pipeline. Sometimes pipeline works sometime pipeline fails. Not only 1 job multiple Jenkins servers and Multiple Pipelines. 

          12:39:53  io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS
12:39:53  	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176)
12:39:53  	at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.waitUntilReady(ExecWebSocketListener.java:158)
12:39:53  	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:329)
12:39:53  	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84)
12:39:53  	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:421)
12:39:53  	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:338)
12:39:53  	at hudson.Launcher$ProcStarter.start(Launcher.java:507)
12:39:53  	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
12:39:53  	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
12:39:53  	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
12:39:53  	at jdk.internal.reflect.GeneratedMethodAccessor558.invoke(Unknown Source)
12:39:53  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:39:53  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
12:39:53  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
12:39:53  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
12:39:53  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
12:39:53  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
12:39:53  	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
12:39:53  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
12:39:53  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
12:39:53  	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
12:39:53  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
12:39:53  	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
12:39:53  	at WorkflowScript.run(WorkflowScript:117)
12:39:53  	at ___cps.transform___(Native Method)
12:39:53  	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
12:39:53  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
12:39:53  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
12:39:53  	at jdk.internal.reflect.GeneratedMethodAccessor278.invoke(Unknown Source)
12:39:53  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12:39:53  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
12:39:53  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
12:39:53  	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
12:39:53  	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
12:39:53  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
12:39:53  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
12:39:53  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
12:39:53  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
12:39:53  	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278)
12:39:53  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
12:39:53  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:39:53  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
12:39:53  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
12:39:53  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
12:39:53  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
12:39:53  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:39:53  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
12:39:53  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
12:39:53  	at java.base/java.lang.Thread.run(Thread.java:829)
12:39:53  [Bitbucket] Notifying commit build result
12:39:54  [Bitbucket] Build result notified
12:39:54  Finished: FAILURE

           

           

          SparkC added a comment - Do anyone know how to resolve this. Full Error Log: This is Happening in random places in pipeline. Sometimes pipeline works sometime pipeline fails. Not only 1 job multiple Jenkins servers and Multiple Pipelines.  12:39:53 io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS 12:39:53 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176) 12:39:53 at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.waitUntilReady(ExecWebSocketListener.java:158) 12:39:53 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:329) 12:39:53 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84) 12:39:53 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:421) 12:39:53 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:338) 12:39:53 at hudson.Launcher$ProcStarter.start(Launcher.java:507) 12:39:53 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176) 12:39:53 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132) 12:39:53 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324) 12:39:53 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) 12:39:53 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) 12:39:53 at jdk.internal.reflect.GeneratedMethodAccessor558.invoke(Unknown Source) 12:39:53 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:39:53 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 12:39:53 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) 12:39:53 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 12:39:53 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) 12:39:53 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) 12:39:53 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) 12:39:53 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) 12:39:53 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 12:39:53 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 12:39:53 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:39:53 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 12:39:53 at WorkflowScript.run(WorkflowScript:117) 12:39:53 at ___cps.transform___(Native Method) 12:39:53 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 12:39:53 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 12:39:53 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 12:39:53 at jdk.internal.reflect.GeneratedMethodAccessor278.invoke(Unknown Source) 12:39:53 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:39:53 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 12:39:53 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 12:39:53 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 12:39:53 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 12:39:53 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 12:39:53 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 12:39:53 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) 12:39:53 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) 12:39:53 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 12:39:53 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 12:39:53 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278) 12:39:53 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 12:39:53 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 12:39:53 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 12:39:53 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 12:39:53 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 12:39:53 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 12:39:53 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 12:39:53 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 12:39:53 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 12:39:53 at java.base/java.lang. Thread .run( Thread .java:829) 12:39:53 [Bitbucket] Notifying commit build result 12:39:54 [Bitbucket] Build result notified 12:39:54 Finished: FAILURE    

          Alex B added a comment -

          We are seeing this also.

          Looks like there is a default 5 second timeout from the plugin here:

          https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L78

          Try setting the system property to increase the timeout, which should be
          org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator.containerReadyTimeout=X

          Alex B added a comment - We are seeing this also. Looks like there is a default 5 second timeout from the plugin here: https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L78 Try setting the system property to increase the timeout, which should be org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator.containerReadyTimeout=X

          Alex B added a comment -

          It's also possible there is a problem with the pod, in which case increasing the timeout will not help if for example the pod never becomes ready for some reason.

          We could query/monitor the pod state/events in this case. 

          Alex B added a comment - It's also possible there is a problem with the pod, in which case increasing the timeout will not help if for example the pod never becomes ready for some reason. We could query/monitor the pod state/events in this case. 

          SparkC added a comment -

          Thanks Alex, For getting back on this, In My case container/pod is already created and i'm able to use it and at some point its getting that Error. 
          Whatever you are saying pod itself not getting into ready state right? In my case this is happening during pod running state. 

          SparkC added a comment - Thanks Alex, For getting back on this, In My case container/pod is already created and i'm able to use it and at some point its getting that Error.  Whatever you are saying pod itself not getting into ready state right? In my case this is happening during pod running state. 

          SparkC added a comment - - edited

          1 weeks back i upgraded all my 4 Jenkins servers to Latest version including plugins that i'm using. starting that all my Jenkins servers are reporting this Error. Average 2/5 attempts Jenkins agent pods are failing with this Error. 
          I tried few thing like. 
          1. Down grade Jenkins server + Plugins to previous version. 
          2. use latest Jenkins + previous stable plugin version.
          3. Delete complete Jenkins and PVC and recreate new one. 
          4. manually edit kubernetes plugin in Jenkins configuration and increase values/. 

          Nothing worked!

          SparkC added a comment - - edited 1 weeks back i upgraded all my 4 Jenkins servers to Latest version including plugins that i'm using. starting that all my Jenkins servers are reporting this Error. Average 2/5 attempts Jenkins agent pods are failing with this Error.  I tried few thing like.  1. Down grade Jenkins server + Plugins to previous version.  2. use latest Jenkins + previous stable plugin version. 3. Delete complete Jenkins and PVC and recreate new one.  4. manually edit kubernetes plugin in Jenkins configuration and increase values/.  Nothing worked!

          Alex B added a comment -

          Hi Rohith, you are right - it's not pod readiness, the code is running an exec on a running pod, ie. is running a shell script command in the target container.

          The code waitUntilReady is just waiting for the completion of a task which is the pod exec in this context.

          Alex B added a comment - Hi Rohith, you are right - it's not pod readiness, the code is running an exec on a running pod, ie. is running a shell script command in the target container. The code waitUntilReady is just waiting for the completion of a task which is the pod exec in this context.

          SparkC added a comment -

          Alex, what will be the solution here? what are action items from my end? 
          Same Setup/Jenkinsfile where working fine for more then 2years, what changed now?

          SparkC added a comment - Alex, what will be the solution here? what are action items from my end?  Same Setup/Jenkinsfile where working fine for more then 2years, what changed now?

          Alex B added a comment -

          We investigated this and found that Increasing the "connection timeout" in the Jenkins kubernetes plugin has resolved the issue for us.

          ie. Manage jenkins->manage nodes and clouds->configure clouds->k8s cloud details

          Alex B added a comment - We investigated this and found that Increasing the "connection timeout" in the Jenkins kubernetes plugin has resolved the issue for us. ie. Manage jenkins->manage nodes and clouds->configure clouds->k8s cloud details

            allan_burdajewicz Allan BURDAJEWICZ
            rohithg534 SparkC
            Votes:
            15 Vote for this issue
            Watchers:
            38 Start watching this issue

              Created:
              Updated:
              Resolved: