-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.277.2 running on GKE Kubernetes 1.18.16-gke.302
kubernetes plugin: 1.29.2
-
Powered by SuggestiMate
Since upgrading to the latest LTS version and plugin, noticed this error showing up when running the first command in container { ... } block:
[2021-04-16T21:01:40.345Z] java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error' [2021-04-16T21:01:40.345Z] at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) [2021-04-16T21:01:40.345Z] at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) [2021-04-16T21:01:40.345Z] at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [2021-04-16T21:01:40.345Z] at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [2021-04-16T21:01:40.345Z] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [2021-04-16T21:01:40.345Z] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [2021-04-16T21:01:40.345Z] at java.base/java.lang.Thread.run(Thread.java:834) io.fabric8.kubernetes.client.KubernetesClientException: container not found ("ubuntu")
I was able to reproduce it with this pipeline script:
def imageName = "ubuntu" def imageTag = "latest" def yaml = """ apiVersion: v1 kind: Pod spec: containers: - name: ${imageName} image: ${imageName}:${imageTag} command: ["/bin/sh"] args: ["-c", "sleep 5; cat"] """ Map podTemplateArgs = [:] podTemplateArgs.namespace = "jenkins" podTemplateArgs.serviceAccount = "jenkins" podTemplateArgs.yaml = yaml podTemplateArgs.idleMinutes = 1 def builders = [:] for(def i = 0; i < 10; i++) { def n = i builders["b_${n}"] = { podTemplate(podTemplateArgs) { node(POD_LABEL) { container(imageName) { sh("pwd") } } } } } parallel(builders)
I know that the container command in the yaml looks odd (normally in my pod yaml it's just "cat") but that was the best example I could come up to reproduce the problem somewhat consistently. Sometimes a I need to start more than 10 pods (even 100) sometimes just a few as 5. It does happen when using a plain "cat" as the entrypoint, but that only happens on real production builds, and it's not obvious how to reproduce those.
I suspect that Jenkins attempts to run the sh command before the container is actually ready. What's interesting is that the pod does start and the container does run (set idleMinutes and checked the container on the cluster) and if I use custom label for one of the pods than a new job will happily re-use it without failing.
I'll post more logs in a comment (from a run where 4 containers failed, did my best to select only one).
[JENKINS-65391] Container not found error although it's started in the pod
From the job: Agent k8s-stress-test-66-plx11-m7l9b-5xqf8 is provisioned from template k8s_stress-test_66-plx11-m7l9b 00:01:37 --- 00:01:37 apiVersion: "v1" 00:01:37 kind: "Pod" 00:01:37 metadata: 00:01:37 annotations: 00:01:37 buildUrl: "http://jenkins-ui:8080/job/k8s/job/stress-test/66/" 00:01:37 runUrl: "job/k8s/job/stress-test/66/" 00:01:37 labels: 00:01:37 jenkins: "slave" 00:01:37 jenkins/label-digest: "fbc268b9a0afda7fae67b5bf11e3c740197a110a" 00:01:37 jenkins/label: "k8s_stress-test_66-plx11" 00:01:37 name: "k8s-stress-test-66-plx11-m7l9b-5xqf8" 00:01:37 spec: 00:01:37 containers: 00:01:37 - args: 00:01:37 - "-c" 00:01:37 - "sleep 5; cat" 00:01:37 command: 00:01:37 - "/bin/sh" 00:01:37 image: "ubuntu:latest" 00:01:37 name: "ubuntu" 00:01:37 volumeMounts: 00:01:37 - mountPath: "/home/jenkins/agent" 00:01:37 name: "workspace-volume" 00:01:37 readOnly: false 00:01:37 - env: 00:01:37 - name: "JENKINS_SECRET" 00:01:37 value: "********" 00:01:37 - name: "JENKINS_TUNNEL" 00:01:37 value: "jenkins-agent:50000" 00:01:37 - name: "JENKINS_AGENT_NAME" 00:01:37 value: "k8s-stress-test-66-plx11-m7l9b-5xqf8" 00:01:37 - name: "JENKINS_NAME" 00:01:37 value: "k8s-stress-test-66-plx11-m7l9b-5xqf8" 00:01:37 - name: "JENKINS_AGENT_WORKDIR" 00:01:37 value: "/home/jenkins/agent" 00:01:37 - name: "JENKINS_URL" 00:01:37 value: "http://jenkins-ui:8080/" 00:01:37 image: "jenkins/inbound-agent:latest-jdk11" 00:01:37 name: "jnlp" 00:01:37 resources: 00:01:37 limits: {} 00:01:37 requests: 00:01:37 memory: "256Mi" 00:01:37 cpu: "100m" 00:01:37 volumeMounts: 00:01:37 - mountPath: "/home/jenkins/agent" 00:01:37 name: "workspace-volume" 00:01:37 readOnly: false 00:01:37 nodeSelector: 00:01:37 kubernetes.io/os: "linux" 00:01:37 restartPolicy: "Never" 00:01:37 serviceAccountName: "jenkins" 00:01:37 volumes: 00:01:37 - emptyDir: 00:01:37 medium: "" 00:01:37 name: "workspace-volume" 00:01:37 Running on k8s-stress-test-66-plx11-m7l9b-5xqf8 in /home/jenkins/agent/workspace/k8s/stress-test 00:01:40 java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error' 00:01:40 at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) 00:01:40 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) 00:01:40 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) 00:01:40 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 00:01:40 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 00:01:40 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 00:01:40 at java.base/java.lang.Thread.run(Thread.java:834) 00:01:40 Also: hudson.remoting.ProxyException: java.lang.Throwable: waiting here 00:01:40 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:154) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.waitUntilReady(ExecWebSocketListener.java:188) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:331) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:421) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:338) 00:01:40 at hudson.Launcher$ProcStarter.start(Launcher.java:508) 00:01:40 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234) 00:01:40 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103) 00:01:40 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) 00:01:40 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) 00:01:40 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) 00:01:40 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) 00:01:40 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 00:01:40 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 00:01:40 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 00:01:40 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 00:01:40 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) 00:01:40 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 00:01:40 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 00:01:40 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 00:01:40 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 Also: Also: hudson.remoting.ProxyException: java.lang.Throwable: waiting here 00:01:40 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:154) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.waitUntilReady(ExecWebSocketListener.java:188) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:331) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:421) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:338) 00:01:40 at hudson.Launcher$ProcStarter.start(Launcher.java:508) 00:01:40 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234) 00:01:40 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103) 00:01:40 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) 00:01:40 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) 00:01:40 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) 00:01:40 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) 00:01:40 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 00:01:40 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 00:01:40 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 00:01:40 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 00:01:40 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) 00:01:40 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 00:01:40 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 00:01:40 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 00:01:40 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: container not found ("ubuntu") 00:01:40 Also: Also: hudson.remoting.ProxyException: java.lang.Throwable: waiting here 00:01:40 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:154) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.waitUntilReady(ExecWebSocketListener.java:188) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:331) 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:421) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:338) 00:01:40 at hudson.Launcher$ProcStarter.start(Launcher.java:508) 00:01:40 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234) 00:01:40 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103) 00:01:40 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) 00:01:40 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) 00:01:40 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) 00:01:40 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) 00:01:40 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 00:01:40 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 00:01:40 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 00:01:40 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 00:01:40 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) 00:01:40 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 00:01:40 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 00:01:40 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 00:01:40 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: container not found ("ubuntu") 00:01:40 Also: hudson.remoting.ProxyException: hudson.AbortException: Process exited immediately after creation. Check logs above for more details. 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:459) 00:01:40 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:338) 00:01:40 at hudson.Launcher$ProcStarter.start(Launcher.java:508) 00:01:40 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234) 00:01:40 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103) 00:01:40 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:320) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) 00:01:40 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) 00:01:40 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) 00:01:40 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) 00:01:40 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) 00:01:40 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) 00:01:40 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 00:01:40 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 00:01:40 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 00:01:40 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 00:01:40 at WorkflowScript.run(WorkflowScript:31) 00:01:40 at ___cps.transform___(Native Method) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 00:01:40 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 00:01:40 at jdk.internal.reflect.GeneratedMethodAccessor299.invoke(Unknown Source) 00:01:40 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 00:01:40 at java.base/java.lang.reflect.Method.invoke(Method.java:566) 00:01:40 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 00:01:40 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 00:01:40 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 00:01:40 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) 00:01:40 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) 00:01:40 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 00:01:40 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276) 00:01:40 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 00:01:40 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 00:01:40 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 00:01:40 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 00:01:40 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 00:01:40 hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: container not found ("ubuntu") 00:01:40 at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:239) 00:01:40 at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) 00:01:40 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) 00:01:40 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) 00:01:40 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 00:01:40 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 00:01:40 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 00:01:40 at java.base/java.lang.Thread.run(Thread.java:834) 00:01:40 Finished: FAILURE
Is the container "ubuntu" available and present on your installation?
I'm afraid I don't understand what you mean by available and present on my installation? I can pull it, and I can start pods with it, jenkins can use it, runs builds on it, etc. Well, unless I hit the above case, when it fails because it considers it didn't start. In my above pipeline run, only 4 for the 10 containers started in that loop failed. Also, if I use a stable label instead of POD_LABEL (something like "pod-${n}" for that specific case) the next build will happily re-use the "just failed" container.
I suspect a race when it firsts starts - either kubernetes tells the plugin "yeah, the pod/containers are ready" or the plugin it's too eager to use them.
00:01:32 Created Pod: kubernetes jenkins/k8s-stress-test-66-plx11-m7l9b-5xqf8 00:01:32 Created Pod: kubernetes jenkins/k8s-stress-test-66-plx11-m7l9b-5xqf8 00:01:32 [Normal][jenkins/k8s-stress-test-66-plx11-m7l9b-5xqf8][Scheduled] Successfully assigned jenkins/k8s-stress-test-66-plx11-m7l9b-5xqf8 to gke 00:01:33 [Normal][jenkins/k8s-stress-test-66-plx11-m7l9b-5xqf8][Pulling] Pulling image "ubuntu:latest" 00:01:33 [Normal][jenkins/k8s-stress-test-66-wt2zv-0xpvg-x3lhm][Pulling] Pulling image "ubuntu:latest" 00:01:33 [Normal][jenkins/k8s-stress-test-66-321zn-lq772-phhxq][Pulled] Successfully pulled image "ubuntu:latest" 00:01:33 [Normal][jenkins/k8s-stress-test-66-321zn-lq772-phhxq][Created] Created container ubuntu 00:01:33 [Normal][jenkins/k8s-stress-test-66-321zn-lq772-phhxq][Started] Started container ubuntu 00:01:33 [Normal][jenkins/k8s-stress-test-66-321zn-lq772-phhxq][Pulled] Container image "jenkins/inbound-agent:latest-jdk11" already present on machine 00:01:33 [Normal][jenkins/k8s-stress-test-66-321zn-lq772-phhxq][Created] Created container jnlp 00:01:33 [Normal][jenkins/k8s-stress-test-66-5bxzd-s6gd6-21zzl][Pulled] Successfully pulled image "ubuntu:latest" 00:01:34 [Normal][jenkins/k8s-stress-test-66-321zn-lq772-phhxq][Started] Started container jnlp 00:01:34 [Normal][jenkins/k8s-stress-test-66-265n4-g32f8-d50tt][Pulled] Successfully pulled image "ubuntu:latest" 00:01:34 [Normal][jenkins/k8s-stress-test-66-wx41j-dv4m3-f9053][Pulled] Successfully pulled image "ubuntu:latest" 00:01:34 [Normal][jenkins/k8s-stress-test-66-265n4-g32f8-d50tt][Created] Created container ubuntu 00:01:34 [Normal][jenkins/k8s-stress-test-66-m2bx5-vt0gv-zstx5][Pulled] Successfully pulled image "ubuntu:latest" 00:01:34 [Normal][jenkins/k8s-stress-test-66-m2bx5-vt0gv-zstx5][Created] Created container ubuntu 00:01:34 [Normal][jenkins/k8s-stress-test-66-wx41j-dv4m3-f9053][Created] Created container ubuntu 00:01:35 [Normal][jenkins/k8s-stress-test-66-m2bx5-vt0gv-zstx5][Started] Started container ubuntu 00:01:35 [Normal][jenkins/k8s-stress-test-66-wx41j-dv4m3-f9053][Started] Started container ubuntu 00:01:35 [Normal][jenkins/k8s-stress-test-66-wx41j-dv4m3-f9053][Pulled] Container image "jenkins/inbound-agent:latest-jdk11" already present on machine 00:01:35 [Normal][jenkins/k8s-stress-test-66-m2bx5-vt0gv-zstx5][Pulled] Container image "jenkins/inbound-agent:latest-jdk11" already present on machine 00:01:35 [Normal][jenkins/k8s-stress-test-66-wx41j-dv4m3-f9053][Created] Created container jnlp 00:01:35 [Normal][jenkins/k8s-stress-test-66-m2bx5-vt0gv-zstx5][Created] Created container jnlp 00:01:35 [Normal][jenkins/k8s-stress-test-66-m2bx5-vt0gv-zstx5][Started] Started container jnlp 00:01:35 [Normal][jenkins/k8s-stress-test-66-wx41j-dv4m3-f9053][Started] Started container jnlp
+1 to the above issue, we are also experiencing the same issue, this happens randomly. We are using Jenkins 2.289.3