Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53297

Pod name is detected as `localhost` in Bluemix IKS

      I've Jenkins running on Kubernetes Cluster (Bluemix IKS).

      I am using the kubernetes plugin to create dynamic agents.

      The jenkins connection with agent is made in the standard way:

      • Jenkins URL: jenkins:8080
      • Jenkins Tunnel: jenkins-agent:50000

       

      When i start a job, the agent connect with Jenkins master successfully. But he keeps long time and not execute pipeline steps.

       

      Agent logs: 

      Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behaviorWarning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behaviorAug 29, 2018 12:07:02 AM hudson.remoting.jnlp.Main createEngineINFO: Setting up agent: jenkins-slave-n39mq-9zgzzAug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener <init>INFO: Jenkins agent is running in headless mode.Aug 29, 2018 12:07:03 AM hudson.remoting.Engine startEngineINFO: Using Remoting version: 3.23Aug 29, 2018 12:07:03 AM hudson.remoting.Engine startEngineWARNING: No Working Directory. Using the legacy JAR Cache location: /home/jenkins/.jenkins/cache/jarsAug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [http://jenkins:8080/]Aug 29, 2018 12:07:03 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]Aug 29, 2018 12:07:03 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability checkAug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Agent discovery successful  Agent address: jenkins-agent  Agent port:    50000  Identity:      bb:d3:5f:09:02:5c:56:70:04:7f:8f:72:f5:bd:23:79Aug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: HandshakingAug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Connecting to jenkins-agent:50000Aug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Trying protocol: JNLP4-connectAug 29, 2018 12:07:03 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Remote identity confirmed: bb:d3:5f:09:02:5c:56:70:04:7f:8f:72:f5:bd:23:79Aug 29, 2018 12:07:04 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: ConnectedAug 29, 2018 12:12:09 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector callINFO: Disabled slave engine reconnects.Aug 29, 2018 12:12:09 AM hudson.util.ProcessTree getWARNING: Error while determining if vetoers existhudson.remoting.RequestAbortedException: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. ChannelApplicationLayer reports that the NIO Channel is closed at hudson.remoting.Request.abort(Request.java:340) at hudson.remoting.Channel.terminate(Channel.java:1038) at hudson.remoting.Channel.close(Channel.java:1439) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1270) at hudson.remoting.Channel$1.handle(Channel.java:565) at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:203) at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:187) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:207) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669) at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-agent/172.21.160.215:50000 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741) at hudson.remoting.Request.call(Request.java:202) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) at com.sun.proxy.$Proxy5.fetch3(Unknown Source) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:209) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at hudson.util.ProcessTree.get(ProcessTree.java:399) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 4 moreCaused by: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. ChannelApplicationLayer reports that the NIO Channel is closed at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:302) at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304) at hudson.remoting.Channel.send(Channel.java:721) at hudson.remoting.Channel.close(Channel.java:1436) ... 18 moreCaused by: java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:694) at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:157) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:299) ... 21 more
      Aug 29, 2018 12:12:09 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: TerminatedAug 29, 2018 12:12:09 AM hudson.util.ProcessTree getWARNING: Failed to load winp. Reverting to the defaultjava.lang.NoClassDefFoundError: hudson/util/ProcessTree$Linux$LinuxProcess at hudson.util.ProcessTree.get(ProcessTree.java:415) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1090) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1081) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$Linux$LinuxProcess at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 12 more
      

       

      Jenkins Master logs

      Aug 29, 2018 12:06:51 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provisionAug 29, 2018 12:06:51 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provisionINFO: Excess workload after pending Kubernetes agents: 1Aug 29, 2018 12:06:51 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provisionINFO: Template: Kubernetes Pod TemplateAug 29, 2018 12:06:51 AM okhttp3.internal.platform.Platform logINFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?Aug 29, 2018 12:06:51 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl applyINFO: Started provisioning Kubernetes Pod Template from devops-k8s with 1 executors. Remaining excess workload: 0Aug 29, 2018 12:07:01 AM hudson.slaves.NodeProvisioner$2 runINFO: Kubernetes Pod Template provisioning successfully completed. We have now 2 computer(s)Aug 29, 2018 12:07:01 AM okhttp3.internal.platform.Platform logINFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?Aug 29, 2018 12:07:01 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launchINFO: Created Pod: jenkins-slave-n39mq-9zgzz in namespace jenkinsAug 29, 2018 12:07:01 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launchINFO: Waiting for Pod to be scheduled (0/100): jenkins-slave-n39mq-9zgzzAug 29, 2018 12:07:03 AM hudson.TcpSlaveAgentListener$ConnectionHandler runINFO: Accepted JNLP4-connect connection #3 from /172.30.89.194:44930Aug 29, 2018 12:07:08 AM okhttp3.internal.platform.Platform logINFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?Aug 29, 2018 12:12:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminateINFO: Terminating Kubernetes instance for agent jenkins-slave-n39mq-9zgzzAug 29, 2018 12:12:09 AM okhttp3.internal.platform.Platform logINFO: ALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path?Aug 29, 2018 12:12:09 AM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosedWARNING: Computer.threadPoolForRemoting [#88] for jenkins-slave-n39mq-9zgzz terminatedjava.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800) at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314) at hudson.remoting.Channel.close(Channel.java:1450) at hudson.remoting.Channel.close(Channel.java:1403) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:799) at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:103) at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:715) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Terminated Kubernetes instance for agent jenkins/jenkins-slave-n39mq-9zgzzAug 29, 2018 12:12:09 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePodINFO: Terminated Kubernetes instance for agent jenkins/jenkins-slave-n39mq-9zgzzDisconnected computer jenkins-slave-n39mq-9zgzzAug 29, 2018 12:12:09 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminateINFO: Disconnected computer jenkins-slave-n39mq-9zgzzAug 29, 2018 12:12:09 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.PodTemplateStepExecution$PodTemplateCallback finishedINFO: Removing pod template jenkins-slave-n39mq from cloud devops-k8sAug 29, 2018 12:12:10 AM org.jenkinsci.plugins.workflow.job.WorkflowRun finishINFO: teste-deploy #45 completed: FAILURE
      

       

       

      Console Output 

      Started by user Vinicius Xavier Running in Durability level: MAX_SURVIVABILITY [Pipeline] podTemplate [Pipeline] { [Pipeline] node Still waiting to schedule task There are no nodes with the label ‘jenkins-pipeline’ Agent jenkins-slave-n39mq-9zgzz is provisioned from template Kubernetes Pod Template Agent specification [Kubernetes Pod Template] (jenkins-pipeline): * [helm] lachlanevenson/k8s-helm Running on jenkins-slave-n39mq-9zgzz in /home/jenkins/workspace/teste-deploy [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] sh [teste-deploy] Running shell script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.waitUntilExists(BaseOperation.java:959) at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:219) at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:37) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:417) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:449) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810) at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810) at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) Caused: java.io.IOException: Failed to execute shell script inside container [helm] of pod [localhost]. Timed out waiting for container to become ready! at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:438) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:449) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810) at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810) at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46) at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) at WorkflowScript.run(WorkflowScript:10) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor375.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
      

       

      Jenkinsfile

      podTemplate(label: 'jenkins-pipeline', cloud: 'devops-k8s', 
        containers: [
          containerTemplate(name: 'helm', image: 'lachlanevenson/k8s-helm', command: 'cat', ttyEnabled: true)], 
        volumes:[ 
          hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')]
      ){ 
      node ('jenkins-pipeline') { 
        container('helm') { 
          sh 'helm init --client-only' sh 'helm ls --namespace default' 
        } 
       } 
      }
      

       

      Obs.: This happened after update Kubernetes 1.10 to 1.11. IKS changed RuntimeContainer from Docker to Containerd
       

          [JENKINS-53297] Pod name is detected as `localhost` in Bluemix IKS

          Vinicius Xavier created issue -

          I actived the ALL log level and caught this:

          <-- 404 Not Found https://kubernetes.default/api/v1/namespaces/jenkins/pods/localhost (3ms)Aug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logContent-Type: application/jsonAug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logDate: Wed, 29 Aug 2018 18:32:10 GMTAug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logContent-Length: 186Aug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logAug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform log{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"localhost\" not found","reason":"NotFound","details":{"name":"localhost","kind":"pods"},"code":404}
          

          Why jenkins using 'localhost' with name of the pod???

          Vinicius Xavier added a comment - I actived the ALL log level and caught this: <-- 404 Not Found https: //kubernetes. default /api/v1/namespaces/jenkins/pods/localhost (3ms)Aug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logContent-Type: application/jsonAug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logDate: Wed, 29 Aug 2018 18:32:10 GMTAug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logContent-Length: 186Aug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform logAug 29, 2018 6:32:10 PM INFO okhttp3.internal.platform.Platform log{ "kind" : "Status" , "apiVersion" : "v1" , "metadata" :{}, "status" : "Failure" , "message" : "pods \" localhost\ " not found" , "reason" : "NotFound" , "details" :{ "name" : "localhost" , "kind" : "pods" }, "code" :404} Why jenkins using 'localhost' with name of the pod???

          what are the log lines before that one

          Carlos Sanchez added a comment - what are the log lines before that one

          I noticed that when using for example: 'echo hi', the pipeline runs successfully, jenkins GET pods/<slave name> endpoint.
          But, when using any command 'sh "cat /etc/hosts" ', jenkins try GET pods/localhost endpoint

          --> GET https://kubernetes.default/api/v1/namespaces/jenkins/pods/localhost http/1.1
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJqZW5raW5zIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImplbmtpbnMtdG9rZW4tZnFsZmMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiamVua2lucyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjM3NzU1YTY1LTk1YmEtMTFlOC04MzlkLTJhZGJhODJkZmMxMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpqZW5raW5zOmplbmtpbnMifQ.mfHpadNaF6JhwS7YL171TBg1J_RxOQJwfcyAYUHDjPnc3VI_E6_LwXZ9CRJIIXxtcN7Ny8EWwL09EonaFq20LfZNK5TQaYbu6FIyipMNGaZ1JxA-vPZ-ZYghxOJcbK36JK_ap71g9Jiun7qUVbqGo39yuF6Ga-ELcDil30BOX3FMrZiAH3D6yTy0FUKMLt81KnCzwUEdyKSOO86Z6nNzzZTQ8qw_lVJOJbUPjicbMcOLz1yp19UW8sFYr2smY-BSv6oeHkoXCfbccKu7UsRFSxFKnFOK-5bv8TBwuZm8Wkm93RVUg9dNhrGnYUI5RtzJ1i2ntjqyBSBdYDeinYz5VPPTRhK7zG_k8bmIsw1OAiUdKEQZIsON1EaNYxFEwJ2yd3byQ5N-BUvkfEbOXwM6pWbKDM_RjEfZcFcSXr2XOC_1q6vbwaWBLKtEyD_xvz_ZLq7dbIU5N-FYm8NwAtSYcyxssnHleOqpUfUcDvVepkXJySMs__aarzMjdQYqLhpoCDQtiAZUq6LyFRtW3AezOClj1vweUoUKVx5KCyHm3meEKFc38TUUn9BkYY3lMlz4Q579TLzKzlTmQSFy9sdY1eSvQlikzLx-VSCb5aSc9D6xrINbky0tDmFH1uM7eT7oNRE9ZphctWADz3kZrP_cU-EWW4Ja798jpln93YgrFpY
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Host: kubernetes.default
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Connection: Keep-Alive
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Accept-Encoding: gzip
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          User-Agent: okhttp/3.9.1
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          --> END GET
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          <-- 404 Not Found https://kubernetes.default/api/v1/namespaces/jenkins/pods/localhost (3ms)
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Content-Type: application/json
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Date: Wed, 29 Aug 2018 19:28:34 GMT
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Content-Length: 186
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"localhost\" not found","reason":"NotFound","details":{"name":"localhost","kind":"pods"},"code":404}
          
          Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log
          <-- END HTTP (186-byte body)
          

          Vinicius Xavier added a comment - I noticed that when using for example: 'echo hi', the pipeline runs successfully, jenkins GET pods/<slave name> endpoint. But, when using any command 'sh "cat /etc/hosts" ', jenkins try GET pods/localhost endpoint --> GET https: //kubernetes. default /api/v1/namespaces/jenkins/pods/localhost http/1.1 Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJqZW5raW5zIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImplbmtpbnMtdG9rZW4tZnFsZmMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiamVua2lucyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjM3NzU1YTY1LTk1YmEtMTFlOC04MzlkLTJhZGJhODJkZmMxMSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpqZW5raW5zOmplbmtpbnMifQ.mfHpadNaF6JhwS7YL171TBg1J_RxOQJwfcyAYUHDjPnc3VI_E6_LwXZ9CRJIIXxtcN7Ny8EWwL09EonaFq20LfZNK5TQaYbu6FIyipMNGaZ1JxA-vPZ-ZYghxOJcbK36JK_ap71g9Jiun7qUVbqGo39yuF6Ga-ELcDil30BOX3FMrZiAH3D6yTy0FUKMLt81KnCzwUEdyKSOO86Z6nNzzZTQ8qw_lVJOJbUPjicbMcOLz1yp19UW8sFYr2smY-BSv6oeHkoXCfbccKu7UsRFSxFKnFOK-5bv8TBwuZm8Wkm93RVUg9dNhrGnYUI5RtzJ1i2ntjqyBSBdYDeinYz5VPPTRhK7zG_k8bmIsw1OAiUdKEQZIsON1EaNYxFEwJ2yd3byQ5N-BUvkfEbOXwM6pWbKDM_RjEfZcFcSXr2XOC_1q6vbwaWBLKtEyD_xvz_ZLq7dbIU5N-FYm8NwAtSYcyxssnHleOqpUfUcDvVepkXJySMs__aarzMjdQYqLhpoCDQtiAZUq6LyFRtW3AezOClj1vweUoUKVx5KCyHm3meEKFc38TUUn9BkYY3lMlz4Q579TLzKzlTmQSFy9sdY1eSvQlikzLx-VSCb5aSc9D6xrINbky0tDmFH1uM7eT7oNRE9ZphctWADz3kZrP_cU-EWW4Ja798jpln93YgrFpY Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Host: kubernetes. default Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Connection: Keep-Alive Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Accept-Encoding: gzip Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log User-Agent: okhttp/3.9.1 Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log --> END GET Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log <-- 404 Not Found https: //kubernetes. default /api/v1/namespaces/jenkins/pods/localhost (3ms) Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Content-Type: application/json Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Date: Wed, 29 Aug 2018 19:28:34 GMT Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Content-Length: 186 Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log { "kind" : "Status" , "apiVersion" : "v1" , "metadata" :{}, "status" : "Failure" , "message" : "pods \" localhost\ " not found" , "reason" : "NotFound" , "details" :{ "name" : "localhost" , "kind" : "pods" }, "code" :404} Aug 29, 2018 7:28:34 PM INFO okhttp3.internal.platform.Platform log <-- END HTTP (186- byte body)
          Vinicius Xavier made changes -
          Attachment New: jenkins-master.log [ 43949 ]

          I attached a log, from the beginning of the job to the first 404 in the endpoint pods / localhost.

          Vinicius Xavier added a comment - I attached a log, from the beginning of the job to the first 404 in the endpoint pods / localhost.

          leon gregori added a comment - - edited

          Hello,

          I have also been using the Jenkins Kubernetes Plugin 1.12.4 on IBM Cloud.
          It also stopped working over the Weekend and im currently in investigation.

          Job output is this, but i will provide more soon. As soon as I can identify the issue and see if it is another root cause, I will create a seperate issue.

          java.io.IOException: container [java] does not exist in pod [jenkins-slave-sm0j8-666ng]
           at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:435)
           at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255)
           at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)
           at hudson.Launcher$ProcStarter.start(Launcher.java:454)
           at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186)
           at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86)
           at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182)
           at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
           at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
           at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
           at sun.reflect.GeneratedMethodAccessor906.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
           at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
           at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
           at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
           at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
           at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
           at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
           at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
           at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
           at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
           at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
           at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
           at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
           at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
           at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
           at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
           at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
           at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
           at WorkflowScript.run(WorkflowScript:44)
           at __cps.transform__(Native Method)
           at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
           at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
           at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
           at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
           at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
           at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
           at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
           at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
           at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
           at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
           at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
           at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
           at com.cloudbees.groovy.cps.Next.step(Next.java:83)
           at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
           at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
           at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
           at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
           at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
           at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
           at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
           at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
           at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
           at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
           at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
           at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
           Finished: FAILURE

          leon gregori added a comment - - edited Hello, I have also been using the Jenkins Kubernetes Plugin 1.12.4 on IBM Cloud. It also stopped working over the Weekend and im currently in investigation. Job output is this, but i will provide more soon. As soon as I can identify the issue and see if it is another root cause, I will create a seperate issue. java.io.IOException: container [java] does not exist in pod [jenkins-slave-sm0j8-666ng] at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:435) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:454) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor906.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at WorkflowScript.run(WorkflowScript:44) at __cps.transform__(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Finished: FAILURE

          Alok Rajiv added a comment - - edited

          Hi ,

          So, I think what everyone is facing here is an issue stemming from the runtime dependency change. Thanks to @rtheis from IBM Cloud Network Dev Team on Slack who reminded me of the change.

          IKS on K8S v1.11 now uses containerd runtime instead of docker runtime. I fear this is why our container state cannot be ready by the master. I experimented on 2 new fresh clusters with 1.10 and 1.11 on IBM Cloud and do confirm that 1.10 works just fine and replicated the above issue on 1.11 (The docker mount itself is failing on 1.11)

           IBM migration doc link: https://console.bluemix.net/docs/containers/cs_versions.html#containerd

           

          Do correct me if anyone has another side to this story.

          Alok Rajiv added a comment - - edited Hi , So, I think what everyone is facing here is an issue stemming from the runtime dependency change. Thanks to @rtheis from IBM Cloud Network Dev Team on Slack who reminded me of the change. IKS on K8S v1.11 now uses containerd   runtime instead of docker runtime. I fear this is why our container state cannot be ready by the master. I experimented on 2 new fresh clusters with 1.10 and 1.11 on IBM Cloud and do confirm that 1.10 works just fine and replicated the above issue on 1.11 (The docker mount itself is failing on 1.11)  IBM migration doc link:  https://console.bluemix.net/docs/containers/cs_versions.html#containerd   Do correct me if anyone has another side to this story.

          leon gregori added a comment -

          alokrajiv yes that is the root cause if using IKS 1.11.

          In Jenkins Pipeline it should be just fine to mount containerd socket instead of docker socket.
          I could not yet test this as our cluster is not updated yet to containerd.

          I will keep you updated here.

          Currently (Working on old IBM IKS)

              volumes: [  hostPathVolume(hostPath: '/etc/docker/certs.d', mountPath: '/etc/docker/certs.d'),
                          hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: '/var/run/docker.sock')]
          

          Possible fix (Working on new containerd IBM IKS)

              volumes: [  hostPathVolume(hostPath: '/etc/docker/certs.d', mountPath: 'Certificates'),
                          hostPathVolume(hostPath: '/var/run/docker.sock', mountPath: 'SOCKET FOR CONTAINERD')]

           

          leon gregori added a comment - alokrajiv yes that is the root cause if using IKS 1.11. In Jenkins Pipeline it should be just fine to mount containerd socket instead of docker socket. I could not yet test this as our cluster is not updated yet to containerd. I will keep you updated here. Currently (Working on old IBM IKS) volumes: [ hostPathVolume(hostPath: '/etc/docker/certs.d' , mountPath: '/etc/docker/certs.d' ), hostPathVolume(hostPath: '/ var /run/docker.sock' , mountPath: '/ var /run/docker.sock' )] Possible fix (Working on new containerd IBM IKS) volumes: [ hostPathVolume(hostPath: '/etc/docker/certs.d' , mountPath: 'Certificates' ), hostPathVolume(hostPath: '/ var /run/docker.sock' , mountPath: 'SOCKET FOR CONTAINERD' )]  
          leon gregori made changes -
          Environment Original: Jenkins 2.121.3
          Kubernetes IKS 1.11 (Bluemix)
          Kubernetes Plugin 1.12.4
          New: Jenkins 2.121.3, 2.141
          Kubernetes IKS 1.11 (Bluemix)
          Kubernetes Plugin 1.12.4

            vlatombe Vincent Latombe
            vxavier Vinicius Xavier
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: