Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68234

Kubernetes pod failed with WebSocketHandshakeException

      I had working setup of jenkins integrated with kubernetes, suddenly i started to get below from multicontainer jobs, specifically while using "Sh" in declarative pipeline

      io.fabric8.kubernetes.client.http.WebSocketHandshakeException at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Suppressed: java.lang.Throwable: waiting here at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:168) at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179) at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332) at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331) at hudson.Launcher$ProcStarter.start(Launcher.java:508) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at jdk.internal.reflect.GeneratedMethodAccessor593.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) at jdk.internal.reflect.GeneratedMethodAccessor232.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:403) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:97) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:315) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:279) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ... 3 more Caused by: java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error' at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) ... 5 more 

       

       

       

       

       

       

       

      Note: All plugins and jenkins is in latest version

          [JENKINS-68234] Kubernetes pod failed with WebSocketHandshakeException

          Olexandr Shamin added a comment - - edited

          I am facing similar problem.

          Kubernetes plugin
          Version 3580.v78271e5631dc
          Kubernetes Client API Plugin
          Version 5.12.1-187.v577c3e368fb_6
          

          from the Log:

          12:26:03  
          12:26:03  BUILD SUCCESSFUL in 1m 4s
          12:26:03  142 actionable tasks: 36 executed, 14 from cache, 92 up-to-date
          12:26:03  Unable to list file systems to check whether they can be watched. The whole state of the virtual file system has been discarded. Reason: Could not query file systems: could not open mount file (errno 2: No such file or directory)
          12:26:03  [Pipeline] }
          12:26:06  [Pipeline] {
          12:26:06  [Pipeline] script
          12:26:06  [Pipeline] {
          12:26:06  [Pipeline] sh
          12:26:21  io.fabric8.kubernetes.client.http.WebSocketHandshakeException
          12:26:21  	at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66)
          12:26:21  	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
          12:26:21  	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
          12:26:21  	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
          12:26:21  	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
          12:26:21  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          12:26:21  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          12:26:21  	at java.lang.Thread.run(Thread.java:748)
          12:26:21  	Suppressed: java.lang.Throwable: waiting here
          12:26:21  		at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:168)
          12:26:21  		at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179)
          12:26:21  		at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332)
          12:26:21  		at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85)
          12:26:21  		at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428)
          12:26:21  		at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331)
          12:26:21  		at hudson.Launcher$ProcStarter.start(Launcher.java:509)
          12:26:21  		at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
          12:26:21  		at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
          12:26:21  		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
          12:26:21  		at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
          12:26:21  		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:26:21  		at java.lang.reflect.Method.invoke(Method.java:498)
          12:26:21  		at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
          12:26:21  		at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          12:26:21  		at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
          12:26:21  		at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
          12:26:21  		at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
          12:26:21  		at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
          12:26:21  		at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
          12:26:21  		at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
          12:26:21  		at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
          12:26:21  		at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
          12:26:21  		at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
          12:26:21  		at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
          12:26:21  		at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
          12:26:21  		at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
          12:26:21  		at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
          12:26:21  		at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
          12:26:21  		at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
          12:26:21  		at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
          12:26:21  		at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source)
          12:26:21  		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:26:21  		at java.lang.reflect.Method.invoke(Method.java:498)
          12:26:21  		at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          12:26:21  		at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
          12:26:21  		at com.cloudbees.groovy.cps.Next.step(Next.java:83)
          12:26:21  		at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
          12:26:21  		at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
          12:26:21  		at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
          12:26:21  		at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
          12:26:21  		at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
          12:26:21  		at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
          12:26:21  		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:26:21  		at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
          12:26:21  		at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          12:26:21  		at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
          12:26:21  		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          12:26:21  		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:26:21  		... 3 more
          12:26:21  Caused by: java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error'
          12:26:21  	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
          12:26:21  	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
          12:26:21  	... 5 more
          12:26:21  [Pipeline] }
          12:26:21  [Pipeline] // script
          12:26:21  [Pipeline] }
          12:26:23  [Pipeline] End of Pipeline
          12:26:23  java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error'
          12:26:23  	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
          12:26:23  	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
          12:26:23  Also:   java.lang.Throwable: waiting here
          12:26:23  	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:168)
          12:26:23  	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179)
          12:26:23  	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332)
          12:26:23  	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85)
          12:26:23  	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428)
          12:26:23  	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331)
          12:26:23  	at hudson.Launcher$ProcStarter.start(Launcher.java:509)
          12:26:23  	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
          12:26:23  	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
          12:26:23  	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
          12:26:23  	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
          12:26:23  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:26:23  	at java.lang.reflect.Method.invoke(Method.java:498)
          12:26:23  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
          12:26:23  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          12:26:23  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
          12:26:23  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
          12:26:23  	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
          12:26:23  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
          12:26:23  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
          12:26:23  	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
          12:26:23  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
          12:26:23  	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
          12:26:23  	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
          12:26:23  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
          12:26:23  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
          12:26:23  	at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source)
          12:26:23  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:26:23  	at java.lang.reflect.Method.invoke(Method.java:498)
          12:26:23  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          12:26:23  	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
          12:26:23  	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
          12:26:23  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
          12:26:23  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
          12:26:23  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
          12:26:23  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
          12:26:23  	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
          12:26:23  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:26:23  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
          12:26:23  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          12:26:23  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
          12:26:23  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          12:26:23  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:26:23  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          12:26:23  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          12:26:23  	at java.lang.Thread.run(Thread.java:748)
          12:26:23  Caused: io.fabric8.kubernetes.client.http.WebSocketHandshakeException
          12:26:23  	at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66)
          12:26:23  	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
          12:26:23  	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
          12:26:23  	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
          12:26:23  	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
          12:26:23  Caused: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
          12:26:23  	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:103)
          12:26:23  	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:97)
          12:26:23  	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:169)
          12:26:23  	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179)
          12:26:23  	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332)
          12:26:23  	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85)
          12:26:23  	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428)
          12:26:23  	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331)
          12:26:23  	at hudson.Launcher$ProcStarter.start(Launcher.java:509)
          12:26:23  	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176)
          12:26:23  	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
          12:26:23  	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
          12:26:23  	at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source)
          12:26:23  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:26:23  	at java.lang.reflect.Method.invoke(Method.java:498)
          12:26:23  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
          12:26:23  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          12:26:23  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
          12:26:23  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
          12:26:23  	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
          12:26:23  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
          12:26:23  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
          12:26:23  	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
          12:26:23  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
          12:26:23  	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
          12:26:23  	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
          12:26:23  	at WorkflowScript.run(WorkflowScript:132)
          12:26:23  	at ___cps.transform___(Native Method)
          12:26:23  	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
          12:26:23  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
          12:26:23  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
          12:26:23  	at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source)
          12:26:23  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          12:26:23  	at java.lang.reflect.Method.invoke(Method.java:498)
          12:26:23  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          12:26:23  	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
          12:26:23  	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
          12:26:23  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
          12:26:23  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
          12:26:23  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
          12:26:23  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
          12:26:23  	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
          12:26:23  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
          12:26:23  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:26:23  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
          12:26:23  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          12:26:23  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
          12:26:23  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          12:26:23  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          12:26:23  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          12:26:23  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          12:26:23  	at java.lang.Thread.run(Thread.java:748)
          12:26:23  
          12:26:23  GitHub has been notified of this commit’s build result
          12:26:23  
          12:26:23  Finished: FAILURE 

          Olexandr Shamin added a comment - - edited I am facing similar problem. Kubernetes plugin Version 3580.v78271e5631dc Kubernetes Client API Plugin Version 5.12.1-187.v577c3e368fb_6 from the Log: 12:26:03 12:26:03 BUILD SUCCESSFUL in 1m 4s 12:26:03 142 actionable tasks: 36 executed, 14 from cache, 92 up-to-date 12:26:03 Unable to list file systems to check whether they can be watched. The whole state of the virtual file system has been discarded. Reason: Could not query file systems: could not open mount file (errno 2: No such file or directory) 12:26:03 [Pipeline] } 12:26:06 [Pipeline] { 12:26:06 [Pipeline] script 12:26:06 [Pipeline] { 12:26:06 [Pipeline] sh 12:26:21 io.fabric8.kubernetes.client.http.WebSocketHandshakeException 12:26:21 at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) 12:26:21 at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) 12:26:21 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) 12:26:21 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) 12:26:21 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 12:26:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 12:26:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 12:26:21 at java.lang. Thread .run( Thread .java:748) 12:26:21 Suppressed: java.lang.Throwable: waiting here 12:26:21 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:168) 12:26:21 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179) 12:26:21 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332) 12:26:21 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) 12:26:21 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428) 12:26:21 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331) 12:26:21 at hudson.Launcher$ProcStarter.start(Launcher.java:509) 12:26:21 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176) 12:26:21 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132) 12:26:21 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324) 12:26:21 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322) 12:26:21 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124) 12:26:21 at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source) 12:26:21 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:26:21 at java.lang.reflect.Method.invoke(Method.java:498) 12:26:21 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) 12:26:21 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 12:26:21 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225) 12:26:21 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) 12:26:21 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41) 12:26:21 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) 12:26:21 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) 12:26:21 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 12:26:21 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 12:26:21 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158) 12:26:21 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 12:26:21 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 12:26:21 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:26:21 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:26:21 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 12:26:21 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 12:26:21 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 12:26:21 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 12:26:21 at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) 12:26:21 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:26:21 at java.lang.reflect.Method.invoke(Method.java:498) 12:26:21 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 12:26:21 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 12:26:21 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 12:26:21 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 12:26:21 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 12:26:21 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) 12:26:21 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) 12:26:21 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 12:26:21 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 12:26:21 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294) 12:26:21 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 12:26:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:26:21 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 12:26:21 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 12:26:21 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 12:26:21 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 12:26:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:26:21 ... 3 more 12:26:21 Caused by: java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error' 12:26:21 at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) 12:26:21 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) 12:26:21 ... 5 more 12:26:21 [Pipeline] } 12:26:21 [Pipeline] // script 12:26:21 [Pipeline] } 12:26:23 [Pipeline] End of Pipeline 12:26:23 java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error' 12:26:23 at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) 12:26:23 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) 12:26:23 Also: java.lang.Throwable: waiting here 12:26:23 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:168) 12:26:23 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179) 12:26:23 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332) 12:26:23 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) 12:26:23 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428) 12:26:23 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331) 12:26:23 at hudson.Launcher$ProcStarter.start(Launcher.java:509) 12:26:23 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176) 12:26:23 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132) 12:26:23 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324) 12:26:23 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322) 12:26:23 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124) 12:26:23 at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source) 12:26:23 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:26:23 at java.lang.reflect.Method.invoke(Method.java:498) 12:26:23 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) 12:26:23 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 12:26:23 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225) 12:26:23 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) 12:26:23 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41) 12:26:23 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) 12:26:23 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 12:26:23 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 12:26:23 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:26:23 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 12:26:23 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 12:26:23 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 12:26:23 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 12:26:23 at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) 12:26:23 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:26:23 at java.lang.reflect.Method.invoke(Method.java:498) 12:26:23 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 12:26:23 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 12:26:23 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 12:26:23 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 12:26:23 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 12:26:23 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) 12:26:23 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) 12:26:23 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 12:26:23 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 12:26:23 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 12:26:23 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:26:23 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 12:26:23 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 12:26:23 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 12:26:23 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 12:26:23 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:26:23 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 12:26:23 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 12:26:23 at java.lang. Thread .run( Thread .java:748) 12:26:23 Caused: io.fabric8.kubernetes.client.http.WebSocketHandshakeException 12:26:23 at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:66) 12:26:23 at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) 12:26:23 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) 12:26:23 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) 12:26:23 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 12:26:23 Caused: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred. 12:26:23 at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:103) 12:26:23 at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:97) 12:26:23 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:169) 12:26:23 at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:179) 12:26:23 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:332) 12:26:23 at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:85) 12:26:23 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:428) 12:26:23 at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:331) 12:26:23 at hudson.Launcher$ProcStarter.start(Launcher.java:509) 12:26:23 at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:176) 12:26:23 at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132) 12:26:23 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324) 12:26:23 at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322) 12:26:23 at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124) 12:26:23 at sun.reflect.GeneratedMethodAccessor235.invoke(Unknown Source) 12:26:23 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:26:23 at java.lang.reflect.Method.invoke(Method.java:498) 12:26:23 at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98) 12:26:23 at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) 12:26:23 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225) 12:26:23 at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034) 12:26:23 at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41) 12:26:23 at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) 12:26:23 at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163) 12:26:23 at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) 12:26:23 at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:26:23 at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135) 12:26:23 at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) 12:26:23 at WorkflowScript.run(WorkflowScript:132) 12:26:23 at ___cps.transform___(Native Method) 12:26:23 at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86) 12:26:23 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) 12:26:23 at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) 12:26:23 at sun.reflect.GeneratedMethodAccessor172.invoke(Unknown Source) 12:26:23 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 12:26:23 at java.lang.reflect.Method.invoke(Method.java:498) 12:26:23 at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) 12:26:23 at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) 12:26:23 at com.cloudbees.groovy.cps.Next.step(Next.java:83) 12:26:23 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) 12:26:23 at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) 12:26:23 at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) 12:26:23 at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) 12:26:23 at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) 12:26:23 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) 12:26:23 at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294) 12:26:23 at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) 12:26:23 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:26:23 at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) 12:26:23 at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) 12:26:23 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) 12:26:23 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 12:26:23 at java.util.concurrent.FutureTask.run(FutureTask.java:266) 12:26:23 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 12:26:23 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 12:26:23 at java.lang. Thread .run( Thread .java:748) 12:26:23 12:26:23 GitHub has been notified of this commit’s build result 12:26:23 12:26:23 Finished: FAILURE

          Uziel Sulkies added a comment -

          We also experience this error.

          I suspect it's connected with the other error we see: 

          io.fabric8.kubernetes.client.KubernetesClientException: not ready after 70000 MILLISECONDS (we increased to 70000 from 5000)

          We configured the Max connections to kubernetes API to 10000 (although no clear way I know to make sure the configuration in the UI is the one used by the plugin).

          We can't find any 500 response code on the Kubernetes API logs (on cloudwatch, for EKS), though.

          Uziel Sulkies added a comment - We also experience this error. I suspect it's connected with the other error we see:  io.fabric8.kubernetes.client.KubernetesClientException: not ready after 70000 MILLISECONDS (we increased to 70000 from 5000) We configured the Max connections to kubernetes API to 10000 (although no clear way I know to make sure the configuration in the UI is the one used by the plugin). We can't find any 500 response code on the Kubernetes API logs (on cloudwatch, for EKS), though.

          Allan BURDAJEWICZ added a comment - - edited

          Since an error 500 is reported, there should be some some error showing up in the kube-apiserver logs or maybe the kubelet logs of the node where the agent pod is running.

          Do you have a simple reproduction scenario ? (simple pipeline script, pod yaml)

          Allan BURDAJEWICZ added a comment - - edited Since an error 500 is reported, there should be some some error showing up in the kube-apiserver logs or maybe the kubelet logs of the node where the agent pod is running. Do you have a simple reproduction scenario ? (simple pipeline script, pod yaml)

          Nagavijayan N added a comment -

          I have misconfigured defaultcontainer in agent block, Its resolved after using right container name from pod template.

          Nagavijayan N added a comment - I have misconfigured defaultcontainer in agent block, Its resolved after using right container name from pod template.

          Nagavijayan N added a comment -

          Please raise appropriate error msg while using container name which is not available in pod template.

          Nagavijayan N added a comment - Please raise appropriate error msg while using container name which is not available in pod template.

          nagavijayan 

          Did you solve this?

          Francisco Aguiar added a comment - nagavijayan   Did you solve this?

          Felipe Radins added a comment -

          Hello everyone. I'm having the same issue, and I found a possible, but inappropriate solution.

          My scenario is a brand new kubernetes cluster, with one master node, and two workers nodes. All of them running with k3s(v1.23.8+k3s1). Jenkins runs inside one of the nodes.

          Jenkins: 2.346.2

          Kubernetes plugin: 3663.v1c1e0ec5b_650

          Kubernetes Client API Plugin: 5.12.2-193.v26a_6078f65a_9

           

          First, I noted that the  error

          Expected HTTP 101 response but was '500 Internal Server Error'
          

          occurs only when running 'sh' command. So here is a jenkinsfile I used for test:

          podTemplate(containers: [
              containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true)]) {
                  node (POD_LABEL) {
                      stage ('foo') {
                          container('docker') {
                          while (true) {
                              sh "echo foo"   
                              sh "sleep 1"
                          }
                      }
                  }
              }
          }
          

          with this scenario, it will certainly give the above mentioned error in a matter of seconds.

          Secondly, I noted that the jnlp container doesn't display any error message, so I went to look for something on my kubernetes cluster.
          To do it, I ran the command 'journalctl -n 20' on the worker node, where the jenkins agent was been executed, after the build fail.
          This gave me an interesting error message, which was the following:

          Jul 15 14:19:32 k3s-worker-node02 k3s[803]: I0715 14:19:32.912086     803 log.go:184] http: TLS handshake error from 127.0.0.1:43742: tls: first record does not look like a TLS handshake
          

          note:

          • the port after the ip isn't any fixed port on my cluster, so i assume it's some container port (or the docker container, or the jnlp container, but I might be wrong)

           

          Honestly, this didn't gave me much of a clue of what the error could be. But i assume it might be something important...

           

          Well, after some more research, I tried a new scenario, instead of using an 'external' container to run my build, I tried using the default jnlp container, so my jenkinsfile looked something like this:

          podTemplate() {
              node (POD_LABEL) {
                  stage ('foo') {
                      while (true) {
                          sh "echo foo"   
                          sh "sleep 1"
                      }
                  }
              }
          }
          

          and now, for some reason, the build won't fail

           

          After this I've made the following tests:
           1. Overwrite the default jnlp container, and use resources to allocate more resources to de container. This didn't work

           
          podTemplate(yaml: """
          apiVersion: v1
          kind: Pod
          spec:
            containers:
            - name: jnlp
              image: 'jenkins/inbound-agent:4.7-1'
              args: ['\$(JENKINS_SECRET)', '\$(JENKINS_NAME)']
              resources:
                requests:
                  memory: "2048Mi"
                  cpu: "2500m"
                limits:
                  memory: "2048Mi"
                  cpu: "2500m"
            - name: docker
              image: docker
              args: ["cat"]
              tty: true     
          """
          ) {
              node (POD_LABEL) {
                      stage ('foo') {
                          container('docker') {
                          while (true) {
                              sh "echo foo"   
                              sh "sleep 1"
                          }
                      }
                  }
              }
          } 

          2. Overwrite de default jnlp image with my own. This worked

             I've created a dockerfile somewhat like the following:

          FROM jenkins/inbound-agent
          
          USER root
          
          RUN apt-get update
          RUN apt-get upgrade -y
          
          #
          # Docker
          #
          RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
          RUN apt-key fingerprint 0EBFCD88
          RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
          RUN apt-get update
          RUN apt-get install -y docker-ce docker-ce-cli containerd.io
          
          
          RUN echo '/usr/local/bin/jenkins-agent $1 $2' > /init.sh
          
          WORKDIR /app/src/
          
          ENTRYPOINT ["sh"]
          

          after building a pushing it to dockerhub, I re-run my build using this jenkinsfile:

          properties([disableConcurrentBuilds()])
          podTemplate(yaml: """
          apiVersion: v1
          kind: Pod
          spec:
            containers:
            - name: jnlp
              image: 'my-custom-image'
              args: ["/init.sh", '\$(JENKINS_SECRET)', '\$(JENKINS_NAME)']
              tty: true
           """ ) {
              node (POD_LABEL) {
                  stage ('foo') {
                      while (true) {
                          sh "echo foo"   
                          sh "sleep 1"
                      }
                  }
              }
          } 

          This time, everything worked properly.

           

          So here is my conclusion:

          • The problem is with running a second container in the build.
          • A possible solution could be, as demonstrated, create your own jnlp.
            But as described in the Kubernetes plugin for Jenkins page, THIS IS NOT RECOMENDED : 'We do not recommend overriding the jnlp container except under unusual circumstances.'
          • I don't know the reason this issue is happening, and i don't think my solution should be used. 

          Felipe Radins added a comment - Hello everyone. I'm having the same issue, and I found a possible, but inappropriate solution. My scenario is a brand new kubernetes cluster, with one master node, and two workers nodes. All of them running with k3s(v1.23.8+k3s1). Jenkins runs inside one of the nodes. Jenkins: 2.346.2 Kubernetes plugin: 3663.v1c1e0ec5b_650 Kubernetes Client API Plugin: 5.12.2-193.v26a_6078f65a_9   First, I noted that the  error Expected HTTP 101 response but was '500 Internal Server Error' occurs only when running 'sh' command. So here is a jenkinsfile I used for test: podTemplate(containers: [ containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true)]) { node (POD_LABEL) { stage ('foo') { container('docker') { while (true) { sh "echo foo" sh "sleep 1" } } } } } with this scenario, it will certainly give the above mentioned error in a matter of seconds. Secondly, I noted that the jnlp container doesn't display any error message, so I went to look for something on my kubernetes cluster. To do it, I ran the command 'journalctl -n 20' on the worker node, where the jenkins agent was been executed , after the build fail. This gave me an interesting error message, which was the following: Jul 15 14:19:32 k3s-worker-node02 k3s[803]: I0715 14:19:32.912086 803 log.go:184] http: TLS handshake error from 127.0.0.1:43742: tls: first record does not look like a TLS handshake note: the port after the ip isn't any fixed port on my cluster, so i assume it's some container port (or the docker container, or the jnlp container, but I might be wrong)   Honestly, this didn't gave me much of a clue of what the error could be. But i assume it might be something important...   Well, after some more research, I tried a new scenario, instead of using an 'external' container to run my build, I tried using the default jnlp container, so my jenkinsfile looked something like this: podTemplate() { node (POD_LABEL) { stage ('foo') { while (true) { sh "echo foo" sh "sleep 1" } } } } and now, for some reason, the build won't fail .    After this I've made the following tests:  1. Overwrite the default jnlp container, and use resources to allocate more resources to de container. This didn't work   podTemplate(yaml: """ apiVersion: v1 kind: Pod spec:   containers:   - name: jnlp     image: 'jenkins/inbound-agent:4.7-1'     args: ['\$(JENKINS_SECRET)', '\$(JENKINS_NAME)']     resources:       requests:         memory: "2048Mi"         cpu: "2500m"       limits:         memory: "2048Mi"         cpu: "2500m"   - name: docker     image: docker     args: ["cat"]     tty: true      """ ) {     node (POD_LABEL) {             stage ('foo') {                 container('docker') {                 while (true) {                     sh "echo foo"                        sh "sleep 1"                 }             }         }     } } 2. Overwrite de default jnlp image with my own. This worked    I've created a dockerfile somewhat like the following: FROM jenkins/inbound-agent USER root RUN apt-get update RUN apt-get upgrade -y # # Docker # RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - RUN apt-key fingerprint 0EBFCD88 RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" RUN apt-get update RUN apt-get install -y docker-ce docker-ce-cli containerd.io RUN echo '/usr/local/bin/jenkins-agent $1 $2' > /init.sh WORKDIR /app/src/ ENTRYPOINT ["sh"] after building a pushing it to dockerhub, I re-run my build using this jenkinsfile: properties([disableConcurrentBuilds()]) podTemplate(yaml: """ apiVersion: v1 kind: Pod spec: containers: - name: jnlp image: 'my-custom-image' args: ["/init.sh", '\$(JENKINS_SECRET)', '\$(JENKINS_NAME)'] tty: true """ ) {     node (POD_LABEL) {         stage ('foo') {             while (true) {                 sh "echo foo"                    sh "sleep 1"             }         }     } } This time, everything worked properly.   So here is my conclusion: The problem is with running a second container in the build. A possible solution could be, as demonstrated, create your own jnlp. But as described in the Kubernetes plugin for Jenkins page, THIS IS NOT RECOMENDED : 'We do not recommend overriding the jnlp container except under unusual circumstances.' I don't know the reason this issue is happening, and i don't think my solution should be used. 

          Fred G added a comment - - edited

          Several Eclipse projects report the same issue, for example Jakarta EE TCK, see https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1685 .

           

          This was first reported with the following versions:

          Jenkins                            2.332.3
          kubernetes plugin                  3580.v78271e5631dc
          kubernetes-client-api plugin       5.12.1-187.v577c3e368fb_6
          

           

          It also appears with:

          Jenkins                            2.346.3
          kubernetes plugin                  3697.v771155683e38
          kubernetes-client-api plugin       5.12.2-193.v26a_6078f65a_9

           

          This potentially affects all (~250) Eclipse projects that use Jenkins instances hosted at the Eclipse Foundation here https://ci.eclipse.org.

           

          vlatombe can you take a look at this?

          Fred G added a comment - - edited Several Eclipse projects report the same issue, for example Jakarta EE TCK, see https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/1685 .   This was first reported with the following versions: Jenkins 2.332.3 kubernetes plugin 3580.v78271e5631dc kubernetes-client-api plugin 5.12.1-187.v577c3e368fb_6   It also appears with: Jenkins 2.346.3 kubernetes plugin               3697.v771155683e38 kubernetes-client-api plugin    5.12.2-193.v26a_6078f65a_9   This potentially affects all (~250) Eclipse projects that use Jenkins instances hosted at the Eclipse Foundation here https://ci.eclipse.org .   vlatombe can you take a look at this?

          Vincent Latombe added a comment - - edited

          HTTP 500 error indicates an internal server error has happened. So you should check logs of the kube apiserver at the same time to understand more about the issue.

          Another possible problem would be that the container that you want to execute commands into doesn't exist, and for some reason your Kubernetes issues a 500 error instead of a 400.

          To confirm that, you would need to keep pods around using pod retention settings for example and inspect them to verify they have the expected containers declared.

          Vincent Latombe added a comment - - edited HTTP 500 error indicates an internal server error has happened. So you should check logs of the kube apiserver at the same time to understand more about the issue. Another possible problem would be that the container that you want to execute commands into doesn't exist, and for some reason your Kubernetes issues a 500 error instead of a 400. To confirm that, you would need to keep pods around using pod retention settings for example and inspect them to verify they have the expected containers declared.

            kylecronin Kyle Cronin
            nagavijayan Nagavijayan N
            Votes:
            8 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: