-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Kubernetes 1.9
Jenkins ver. 2.121.3
Kubernetes Pluin 1.12.4
The problem is that I need to configure timeout time for fabric9 Kubernetes API to access the Kubernetes API. The Pod is created and is running, but in some cases fabric9 Kubernetes API takes a little bit more time to access Kubernetes, resulting in the described error.
Looking in the code, there should be a default timeout of 5 minutes.
Since CONTAINER_READY_TIMEOUT=5 and TimeUnit.MINUTES.
But the Pipeline is definitely not waiting 5 minutes.
I know that my setup is working, because the Pipeline works 9/10 times. It just throws this error in the rare cases if Kubernetes needs a little bit longer to respond.
The step takes 1min 9s before the exception occurs. The Pipeline have a timeout of 1 hour.
// code placeholder java.net.SocketException: Socket closed at java.net.SocketInputStream.read(SocketInputStream.java:204) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) Caused: java.net.SocketTimeoutException: timeout at okio.Okio$4.newTimeoutException(Okio.java:230) at okio.AsyncTimeout.exit(AsyncTimeout.java:285) at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:104) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:56) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:313) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:296) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:780) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:196) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:163) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.waitUntilExists(BaseOperation.java:953) at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:219) at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:37) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:417) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:449) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor1651.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [maven-3-jdk-8-hgdv3] in namespace: [jenkins] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:207) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:163) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.waitUntilExists(BaseOperation.java:953) at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:219) at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:37) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:417) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:449) at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186) at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor1651.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157) at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17) at WorkflowScript.run(WorkflowScript:73) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor207.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Finished: FAILURE
- relates to
-
-
- Open
-
- links to
[JENKINS-53532] Kubernetes java.net.SocketTimeoutException: timeout
Daniel Watrous
added a comment - I created PR https://github.com/jenkinsci/kubernetes-plugin/pull/442 to fix the issue of these always reverting to zero.
Daniel Watrous
added a comment - I created PR https://github.com/jenkinsci/kubernetes-plugin/pull/442 to fix the issue of these always reverting to zero. 
Matthew Ludlum
added a comment - I'm guessing a change in the fabric8 client 4.1.2 upgrade which included okHttp 3.12.0 caused this regression to pop up in 1.14.8 vs . prior versions.
Matthew Ludlum
added a comment - I'm guessing a change in the fabric8 client 4.1.2 upgrade which included okHttp 3.12.0 caused this regression to pop up in 1.14.8 vs . prior versions. **setting the timeout will probably not fix the ping issue**
To confirm, this is likely an issue with updating to the latest fabric8 client, which went from 3.9.1 to 3.12.0 of okhttp as of the latest release.
Starting with okhttp 3.10(https://github.com/square/okhttp/blob/master/CHANGELOG.md#version-3100 )- it started aggressively checking connectivity with the pingInterval as well.
To fix this, adding -Dkubernetes.websocket.ping.interval=<number> to the jenkins startup or system file should change the ping interval.
The okhttp docs recommend 30s :
If you have a configured ping interval, you should confirm that it is long enough for a roundtrip from client to server. If your ping interval is too short, slow connections may be misinterpreted as failed connections. A ping interval of 30 seconds is reasonable for most use cases.
Matthew Ludlum
added a comment - - edited ** setting the timeout will probably not fix the ping issue **
To confirm, this is likely an issue with updating to the latest fabric8 client, which went from 3.9.1 to 3.12.0 of okhttp as of the latest release.
https://github.com/fabric8io/kubernetes-client/commit/9a40df1346876764fecf1cc170f482cc3722cd7a#diff-600376dffeb79835ede4a0b285078036L82
Starting with okhttp 3.10( https://github.com/square/okhttp/blob/master/CHANGELOG.md#version-3100 )- it started aggressively checking connectivity with the pingInterval as well.
To fix this, adding -Dkubernetes.websocket.ping.interval=<number> to the jenkins startup or system file should change the ping interval.
The okhttp docs recommend 30s :
If you have a configured ping interval, you should confirm that it is long enough for a roundtrip from client to server. If your ping interval is too short, slow connections may be misinterpreted as failed connections. A ping interval of 30 seconds is reasonable for most use cases. Daniel Watrous
added a comment - seakip18 Are you suggesting that this might be better moved out of the plugin and handled by passing in a parameter when starting Jenkins? My other question is: Does the current connectTimeout value set that ping interval?
Daniel Watrous
added a comment - seakip18 Are you suggesting that this might be better moved out of the plugin and handled by passing in a parameter when starting Jenkins? My other question is: Does the current connectTimeout value set that ping interval? Matthew Ludlum
added a comment - The ping setting is only settable via parameter when starting jenkins as of right now, per my reading through the source and docs.
Matthew Ludlum
added a comment - The ping setting is only settable via parameter when starting jenkins as of right now, per my reading through the source and docs. If you want to be able to set the pingTimeout in the Jenkins config it needs to be added the same way the connectTimeout and readTimeout
I'm closing this issue, as the correct solution is to increase the connectTimeout and readTimeout and in the next release they will be correctly saved. Feel free to open other issue if you want to add pingTimeout
mu qiao
added a comment - As seakip18 (thx a lot, you save my days) said, adding -Dkubernetes.websocket.ping.interval=<number> fix the ping timeout issue
More information is: this so-called 'interval' option changes ping/pong timeout and uses millisecond as unit. I set it to 30 and got log below
mu qiao
added a comment - As seakip18 (thx a lot, you save my days) said, adding -Dkubernetes.websocket.ping.interval=<number> fix the ping timeout issue
More information is: this so-called 'interval' option changes ping/pong timeout and uses millisecond as unit. I set it to 30 and got log below
elhay efrat
added a comment - seakip18, there is not a configuration that we can change in the plugin itself, we really need to restart Jenkins for it?
elhay efrat
added a comment - seakip18 , there is not a configuration that we can change in the plugin itself, we really need to restart Jenkins for it? 
Allan BURDAJEWICZ
added a comment - The original issue java.net.SocketTimeoutException: timeout is due to a Socket Read Timeout. The value can be changed in the kubernetes cloud configuration since version 1.14.9. Before this version, there was a persistent issue and the value was reset to 0 every time the configuration was saved from the UI.
Note that until version 1.22.3 a value of 0 does not mean no timeout. If a value of 0 is provided, the fabric8 kubernetes client does not explicitly set a timeout. But the okhttp client defaults to 10s. *A value of 0 actually means 10s*.
Since version 1.22.3 of the kubernetes plugin:
- the minimum value for Read Timeout is 15s
- the minimum value of the connection timeout is 5s.
****
The SocketTimeoutException: sent ping but didn't receive pong within XXXXms is related to the Socket Ping Timeout interval and should be fixed since version 1.19.3 of the kubernetes plugin: https://issues.jenkins-ci.org/browse/JENKINS-58301.
Allan BURDAJEWICZ
added a comment - The original issue java.net.SocketTimeoutException: timeout is due to a Socket Read Timeout . The value can be changed in the kubernetes cloud configuration since version 1.14.9 . Before this version, there was a persistent issue and the value was reset to 0 every time the configuration was saved from the UI.
Note that until version 1.22.3 a value of 0 does not mean no timeout . If a value of 0 is provided, the fabric8 kubernetes client does not explicitly set a timeout . But the okhttp client defaults to 10s . * A value of 0 actually means 10s *.
Since version 1.22.3 of the kubernetes plugin:
the minimum value for Read Timeout is 15s
the minimum value of the connection timeout is 5s.
****
The SocketTimeoutException: sent ping but didn't receive pong within XXXXms is related to the Socket Ping Timeout interval and should be fixed since version 1.19.3 of the kubernetes plugin: https://issues.jenkins-ci.org/browse/JENKINS-58301 . 
It looks like this change does indeed work.
However, it also looks like there's something incorrectly wired in the UI, because as soon as I change any values via the UI, the timeout settings are reset back to 0:
<connectTimeout>0</connectTimeout>
<readTimeout>0</readTimeout>
I can only get it to work again by changing the values from the config.xml file itself, and then reloading the config from disk.