Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53532

Kubernetes java.net.SocketTimeoutException: timeout

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Kubernetes 1.9
      Jenkins ver. 2.121.3
      Kubernetes Pluin 1.12.4
    • Similar Issues:

      Description

      The problem is that I need to configure timeout time for fabric9 Kubernetes API to access the Kubernetes API. The Pod is created and is running, but in some cases fabric9 Kubernetes API takes a little bit more time to access Kubernetes, resulting in the described error.

      Looking in the code, there should be a default timeout of 5 minutes.

      https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L416

      Since CONTAINER_READY_TIMEOUT=5 and TimeUnit.MINUTES.

      But the Pipeline is definitely not waiting 5 minutes.

      I know that my setup is working, because the Pipeline works 9/10 times. It just throws this error in the rare cases if Kubernetes needs a little bit longer to respond.

      The step takes 1min 9s before the exception occurs. The Pipeline have a timeout of 1 hour.

       

      // code placeholder
      java.net.SocketException: Socket closed
      	at java.net.SocketInputStream.read(SocketInputStream.java:204)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
      	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
      	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
      	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
      	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
      	at okio.Okio$2.read(Okio.java:139)
      	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
      Caused: java.net.SocketTimeoutException: timeout
      	at okio.Okio$4.newTimeoutException(Okio.java:230)
      	at okio.AsyncTimeout.exit(AsyncTimeout.java:285)
      	at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
      	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
      	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
      	at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
      	at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
      	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
      	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
      	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
      	at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:104)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
      	at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:56)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
      	at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
      	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
      	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
      	at okhttp3.RealCall.execute(RealCall.java:77)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:313)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:296)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:780)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:196)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:163)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.waitUntilExists(BaseOperation.java:953)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:219)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:37)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:417)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:449)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at sun.reflect.GeneratedMethodAccessor1651.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [maven-3-jdk-8-hgdv3]  in namespace: [jenkins]  failed.
      	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:62)
      	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:71)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:207)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:163)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.waitUntilExists(BaseOperation.java:953)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:219)
      	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.waitUntilReady(HasMetadataOperation.java:37)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.waitUntilContainerIsReady(ContainerExecDecorator.java:417)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:255)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:449)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:186)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:86)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:182)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at sun.reflect.GeneratedMethodAccessor1651.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      	at WorkflowScript.run(WorkflowScript:73)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor207.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

        Attachments

          Issue Links

            Activity

            Hide
            seakip18 Matthew Ludlum added a comment -

            The ping setting is only settable via parameter when starting jenkins as of right now, per my reading through the source and docs.

            Show
            seakip18 Matthew Ludlum added a comment - The ping setting is only settable via parameter when starting jenkins as of right now, per my reading through the source and docs.
            Hide
            csanchez Carlos Sanchez added a comment -

            If you want to be able to set the pingTimeout in the Jenkins config it needs to be added the same way the connectTimeout and readTimeout

            I'm closing this issue, as the correct solution is to increase the connectTimeout and readTimeout and in the next release they will be correctly saved. Feel free to open other issue if you want to add pingTimeout

            Show
            csanchez Carlos Sanchez added a comment - If you want to be able to set the pingTimeout in the Jenkins config it needs to be added the same way the connectTimeout and readTimeout I'm closing this issue, as the correct solution is to increase the connectTimeout and readTimeout and in the next release they will be correctly saved. Feel free to open other issue if you want to add pingTimeout
            Hide
            hippoknight mu qiao added a comment -

            As Matthew Ludlum (thx a lot, you save my days) said, adding -Dkubernetes.websocket.ping.interval=<number> fix the ping timeout issue

            More information is: this so-called 'interval' option changes ping/pong timeout and uses millisecond as unit. I set it to 30 and got log below

             

            Show
            hippoknight mu qiao added a comment - As Matthew Ludlum  (thx a lot, you save my days) said, adding -Dkubernetes.websocket.ping.interval=<number>  fix the ping timeout issue More information is: this so-called 'interval' option changes ping/pong timeout and uses millisecond as unit. I set it to 30 and got log below  
            Hide
            elhay elhay efrat added a comment -

            Matthew Ludlum, there is not a configuration that we can change in the plugin itself, we really need to restart Jenkins for it? 

            Show
            elhay elhay efrat added a comment - Matthew Ludlum , there is not a configuration that we can change in the plugin itself, we really need to restart Jenkins for it? 
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment -

            The original issue java.net.SocketTimeoutException: timeout is due to a Socket Read Timeout. The value can be changed in the kubernetes cloud configuration since version 1.14.9. Before this version, there was a persistent issue and the value was reset to 0 every time the configuration was saved from the UI.

            Note that until version 1.22.3 a value of 0 does not mean no timeout. If a value of 0 is provided, the fabric8 kubernetes client does not explicitly set a timeout. But the okhttp client defaults to 10s. *A value of 0 actually means 10s*.

            Since version 1.22.3 of the kubernetes plugin:

            • the minimum value for Read Timeout is 15s
            • the minimum value of the connection timeout is 5s.

            ****

            The SocketTimeoutException: sent ping but didn't receive pong within XXXXms is related to the Socket Ping Timeout interval and should be fixed since version 1.19.3 of the kubernetes plugin: https://issues.jenkins-ci.org/browse/JENKINS-58301.

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - The original issue java.net.SocketTimeoutException: timeout is due to a Socket Read Timeout . The value can be changed in the kubernetes cloud configuration since version 1.14.9 . Before this version, there was a persistent issue and the value was reset to 0 every time the configuration was saved from the UI. Note that until version 1.22.3 a value of 0 does not mean no timeout . If a value of 0 is provided, the fabric8 kubernetes client does not explicitly set a timeout . But the okhttp client defaults to 10s . * A value of 0 actually means 10s *. Since version 1.22.3 of the kubernetes plugin: the minimum value for Read Timeout is 15s the minimum value of the connection timeout is 5s. **** The SocketTimeoutException: sent ping but didn't receive pong within XXXXms is related to the Socket Ping Timeout interval and should be fixed since version 1.19.3 of the kubernetes plugin: https://issues.jenkins-ci.org/browse/JENKINS-58301 .

              People

              Assignee:
              csanchez Carlos Sanchez
              Reporter:
              erwin_ntt Erwin Müller
              Votes:
              7 Vote for this issue
              Watchers:
              18 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: