Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55138

Exec Failure: HTTP:0. Message:No response

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins version: 2.150.1
      Kubernetes plugin version: 1.13.7
      Kubernetes cluster version: 1.10.11
      JNLP container: jenkins/jnlp-slave:3.27-1

      Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:
       

      14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141) 
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) 
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) 
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) 
14:56:27 at okio.Okio$2.read(Okio.java:139) 
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) 
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67) 
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60) 
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97) 
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265) 
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204) 
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) 
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
14:56:27 at java.lang.Thread.run(Thread.java:748)

       
      The job continues until the end of the stage and fails there after this message is encountered.

      In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:
       

      ..
Dec 12, 2018 1:22:05 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [clitest-rcc-30831-mhzxr-xrsf6], container: [clitest] with pid:[-1]
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label py36: Kubernetes Pod Template
Dec 12, 2018 1:22:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 12, 2018 1:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent clitest-3415-8zdzg-646kj
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

      Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

       

        1. jenkins-master.log
          2.07 MB

          [JENKINS-55138] Exec Failure: HTTP:0. Message:No response

          Juha Tiensyrjä added a comment -

          I attached full Jenkins master log since the previous restart.

          Juha Tiensyrjä added a comment - I attached full Jenkins master log since the previous restart.

          Juha Tiensyrjä added a comment - - edited

          I downgraded Jenkins master to version 2.138.4, still seeing this issue.

          Upgraded back to 2.150.1 and downgraded the K8s plugin to 1.13.6, it still reproduces. Will try 1.13.5 next.

          Juha Tiensyrjä added a comment - - edited I downgraded Jenkins master to version 2.138.4, still seeing this issue. Upgraded back to 2.150.1 and downgraded the K8s plugin to 1.13.6, it still reproduces. Will try 1.13.5 next.

          Stephan Pauxberger added a comment - - edited

          we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour.

           

          As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version.

           

          One more thing that I noticed: Once this happens, it seems to affect more than one job.

           

          Also, after that there is always the exception later:

           

          java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@512db46f rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:130)
	at okhttp3.RealCall.enqueue(RealCall.java:100)
	at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183)
	at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)
	at hudson.Launcher$ProcStarter.start(Launcher.java:455)

          So obviously, the error seems to not really affect the execution, but leads to the ThreadPool bein terminated.

           

          Stephan Pauxberger added a comment - - edited we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour.   As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version.   One more thing that I noticed: Once this happens, it seems to affect more than one job.   Also, after that there is always the exception later:   java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@512db46f rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at okhttp3.Dispatcher.enqueue(Dispatcher.java:130) at okhttp3.RealCall.enqueue(RealCall.java:100) at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183) at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436) at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267) at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:455) So obviously, the error seems to not really affect the execution, but leads to the ThreadPool bein terminated.  

          Carlos Sanchez added a comment -

          Downgrade to 1.13.5 for now to solve the issue

          Carlos Sanchez added a comment - Downgrade to 1.13.5 for now to solve the issue

          Carlos Sanchez added a comment -

          should be fixed in 1.13.8

          Carlos Sanchez added a comment - should be fixed in 1.13.8

          Yo-An Lin added a comment - - edited

          Same here, 1.13.7 is broken.  this issue is very critical, it happens 10+ times a day, 80% builds failed due to this issue.  Now Trying 1.13.8.

          Yo-An Lin added a comment - - edited Same here, 1.13.7 is broken.  this issue is very critical, it happens 10+ times a day, 80% builds failed due to this issue.  Now Trying 1.13.8.

          junaid mukhtar added a comment -

          We upgraded the plugin to 1.18.3 but are still hitting the same issue although a bit different error message this time

           

          ```Cannot contact ugc-test-ff643992-51da-478c-b63c-36f16f9ab9eb-0kz5p-v4642: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException```

          junaid mukhtar added a comment - We upgraded the plugin to 1.18.3 but are still hitting the same issue although a bit different error message this time   ```Cannot contact ugc-test-ff643992-51da-478c-b63c-36f16f9ab9eb-0kz5p-v4642: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException```

            vlatombe Vincent Latombe
            juhtie01 Juha Tiensyrjä
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: