[JENKINS-55138] Exec Failure: HTTP:0. Message:No response

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: kubernetes-plugin
Labels:
None
Environment:
Jenkins version: 2.150.1
Kubernetes plugin version: 1.13.7
Kubernetes cluster version: 1.10.11
JNLP container: jenkins/jnlp-slave:3.27-1

Similar Issues:
Powered by SuggestiMate

Show

Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141) 
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) 
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) 
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) 
14:56:27 at okio.Okio$2.read(Okio.java:139) 
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) 
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67) 
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60) 
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97) 
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265) 
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204) 
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) 
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
14:56:27 at java.lang.Thread.run(Thread.java:748)

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

..
Dec 12, 2018 1:22:05 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [clitest-rcc-30831-mhzxr-xrsf6], container: [clitest] with pid:[-1]
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label py36: Kubernetes Pod Template
Dec 12, 2018 1:22:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 12, 2018 1:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent clitest-3415-8zdzg-646kj
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

jenkins-master.log
2.07 MB
2018-12-12 11:02

is related to

JENKINS-42136 shared-library abstraction causing RejectedExecutionException when running sh() commands

Resolved

links to

PR #407

Juha Tiensyrjä created issue - 2018-12-12 08:24

Juha Tiensyrjä made changes - 2018-12-12 11:01

Description

Original: Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

{noformat}
14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141)
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503)
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
14:56:27 at okio.Okio$2.read(Okio.java:139)
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67)
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60)
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:56:27 at java.lang.Thread.run(Thread.java:748)
{noformat}

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

{noformat}
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748){noformat}
Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

New: Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

{noformat}
14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141)
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503)
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
14:56:27 at okio.Okio$2.read(Okio.java:139)
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67)
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60)
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:56:27 at java.lang.Thread.run(Thread.java:748)
{noformat}

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

{noformat}
..
Dec 12, 2018 1:22:05 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [clitest-rcc-30831-mhzxr-xrsf6], container: [clitest] with pid:[-1]
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label py36: Kubernetes Pod Template
Dec 12, 2018 1:22:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 12, 2018 1:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent clitest-3415-8zdzg-646kj
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748){noformat}
Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

Juha Tiensyrjä made changes - 2018-12-12 11:02

Attachment

New: jenkins-master.log [ 45434 ]

Juha Tiensyrjä added a comment - 2018-12-12 11:03

I attached full Jenkins master log since the previous restart.

Juha Tiensyrjä added a comment - 2018-12-12 11:03 I attached full Jenkins master log since the previous restart.

Juha Tiensyrjä added a comment - 2018-12-13 07:07 - edited

I downgraded Jenkins master to version 2.138.4, still seeing this issue.

Upgraded back to 2.150.1 and downgraded the K8s plugin to 1.13.6, it still reproduces. Will try 1.13.5 next.

Juha Tiensyrjä added a comment - 2018-12-13 07:07 - edited I downgraded Jenkins master to version 2.138.4, still seeing this issue. Upgraded back to 2.150.1 and downgraded the K8s plugin to 1.13.6, it still reproduces. Will try 1.13.5 next.

Stephan Pauxberger added a comment - 2018-12-13 12:05 - edited

we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour.

As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version.

One more thing that I noticed: Once this happens, it seems to affect more than one job.

Also, after that there is always the exception later:

java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@512db46f rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:130)
	at okhttp3.RealCall.enqueue(RealCall.java:100)
	at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183)
	at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236)
	at hudson.Launcher$ProcStarter.start(Launcher.java:455)

So obviously, the error seems to not really affect the execution, but leads to the ThreadPool bein terminated.

Stephan Pauxberger added a comment - 2018-12-13 12:05 - edited we are encountering this error about 10 times a day which is quite critical, since it also breaks jobs which run for over an hour. As with OP, this occured after updating to 1.13.7. Before the upgrade, we had a lot of memory issues in OpenShift, therefore I would like to prevent having to roll back the version. One more thing that I noticed: Once this happens, it seems to affect more than one job. Also, after that there is always the exception later: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@512db46f rejected from java.util.concurrent.ThreadPoolExecutor@2e02dc25[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 258] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at okhttp3.Dispatcher.enqueue(Dispatcher.java:130) at okhttp3.RealCall.enqueue(RealCall.java:100) at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183) at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436) at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267) at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:318) at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:236) at hudson.Launcher$ProcStarter.start(Launcher.java:455) So obviously, the error seems to not really affect the execution, but leads to the ThreadPool bein terminated.

Stephan Pauxberger made changes - 2018-12-13 12:05

Priority

Original: Minor [ 4 ]

New: Major [ 3 ]

Vincent Latombe made changes - 2018-12-13 15:50

Assignee

Original: Carlos Sanchez [ csanchez ]

New: Vincent Latombe [ vlatombe ]

Vincent Latombe made changes - 2018-12-13 15:50

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Vincent Latombe made changes - 2018-12-13 15:50

Remote Link

New: This issue links to "PR #407 (Web Link)" [ 22102 ]

Assignee:: Vincent Latombe

Reporter:: Juha Tiensyrjä

Votes:: 1 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: 2018-12-12 08:24

Updated:: 2019-09-09 09:21

Resolved:: 2018-12-13 21:56

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Juha Tiensyrjä added a comment - 2018-12-12 11:03

Expand comment: Juha Tiensyrjä added a comment - 2018-12-12 11:03

Collapse comment: Juha Tiensyrjä added a comment - 2018-12-13 07:07, Edited by Juha Tiensyrjä - 2018-12-13 15:08

Expand comment: Juha Tiensyrjä added a comment - 2018-12-13 07:07, Edited by Juha Tiensyrjä - 2018-12-13 15:08

Collapse comment: Stephan Pauxberger added a comment - 2018-12-13 12:05, Edited by Stephan Pauxberger - 2018-12-13 18:41

Expand comment: Stephan Pauxberger added a comment - 2018-12-13 12:05, Edited by Stephan Pauxberger - 2018-12-13 18:41

People

Dates