[JENKINS-55138] Exec Failure: HTTP:0. Message:No response

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: kubernetes-plugin
Labels:
None
Environment:
Jenkins version: 2.150.1
Kubernetes plugin version: 1.13.7
Kubernetes cluster version: 1.10.11
JNLP container: jenkins/jnlp-slave:3.27-1

Similar Issues:
Powered by SuggestiMate

Show

Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171) 
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141) 
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) 
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) 
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) 
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) 
14:56:27 at okio.Okio$2.read(Okio.java:139) 
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) 
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67) 
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60) 
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113) 
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97) 
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265) 
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204) 
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) 
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
14:56:27 at java.lang.Thread.run(Thread.java:748)

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

..
Dec 12, 2018 1:22:05 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [clitest-rcc-30831-mhzxr-xrsf6], container: [clitest] with pid:[-1]
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label py36: Kubernetes Pod Template
Dec 12, 2018 1:22:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 12, 2018 1:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent clitest-3415-8zdzg-646kj
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)

Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

jenkins-master.log
2.07 MB
2018-12-12 11:02

is related to

JENKINS-42136 shared-library abstraction causing RejectedExecutionException when running sh() commands

Resolved

links to

PR #407

Juha Tiensyrjä created issue - 2018-12-12 08:24

Juha Tiensyrjä made changes - 2018-12-12 11:01

Description

Original: Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

{noformat}
14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141)
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503)
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
14:56:27 at okio.Okio$2.read(Okio.java:139)
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67)
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60)
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:56:27 at java.lang.Thread.run(Thread.java:748)
{noformat}

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

{noformat}
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748){noformat}
Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

New: Since upgrading our Jenkins version from 2.138.3 to 2.150.1 and K8s plugin from 1.13.5 to 1.13.7, some of our jobs have started to fail with the following error present in the job console log:

{noformat}
14:56:27 java.net.SocketException: Socket closed
14:56:27 at java.net.SocketInputStream.socketRead0(Native Method)
14:56:27 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:171)
14:56:27 at java.net.SocketInputStream.read(SocketInputStream.java:141)
14:56:27 at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
14:56:27 at sun.security.ssl.InputRecord.read(InputRecord.java:503)
14:56:27 at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
14:56:27 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
14:56:27 at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
14:56:27 at okio.Okio$2.read(Okio.java:139)
14:56:27 at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
14:56:27 at okio.RealBufferedSource.request(RealBufferedSource.java:67)
14:56:27 at okio.RealBufferedSource.require(RealBufferedSource.java:60)
14:56:27 at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
14:56:27 at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
14:56:27 at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
14:56:27 at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
14:56:27 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
14:56:27 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
14:56:27 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
14:56:27 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
14:56:27 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
14:56:27 at java.lang.Thread.run(Thread.java:748)
{noformat}

The job continues until the end of the stage and fails there after this message is encountered.

In Jenkins master logs we have found these kinds of errors (usually more than one at exactly the same time) which correspond to the timing in the job console logs:

{noformat}
..
Dec 12, 2018 1:22:05 AM org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1 doLaunch
INFO: Created process inside pod: [clitest-rcc-30831-mhzxr-xrsf6], container: [clitest] with pid:[-1]
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Excess workload after pending Kubernetes agents: 1
Dec 12, 2018 1:22:06 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
INFO: Template for label py36: Kubernetes Pod Template
Dec 12, 2018 1:22:06 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
INFO: Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Dec 12, 2018 1:22:08 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent clitest-3415-8zdzg-646kj
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
  at okio.RealBufferedSource.request(RealBufferedSource.java:67)
  at okio.RealBufferedSource.require(RealBufferedSource.java:60)
  at okio.RealBufferedSource.readByte(RealBufferedSource.java:73)
  at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:113)
  at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:97)
  at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265)
  at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204)
  at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
  at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748){noformat}
Both the Jenkins master and the slave pods are running in the same K8s cluster in the default namespace.

Juha Tiensyrjä made changes - 2018-12-12 11:02

Attachment

New: jenkins-master.log [ 45434 ]

Stephan Pauxberger made changes - 2018-12-13 12:05

Priority

Original: Minor [ 4 ]

New: Major [ 3 ]

Vincent Latombe made changes - 2018-12-13 15:50

Assignee

Original: Carlos Sanchez [ csanchez ]

New: Vincent Latombe [ vlatombe ]

Vincent Latombe made changes - 2018-12-13 15:50

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Vincent Latombe made changes - 2018-12-13 15:50

Remote Link

New: This issue links to "PR #407 (Web Link)" [ 22102 ]

Vincent Latombe made changes - 2018-12-13 15:50

Status

Original: In Progress [ 3 ]

New: In Review [ 10005 ]

Carlos Sanchez made changes - 2018-12-13 18:51

Link

New: This issue is related to ~~JENKINS-42136~~ [ ~~JENKINS-42136~~ ]

Carlos Sanchez made changes - 2018-12-13 21:56

Resolution		New: Fixed [ 1 ]
Status	Original: In Review [ 10005 ]	New: Resolved [ 5 ]

Assignee:: Vincent Latombe

Reporter:: Juha Tiensyrjä

Votes:: 1 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: 2018-12-12 08:24

Updated:: 2019-09-09 09:21

Resolved:: 2018-12-13 21:56

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates