Loading...

XML

Word

Printable

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: kubernetes-plugin, remoting
Labels:
None
Environment:
Jenkins 2.401.1
Jenkins inbound agent 3131.vf2b_b_798b_ce99-3
Jenkins Kubernetes plugin 3971.v94b_4c914ca_75

Similar Issues:

Show

Hello,

We upgraded our Jenkins environment from 2.375.1 and the agents created via the Kubernetes plugin suddenly became unable to connect back to the controller. We can see a lot of logs but nothing that clearly states what is going on.

The agent is connecting to the controller via WebSocket over a tunnel to the http port of the controller. It is able to connect and it's getting closed:

Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Performing onReconnect operation.
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: onReconnect operation completed.
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: WebSocket connection open
Jul 17, 2023 8:29:10 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Jul 17, 2023 8:29:12 AM hudson.remoting.ProxyOutputStream$Chunk lambda$execute$0
WARNING: Failed to ack the stream
java.io.IOException: java.lang.IllegalStateException: The connection has been closed.
    at hudson.remoting.Engine$1AgentEndpoint$Transport.write(Engine.java:661)
    at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:335)
    at hudson.remoting.Channel.send(Channel.java:765)
    at hudson.remoting.ProxyOutputStream$Chunk.lambda$execute$0(ProxyOutputStream.java:275)
    at hudson.remoting.PipeWriter$1.run(PipeWriter.java:159)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: The connection has been closed.
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusSession.checkConnectionState(TyrusSession.java:545)
    at io.jenkins.remoting.shaded.org.glassfish.tyrus.core.TyrusSession.getAsyncRemote(TyrusSession.java:201)
    at hudson.remoting.Engine$1AgentEndpoint$Transport.write(Engine.java:659)
    ... 13 more

Any idea what could be wrong ?

Logs from the controller:

Jul 17, 2023 1:01:01 PM WARNING hudson.util.ExceptionCatchingThreadFactory uncaughtExceptionThread Computer.threadPoolForRemoting [#365] terminated unexpectedly
java.nio.channels.ClosedChannelException
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from localhost/127.0.0.1:42350
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
		at hudson.remoting.Request.call(Request.java:199)
		at hudson.remoting.Channel.call(Channel.java:999)
		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:714)
		at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:176)
		at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:337)
		at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:428)
		at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler.lambda$onChannel$0(JnlpProtocol4Handler.java:335)
		at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
		at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
		at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
Caused: hudson.remoting.RequestAbortedException
	at hudson.remoting.Request.abort(Request.java:346)
	at hudson.remoting.Channel.terminate(Channel.java:1080)
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:700)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:156)
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:328)
	at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:335)
	at hudson.remoting.Channel.send(Channel.java:765)
	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:146)
	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:112)
	at hudson.remoting.Util.copy(Util.java:58)
	at hudson.remoting.JarLoaderImpl.writeJarTo(JarLoaderImpl.java:57)
	at jdk.internal.reflect.GeneratedMethodAccessor2287.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:924)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:902)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
	at hudson.remoting.CallableDecoratorList.lambda$applyDecorator$0(CallableDecoratorList.java:19)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Jul 17, 2023 1:01:01 PM WARNING hudson.remoting.Request$2 runFailed to send back a reply to the request UserRequest:UserRPCRequest:hudson.remoting.JarLoader.writeJarTo[long,long,java.io.OutputStream](3)
java.nio.channels.ClosedChannelException
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:177)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:279)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:501)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:244)
	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:196)
	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doSend(ProtocolStack.java:700)
	at org.jenkinsci.remoting.protocol.ApplicationLayer.write(ApplicationLayer.java:156)
	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:328)
	at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:335)
	at hudson.remoting.Channel.send(Channel.java:765)
	at hudson.remoting.ProxyOutputStream.write(ProxyOutputStream.java:146)
	at hudson.remoting.RemoteOutputStream.write(RemoteOutputStream.java:112)
	at hudson.remoting.Util.copy(Util.java:58)
	at hudson.remoting.JarLoaderImpl.writeJarTo(JarLoaderImpl.java:57)
	at jdk.internal.reflect.GeneratedMethodAccessor2287.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:924)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:902)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:853)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
Caused: hudson.remoting.ChannelClosedException: Channel "JNLP4-connect connection from localhost/127.0.0.1:42350": channel is already closed
	at hudson.remoting.Channel.send(Channel.java:761)
	at hudson.remoting.Request$2.run(Request.java:390)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
	at hudson.remoting.CallableDecoratorList.lambda$applyDecorator$0(CallableDecoratorList.java:19)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Jul 17, 2023 1:01:09 PM WARNING hudson.remoting.AbstractByteBufferCommandTransport processCommandFailed to construct Command in channel xxxx java.io.UTFDataFormatException at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3728) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3672) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3476) at java.base/java.io.ObjectInputStream.readUTF(ObjectInputStream.java:1212) at java.base/java.io.ObjectStreamClass.readNonProxy(ObjectStreamClass.java:629) at java.base/java.io.ObjectInputStream.readClassDescriptor(ObjectInputStream.java:961) at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1992) at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870) at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447) at hudson.remoting.Command.readFromObjectStream(Command.java:155) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:233) at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:220) at jenkins.agents.WebSocketAgents$Session.binary(WebSocketAgents.java:148) at jenkins.websocket.WebSockets$1.onWebSocketBinary(WebSockets.java:103) at jenkins.websocket.Jetty10Provider$2.onWebSocketBinary(Jetty10Provider.java:154) at org.eclipse.jetty.websocket.core.internal.messages.ByteArrayMessageSink.accept(ByteArrayMessageSink.java:89) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.acceptMessage(JettyWebSocketFrameHandler.java:334) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onContinuationFrame(JettyWebSocketFrameHandler.java:349) at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onFrame(JettyWebSocketFrameHandler.java:246) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$IncomingAdaptor.lambda$onFrame$1(WebSocketCoreSession.java:671) at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1468) at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1487) at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession$IncomingAdaptor.onFrame(WebSocketCoreSession.java:671) at org.eclipse.jetty.websocket.core.internal.ExtensionStack.onFrame(ExtensionStack.java:120) at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.onFrame(WebSocketCoreSession.java:481) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFrame(WebSocketConnection.java:262) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:455) at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFillable(WebSocketConnection.java:340) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078) at java.base/java.lang.Thread.run(Thread.java:829)

Jul 17, 2023 1:01:09 PM WARNING jenkins.agents.WebSocketAgents$Session errornull
org.eclipse.jetty.websocket.core.exception.ProtocolException: Unknown opcode: 14
	at org.eclipse.jetty.websocket.core.internal.Parser.newFrame(Parser.java:271)
	at org.eclipse.jetty.websocket.core.internal.Parser.parsePayload(Parser.java:352)
	at org.eclipse.jetty.websocket.core.internal.Parser.parse(Parser.java:213)
	at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:448)
Caused: org.eclipse.jetty.websocket.api.exceptions.ProtocolException: Unknown opcode: 14
	at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.convertCause(JettyWebSocketFrameHandler.java:515)
	at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onError(JettyWebSocketFrameHandler.java:258)
	at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$2(WebSocketCoreSession.java:284)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1468)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1487)
	at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
	at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.closeConnection(WebSocketCoreSession.java:284)
	at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$sendFrame$7(WebSocketCoreSession.java:519)
	at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:155)
	at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.notifyCallbackSuccess(TransformingFlusher.java:197)
	at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:154)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:232)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:214)
	at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77)
	at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:522)
	at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.close(WebSocketCoreSession.java:239)
	at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.processConnectionError(WebSocketCoreSession.java:335)
	at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.fillAndParse(WebSocketConnection.java:506)
	at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onFillable(WebSocketConnection.java:340)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
	at java.base/java.lang.Thread.run(Thread.java:829)

Assignee:: Unassigned

Reporter:: Gaylor

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Created:: 2023-07-17 11:38

Updated:: 2023-07-28 05:42

Resolved:: 2023-07-28 05:42

Details

Description

Attachments

Activity

People

Dates