Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61253

Remoting using WebSocket fails with "Handshake response not received" Exception

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Won't Fix
    • Component/s: remoting
    • Labels:
      None
    • Environment:
      - Remoting v4.2
      - AdoptOpenJDK v11.0.5 (Windows 10)
      - Jenkins v2.204.2
    • Similar Issues:
    • Released As:
      Jenkins v2.222.1 / Remoting v4.2

      Description

      Using remoting v4.2 with WebSocket crashes after ~ 30 seconds with a "Handshake response not received" exception:
       

      io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager <init>
      CONFIG: Provider class loaded: io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.JdkClientContainer
      hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Handshake response not received.
      io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
      	at hudson.remoting.Engine.runWebSocket(Engine.java:624)
      	at hudson.remoting.Engine.run(Engine.java:469)
      

       

      Remoting is launched through hudson.remoting.jnlp.Main.main() with arguments: 

      [<Secret ID>, <Agent>, -url, https://<Master Host>:<Port>/, -webSocket, -workDir, <WorkDir Path> -headless, -noreconnect]

        Attachments

          Issue Links

            Activity

            Hide
            ethorsa ethorsa added a comment -

            Cli seems to have a similar issue (JENKINS-61212).

            Show
            ethorsa ethorsa added a comment - Cli seems to have a similar issue ( JENKINS-61212 ).
            Hide
            jthompson Jeff Thompson added a comment -

            You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217.

            Show
            jthompson Jeff Thompson added a comment - You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217 .
            Hide
            ethorsa ethorsa added a comment -

            I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?

            Show
            ethorsa ethorsa added a comment - I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?
            Hide
            jthompson Jeff Thompson added a comment -

            As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.

            Show
            jthompson Jeff Thompson added a comment - As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.
            Hide
            ethorsa ethorsa added a comment -

            Great, thanks!

            Show
            ethorsa ethorsa added a comment - Great, thanks!
            Hide
            ethorsa ethorsa added a comment - - edited

            Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error:

            1. Create a new static agent on the master (test-websocket in this example)
            2. Enable "Use WebSocket" in it's configuration
            3. Start the agent:
            java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test"

            After ~ 30 sec the agent fails by exception:

            Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error
            SEVERE: Handshake response not received.
            io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
            	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
            	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
            	at hudson.remoting.Engine.runWebSocket(Engine.java:627)
            	at hudson.remoting.Engine.run(Engine.java:469)
            

            The master log shows a single related entry:

            INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
            Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException
            

            Error on the agent after connection failed:

            java.nio.channels.ClosedChannelException
            	at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141)
            	at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90)
            	at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source)
            	at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
            	at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317)
            	at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
            [...]
            
            Show
            ethorsa ethorsa added a comment - - edited Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error: Create a new static agent on the master ( test-websocket in this example) Enable "Use WebSocket" in it's configuration Start the agent: java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test" After ~ 30 sec the agent fails by exception: Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Handshake response not received. io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received. at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355) at hudson.remoting.Engine.runWebSocket(Engine.java:627) at hudson.remoting.Engine.run(Engine.java:469) The master log shows a single related entry: INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException Error on the agent after connection failed: java.nio.channels.ClosedChannelException at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141) at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90) at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119) at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317) at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42) [...]
            Hide
            ethorsa ethorsa added a comment -

            I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main(). The agent launches successfully, but it's marked as offline on the master.

             

            Remoting log (on Agent):

            hudson.remoting.jnlp.Main$CuiListener status
            INFO: WebSocket connection open
            hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connected

            Agent log (Icon of the node):

            Inbound agent connected from <Ip>
            

            Master Log:

            WARNING jenkins.agents.WebSocketAgents$Session error
            null
            java.io.IOException: Broken pipe
            	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
            	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
            	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
            	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
            	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496)
            	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
            Caused: org.eclipse.jetty.io.EofException
            	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
            	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
            	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
            	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
            	at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
            	at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
            	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
            	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
            	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
            	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
            	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
            	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
            	at java.base/java.lang.Thread.run(Thread.java:834)
            
            INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
            Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
            
            Show
            ethorsa ethorsa added a comment - I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main() . The agent launches successfully, but it's marked as offline on the master.   Remoting log (on Agent): hudson.remoting.jnlp.Main$CuiListener status INFO: WebSocket connection open hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Agent log (Icon of the node): Inbound agent connected from <Ip> Master Log: WARNING jenkins.agents.WebSocketAgents$Session error null java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263) Caused: org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264) at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193) at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Thread.java:834) INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
            Hide
            jthompson Jeff Thompson added a comment -

            Try Remoting (agent) 4.2.

            You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh.

            Remoting 4.2:

               Weekly releases 2.217 through 2.228

               LTS release 2.222.1

            Remoting 4.3:

              Weekly releases 2.229+

              Upcoming LTS

             

            If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.

            Show
            jthompson Jeff Thompson added a comment - Try Remoting (agent) 4.2. You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh. Remoting 4.2:    Weekly releases 2.217 through 2.228    LTS release 2.222.1 Remoting 4.3:   Weekly releases 2.229+   Upcoming LTS   If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.
            Hide
            ethorsa ethorsa added a comment -

            Thanks Jeff Thompson, downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets. 

             

            The WebSocket implementation is still in Beta, because we need to get extended real-world testing.

            I'll let you know if there are other issues.

            Show
            ethorsa ethorsa added a comment - Thanks Jeff Thompson , downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets.    The WebSocket implementation is still in Beta, because we need to get extended real-world testing. I'll let you know if there are other issues.
            Hide
            jglick Jesse Glick added a comment -

            Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.

            Show
            jglick Jesse Glick added a comment - Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.
            Hide
            jglick Jesse Glick added a comment -

            AFAICT this is all just about JENKINS-61409, which as indicated in that issue requires a matching upgrade on the controller and agent sides.

            Show
            jglick Jesse Glick added a comment - AFAICT this is all just about JENKINS-61409 , which as indicated in that issue requires a matching upgrade on the controller and agent sides.

              People

              Assignee:
              jthompson Jeff Thompson
              Reporter:
              ethorsa ethorsa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: