Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61253

Remoting using WebSocket fails with "Handshake response not received" Exception

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • remoting
    • None
    • - Remoting v4.2
      - AdoptOpenJDK v11.0.5 (Windows 10)
      - Jenkins v2.204.2
    • Jenkins v2.222.1 / Remoting v4.2

      Using remoting v4.2 with WebSocket crashes after ~ 30 seconds with a "Handshake response not received" exception:
       

      io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager <init>
      CONFIG: Provider class loaded: io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.JdkClientContainer
      hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Handshake response not received.
      io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
      	at hudson.remoting.Engine.runWebSocket(Engine.java:624)
      	at hudson.remoting.Engine.run(Engine.java:469)
      

       

      Remoting is launched through hudson.remoting.jnlp.Main.main() with arguments: 

      [<Secret ID>, <Agent>, -url, https://<Master Host>:<Port>/, -webSocket, -workDir, <WorkDir Path> -headless, -noreconnect]

          [JENKINS-61253] Remoting using WebSocket fails with "Handshake response not received" Exception

          ethorsa added a comment -

          I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?

          ethorsa added a comment - I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?

          Jeff Thompson added a comment -

          As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.

          Jeff Thompson added a comment - As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.

          ethorsa added a comment -

          Great, thanks!

          ethorsa added a comment - Great, thanks!

          ethorsa added a comment - - edited

          Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error:

          1. Create a new static agent on the master (test-websocket in this example)
          2. Enable "Use WebSocket" in it's configuration
          3. Start the agent:
          java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test"

          After ~ 30 sec the agent fails by exception:

          Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error
          SEVERE: Handshake response not received.
          io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
          	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
          	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
          	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
          	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
          	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
          	at hudson.remoting.Engine.runWebSocket(Engine.java:627)
          	at hudson.remoting.Engine.run(Engine.java:469)
          

          The master log shows a single related entry:

          INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
          Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException
          

          Error on the agent after connection failed:

          java.nio.channels.ClosedChannelException
          	at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141)
          	at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90)
          	at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source)
          	at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
          	at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389)
          	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317)
          	at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
          [...]
          

          ethorsa added a comment - - edited Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error: Create a new static agent on the master ( test-websocket in this example) Enable "Use WebSocket" in it's configuration Start the agent: java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test" After ~ 30 sec the agent fails by exception: Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Handshake response not received. io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received. at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355) at hudson.remoting.Engine.runWebSocket(Engine.java:627) at hudson.remoting.Engine.run(Engine.java:469) The master log shows a single related entry: INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException Error on the agent after connection failed: java.nio.channels.ClosedChannelException at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141) at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90) at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119) at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317) at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42) [...]

          ethorsa added a comment -

          I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main(). The agent launches successfully, but it's marked as offline on the master.

           

          Remoting log (on Agent):

          hudson.remoting.jnlp.Main$CuiListener status
          INFO: WebSocket connection open
          hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connected

          Agent log (Icon of the node):

          Inbound agent connected from <Ip>
          

          Master Log:

          WARNING jenkins.agents.WebSocketAgents$Session error
          null
          java.io.IOException: Broken pipe
          	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
          	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
          	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
          	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
          	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496)
          	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
          Caused: org.eclipse.jetty.io.EofException
          	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
          	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
          	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
          	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
          	at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
          	at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
          	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
          	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
          	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
          	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
          	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
          	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
          	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
          	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
          	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
          	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
          	at java.base/java.lang.Thread.run(Thread.java:834)
          
          INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
          Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
          

          ethorsa added a comment - I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main() . The agent launches successfully, but it's marked as offline on the master.   Remoting log (on Agent): hudson.remoting.jnlp.Main$CuiListener status INFO: WebSocket connection open hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Agent log (Icon of the node): Inbound agent connected from <Ip> Master Log: WARNING jenkins.agents.WebSocketAgents$Session error null java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263) Caused: org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264) at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193) at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Thread.java:834) INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException

          Jeff Thompson added a comment -

          Try Remoting (agent) 4.2.

          You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh.

          Remoting 4.2:

             Weekly releases 2.217 through 2.228

             LTS release 2.222.1

          Remoting 4.3:

            Weekly releases 2.229+

            Upcoming LTS

           

          If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.

          Jeff Thompson added a comment - Try Remoting (agent) 4.2. You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh. Remoting 4.2:    Weekly releases 2.217 through 2.228    LTS release 2.222.1 Remoting 4.3:   Weekly releases 2.229+   Upcoming LTS   If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.

          ethorsa added a comment -

          Thanks jthompson, downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets. 

           

          The WebSocket implementation is still in Beta, because we need to get extended real-world testing.

          I'll let you know if there are other issues.

          ethorsa added a comment - Thanks jthompson , downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets.    The WebSocket implementation is still in Beta, because we need to get extended real-world testing. I'll let you know if there are other issues.

          Jesse Glick added a comment -

          Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.

          Jesse Glick added a comment - Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.

          Jesse Glick added a comment -

          AFAICT this is all just about JENKINS-61409, which as indicated in that issue requires a matching upgrade on the controller and agent sides.

          Jesse Glick added a comment - AFAICT this is all just about JENKINS-61409 , which as indicated in that issue requires a matching upgrade on the controller and agent sides.

          Basil Crow added a comment -

          Duplicates JENKINS-61212.

          Basil Crow added a comment - Duplicates JENKINS-61212 .

            Unassigned Unassigned
            ethorsa ethorsa
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: