Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61253

Remoting using WebSocket fails with "Handshake response not received" Exception

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Won't Fix
    • Component/s: remoting
    • Labels:
      None
    • Environment:
      - Remoting v4.2
      - AdoptOpenJDK v11.0.5 (Windows 10)
      - Jenkins v2.204.2
    • Similar Issues:
    • Released As:
      Jenkins v2.222.1 / Remoting v4.2

      Description

      Using remoting v4.2 with WebSocket crashes after ~ 30 seconds with a "Handshake response not received" exception:
       

      io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager <init>
      CONFIG: Provider class loaded: io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.JdkClientContainer
      hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Handshake response not received.
      io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
      	at hudson.remoting.Engine.runWebSocket(Engine.java:624)
      	at hudson.remoting.Engine.run(Engine.java:469)
      

       

      Remoting is launched through hudson.remoting.jnlp.Main.main() with arguments: 

      [<Secret ID>, <Agent>, -url, https://<Master Host>:<Port>/, -webSocket, -workDir, <WorkDir Path> -headless, -noreconnect]

        Attachments

          Issue Links

            Activity

            ethorsa ethorsa created issue -
            ethorsa ethorsa made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-61212 [ JENKINS-61212 ]
            ethorsa ethorsa made changes -
            Link This issue relates to JENKINS-22877 [ JENKINS-22877 ]
            Hide
            ethorsa ethorsa added a comment -

            Cli seems to have a similar issue (JENKINS-61212).

            Show
            ethorsa ethorsa added a comment - Cli seems to have a similar issue ( JENKINS-61212 ).
            ethorsa ethorsa made changes -
            Environment - Remoting v4.2
            - AdoptOpenJDK v11.0.5 (Windows 10)
            - Jenkins v2.204.2
            - Remoting v4.2
            - AdoptOpenJDK v11.0.5, Oracle Java v1.8.0-181 (Windows 10)
            - Jenkins v2.204.2
            ethorsa ethorsa made changes -
            Environment - Remoting v4.2
            - AdoptOpenJDK v11.0.5, Oracle Java v1.8.0-181 (Windows 10)
            - Jenkins v2.204.2
            - Remoting v4.2
            - AdoptOpenJDK v11.0.5 (Windows 10)
            - Jenkins v2.204.2
            Hide
            jthompson Jeff Thompson added a comment -

            You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217.

            Show
            jthompson Jeff Thompson added a comment - You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217 .
            jthompson Jeff Thompson made changes -
            Resolution Not A Defect [ 7 ]
            Status Open [ 1 ] Closed [ 6 ]
            Hide
            ethorsa ethorsa added a comment -

            I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?

            Show
            ethorsa ethorsa added a comment - I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?
            Hide
            jthompson Jeff Thompson added a comment -

            As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.

            Show
            jthompson Jeff Thompson added a comment - As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.
            Hide
            ethorsa ethorsa added a comment -

            Great, thanks!

            Show
            ethorsa ethorsa added a comment - Great, thanks!
            ethorsa ethorsa made changes -
            Resolution Not A Defect [ 7 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            Hide
            ethorsa ethorsa added a comment - - edited

            Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error:

            1. Create a new static agent on the master (test-websocket in this example)
            2. Enable "Use WebSocket" in it's configuration
            3. Start the agent:
            java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test"

            After ~ 30 sec the agent fails by exception:

            Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error
            SEVERE: Handshake response not received.
            io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
            	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
            	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
            	at hudson.remoting.Engine.runWebSocket(Engine.java:627)
            	at hudson.remoting.Engine.run(Engine.java:469)
            

            The master log shows a single related entry:

            INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
            Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException
            

            Error on the agent after connection failed:

            java.nio.channels.ClosedChannelException
            	at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141)
            	at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90)
            	at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source)
            	at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
            	at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317)
            	at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
            [...]
            
            Show
            ethorsa ethorsa added a comment - - edited Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error: Create a new static agent on the master ( test-websocket in this example) Enable "Use WebSocket" in it's configuration Start the agent: java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test" After ~ 30 sec the agent fails by exception: Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Handshake response not received. io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received. at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355) at hudson.remoting.Engine.runWebSocket(Engine.java:627) at hudson.remoting.Engine.run(Engine.java:469) The master log shows a single related entry: INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException Error on the agent after connection failed: java.nio.channels.ClosedChannelException at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141) at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90) at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119) at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317) at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42) [...]
            Hide
            ethorsa ethorsa added a comment -

            I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main(). The agent launches successfully, but it's marked as offline on the master.

             

            Remoting log (on Agent):

            hudson.remoting.jnlp.Main$CuiListener status
            INFO: WebSocket connection open
            hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connected

            Agent log (Icon of the node):

            Inbound agent connected from <Ip>
            

            Master Log:

            WARNING jenkins.agents.WebSocketAgents$Session error
            null
            java.io.IOException: Broken pipe
            	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
            	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
            	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
            	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
            	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496)
            	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
            Caused: org.eclipse.jetty.io.EofException
            	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
            	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
            	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
            	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
            	at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
            	at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
            	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
            	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
            	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
            	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
            	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
            	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
            	at java.base/java.lang.Thread.run(Thread.java:834)
            
            INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
            Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
            
            Show
            ethorsa ethorsa added a comment - I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main() . The agent launches successfully, but it's marked as offline on the master.   Remoting log (on Agent): hudson.remoting.jnlp.Main$CuiListener status INFO: WebSocket connection open hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Agent log (Icon of the node): Inbound agent connected from <Ip> Master Log: WARNING jenkins.agents.WebSocketAgents$Session error null java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263) Caused: org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264) at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193) at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Thread.java:834) INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
            Hide
            jthompson Jeff Thompson added a comment -

            Try Remoting (agent) 4.2.

            You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh.

            Remoting 4.2:

               Weekly releases 2.217 through 2.228

               LTS release 2.222.1

            Remoting 4.3:

              Weekly releases 2.229+

              Upcoming LTS

             

            If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.

            Show
            jthompson Jeff Thompson added a comment - Try Remoting (agent) 4.2. You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh. Remoting 4.2:    Weekly releases 2.217 through 2.228    LTS release 2.222.1 Remoting 4.3:   Weekly releases 2.229+   Upcoming LTS   If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.
            Hide
            ethorsa ethorsa added a comment -

            Thanks Jeff Thompson, downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets. 

             

            The WebSocket implementation is still in Beta, because we need to get extended real-world testing.

            I'll let you know if there are other issues.

            Show
            ethorsa ethorsa added a comment - Thanks Jeff Thompson , downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets.    The WebSocket implementation is still in Beta, because we need to get extended real-world testing. I'll let you know if there are other issues.
            ethorsa ethorsa made changes -
            Released As Jenkins v2.222.1 / Remoting v4.2
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
            ethorsa ethorsa made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            Hide
            jglick Jesse Glick added a comment -

            Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.

            Show
            jglick Jesse Glick added a comment - Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            jglick Jesse Glick added a comment -

            AFAICT this is all just about JENKINS-61409, which as indicated in that issue requires a matching upgrade on the controller and agent sides.

            Show
            jglick Jesse Glick added a comment - AFAICT this is all just about JENKINS-61409 , which as indicated in that issue requires a matching upgrade on the controller and agent sides.
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-61409 [ JENKINS-61409 ]
            jglick Jesse Glick made changes -
            Resolution Won't Fix [ 2 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

              People

              Assignee:
              jthompson Jeff Thompson
              Reporter:
              ethorsa ethorsa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: