Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61253

Remoting using WebSocket fails with "Handshake response not received" Exception

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • remoting
    • None
    • - Remoting v4.2
      - AdoptOpenJDK v11.0.5 (Windows 10)
      - Jenkins v2.204.2
    • Jenkins v2.222.1 / Remoting v4.2

    Description

      Using remoting v4.2 with WebSocket crashes after ~ 30 seconds with a "Handshake response not received" exception:
       

      io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager <init>
      CONFIG: Provider class loaded: io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.JdkClientContainer
      hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Handshake response not received.
      io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
      	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
      	at hudson.remoting.Engine.runWebSocket(Engine.java:624)
      	at hudson.remoting.Engine.run(Engine.java:469)
      

       

      Remoting is launched through hudson.remoting.jnlp.Main.main() with arguments: 

      [<Secret ID>, <Agent>, -url, https://<Master Host>:<Port>/, -webSocket, -workDir, <WorkDir Path> -headless, -noreconnect]

      Attachments

        Issue Links

          Activity

            ethorsa ethorsa created issue -
            ethorsa ethorsa made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-61212 [ JENKINS-61212 ]
            ethorsa ethorsa made changes -
            Link This issue relates to JENKINS-22877 [ JENKINS-22877 ]
            ethorsa ethorsa added a comment -

            Cli seems to have a similar issue (JENKINS-61212).

            ethorsa ethorsa added a comment - Cli seems to have a similar issue ( JENKINS-61212 ).
            ethorsa ethorsa made changes -
            Environment - Remoting v4.2
            - AdoptOpenJDK v11.0.5 (Windows 10)
            - Jenkins v2.204.2
            - Remoting v4.2
            - AdoptOpenJDK v11.0.5, Oracle Java v1.8.0-181 (Windows 10)
            - Jenkins v2.204.2
            ethorsa ethorsa made changes -
            Environment - Remoting v4.2
            - AdoptOpenJDK v11.0.5, Oracle Java v1.8.0-181 (Windows 10)
            - Jenkins v2.204.2
            - Remoting v4.2
            - AdoptOpenJDK v11.0.5 (Windows 10)
            - Jenkins v2.204.2
            jthompson Jeff Thompson added a comment -

            You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217.

            jthompson Jeff Thompson added a comment - You need to upgrade to a later Jenkins version. The server side components went into Jenkins 2.217 .
            jthompson Jeff Thompson made changes -
            Resolution Not A Defect [ 7 ]
            Status Open [ 1 ] Closed [ 6 ]
            ethorsa ethorsa added a comment -

            I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?

            ethorsa ethorsa added a comment - I see, thanks. What LTS version ships the WS support – 2.204.3 or the one after that?
            jthompson Jeff Thompson added a comment -

            As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.

            jthompson Jeff Thompson added a comment - As WebSocket is a beta feature, there is no expectation that it would be backported from 2.217 to an earlier LTS. The first LTS to have WebSocket will be the one following 2.217. It looks like that is going to be based on 2.222 and should be out within the coming weeks.
            ethorsa ethorsa added a comment -

            Great, thanks!

            ethorsa ethorsa added a comment - Great, thanks!
            ethorsa ethorsa made changes -
            Resolution Not A Defect [ 7 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            ethorsa ethorsa added a comment - - edited

            Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error:

            1. Create a new static agent on the master (test-websocket in this example)
            2. Enable "Use WebSocket" in it's configuration
            3. Start the agent:
            java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test"

            After ~ 30 sec the agent fails by exception:

            Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error
            SEVERE: Handshake response not received.
            io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received.
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
            	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
            	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
            	at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
            	at hudson.remoting.Engine.runWebSocket(Engine.java:627)
            	at hudson.remoting.Engine.run(Engine.java:469)
            

            The master log shows a single related entry:

            INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
            Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException
            

            Error on the agent after connection failed:

            java.nio.channels.ClosedChannelException
            	at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141)
            	at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90)
            	at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source)
            	at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
            	at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317)
            	at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42)
            [...]
            
            ethorsa ethorsa added a comment - - edited Unfortunately I run into the same problem with Jenkins 2.222.1 (Remoting 4.3). My minimal steps to reproduce the error: Create a new static agent on the master ( test-websocket in this example) Enable "Use WebSocket" in it's configuration Start the agent: java -jar remoting-4.3.jar -jnlpUrl https://<Master>:<Port>/computer/test-websocket/slave-agent.jnlp -secret <SecretID> -workDir "C:\temp\jenkins-test" After ~ 30 sec the agent fails by exception: Mar 30, 2020 9:35:24 AM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Handshake response not received. io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake response not received. at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866) at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355) at hudson.remoting.Engine.runWebSocket(Engine.java:627) at hudson.remoting.Engine.run(Engine.java:469) The master log shows a single related entry: INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-2975 for test-websocket terminated: java.nio.channels.ClosedChannelException Error on the agent after connection failed: java.nio.channels.ClosedChannelException at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:141) at jenkins.websocket.WebSocketSession.onWebSocketSomething(WebSocketSession.java:90) at com.sun.proxy.$Proxy118.onWebSocketClose(Unknown Source) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119) at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:389) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:317) at org.eclipse.jetty.websocket.common.io.DisconnectCallback.succeeded(DisconnectCallback.java:42) [...]
            ethorsa ethorsa added a comment -

            I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main(). The agent launches successfully, but it's marked as offline on the master.

             

            Remoting log (on Agent):

            hudson.remoting.jnlp.Main$CuiListener status
            INFO: WebSocket connection open
            hudson.remoting.jnlp.Main$CuiListener status
            INFO: Connected

            Agent log (Icon of the node):

            Inbound agent connected from <Ip>
            

            Master Log:

            WARNING jenkins.agents.WebSocketAgents$Session error
            null
            java.io.IOException: Broken pipe
            	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
            	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
            	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
            	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
            	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496)
            	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263)
            Caused: org.eclipse.jetty.io.EofException
            	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283)
            	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
            	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
            	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
            	at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
            	at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
            	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
            	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511)
            	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
            	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
            	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
            	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
            	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
            	at java.base/java.lang.Thread.run(Thread.java:834)
            
            INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
            Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
            
            ethorsa ethorsa added a comment - I've made a test on a Linux based Master (Jenkins v2.2221, Remoting v4.3, OpenJDK 11) with an agent configured as above but passing the arguments directly to remoting main() . The agent launches successfully, but it's marked as offline on the master.   Remoting log (on Agent): hudson.remoting.jnlp.Main$CuiListener status INFO: WebSocket connection open hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Agent log (Icon of the node): Inbound agent connected from <Ip> Master Log: WARNING jenkins.agents.WebSocketAgents$Session error null java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182) at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:263) Caused: org.eclipse.jetty.io.EofException at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:283) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264) at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193) at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:584) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:511) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.base/java.lang.Thread.run(Thread.java:834) INFO jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed Jetty (winstone)-1001 for <Agent Host> terminated: java.nio.channels.ClosedChannelException
            jthompson Jeff Thompson added a comment -

            Try Remoting (agent) 4.2.

            You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh.

            Remoting 4.2:

               Weekly releases 2.217 through 2.228

               LTS release 2.222.1

            Remoting 4.3:

              Weekly releases 2.229+

              Upcoming LTS

             

            If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.

            jthompson Jeff Thompson added a comment - Try Remoting (agent) 4.2. You're experiencing the fun of being on the bleeding edge. The WebSocket implementation is still in Beta, because we need to get extended real-world testing. It first came out in an LTS release last week, with Remoting 4.2. Unfortunately, at the same time someone reported a problem with large payloads, which we fixed last week. Remoting 4.3 addresses that problem, but, because it's still in Beta, also requires an upgrade to the master, Jenkins 2.229. That fix should get merged back to the next Jenkins LTS refresh. Remoting 4.2:    Weekly releases 2.217 through 2.228    LTS release 2.222.1 Remoting 4.3:   Weekly releases 2.229+   Upcoming LTS   If you're not using the experimental WebSocket feature, then Remoting 4.2 and 4.3 are equivalent.
            ethorsa ethorsa added a comment -

            Thanks jthompson, downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets. 

             

            The WebSocket implementation is still in Beta, because we need to get extended real-world testing.

            I'll let you know if there are other issues.

            ethorsa ethorsa added a comment - Thanks jthompson , downgrading to v4.2 solved the issue and I was able to connect agents over WebSockets.    The WebSocket implementation is still in Beta, because we need to get extended real-world testing. I'll let you know if there are other issues.
            ethorsa ethorsa made changes -
            Released As Jenkins v2.222.1 / Remoting v4.2
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
            ethorsa ethorsa made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            jglick Jesse Glick added a comment -

            Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.

            jglick Jesse Glick added a comment - Nothing here was “fixed” that I can see. A FIXED bug should link to a PR merged in a particular release.
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick added a comment -

            AFAICT this is all just about JENKINS-61409, which as indicated in that issue requires a matching upgrade on the controller and agent sides.

            jglick Jesse Glick added a comment - AFAICT this is all just about JENKINS-61409 , which as indicated in that issue requires a matching upgrade on the controller and agent sides.
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-61409 [ JENKINS-61409 ]
            jglick Jesse Glick made changes -
            Resolution Won't Fix [ 2 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            basil Basil Crow made changes -
            Link This issue relates to JENKINS-61212 [ JENKINS-61212 ]
            basil Basil Crow made changes -
            Assignee Jeff Thompson [ jthompson ]
            Resolution Won't Fix [ 2 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            basil Basil Crow added a comment -

            Duplicates JENKINS-61212.

            basil Basil Crow added a comment - Duplicates JENKINS-61212 .
            basil Basil Crow made changes -
            Link This issue duplicates JENKINS-61212 [ JENKINS-61212 ]
            basil Basil Crow made changes -
            Resolution Duplicate [ 3 ]
            Status Reopened [ 4 ] Closed [ 6 ]

            People

              Unassigned Unassigned
              ethorsa ethorsa
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: