Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61212

CLI, Agent -websockets DeploymentException: Handshake response not received on jdk-11

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      2.317

      Description

      Testing CLI, agent connections with the new '-websocket' functionality added by
      JEP-222.

       

      Jetty access log shows:

      172.18.0.3 - - [25/Feb/2020:01:48:06 +0000] "GET /cli/ws HTTP/1.1" 101 0 "-" "-" 
      

      CLI output:

      javax.websocket.DeploymentException: Handshake response not received.
      	at org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:694)
      	at org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
      ...
      	at org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
      	at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
      	at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
      	at hudson.cli.CLI.webSocketConnection(CLI.java:323)
      	at hudson.cli.CLI._main(CLI.java:301)
      	at hudson.cli.CLI.main(CLI.java:95)
      

      When I attach a debugger to the Jenkins server it seems to get stuck here:
      https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/cli/CLIAction.java#L255

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - CC Jesse Glick
            Hide
            jglick Jesse Glick added a comment -

            Probably something specific to your environment. First try using JDK 8 on both client and server (11 is in general poorly tested in Jenkins). Then check your service setup. You are using the built-in Winstone/Jetty servlet container? How is HTTPS being terminated—by Jetty’s built-in options (never before tested with WebSocket AFAIK), or using some sort of reverse proxy (which)?

            Show
            jglick Jesse Glick added a comment - Probably something specific to your environment. First try using JDK 8 on both client and server (11 is in general poorly tested in Jenkins). Then check your service setup. You are using the built-in Winstone/Jetty servlet container? How is HTTPS being terminated—by Jetty’s built-in options (never before tested with WebSocket AFAIK), or using some sort of reverse proxy (which)?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            UPD: deleted comment, wrong ticket

            Show
            oleg_nenashev Oleg Nenashev added a comment - UPD: deleted comment, wrong ticket
            Hide
            build_admiral Fred Vogt added a comment -

            Jesse Glick - thanks  for the prompt response.

            You are probably right.  I'll do testing with different combinations of jdk-8 and without HTTPS today.

            It will take me a bit of time to come up with the smallest reproducible setup and test with jdk-8.

            server-*.pem is a self-signed cert with :

            localhost
            127.0.0.1
            jenkins-server
            

            google chrome likes it when imported (Mac/Linux)

            The client / agent are connecting using a different container over a bridge network.

            server:

            # winstone args
            --httpsPort=${HTTPS_PORT-8443} \
            --httpsPrivateKey=/var/jenkins_home/tls/server-rsa.pem \
            --httpsCertificate=/var/jenkins_home/tls/server-cert.pem} \
            --extraLibFolder=/var/jenkins_home/extraLibs \
            --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger \
            --simpleAccessLogger.format=combined \
            --simpleAccessLogger.file=/var/jenkins_home/logs/access.log  \
            

            client:

            java \
              -Djavax.net.ssl.trustStore=/var/lib/jenkins/truststore.jks \
              -Djavax.net.ssl.trustStorePassword=jenkins \
              -Djava.util.logging.config.file=/var/lib/jenkins/debug-logging.properties  \
              -jar "../jenkins-cli.jar" \
              -s "https://jenkins-server:8443" \
              -webSocket \
              -logger FINE \
              -auth "$AGENT_AUTH" \
              <args>
            
            Show
            build_admiral Fred Vogt added a comment - Jesse Glick - thanks  for the prompt response. You are probably right.  I'll do testing with different combinations of jdk-8 and without HTTPS today. It will take me a bit of time to come up with the smallest reproducible setup and test with jdk-8. server-*.pem is a self-signed cert with : localhost 127.0.0.1 jenkins-server google chrome likes it when imported (Mac/Linux) The client / agent are connecting using a different container over a bridge network. server: # winstone args --httpsPort=${HTTPS_PORT-8443} \ --httpsPrivateKey=/ var /jenkins_home/tls/server-rsa.pem \ --httpsCertificate=/ var /jenkins_home/tls/server-cert.pem} \ --extraLibFolder=/ var /jenkins_home/extraLibs \ --accessLoggerClassName=winstone.accesslog.SimpleAccessLogger \ --simpleAccessLogger.format=combined \ --simpleAccessLogger.file=/ var /jenkins_home/logs/access.log \ client: java \ -Djavax.net.ssl.trustStore=/ var /lib/jenkins/truststore.jks \ -Djavax.net.ssl.trustStorePassword=jenkins \ -Djava.util.logging.config.file=/ var /lib/jenkins/debug-logging.properties \ -jar "../jenkins-cli.jar" \ -s "https: //jenkins-server:8443" \ -webSocket \ -logger FINE \ -auth "$AGENT_AUTH" \ <args>
            Hide
            jglick Jesse Glick added a comment -

            Definitely I never tried to test against Winstone’s built-in --httpsPort, only against TLS termination done by reverse proxies (specifically Kubernetes ingress). In principle it ought to work to the extent that Jetty supports TLS + WebSocket.

            Offhand I suspect that the system properties you are passing to the client are not sufficient for the Tyrus library to recognize the custom certificate. Do you get the same error if you simply omit those properties? If so, this may just be a matter of inadequate error logging.

            Show
            jglick Jesse Glick added a comment - Definitely I never tried to test against Winstone’s built-in --httpsPort , only against TLS termination done by reverse proxies (specifically Kubernetes ingress). In principle it ought to work to the extent that Jetty supports TLS + WebSocket. Offhand I suspect that the system properties you are passing to the client are not sufficient for the Tyrus library to recognize the custom certificate. Do you get the same error if you simply omit those properties? If so, this may just be a matter of inadequate error logging.
            Hide
            build_admiral Fred Vogt added a comment - - edited

            Jesse Glick - I retested with jdk8 and the web-socket inbound cli/agent are working.

            I'll update with more findings later today when I have more info - and you can decide if this is a bug or a jdk-11 compat issue to be addressed as described at jenkins-on-java-11

            BTW - I was https with jdk-8 also.

            Show
            build_admiral Fred Vogt added a comment - - edited Jesse Glick - I retested with jdk8 and the web-socket inbound cli/agent are working. I'll update with more findings later today when I have more info - and you can decide if this is a bug or a jdk-11 compat issue to be addressed as described at jenkins-on-java-11 BTW - I was https with jdk-8 also.
            Hide
            jglick Jesse Glick added a comment -

            Interesting. Is it the client or the server that matters? There are smoke tests of WebSocket modes in the Jenkins core tree which get run on JDK 11 in CI, but this is not using Jetty-terminated TLS or custom certificates.

            Show
            jglick Jesse Glick added a comment - Interesting. Is it the client or the server that matters? There are smoke tests of WebSocket modes in the Jenkins core tree which get run on JDK 11 in CI, but this is not using Jetty-terminated TLS or custom certificates.
            Hide
            build_admiral Fred Vogt added a comment -

            Jesse Glick - I'll try to distill it down to the simplest reproducible jdk-11 setup (without TLS), and test it with your smoke test and websocat.

            Show
            build_admiral Fred Vogt added a comment - Jesse Glick - I'll try to distill it down to the simplest reproducible jdk-11 setup (without TLS), and test it with your smoke test and websocat.
            Hide
            build_admiral Fred Vogt added a comment -

            Jesse Glick -  hm... you are correct.

            jdk-11 with HTTP is working.

            TLS+websocket inbound cli/agent works with jdk-8, but NOT jdk-11.

             

            Show
            build_admiral Fred Vogt added a comment - Jesse Glick -  hm... you are correct. jdk-11 with HTTP is working. TLS+websocket inbound cli/agent works with jdk-8, but NOT jdk-11.  
            Hide
            build_admiral Fred Vogt added a comment -

            Very nice.  Using jdk-8 server / agent I can now use an AWS ALB for all Jenkins server ingress server-UI / CLI / Agents.

             

            Show
            build_admiral Fred Vogt added a comment - Very nice.  Using jdk-8 server / agent I can now use an AWS ALB for all Jenkins server ingress server-UI / CLI / Agents.  
            Hide
            jglick Jesse Glick added a comment -

            Still unclear from comments whether the issue is caused by using Java 11 for the client or the server (or both).

            Show
            jglick Jesse Glick added a comment - Still unclear from comments whether the issue is caused by using Java 11 for the client or the server (or both).
            Hide
            build_admiral Fred Vogt added a comment - - edited

            Jesse Glick just tested a bunch combinations, results below.

            https+websockets

            • jdk-11 server - the CLI/agent to hang (jdk-8 and jdk-11 agent).
            • jdk-8 server - the CLI/agent connect as expected (jdk-8 and jdk-11 agent)

            http+websockets

            • all combinations of server / CLI/agent jdk work as expected.

            TLDR - only jdk-11 server w/HTTPS port SERVER_URL hangs (client jdk doesn't matter in this case).

             

            To get a self-signed cert that google chrome likes, while using docker for mac, and being limited to the 1 server url is a bit tricky.

            When I can I'll try and extract out the simplest setup to reproduce the behavior. 

            Show
            build_admiral Fred Vogt added a comment - - edited Jesse Glick just tested a bunch combinations, results below. https+websockets jdk-11 server - the CLI/agent to hang (jdk-8 and jdk-11 agent). jdk-8 server - the CLI/agent connect as expected (jdk-8 and jdk-11 agent) http+websockets all combinations of server / CLI/agent jdk work as expected. TLDR - only jdk-11 server w/ HTTPS port SERVER_URL hangs (client jdk doesn't matter in this case).   To get a self-signed cert that google chrome likes, while using docker for mac, and being limited to the 1 server url is a bit tricky. When I can I'll try and extract out the simplest setup to reproduce the behavior. 
            Hide
            peterloron Peter Loron added a comment -

            Has there been any movement on this? We can't run JDK-8 and must use TLS for the traffic between agent and server. Forcing people to use an old JDK or NOT use encryption doesn't seem like a minor bug. It's a security issue. It's a hard blocker for us.

            Show
            peterloron Peter Loron added a comment - Has there been any movement on this? We can't run JDK-8 and must use TLS for the traffic between agent and server. Forcing people to use an old JDK or NOT use encryption doesn't seem like a minor bug. It's a security issue. It's a hard blocker for us.
            Hide
            build_admiral Fred Vogt added a comment -

            Peter Loron - I haven't gotten around around to creating a github repo with a small reproducible case yet.

            Once we have that - the Jenkins core contributors devs are fairly responsive.

            I can take stab at this later today.

            Show
            build_admiral Fred Vogt added a comment - Peter Loron - I haven't gotten around around to creating a github repo with a small reproducible case yet. Once we have that - the Jenkins core contributors devs are fairly responsive. I can take stab at this later today.
            Hide
            jglick Jesse Glick added a comment -

            I have no immediate plans to work on this—Java 11 support in Jenkins remains more or less experimental. I would recommend running on Java 8 unless you have a particular reason you must run on 11. (In which case you can continue to use TCP inbound agents.)

            Show
            jglick Jesse Glick added a comment - I have no immediate plans to work on this—Java 11 support in Jenkins remains more or less experimental. I would recommend running on Java 8 unless you have a particular reason you must run on 11. (In which case you can continue to use TCP inbound agents.)
            Hide
            danielbeck Daniel Beck added a comment -

            Jesse Glick

            Java 11 support in Jenkins remains more or less experimental

            More or less experimental than remoting over websocket?

            FWIW we announced full Java 11 support more than a year ago, so even if there are some kinks due to very low usage numbers relative to Java 8, admins have a reasonable expectation of it working.

            Show
            danielbeck Daniel Beck added a comment - Jesse Glick Java 11 support in Jenkins remains more or less experimental More or less experimental than remoting over websocket? FWIW we announced full Java 11 support more than a year ago, so even if there are some kinks due to very low usage numbers relative to Java 8, admins have a reasonable expectation of it working.
            Hide
            build_admiral Fred Vogt added a comment - - edited

            Jesse Glick, Daniel Beck, Peter Loron - Ok, I did put together a "minimal" Github repo to help reproduce the issue.

            https://github.com/fred-vogt/jenkins-websockets-tester

            However it doesn't exhibit the hanging issue. Hmm.
            One config item missing in the sample here is global security.
            I'll add that in and retest when I get a chance and report back.

            This setup uses local HTTPS, with default host web browser.
            Jenkins server, agent running in docker.

            Show
            build_admiral Fred Vogt added a comment - - edited Jesse Glick , Daniel Beck , Peter Loron - Ok, I did put together a "minimal" Github repo to help reproduce the issue. https://github.com/fred-vogt/jenkins-websockets-tester However it doesn't exhibit the hanging issue. Hmm. One config item missing in the sample here is global security. I'll add that in and retest when I get a chance and report back. This setup uses local HTTPS, with default host web browser. Jenkins server, agent running in docker.
            Hide
            jglick Jesse Glick added a comment -

            I should add that the motivating use case for JEP-222 is Jenkins running behind a reverse proxy (such as Kubernetes ingress), in which case TLS is typically terminated at the proxy. If the problem involves some sort of incompatibility between the TLS implementation in Tyrus (or whatever library it is using for that) on the client side vs. the TLS implementation in Jetty running on Java 11 on the server side, then it would presumably not affect this environment.

            Show
            jglick Jesse Glick added a comment - I should add that the motivating use case for JEP-222 is Jenkins running behind a reverse proxy (such as Kubernetes ingress), in which case TLS is typically terminated at the proxy. If the problem involves some sort of incompatibility between the TLS implementation in Tyrus (or whatever library it is using for that) on the client side vs. the TLS implementation in Jetty running on Java 11 on the server side, then it would presumably not affect this environment.
            Hide
            jpschewe jpschewe added a comment -

            I've tried setting up a node with the websocket option and got the handshake error specified here. Both the node and the Jenkins host are using openJDK 11. The Jenkins host is behind Apache using mod_proxy. Apache is terminating the TLS connection. Switching off the websocket option and opening the JNLP port allows my node to connect to the host.

             

            Show
            jpschewe jpschewe added a comment - I've tried setting up a node with the websocket option and got the handshake error specified here. Both the node and the Jenkins host are using openJDK 11. The Jenkins host is behind Apache using mod_proxy. Apache is terminating the TLS connection. Switching off the websocket option and opening the JNLP port allows my node to connect to the host.  
            Hide
            build_admiral Fred Vogt added a comment -

            jpschewe - I haven't isolated the smallest reproducible case.

            You encountering this issue is at least another data point.

            Show
            build_admiral Fred Vogt added a comment - jpschewe - I haven't isolated the smallest reproducible case. https://github.com/fred-vogt/jenkins-websockets-tester - [no authentication] + [TLS] - no hang https://github.com/cicdenv/cicdenv/tree/master/jenkins - [Github OAuth] + [TLS] - hangs You encountering this issue is at least another data point.
            Hide
            jpschewe jpschewe added a comment -

            I'm starting my node on Linux with the following command inside a shell script. The shell script is launched as an unprivileged user from systemd.

            nohup java -jar agent.jar -jnlpUrl https://${jenkins_host}/computer/${jenkins_node_name}/slave-agent.jnlp -secret ${jenkins_node_secret} -workDir "${HOME}" > "${HOME}"/jenkins-node.log 2>&1 
            Show
            jpschewe jpschewe added a comment - I'm starting my node on Linux with the following command inside a shell script. The shell script is launched as an unprivileged user from systemd. nohup java -jar agent.jar -jnlpUrl https: //${jenkins_host}/computer/${jenkins_node_name}/slave-agent.jnlp -secret ${jenkins_node_secret} -workDir "${HOME}" > "${HOME}" /jenkins-node.log 2>&1
            Hide
            jpschewe jpschewe added a comment -

            I have the same error when I enable the websocket option for a node that is running on Windows 10 with Oracle Java 1.8.

            Show
            jpschewe jpschewe added a comment - I have the same error when I enable the websocket option for a node that is running on Windows 10 with Oracle Java 1.8.
            Hide
            benbrummer Benjamin Brummer added a comment - - edited

            UPD 2020-10-16: Caused by wrong configured Websocket on Apache Reverse Proxy

            Jenkins Master on Ubuntu Server with OpenJDK 8 + Agent on Windows 10 with AdoptOpenJDK 8

            Okt 01, 2020 1:01:04 PM hudson.remoting.jnlp.Main$CuiListener error
            SEVERE: Handshake error.
            io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake error.
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:674)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712)
                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866)
                    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355)
                    at hudson.remoting.Engine.runWebSocket(Engine.java:628)
                    at hudson.remoting.Engine.run(Engine.java:470)
            Caused by: io.jenkins.remoting.shaded.org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 400.
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.TyrusClientEngine.processResponse(TyrusClientEngine.java:320)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.ClientFilter.processRead(ClientFilter.java:189)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:134)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.SslFilter.handleRead(SslFilter.java:406)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.SslFilter.processRead(SslFilter.java:368)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:134)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:299)
                    at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:283)
                    at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
                    at sun.nio.ch.Invoker$2.run(Invoker.java:218)
                    at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                    at java.lang.Thread.run(Thread.java:748)
            Show
            benbrummer Benjamin Brummer added a comment - - edited UPD 2020-10-16: Caused by wrong configured Websocket on Apache Reverse Proxy Jenkins Master on Ubuntu Server with OpenJDK 8 + Agent on Windows 10 with AdoptOpenJDK 8 Okt 01, 2020 1:01:04 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: Handshake error. io.jenkins.remoting.shaded.javax.websocket.DeploymentException: Handshake error. at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:674) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:712) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:866) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:511) at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:355) at hudson.remoting.Engine.runWebSocket(Engine.java:628) at hudson.remoting.Engine.run(Engine.java:470) Caused by: io.jenkins.remoting.shaded.org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 400. at io.jenkins.remoting.shaded.org.glassfish.tyrus.client.TyrusClientEngine.processResponse(TyrusClientEngine.java:320) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.ClientFilter.processRead(ClientFilter.java:189) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:134) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.SslFilter.handleRead(SslFilter.java:406) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.SslFilter.processRead(SslFilter.java:368) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:134) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.Filter.onRead(Filter.java:136) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:299) at io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.TransportFilter$4.completed(TransportFilter.java:283) at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) at sun.nio.ch.Invoker$2.run(Invoker.java:218) at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)
            Hide
            jglick Jesse Glick added a comment -

            Benjamin Brummer this issue is about a specific condition (Java 11 server, TLS terminated at Jenkins rather than a reverse proxy). “Handshake errors” under other conditions are likely unrelated problems, possibly something with your environment.

            Show
            jglick Jesse Glick added a comment - Benjamin Brummer this issue is about a specific condition (Java 11 server, TLS terminated at Jenkins rather than a reverse proxy). “Handshake errors” under other conditions are likely unrelated problems, possibly something with your environment.
            Hide
            jglick Jesse Glick added a comment -

            At least superficially sounds like https://github.com/eclipse-ee4j/tyrus/issues/613 but could probably be unrelated.

            Ought to try upgrading Tyrus as a first step.

            Show
            jglick Jesse Glick added a comment - At least superficially sounds like https://github.com/eclipse-ee4j/tyrus/issues/613 but could probably be unrelated. Ought to try upgrading Tyrus as a first step.
            Hide
            jglick Jesse Glick added a comment -

            https://github.com/eclipse-ee4j/tyrus/issues/676 also sounds similar, though that is about the Java version of the client.

            Show
            jglick Jesse Glick added a comment - https://github.com/eclipse-ee4j/tyrus/issues/676 also sounds similar, though that is about the Java version of the client .
            Hide
            build_admiral Fred Vogt added a comment -

            I'll be revisiting this issue once I've made some updates to my setup.

            Show
            build_admiral Fred Vogt added a comment - I'll be revisiting this issue once I've made some updates to my setup.
            Show
            jglick Jesse Glick added a comment - Please try https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/remoting/4.6-rc2798.e8ee0de5c872/remoting-4.6-rc2798.e8ee0de5c872.jar from https://github.com/jenkinsci/remoting/pull/403 to see if that makes a difference.
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            A quick note: in most cases that I have seen with the error "Response code was not 101: 400", it was due to a front load balancer not supporting Websockets. A concrete example is when using an AWS Classic Load Balancer with Layer 7 listeners (HTTP / HTTPS) doing TLS Termination: that particular load balancer when termintaing TLS just override headers and does not forward the websocket request headers (such as sec-websocket-key, sec-websocket-version, Upgrade: websocket, ...).

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited A quick note: in most cases that I have seen with the error "Response code was not 101: 400", it was due to a front load balancer not supporting Websockets. A concrete example is when using an AWS Classic Load Balancer with Layer 7 listeners (HTTP / HTTPS) doing TLS Termination: that particular load balancer when termintaing TLS just override headers and does not forward the websocket request headers (such as sec-websocket-key , sec-websocket-version , Upgrade: websocket , ...).
            Hide
            build_admiral Fred Vogt added a comment - - edited

            Jesse Glick I can confirm that that the issue is still present with:

            • core `v2.263` / remoting `4.5` + jdk11
            • core `v2.263` / remoting `4.6-rc2798.e8ee0de5c872` + jdk11

            I tested this both as `4.5` (server) `4.6` (agent) , and `4.6` (server+agent).

            Show
            build_admiral Fred Vogt added a comment - - edited Jesse Glick I can confirm that that the issue is still present with: core `v2.263` / remoting `4.5` + jdk11 core `v2.263` / remoting `4.6-rc2798.e8ee0de5c872` + jdk11 I tested this both as `4.5` (server) `4.6` (agent) , and `4.6` (server+agent).
            Hide
            felipecassiors Felipe Santos added a comment -

            I can confirm this issue is still present with AdoptOpenJDK 11.0.1+9 and Remoting 4.9.

            Show
            felipecassiors Felipe Santos added a comment - I can confirm this issue is still present with AdoptOpenJDK 11.0.1+9 and Remoting 4.9.
            Hide
            gturek Gabriella added a comment -

            Also confirm that this bug is present in Remoting 4.9. Tested on both CentOS 8.2 (openjdk version "1.8.0_252") and Mac OS X Big Bear (java 11.0.9 2020-10-20 LTS)

            Show
            gturek Gabriella added a comment - Also confirm that this bug is present in Remoting 4.9. Tested on both CentOS 8.2 (openjdk version "1.8.0_252") and Mac OS X Big Bear (java 11.0.9 2020-10-20 LTS)
            Hide
            vlatombe Vincent Latombe added a comment - - edited

            You can pass -Djdk.tls.client.protocols=TLSv1.2 to the agent JVM to workaround.

            The root cause is documented in https://github.com/eclipse-ee4j/tyrus/issues/676, and apparently fixed by https://github.com/eclipse-ee4j/tyrus/pull/707 but not in a release yet.

            Show
            vlatombe Vincent Latombe added a comment - - edited You can pass -Djdk.tls.client.protocols=TLSv1.2 to the agent JVM to workaround. The root cause is documented in https://github.com/eclipse-ee4j/tyrus/issues/676 , and apparently fixed by https://github.com/eclipse-ee4j/tyrus/pull/707 but not in a release yet.
            Hide
            jglick Jesse Glick added a comment -

            If https://github.com/jenkinsci/remoting/pull/478 gets a green build, https://repo.jenkins-ci.org/native/incrementals/org/jenkins-ci/main/remoting/4.11-rc2954.2fca0f20eb1d/remoting-4.11-rc2954.2fca0f20eb1d.jar should be published which would be an agent JAR that (if I understand correctly) would solve this issue. Anyone able to reproduce this issue who cares to try it?

            Show
            jglick Jesse Glick added a comment - If https://github.com/jenkinsci/remoting/pull/478 gets a green build, https://repo.jenkins-ci.org/native/incrementals/org/jenkins-ci/main/remoting/4.11-rc2954.2fca0f20eb1d/remoting-4.11-rc2954.2fca0f20eb1d.jar should be published which would be an agent JAR that (if I understand correctly) would solve this issue. Anyone able to reproduce this issue who cares to try it?
            Hide
            felipecassiors Felipe Santos added a comment -

            I'm willing to test, but sadly, it seems that the build failed.

            Show
            felipecassiors Felipe Santos added a comment - I'm willing to test, but sadly, it seems that the build failed.
            Hide
            jglick Jesse Glick added a comment - - edited

            Indeed. CI seems to be broken in trunk—some sort of environmental issue (lack of RAM?), pending some work from a maintainer. If you are set up to do so, you could run a Maven build locally:

            mvn -Pquick-build package
            

            using the resulting target/remoting-4.11-SNAPSHOT.jar. I could of course attach a binary here but you should not be in the habit of running binaries built from my laptop!

            Show
            jglick Jesse Glick added a comment - - edited Indeed. CI seems to be broken in trunk—some sort of environmental issue (lack of RAM?), pending some work from a maintainer. If you are set up to do so, you could run a Maven build locally: mvn -Pquick-build package using the resulting target/remoting-4.11-SNAPSHOT.jar . I could of course attach a binary here but you should not be in the habit of running binaries built from my laptop!
            Hide
            oxygenxo Andrey Babushkin added a comment -

            I've faced this when tried to use jenkins/inbound-agent:4.10-2-alpine based Docker image for jnlp containers in Kubernetes pods, but, in my case, switching to Ubuntu 20.04 based image helped. Its configuration includes:

            $ java -version
            openjdk version "11.0.11" 2021-04-20
            OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
            OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
            

            and quite old agent.jar version:

            $ java -jar /usr/share/jenkins/agent.jar -version
            3.36
            
            Show
            oxygenxo Andrey Babushkin added a comment - I've faced this when tried to use jenkins/inbound-agent:4.10-2-alpine based Docker image for jnlp containers in Kubernetes pods, but, in my case, switching to Ubuntu 20.04 based image helped. Its configuration includes: $ java -version openjdk version "11.0.11" 2021-04-20 OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04) OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing) and quite old agent.jar version: $ java -jar /usr/share/jenkins/agent.jar -version 3.36
            Hide
            jeffret Jeff Thompson added a comment -

            Andrey Babushkin keep in mind that 3.36 lacks security fixes that have been published.

            Show
            jeffret Jeff Thompson added a comment - Andrey Babushkin keep in mind that 3.36 lacks security fixes that have been published.
            Hide
            danielbeck Daniel Beck added a comment -

            Mark Waite Have you seen confirmation that this issue is resolved?

            Show
            danielbeck Daniel Beck added a comment - Mark Waite Have you seen confirmation that this issue is resolved?
            Hide
            build_admiral Fred Vogt added a comment -

            Daniel Beck Mark Waite - I noticed this was mentioned in the `v2.313` https://www.jenkins.io/changelog/.

             

            I can update my original test environment to confirm the fix is in `v2.313`.

            Show
            build_admiral Fred Vogt added a comment - Daniel Beck Mark Waite  - I noticed this was mentioned in the `v2.313` https://www.jenkins.io/changelog/.   I can update my original test environment to confirm the fix is in `v2.313`.
            Hide
            markewaite Mark Waite added a comment -

            Thanks Fred Vogt. Daniel Beck I set this as "Closed" because it was mentioned in the 2.313 changelog and did not seem to have obvious other cases which would prompt deeper testing before marking it as "Fixed in 2.313".

            Should I be doing more rigorous verification of fixes that are in Jenkins core weekly releases before they are listed as "Fixed in release x.yyy"?

            Show
            markewaite Mark Waite added a comment - Thanks Fred Vogt . Daniel Beck I set this as "Closed" because it was mentioned in the 2.313 changelog and did not seem to have obvious other cases which would prompt deeper testing before marking it as "Fixed in 2.313". Should I be doing more rigorous verification of fixes that are in Jenkins core weekly releases before they are listed as "Fixed in release x.yyy"?
            Hide
            danielbeck Daniel Beck added a comment -

            I think this one is unusual, as even Jesse as PR author wrote he didn't test and we have no other confirmation yet. No need to build process around this.

            Fred Vogt That would be very helpful.

            Show
            danielbeck Daniel Beck added a comment - I think this one is unusual, as even Jesse as PR author wrote he didn't test and we have no other confirmation yet. No need to build process around this. Fred Vogt That would be very helpful.
            Hide
            jglick Jesse Glick added a comment -

            This was certainly not fixed for agents; https://github.com/jenkinsci/remoting/pull/478 was just merged, and has not yet been released, much less incorporated into core and that released. It may have been fixed for the CLI.

            Show
            jglick Jesse Glick added a comment - This was certainly not fixed for agents; https://github.com/jenkinsci/remoting/pull/478 was just merged, and has not yet been released, much less incorporated into core and that released. It may have been fixed for the CLI.
            Show
            jglick Jesse Glick added a comment - You can try running https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/remoting/4.11-rc2954.2fca0f20eb1d/remoting-4.11-rc2954.2fca0f20eb1d.jar as an agent JAR.
            Hide
            felipecassiors Felipe Santos added a comment -

            I was able to successfully launch WebSocket JDK11 agents using self-signed CA with the custom agent.jar posted above, via https://github.com/felipecrs/jenkins-agent-dind/pull/34 which can be tested with:

            docker pull ghcr.io/felipecrs/jenkins-agent-dind:pr-34
            

            In case someone else wants to try.

            Show
            felipecassiors Felipe Santos added a comment - I was able to successfully launch WebSocket JDK11 agents using self-signed CA with the custom agent.jar posted above, via https://github.com/felipecrs/jenkins-agent-dind/pull/34 which can be tested with: docker pull ghcr.io/felipecrs/jenkins-agent-dind:pr-34 In case someone else wants to try.
            Hide
            jglick Jesse Glick added a comment -

            launch WebSocket JDK11 agents

            Remember that, as per comment of 2020-03-02, the crucial issue seems to be whether the controller was running 8 or 11. So please test that specifically, if you have not done so already.

            Show
            jglick Jesse Glick added a comment - launch WebSocket JDK11 agents Remember that, as per comment of 2020-03-02, the crucial issue seems to be whether the controller was running 8 or 11. So please test that specifically, if you have not done so already.
            Hide
            felipecassiors Felipe Santos added a comment -

            My controller is using JDK11 always. But regardless, I previously tested the Agent with JDK11 (without the fix) and it was not working.

            Show
            felipecassiors Felipe Santos added a comment - My controller is using JDK11 always. But regardless, I previously tested the Agent with JDK11 (without the fix) and it was not working.
            Hide
            jglick Jesse Glick added a comment -

            Good news—that means we can have some confidence in marking this fixed once the remoting change is integrated into a core weekly.

            Show
            jglick Jesse Glick added a comment - Good news—that means we can have some confidence in marking this fixed once the remoting change is integrated into a core weekly.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            *  The fix was released in Remoting 4.11

            Show
            oleg_nenashev Oleg Nenashev added a comment - *  The fix was released in Remoting 4.11 https://github.com/jenkinsci/jenkins/pull/5821  picks up the fix into the Weekly. Should be released next week in 2.317

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              build_admiral Fred Vogt
              Votes:
              9 Vote for this issue
              Watchers:
              24 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: