Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62738

Can't connect JNLP agent trhough traefik2 TCP SNI ingressroutetcp

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We're trying to connect remote agents to a jenkins master deployed on k8s infra by using JNLP4 protocol. Traefik2 ingressroutetcp is like this:

      apiVersion: traefik.containo.us/v1alpha1
      kind: IngressRouteTCP
      metadata:
      name: jenkins-agent
      namespace: default
      spec:
      entryPoints:

      • tcp-ep
        routes:
      • match: HostSNI(`tcp.localhost.test`)
        kind: Rule
        services:
      • name: jenkins-test-agent
        port: 50000
        tls:
        secretName: tcp-localhost-test

      The connection is succesfull if we avoid the host SNI verification but if we want to evaluate the target host the connection never reaches jenkins agent port service. We also tried (-Djsse.enableSNIExtension=true):

      java -Djsse.enableSNIExtension=true -jar agent.jar -jnlpUrl http://jenkins.localhost.test:8080/computer/test-agent-tcp/slave-agent.jnlp -secret @secret-file -workDir "/tmp"

      but connection is not performed:

      Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirJun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /tmp/remoting as a remoting work directoryJun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager setupLoggingINFO: Both error and output logs will be printed to /tmp/remotingJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main createEngineINFO: Setting up agent: test-agent-tcpJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener <init>INFO: Jenkins agent is running in headless mode.Jun 21, 2020 5:33:10 PM hudson.remoting.Engine startEngineINFO: Using Remoting version: 4.3Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /tmp/remoting as a remoting work directoryJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among http://jenkins-test:8080/, http://jenkins.localhost.test:8080/Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]Jun 21, 2020 5:33:10 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability checkJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Agent discovery successful  Agent address: tcp.localhost.test  Agent port:    8081  Identity:      0e:89:a3:be:c8:76:25:b1:3d:36:74:68:9a:a6:63:fbJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: HandshakingJun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Connecting to tcp.localhost.test:8081Jun 21, 2020 5:33:10 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Trying protocol: JNLP4-connectJun 21, 2020 5:33:21 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Protocol JNLP4-connect encountered an unexpected exceptionjava.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223) at hudson.remoting.Engine.innerRun(Engine.java:743) at hudson.remoting.Engine.run(Engine.java:518)Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:283) at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164) at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.access$000(AckFilterLayer.java:45) at org.jenkinsci.remoting.protocol.impl.AckFilterLayer$1.run(AckFilterLayer.java:179) at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:964) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) at java.base/java.lang.Thread.run(Thread.java:834)
      Jun 21, 2020 5:33:21 PM hudson.remoting.jnlp.Main$CuiListener errorSEVERE: The server rejected the connection: None of the protocols were acceptedjava.lang.Exception: The server rejected the connection: None of the protocols were accepted at hudson.remoting.Engine.onConnectionRejected(Engine.java:828) at hudson.remoting.Engine.innerRun(Engine.java:768) at hudson.remoting.Engine.run(Engine.java:518)

      Can you please confirm if agent.jar is already prepared to send SNI information?

      Regards

       

        Attachments

          Activity

          Hide
          danielmurga daniel murga added a comment -

          Hi Jeff Thompson can you please a look, we've just changed the component to the right one (remoting)

          Best regards,

          Show
          danielmurga daniel murga added a comment - Hi Jeff Thompson can you please a look, we've just changed the component to the right one (remoting) Best regards,
          Hide
          jthompson Jeff Thompson added a comment -

          [Please note that Jenkins has deprecated the term "slaves" and replaced it with "agents". The replacements are ongoing in various locations. Please use the accepted terms in new usages.]

          Show
          jthompson Jeff Thompson added a comment - [Please note that Jenkins has deprecated the term "slaves" and replaced it with "agents". The replacements are ongoing in various locations. Please use the accepted terms in new usages.]
          Hide
          jthompson Jeff Thompson added a comment -

          I'm afraid I don't have any information on using SNI with Remoting. I wouldn't have high hopes for it working, though maybe someone has figured it out.

          You might try the new WebSockets implementation for agent-initiated connections. As it works over existing HTTPS mechanisms it may provide better results with SNI.

          Show
          jthompson Jeff Thompson added a comment - I'm afraid I don't have any information on using SNI with Remoting. I wouldn't have high hopes for it working, though maybe someone has figured it out. You might try the new WebSockets implementation for agent-initiated connections. As it works over existing HTTPS mechanisms it may provide better results with SNI.
          Hide
          alexander_kazakov Aleksandr Kazakov added a comment - - edited

          I found the same issue with envoy, dug in and found the actual reason. I think it's applicable to Traefik2 too.

          Long story short, Jenkins agent connects to a Jenkins instance over tcp, sends a few bytes with the protocol version it's going to use and upgrades the connection to SSL for JNLP4 protocol after that. Load balancer accepts connection, expects it to be secured (when you enable SNI check), sees that the client sends something unexpected (no ssl handshake at this point, unencrypted data with the protocol version goes first) and rejects the connection. Jenkins agent uses ssl tools provided by the java sdk that fully supports SNI. The problem is that jenkins agent doesn't use SSL in the beginning and if the load balancer doesn't support unsecured tcp connections (SNI check enabled) then the agent cannot connect to the Jenkins instance.

          What to do:
          1. Allow unsecured connections in load balancer.
          2. Use websockets instead of jnlp agents if possible as Jeff suggested.
          3. I couldn't use websockets for some reasons so I ended up patching the agent to wrap the connection in a ssl layer as the first step. Not ideal as it encrypts data twice but it works tho. Also jenkins does some weird stuff with certificates (agents get them from the http endpoint and use them to connect to the jnlp port, it doesn't work the jnlp and web port for some reason have different hostnames)

          Show
          alexander_kazakov Aleksandr Kazakov added a comment - - edited I found the same issue with  envoy , dug in and found the actual reason. I think it's applicable to Traefik2 too. Long story short, Jenkins agent connects to a Jenkins instance over tcp, sends a few bytes with the protocol version it's going to use and upgrades the connection to SSL for JNLP4 protocol after that. Load balancer accepts connection, expects it to be secured (when you enable SNI check), sees that the client sends something unexpected (no ssl handshake at this point, unencrypted data with the protocol version goes first) and rejects the connection. Jenkins agent uses ssl tools provided by the java sdk that fully supports SNI. The problem is that jenkins agent doesn't use SSL in the beginning and if the load balancer doesn't support unsecured tcp connections (SNI check enabled) then the agent cannot connect to the Jenkins instance. What to do: 1. Allow unsecured connections in load balancer. 2. Use websockets instead of jnlp agents if possible as Jeff suggested. 3. I couldn't use websockets for some reasons so I ended up patching the agent to wrap the connection in a ssl layer as the first step. Not ideal as it encrypts data twice but it works tho. Also jenkins does some weird stuff with certificates (agents get them from the http endpoint and use them to connect to the jnlp port, it doesn't work the jnlp and web port for some reason have different hostnames)

            People

            Assignee:
            peppe Giuseppe Landolfi
            Reporter:
            danielmurga daniel murga
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: