Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64352

Unable to connect to jnlp via ssh tunnel port

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Won't Do
    • Component/s: remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.235.2 (OK) and 2.249.3 (broken)
      Jenkins running on Ubuntu 5.4 / Java 1.8.0_212 / Tomcat 8.5.40 (bitnami image on AWS)
      Agent running on: Red Hat 4.8 / Java 1.8.0_262
    • Similar Issues:

      Description

      Running with jenkins 2.235.2 we could connect between firewalled machines using ssh tunnels:

      ssh -M -S ${JENKINS_HOME}/jenkins-ssh-control -N -f \
       -L 8080:jenkins.build.example.com:80 \
       -L 8081:jenkins.build.example.com:8081 \
       jumpbox.build.example.com

      and then, downloading the agent.jar from jenkins to keep it up-to-date:

      curl -f -s -o ${JENKINS_HOME}/agent.jar http://localhost:8080/jenkins/jnlpJars/agent.jar
      

      and starting the agent:

      java -jar ${JENKINS_HOME}/agent.jar \
       -jnlpUrl http://localhost:8080/jenkins/computer/${AGENT_NAME}/slave-agent.jnlp \
       -secret ${AGENT_SECRET} -workDir "${JENKINS_HOME}" > ${JENKINS_HOME}/agent.log 2>&1 &

      Sometime after that version, this no longer works:

      Dec 02, 2020 1:44:03 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Failed to connect to http://jenkins.build.example.com/jenkins/tcpSlaveAgentListener/: jenkins.build.example.com
      java.io.IOException: Failed to connect to http://jenkins.build.example.com/jenkins/tcpSlaveAgentListener/: jenkins.build.example.com
       at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:217)
       at hudson.remoting.Engine.innerRun(Engine.java:694)
       at hudson.remoting.Engine.run(Engine.java:519)
      Caused by: java.net.UnknownHostException: jenkins.build.example.com
       at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
       at java.net.Socket.connect(Socket.java:607)
       at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
       at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
       at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
       at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) at sun.net.www.http.HttpClient.New(HttpClient.java:339)
       at sun.net.www.http.HttpClient.New(HttpClient.java:357)
       at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
       at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
       at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
       at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
       at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:214) ... 2 more
      

       

      I believe this is due to https://github.com/jenkinsci/remoting/commit/9ce46eb9be1b35b5d6cb66e6c63b28a2e3798b31

      That change no longer uses the agentJnlpURL to parse/resolve the DOM for jnlp, so the result is the remote trying to connect to the "real" host, rather than the tunnel as specified in the agent's --jnlpUrl parameter.

       

      We are unable to upgrade jenkins (or alternatively we must maintain an old version of the agent.jar).

        Attachments

          Activity

          landers Dave Landers created issue -
          Hide
          jthompson Jeff Thompson added a comment -

          Hmm ... sounds like you've got a fairly unusual connection setup that has no tests or documentation. Support for tunneling has been piecemeal at best. Let's see what we can do address your situation.

          First, can you share information about your "jnlp" file? I can use this for analysis and might be able to create a test for this situation based on it.

          Second, could the direct connection mechanism introduced in Remoting 3.34 help in your situation?

          Show
          jthompson Jeff Thompson added a comment - Hmm ... sounds like you've got a fairly unusual connection setup that has no tests or documentation. Support for tunneling has been piecemeal at best. Let's see what we can do address your situation. First, can you share information about your "jnlp" file? I can use this for analysis and might be able to create a test for this situation based on it. Second, could the direct connection mechanism introduced in Remoting 3.34 help in your situation?
          Hide
          landers Dave Landers added a comment -

          First:  jnlp from jenkins 2.235.2:

          <jnlp codebase="http://jenkins.build.example.com/jenkins/computer/my-agent/" spec="1.0+">
            <information>
              <title>Agent for my-agent</title>
              <vendor>Jenkins project</vendor>
              <homepage href="https://jenkins-ci.org/"></homepage>
            </information>
            <security>
              <all-permissions></all-permissions>
            </security>
            <resources>
              <j2se version="1.8+"></j2se>
              <jar href="http://jenkins.build.example.com/jenkins/jnlpJars/remoting.jar"></jar>
            </resources>
              <application-desc main-class="hudson.remoting.jnlp.Main">
              <argument>2f8dd268f2dff2d1c9663a70bcf2f4c68e8c886054eb08c40cdaabc5780dd16c</argument>
              <argument>my-agent</argument>
              <argument>-workDir</argument>
              <argument>/home/jenkins/jenkins_home</argument>
              <argument>-internalDir</argument>
              <argument>remoting</argument>
              <argument>-url</argument>
              <argument>http://jenkins.build.example.com/jenkins/</argument>
              <argument>-url</argument>
              <argument>http://localhost:8080/jenkins/</argument>
            </application-desc>
          </jnlp>

          jnlp from jenkins 2.263.1:

           

          <jnlp codebase="http://jenkins.build.example.com/jenkins/computer/my-agent/" spec="1.0+">
            <information>
              <title>Agent for my-agent</title>
              <vendor>Jenkins project</vendor>
              <homepage href="https://jenkins-ci.org/"></homepage>
            </information>
            <security>
              <all-permissions></all-permissions>
              </security>
            <resources>
              <j2se version="1.8+"></j2se>
              <jar href="http://jenkins.build.example.com/jenkins/jnlpJars/remoting.jar"></jar>
            </resources>
            <application-desc main-class="hudson.remoting.jnlp.Main">
              <argument>2f8dd268f2dff2d1c9663a70bcf2f4c68e8c886054eb08c40cdaabc5780dd16c</argument>
              <argument>my-agent</argument>
              <argument>-workDir</argument>
              <argument>/home/jenkins/jenkins_home</argument>
              <argument>-internalDir</argument>
              <argument>remoting</argument>
              <argument>-url</argument>
              <argument>http://jenkins.build.example.com/jenkins/</argument>
            </application-desc>
          </jnlp>

          Note the url argument from the newer version is different - this is I believe what is used to make the remote connection, and the difference in how the DOM is resolved (I did a simple test with DocumentBuilder.parse() and it does make this difference).

           

           

          Second:  The direct connection does seem to be working. I had trouble finding docs - was unable to find anything relevant in the docs on jenkins.io, there is nothing on the node config page (in jenkins server). Fortunately (I thought) you mentioned the exact version number for remoting so I dig thru release notes and found issue JENKINS-53461 which Iinks to docs that 404 (https://github.com/jenkinsci/remoting/blob/master/docs/tcpAgent.md) . Aargh.

          Finally dug around in github and found https://github.com/jenkinsci/remoting/blob/master/docs/inbound-agent.md, which got things working.

          Given the difficulty I had in finding this, makes me wonder if it's a real/supported feature or not.  IIRC, I've been doing some form of the jnlp-via-ssh-tunnel thing on and off since hudson.

           

           

           

           

          Show
          landers Dave Landers added a comment - First:  jnlp from jenkins 2.235.2: <jnlp codebase= "http: //jenkins.build.example.com/jenkins/computer/my-agent/" spec= "1.0+" > <information> <title>Agent for my-agent</title> <vendor>Jenkins project</vendor> <homepage href= "https: //jenkins-ci.org/" ></homepage> </information> <security> <all-permissions></all-permissions> </security> <resources> <j2se version= "1.8+" ></j2se> <jar href= "http: //jenkins.build.example.com/jenkins/jnlpJars/remoting.jar" ></jar> </resources> <application-desc main-class= "hudson.remoting.jnlp.Main" > <argument>2f8dd268f2dff2d1c9663a70bcf2f4c68e8c886054eb08c40cdaabc5780dd16c</argument> <argument>my-agent</argument> <argument>-workDir</argument> <argument>/home/jenkins/jenkins_home</argument> <argument>-internalDir</argument> <argument>remoting</argument> <argument>-url</argument> <argument>http: //jenkins.build.example.com/jenkins/</argument> <argument>-url</argument> <argument>http: //localhost:8080/jenkins/</argument> </application-desc> </jnlp> jnlp from jenkins 2.263.1:   <jnlp codebase= "http: //jenkins.build.example.com/jenkins/computer/my-agent/" spec= "1.0+" > <information> <title>Agent for my-agent</title> <vendor>Jenkins project</vendor> <homepage href= "https: //jenkins-ci.org/" ></homepage> </information> <security> <all-permissions></all-permissions> </security> <resources> <j2se version= "1.8+" ></j2se> <jar href= "http: //jenkins.build.example.com/jenkins/jnlpJars/remoting.jar" ></jar> </resources> <application-desc main-class= "hudson.remoting.jnlp.Main" > <argument>2f8dd268f2dff2d1c9663a70bcf2f4c68e8c886054eb08c40cdaabc5780dd16c</argument> <argument>my-agent</argument> <argument>-workDir</argument> <argument>/home/jenkins/jenkins_home</argument> <argument>-internalDir</argument> <argument>remoting</argument> <argument>-url</argument> <argument>http: //jenkins.build.example.com/jenkins/</argument> </application-desc> </jnlp> Note the url argument from the newer version is different - this is I believe what is used to make the remote connection, and the difference in how the DOM is resolved (I did a simple test with DocumentBuilder.parse() and it does make this difference).     Second:  The direct connection does seem to be working. I had trouble finding docs - was unable to find anything relevant in the docs on jenkins.io, there is nothing on the node config page (in jenkins server). Fortunately (I thought) you mentioned the exact version number for remoting so I dig thru release notes and found issue JENKINS-53461 which Iinks to docs that 404 ( https://github.com/jenkinsci/remoting/blob/master/docs/tcpAgent.md)  . Aargh. Finally dug around in github and found  https://github.com/jenkinsci/remoting/blob/master/docs/inbound-agent.md , which got things working. Given the difficulty I had in finding this, makes me wonder if it's a real/supported feature or not.  IIRC, I've been doing some form of the jnlp-via-ssh-tunnel thing on and off since hudson.        
          Hide
          jthompson Jeff Thompson added a comment -

          My apologies for not linking directly to that documentation. It escaped my mind. I've been wanting to get better documentation somewhere about configuring agents, but it never rises high enough on the priority list. There are various different configurations and options, including yours, which makes it more difficult.

          I'm looking at the information you provided.

          Show
          jthompson Jeff Thompson added a comment - My apologies for not linking directly to that documentation. It escaped my mind. I've been wanting to get better documentation somewhere about configuring agents, but it never rises high enough on the priority list. There are various different configurations and options, including yours, which makes it more difficult. I'm looking at the information you provided.
          Hide
          jthompson Jeff Thompson added a comment -

          If you can use the "Direct inbound TCP agent connection" mechanism, I recommend going that way. This is a official, supported mechanism that was added to support a variety of scenarios, particularly a number of new, popular ones.

          Digging through the changes and behavior, it looks like the change that broke your scenario is https://github.com/jenkinsci/jenkins/pull/4839 . This removed the extra "-url" argument from the JNLP file. It's unlikely that change would be reverted without strong motivation.

          Show
          jthompson Jeff Thompson added a comment - If you can use the "Direct inbound TCP agent connection" mechanism, I recommend going that way. This is a official, supported mechanism that was added to support a variety of scenarios, particularly a number of new, popular ones. Digging through the changes and behavior, it looks like the change that broke your scenario is https://github.com/jenkinsci/jenkins/pull/4839 . This removed the extra "-url" argument from the JNLP file. It's unlikely that change would be reverted without strong motivation.
          Hide
          jthompson Jeff Thompson added a comment -

          Dave Landers, is there any reason not to close this, given that a workaround (or rather, a different solution) exists?

          Show
          jthompson Jeff Thompson added a comment - Dave Landers , is there any reason not to close this, given that a workaround (or rather, a different solution) exists?
          Hide
          landers Dave Landers added a comment -

          can be closed

          Show
          landers Dave Landers added a comment - can be closed
          Hide
          jthompson Jeff Thompson added a comment -

          Thanks for confirming

          Show
          jthompson Jeff Thompson added a comment - Thanks for confirming
          jthompson Jeff Thompson made changes -
          Field Original Value New Value
          Resolution Won't Do [ 10001 ]
          Status Open [ 1 ] Closed [ 6 ]

            People

            Assignee:
            jthompson Jeff Thompson
            Reporter:
            landers Dave Landers
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: