Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59094

Agent tunnel connection issues after ugrading Jenkins to 2.191

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • remoting
    • Windows 7
      Jenkins 2.191
      Oracle JRE jre1.8.0_171
      AdoptOpenJDK jre-8.0.222.10-hotspot
    • Remoting 3.35

      After updating Jenkins master to 2.191 windows slave thows following exception:

      Aug 27, 2019 8:41:10 AM hudson.remoting.jnlp.Main createEngine
      INFORMATION: Setting up agent: spsde05ws016
      Aug 27, 2019 8:41:10 AM hudson.remoting.jnlp.Main$CuiListener <init>
      INFORMATION: Jenkins agent is running in headless mode.
      Aug 27, 2019 8:41:11 AM hudson.remoting.jnlp.Main createEngine
      WARNUNG: Certificate validation for HTTPs endpoints is disabled
      Aug 27, 2019 8:41:11 AM hudson.remoting.Engine startEngine
      INFORMATION: Using Remoting version: 3.34
      Aug 27, 2019 8:41:11 AM hudson.remoting.Engine startEngine
      WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
      Aug 27, 2019 8:41:11 AM hudson.remoting.jnlp.Main$CuiListener status
      INFORMATION: Locating server among [https://tools.xxxxx.local/devjenkins/]
      Aug 27, 2019 8:41:11 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
      WARNUNG: HTTPs certificate check is disabled for the endpoint.
      Aug 27, 2019 8:41:11 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Aug 27, 2019 8:41:11 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFORMATION: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
      Aug 27, 2019 8:41:11 AM hudson.remoting.jnlp.Main$CuiListener error
      SCHWERWIEGEND: For input string: ""
      java.lang.NumberFormatException: For input string: ""
      	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
      	at java.lang.Integer.parseInt(Integer.java:592)
      	at java.lang.Integer.parseInt(Integer.java:615)
      	at org.jenkinsci.remoting.engine.HostPort.splitHostPort(HostPort.java:25)
      	at org.jenkinsci.remoting.engine.HostPort.<init>(HostPort.java:8)
      	at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:329)
      	at hudson.remoting.Engine.innerRun(Engine.java:529)
      	at hudson.remoting.Engine.run(Engine.java:490)
      

       

       

          [JENKINS-59094] Agent tunnel connection issues after ugrading Jenkins to 2.191

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - It was caused by  https://github.com/jenkinsci/remoting/pull/338/files  . CC jthompson

          Jeff Thompson added a comment -

          I tested on Windows and Unix under a few configurations. Another contributor also tested a variety of other configurations. Somehow this one got missed.

          I'm looking into it.

          Jeff Thompson added a comment - I tested on Windows and Unix under a few configurations. Another contributor also tested a variety of other configurations. Somehow this one got missed. I'm looking into it.

          Jeff Thompson added a comment -

          dmazuronak, could you clarify your configuration? The log messages indicate that you are using tunneling. Based upon the error messages, it looks like your tunnel parameter looks something like "myjenkins:", in other words, no port number is specified. Or it's specified in an unusual way I didn't account for. You could provide the parameters you use in your launch command, particularly anything relating to tunneling.

          You could also provide log messages from an agent connection sequence from a working Jenkins, before 2.191 that shows how the port is handled.

          I've got a change that might make it work like it did before but I want to make sure I'm addressing the problem before submitting.

          On my Windows test system I wasn't able to reproduce the problem with a basic installation.

           

          Jeff Thompson added a comment - dmazuronak , could you clarify your configuration? The log messages indicate that you are using tunneling. Based upon the error messages, it looks like your tunnel parameter looks something like "myjenkins:", in other words, no port number is specified. Or it's specified in an unusual way I didn't account for. You could provide the parameters you use in your launch command, particularly anything relating to tunneling. You could also provide log messages from an agent connection sequence from a working Jenkins, before 2.191 that shows how the port is handled. I've got a change that might make it work like it did before but I want to make sure I'm addressing the problem before submitting. On my Windows test system I wasn't able to reproduce the problem with a basic installation.  

          Hello jthompson

          here is my configuration:

          Jenkins is running on the host J (x.x.3.9). The Jenkins web interface is available over reverse proxy running on the different host P (x.x.3.2). There are DNS alias tools.xxxxx.local, pointing to the host P. Jenkins windows slave runs on the host xxxxx05ws016 (x.x.13.82) as Windows Service, initially installed over Java Web Start.

          Here are some interesting sections from Jenkins Global Security Configuration page:

           

          Agents:
              TCP port for inbound agents:    Random
              Agent protocols:                Java-Web-Start-Agent protocol Version 4 (TLS-encryption)
          SSH Server:
              SSHD Port:         Deactivated
          

          And here are part of the windows agent log, after Jenkins was restored to the version 2.190:

          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main createEngine
          INFORMATION: Setting up agent: xxxxx05ws016
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener <init>
          INFORMATION: Jenkins agent is running in headless mode.
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main createEngine
          WARNUNG: Certificate validation for HTTPs endpoints is disabled
          Aug 27, 2019 8:43:45 AM hudson.remoting.Engine startEngine
          INFORMATION: Using Remoting version: 3.33
          Aug 27, 2019 8:43:45 AM hudson.remoting.Engine startEngine
          WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Locating server among [https://tools.xxx.local/devjenkins/]
          Aug 27, 2019 8:43:45 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
          WARNUNG: HTTPs certificate check is disabled for the endpoint.
          Aug 27, 2019 8:43:45 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Aug 27, 2019 8:43:45 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFORMATION: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Agent discovery successful
            Agent address: x.x.3.9
            Agent port:    33773
            Identity:      f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Handshaking
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Connecting to x.x.3.9:33773
          Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Trying protocol: JNLP4-connect
          Aug 27, 2019 8:43:46 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Remote identity confirmed: f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11
          Aug 27, 2019 8:43:48 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Connected
          
          

           

          Dzianis Mazuronak added a comment - Hello jthompson ,  here is my configuration: Jenkins is running on the host J (x.x.3.9). The Jenkins web interface is available over reverse proxy running on the different host P (x.x.3.2). There are DNS alias tools.xxxxx.local, pointing to the host P. Jenkins windows slave runs on the host xxxxx05ws016 (x.x.13.82) as Windows Service, initially installed over Java Web Start. Here are some interesting sections from Jenkins Global Security Configuration page:   Agents: TCP port for inbound agents: Random    Agent protocols:                Java-Web-Start-Agent protocol Version 4 (TLS-encryption) SSH Server:     SSHD Port:         Deactivated And here are part of the windows agent log, after Jenkins was restored to the version 2.190: Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main createEngine INFORMATION: Setting up agent: xxxxx05ws016 Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener <init> INFORMATION: Jenkins agent is running in headless mode. Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main createEngine WARNUNG: Certificate validation for HTTPs endpoints is disabled Aug 27, 2019 8:43:45 AM hudson.remoting.Engine startEngine INFORMATION: Using Remoting version: 3.33 Aug 27, 2019 8:43:45 AM hudson.remoting.Engine startEngine WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Locating server among [https: //tools.xxx.local/devjenkins/] Aug 27, 2019 8:43:45 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection WARNUNG: HTTPs certificate check is disabled for the endpoint. Aug 27, 2019 8:43:45 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Aug 27, 2019 8:43:45 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFORMATION: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Agent discovery successful Agent address: x.x.3.9 Agent port: 33773 Identity: f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11 Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Handshaking Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Connecting to x.x.3.9:33773 Aug 27, 2019 8:43:45 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Trying protocol: JNLP4-connect Aug 27, 2019 8:43:46 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Remote identity confirmed: f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11 Aug 27, 2019 8:43:48 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Connected  

          Jeff Thompson added a comment -

          Thanks a lot! I wondered if this might have something to do with the random port setting. I'll see if I can reproduce the problem setting up a system like that.

          If I create a PR for this, would you have any capability for trying it out before it's added into Jenkins?

          Jeff Thompson added a comment - Thanks a lot! I wondered if this might have something to do with the random port setting. I'll see if I can reproduce the problem setting up a system like that. If I create a PR for this, would you have any capability for trying it out before it's added into Jenkins?

          Jeff Thompson added a comment -

          Hmmm .... Actually one of my test systems does use a random port. But none are set up for tunneling, proxy, etc. Can you find information on the command-line arguments used to launch the agent? Depending on how you have things configured on the agent, it might be in jenkins-slave.xml. Or in some other file where the agent is installed.

          Jeff Thompson added a comment - Hmmm .... Actually one of my test systems does use a random port. But none are set up for tunneling, proxy, etc. Can you find information on the command-line arguments used to launch the agent? Depending on how you have things configured on the agent, it might be in jenkins-slave.xml. Or in some other file where the agent is installed.

          Dzianis Mazuronak added a comment - - edited

          Here is the jenkins-slave.xml file from my windows slave xxxxxx05ws016:

          <service>
            <id>jenkinsslave-C__projects_jenkins</id>
            <name>Jenkins agent (jenkinsslave-C__projects_jenkins)</name>
            <description>This service runs an agent for Jenkins automation server.</description>
            <executable>C:\Program Files (x86)\AdoptOpenJDK\jre-8.0.222.10-hotspot\bin\java.exe</executable>
            <arguments>-Xrs  -jar "%BASE%\slave.jar" -jnlpUrl https://tools.xxxxx.local/devjenkins/computer/xxxxx05ws016/slave-agent.jnlp -secret XXXXXXXXXXX -noCertificateCheck</arguments>
            <logmode>rotate</logmode>
            <onfailure action="restart" />
            <download from="https://tools.xxxxx.local/devjenkins/jnlpJars/slave.jar" to="%BASE%\slave.jar"/>
            <extensions>
              <!-- This is a sample configuration for the RunawayProcessKiller extension. -->
              <extension enabled="true" 
                         className="winsw.Plugins.RunawayProcessKiller.RunawayProcessKillerExtension"
                         id="killOnStartup">
                <pidfile>%BASE%\jenkins_agent.pid</pidfile>
                <stopTimeout>5000</stopTimeout>
                <stopParentFirst>false</stopParentFirst>
              </extension>
            </extensions>
          </service>
          

          Unfortunately, I'll not be able to test your PR. Compiling Jenkins from the source code is not a easy task.

          Dzianis Mazuronak added a comment - - edited Here is the jenkins-slave.xml file from my windows slave xxxxxx05ws016: <service> <id> jenkinsslave-C__projects_jenkins </id> <name> Jenkins agent (jenkinsslave-C__projects_jenkins) </name> <description> This service runs an agent for Jenkins automation server. </description> <executable> C:\Program Files (x86)\AdoptOpenJDK\jre-8.0.222.10-hotspot\bin\java.exe </executable> <arguments> -Xrs -jar "%BASE%\slave.jar" -jnlpUrl https://tools.xxxxx.local/devjenkins/computer/xxxxx05ws016/slave-agent.jnlp -secret XXXXXXXXXXX -noCertificateCheck </arguments> <logmode> rotate </logmode> <onfailure action= "restart" /> <download from= "https://tools.xxxxx.local/devjenkins/jnlpJars/slave.jar" to= "%BASE%\slave.jar" /> <extensions> <!-- This is a sample configuration for the RunawayProcessKiller extension. --> <extension enabled= "true" className= "winsw.Plugins.RunawayProcessKiller.RunawayProcessKillerExtension" id= "killOnStartup" > <pidfile> %BASE%\jenkins_agent.pid </pidfile> <stopTimeout> 5000 </stopTimeout> <stopParentFirst> false </stopParentFirst> </extension> </extensions> </service> Unfortunately, I'll not be able to test your PR. Compiling Jenkins from the source code is not a easy task.

          Jeff Thompson added a comment -

          PR uploaded at https://github.com/jenkinsci/remoting/pull/345 . This PR restores the parsing behavior for empty port and for empty host. The separator character is still required.

          I haven't been able to reproduce or test these other scenarios. It would be great if someone else with those setups could give it a try.

          If we can get some more reviews or testing, I should be able to get a new version out quickly.

          Jeff Thompson added a comment - PR uploaded at  https://github.com/jenkinsci/remoting/pull/345  . This PR restores the parsing behavior for empty port and for empty host. The separator character is still required. I haven't been able to reproduce or test these other scenarios. It would be great if someone else with those setups could give it a try. If we can get some more reviews or testing, I should be able to get a new version out quickly.

          Jeff Thompson added a comment -

          I updated the title to something that I think is a little clearer and more accurate.

          Jeff Thompson added a comment - I updated the title to something that I think is a little clearer and more accurate.

          Jeff Thompson added a comment -

          dmazuronak, I had planned to have an incrementals version published to the Jenkins Artifactory repository that you could consider testing. Unfortunately, it's not fully set up for that yet, but there are changes in the works to enable it. Without that, it is definitely harder to test.

          I'll see if anyone else is able to perform any reviews or tests. I hope to get Remoting released early next and included in the following Jenkins weekly.

          Jeff Thompson added a comment - dmazuronak , I had planned to have an incrementals version published to the Jenkins Artifactory repository that you could consider testing. Unfortunately, it's not fully set up for that yet, but there are changes in the works to enable it. Without that, it is definitely harder to test. I'll see if anyone else is able to perform any reviews or tests. I hope to get Remoting released early next and included in the following Jenkins weekly.

          Oleg Nenashev added a comment -

          https://github.com/jenkinsci/jenkins/pull/4186 for a temporary regression fix until the new version of Remoting is ready

          Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/4186  for a temporary regression fix until the new version of Remoting is ready

          Oleg Nenashev added a comment -

          A workaround has been released in 2.193. We hope to have a Remoting fix in the next releases

          Oleg Nenashev added a comment - A workaround has been released in 2.193. We hope to have a Remoting fix in the next releases

          Jeff Thompson added a comment -

          The problem turned out to be related to tunneling configurations, when the tunnel HOST:PORT is missing the HOST and / or PORT specification. In my change for Remoting 3.34, I had missed some magic whereby the values are filled in from prior sources. Remoting 3.3.5 restores these configuration paths / options and retains the new capabilities introduced in 3.34. I couldn't find any connection between this issue and the Windows platform, as the original title stated, so I've changed the title to better reflect the actual issue.

          Jeff Thompson added a comment - The problem turned out to be related to tunneling configurations, when the tunnel HOST:PORT is missing the HOST and / or PORT specification. In my change for Remoting 3.34, I had missed some magic whereby the values are filled in from prior sources. Remoting 3.3.5 restores these configuration paths / options and retains the new capabilities introduced in 3.34. I couldn't find any connection between this issue and the Windows platform, as the original title stated, so I've changed the title to better reflect the actual issue.

          Update to Jenkins version 2.194 was successful. Thank you very much for this fix.

          Dzianis Mazuronak added a comment - Update to Jenkins version 2.194 was successful. Thank you very much for this fix.

          Jeff Thompson added a comment -

          Great! I'm glad that solved it for you.

          Jeff Thompson added a comment - Great! I'm glad that solved it for you.

            jthompson Jeff Thompson
            dmazuronak Dzianis Mazuronak
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: