Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63656

Jenkins 2.249.1 agents unable to connect to alternate URL

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • Jenkins running on Windows Server 2016
      New version of Jenkins (2.249.1)
      Java jdk1.8.0_101\bin\java.exe

      After upgrade all workers fail to connect:

      Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener <init>
      Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Sep 11, 2020 7:54:48 PM hudson.remoting.Engine startEngine
      INFO: Using Remoting version: 3.29
      Sep 11, 2020 7:54:48 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using D:\Jenkins\remoting as a remoting work directory
      Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [https://jenkins.cdl.af.mil/]
      Sep 11, 2020 7:54:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Sep 11, 2020 7:54:53 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
      WARNING: connect timed out
      Sep 11, 2020 7:54:53 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: https://jenkins.cdl.af.mil/ provided port:51001 is not reachable
      java.io.IOException: https://jenkins.cdl.af.mil/ provided port:51001 is not reachable 
      at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287) 
      at hudson.remoting.Engine.innerRun(Engine.java:523) 
      at hudson.remoting.Engine.run(Engine.java:474)
      Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main createEngine
      INFO: Setting up agent: JenkinsWorkerOne
      Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main$CuiListener <init>
      INFO: Jenkins agent is running in headless mode.
      Sep 11, 2020 7:54:55 PM hudson.remoting.Engine startEngine
      INFO: Using Remoting version: 3.29
      Sep 11, 2020 7:54:55 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
      INFO: Using D:\Jenkins\remoting as a remoting work directory
      Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [https://jenkins.cdl.af.mil/]
      

       
      After reverting to previous version (Jenkins 2.235.5) they all connect again.

       

      Sep 11, 2020 8:37:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
      Agent address: 10.101.0.203
      Agent port: 52142
      Identity: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25
      Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to 10.101.0.203:52142
      Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25
      Sep 11, 2020 8:37:01 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connected

        1. image-2020-09-13-10-25-16-349.png
          image-2020-09-13-10-25-16-349.png
          5 kB
        2. jenkins-slave.xml
          4 kB
        3. remoting.log.0
          1 kB
        4. remoting.log.4
          0.8 kB

          [JENKINS-63656] Jenkins 2.249.1 agents unable to connect to alternate URL

          Larry Charbonneau created issue -
          Mark Waite made changes -
          Description Original: After upgrade all workers fail to connect:

          Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener <init>Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener <init>INFO: Jenkins agent is running in headless mode.Sep 11, 2020 7:54:48 PM hudson.remoting.Engine startEngineINFO: Using Remoting version: 3.29Sep 11, 2020 7:54:48 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using D:\Jenkins\remoting as a remoting work directorySep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [https://jenkins.cdl.af.mil/]Sep 11, 2020 7:54:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolveINFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]Sep 11, 2020 7:54:53 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisibleWARNING: connect timed outSep 11, 2020 7:54:53 PM hudson.remoting.jnlp.Main$CuiListener errorSEVERE: https://jenkins.cdl.af.mil/ provided port:51001 is not reachablejava.io.IOException: https://jenkins.cdl.af.mil/ provided port:51001 is not reachable at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287) at hudson.remoting.Engine.innerRun(Engine.java:523) at hudson.remoting.Engine.run(Engine.java:474)
          Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main createEngineINFO: Setting up agent: JenkinsWorkerOneSep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main$CuiListener <init>INFO: Jenkins agent is running in headless mode.Sep 11, 2020 7:54:55 PM hudson.remoting.Engine startEngineINFO: Using Remoting version: 3.29Sep 11, 2020 7:54:55 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using D:\Jenkins\remoting as a remoting work directorySep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [https://jenkins.cdl.af.mil/]

           

          After reverting to previous version ([Jenkins 2.235.5|https://jenkins.io/]) they all connect again.

           

          Sep 11, 2020 8:37:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Agent discovery successful
           Agent address: 10.101.0.203
           Agent port: 52142
           Identity: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Handshaking
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to 10.101.0.203:52142
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Trying protocol: JNLP4-connect
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Remote identity confirmed: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25
          Sep 11, 2020 8:37:01 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connected
          New: After upgrade all workers fail to connect:

          {noformat}
          Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener <init>
          Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener <init>
          INFO: Jenkins agent is running in headless mode.
          Sep 11, 2020 7:54:48 PM hudson.remoting.Engine startEngine
          INFO: Using Remoting version: 3.29
          Sep 11, 2020 7:54:48 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using D:\Jenkins\remoting as a remoting work directory
          Sep 11, 2020 7:54:48 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Locating server among [https://jenkins.cdl.af.mil/]
          Sep 11, 2020 7:54:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Sep 11, 2020 7:54:53 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
          WARNING: connect timed out
          Sep 11, 2020 7:54:53 PM hudson.remoting.jnlp.Main$CuiListener error
          SEVERE: https://jenkins.cdl.af.mil/ provided port:51001 is not reachable
          java.io.IOException: https://jenkins.cdl.af.mil/ provided port:51001 is not reachable
          at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:287)
          at hudson.remoting.Engine.innerRun(Engine.java:523)
          at hudson.remoting.Engine.run(Engine.java:474)
          Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main createEngine
          INFO: Setting up agent: JenkinsWorkerOne
          Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main$CuiListener <init>
          INFO: Jenkins agent is running in headless mode.
          Sep 11, 2020 7:54:55 PM hudson.remoting.Engine startEngine
          INFO: Using Remoting version: 3.29
          Sep 11, 2020 7:54:55 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
          INFO: Using D:\Jenkins\remoting as a remoting work directory
          Sep 11, 2020 7:54:55 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Locating server among [https://jenkins.cdl.af.mil/]
          {noformat}
           
          After reverting to previous version ([Jenkins 2.235.5|https://jenkins.io/]) they all connect again.

           

          Sep 11, 2020 8:37:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Agent discovery successful
           Agent address: 10.101.0.203
           Agent port: 52142
           Identity: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Handshaking
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connecting to 10.101.0.203:52142
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Trying protocol: JNLP4-connect
          Sep 11, 2020 8:37:00 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Remote identity confirmed: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25
          Sep 11, 2020 8:37:01 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Connected

          Mark Waite added a comment -

          Some ideas that may help as you explore the failure.

          The failing console output reports that port 51001 is not reachable, while in the successful case it reports that agent port 52142 is used. That may indicate that Jenkins is configured to use a random port for inbound agents. It might also indicate that Jenkins was configured to open port 51001 but the port is somehow blocked on that computer.

          The remoting version on the agent is 3.29. That was released in Jan 2019. Jenkins 2.249.1 includes remoting 4.5, released two months ago. There may be an incompatibility between the port negotiation being done by remoting 3.29 and Jenkins 2.249.1. You could try upgrading the agent.jar file from the older version you're running to see if that helps.

          The Java version 8u121 was released in 2017. Oracle expired that release with the April 2017 release of the next Oracle critical patch update. The current AdoptOpenJDK release is 8u262.

          Mark Waite added a comment - Some ideas that may help as you explore the failure. The failing console output reports that port 51001 is not reachable, while in the successful case it reports that agent port 52142 is used. That may indicate that Jenkins is configured to use a random port for inbound agents. It might also indicate that Jenkins was configured to open port 51001 but the port is somehow blocked on that computer. The remoting version on the agent is 3.29. That was released in Jan 2019. Jenkins 2.249.1 includes remoting 4.5, released two months ago. There may be an incompatibility between the port negotiation being done by remoting 3.29 and Jenkins 2.249.1. You could try upgrading the agent.jar file from the older version you're running to see if that helps. The Java version 8u121 was released in 2017. Oracle expired that release with the April 2017 release of the next Oracle critical patch update. The current AdoptOpenJDK release is 8u262.
          Larry Charbonneau made changes -
          Attachment New: remoting.log.0 [ 52586 ]
          Attachment New: remoting.log.4 [ 52587 ]

          So, I updated the agent.jar and got same result.

           

          I added in two remoting logs .... look at them.  I see a difference, the new version of Jenkins is the remoting.log.4 and the old version is remoting.log.0.

          The difference is 

          New : INFO: Locating server among https://jenkins.cdl.af.mil/

          Old: INFO: Locating server among https://jenkins.cdl.af.mil/, http://10.101.0.203:8090/

          it has two URL's and the second is the one that connects.  We need to connect via IP based on location of the worker server and not come in the front door based on security.

          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Sep 12, 2020 12:10:57 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
          WARNING: connect timed out
          Sep 12, 2020 12:10:57 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Sep 12, 2020 12:10:57 PM hudson.remoting.jnlp.Main$CuiListener status
          INFO: Agent discovery successful
          Agent address: 10.101.0.203
          Agent port: 61926
          Identity: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25 

           

          Notice it fails on first URL but success on second.  The new version only has the one URL and it just fails like old but does not have second URL to try.

           

           

          Larry Charbonneau added a comment - So, I updated the agent.jar and got same result.   I added in two remoting logs .... look at them.  I see a difference, the new version of Jenkins is the remoting.log.4 and the old version is remoting.log.0. The difference is  New : INFO: Locating server among https://jenkins.cdl.af.mil/ Old: INFO: Locating server among https://jenkins.cdl.af.mil/, http://10.101.0.203:8090/ it has two URL's and the second is the one that connects.  We need to connect via IP based on location of the worker server and not come in the front door based on security. INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Sep 12, 2020 12:10:57 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible WARNING: connect timed out Sep 12, 2020 12:10:57 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Sep 12, 2020 12:10:57 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: 10.101.0.203 Agent port: 61926 Identity: 40:97:48:b3:eb:09:11:33:91:c9:18:ab:c5:f2:91:25    Notice it fails on first URL but success on second.  The new version only has the one URL and it just fails like old but does not have second URL to try.    
          Larry Charbonneau made changes -
          Attachment New: jenkins-slave.xml [ 52588 ]

          Mark Waite added a comment -

          Good insight. I wonder if that is related to the comment in the Jenkins 2.245.1 changelog that says:

          Remove the fallback Jenkins URL from the JNLP launch file so that WebSocket agents can be connected over Java Web Start

          The agent.jar still accepts the -jnlpUrl argument, but possibly the Jenkins controller is ignoring the argument?

          JENKINS-63014 seems like it might be related

          Mark Waite added a comment - Good insight. I wonder if that is related to the comment in the Jenkins 2.245.1 changelog that says: Remove the fallback Jenkins URL from the JNLP launch file so that WebSocket agents can be connected over Java Web Start The agent.jar still accepts the -jnlpUrl argument, but possibly the Jenkins controller is ignoring the argument? JENKINS-63014 seems like it might be related

          So is there anything we can do?  Or does it need a Jenkins change?

          Larry Charbonneau added a comment - So is there anything we can do?  Or does it need a Jenkins change?

          Mark Waite added a comment -

          I think that the comment from jglick in JENKINS-63014 hints that there may be a different set of arguments you can pass to agent.jar that will cause it to connect to the 10.101.0.203 address and port.

          You might try configuring your Jenkins server to use a fixed port for inbound agents ("Manage Jenkins" -> "Configure Global Security"), then use that fixed port in the agent.jar command line.

          Mark Waite added a comment - I think that the comment from jglick in JENKINS-63014 hints that there may be a different set of arguments you can pass to agent.jar that will cause it to connect to the 10.101.0.203 address and port. You might try configuring your Jenkins server to use a fixed port for inbound agents ("Manage Jenkins" -> "Configure Global Security"), then use that fixed port in the agent.jar command line.

          The port is not the issue.  I need for the URL to be the IP 10.101.0.203 ..... that's whats in the agent config.  How else can I tell it to use the correct URL?

           

          I don't know what Jesse Glick is referring to "you can still do that with TCP agents, using the lower-level and more explicit launch mode" .... any insight?

          Larry Charbonneau added a comment - The port is not the issue.  I need for the URL to be the IP 10.101.0.203 ..... that's whats in the agent config.  How else can I tell it to use the correct URL?   I don't know what Jesse Glick is referring to "you can still do that with TCP agents, using the lower-level and more explicit launch mode" .... any insight?

            Unassigned Unassigned
            charbl2007 Larry Charbonneau
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: