Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52204

Jenkins Windows Agent failed to start after upgrade to 2.1.129

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core, remoting
    • None
    • Jenkins 2.1.129
    • Jenkins 2.130, Remoting 3.23

      Jenkins Windows Agent failed to start after upgrade to version 2.1.129: Jenkins is running behind apache reverse proxy on different host that proxy itself. Parameter “tunnel connection through” is set correctly. From the error message we can see that remote agent was trying to connect to reverse proxy host instead of Jenkins master host.

      The version 2.1.128 works without issues.

          [JENKINS-52204] Jenkins Windows Agent failed to start after upgrade to 2.1.129

          Oleg Nenashev added a comment -

          Most likely it is related to JENKINS-51818, CC denami
          dmazuronak have you updated Remoting on agents to 3.22?

          Oleg Nenashev added a comment - Most likely it is related to JENKINS-51818 , CC denami dmazuronak have you updated Remoting on agents to 3.22?

          Remoting 3.22 has the problem. Remoting 3.21 works OK:

          INFORMATION: Setting up agent: MY-CLIENT
          Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener <init>
          INFORMATION: Jenkins agent is running in headless mode.
          Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main createEngine
          WARNUNG: Certificate validation for HTTPs endpoints is disabled
          Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine
          INFORMATION: Using Remoting version: 3.22
          Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine
          WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
          Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Locating server among [https://tools.domain.local/devjenkins/]
          Jun 26, 2018 11:07:54 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
          WARNUNG: HTTPs certificate check is disabled for the endpoint.
          Jun 26, 2018 11:07:55 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Jun 26, 2018 11:07:56 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
          WARNUNG: Connection refused: connect
          Jun 26, 2018 11:07:56 PM hudson.remoting.jnlp.Main$CuiListener error
          SCHWERWIEGEND: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable
          java.io.IOException: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable
          	at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:284)
          	at hudson.remoting.Engine.innerRun(Engine.java:518)
          	at hudson.remoting.Engine.run(Engine.java:469)
          
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine
          INFORMATION: Setting up agent: MY-CLIENT
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener <init>
          INFORMATION: Jenkins agent is running in headless mode.
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine
          WARNUNG: Certificate validation for HTTPs endpoints is disabled
          Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine
          INFORMATION: Using Remoting version: 3.21
          Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine
          WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Locating server among [https://tools.domain.local/devjenkins/]
          Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
          WARNUNG: HTTPs certificate check is disabled for the endpoint.
          Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
          INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Agent discovery successful
            Agent address: 10.150.3.9
            Agent port:    44864
            Identity:      f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Handshaking
          Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
          INFORMATION: Connecting to 10.150.3.9:44864
          

           Port 37367 is open on Jenkins server 10.150.3.9 and not on reverse proxy tools.domain.local.

          Dzianis Mazuronak added a comment - Remoting 3.22 has the problem. Remoting 3.21 works OK: INFORMATION: Setting up agent: MY-CLIENT Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener <init> INFORMATION: Jenkins agent is running in headless mode. Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main createEngine WARNUNG: Certificate validation for HTTPs endpoints is disabled Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine INFORMATION: Using Remoting version: 3.22 Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Locating server among [https://tools.domain.local/devjenkins/] Jun 26, 2018 11:07:54 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection WARNUNG: HTTPs certificate check is disabled for the endpoint. Jun 26, 2018 11:07:55 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jun 26, 2018 11:07:56 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible WARNUNG: Connection refused: connect Jun 26, 2018 11:07:56 PM hudson.remoting.jnlp.Main$CuiListener error SCHWERWIEGEND: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable java.io.IOException: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:284) at hudson.remoting.Engine.innerRun(Engine.java:518) at hudson.remoting.Engine.run(Engine.java:469) Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine INFORMATION: Setting up agent: MY-CLIENT Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener <init> INFORMATION: Jenkins agent is running in headless mode. Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine WARNUNG: Certificate validation for HTTPs endpoints is disabled Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine INFORMATION: Using Remoting version: 3.21 Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Locating server among [https://tools.domain.local/devjenkins/] Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection WARNUNG: HTTPs certificate check is disabled for the endpoint. Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Agent discovery successful Agent address: 10.150.3.9 Agent port: 44864 Identity: f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11 Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Handshaking Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Connecting to 10.150.3.9:44864  Port 37367 is open on Jenkins server 10.150.3.9 and not on reverse proxy tools.domain.local.

          Oleg Nenashev added a comment -

          From what I see in the https://github.com/jenkinsci/remoting/pull/275/files code, the logic should be using the resolved hostname for direct connection:

          host = defaultString(first(header(con, "X-Jenkins-JNLP-Host")), salURL.getHost());
          

          Are you sure that "hudson.TcpSlaveAgentListener.hostName" is set on your instance?
          If no, setting it up on the Master side would be a workaround until we figure out the tunneling issue

          Oleg Nenashev added a comment - From what I see in the https://github.com/jenkinsci/remoting/pull/275/files code, the logic should be using the resolved hostname for direct connection: host = defaultString(first(header(con, "X-Jenkins-JNLP-Host" )), salURL.getHost()); Are you sure that "hudson.TcpSlaveAgentListener.hostName" is set on your instance? If no, setting it up on the Master side would be a workaround until we figure out the tunneling issue

          On master side this slave has the parameter "Tunnel connection through" set to 10.150.3.9.

          Dzianis Mazuronak added a comment - On master side this slave has the parameter "Tunnel connection through" set to 10.150.3.9.

          Oleg Nenashev added a comment -

          dmazuronak Could you please set the "-Dhudson.TcpSlaveAgentListener.hostName" system property on master as I suggested above?

          Oleg Nenashev added a comment - dmazuronak Could you please set the "-Dhudson.TcpSlaveAgentListener.hostName" system property on master as I suggested above?

          Chris Rule added a comment - - edited

          I'm having the same issue. I have  jenkins running on host "j-jenkins" and the apache web-server running as reverse proxy on "eddard". Setting the hudson.TcpSlaveAgentListener.hostName property to "j-jenkins" did not correct the problem. I receive the same error as before "IOException: https://eddard/jenkins/ provided port:41829 is not reachable".

          Chris Rule added a comment - - edited I'm having the same issue. I have  jenkins running on host "j-jenkins" and the apache web-server running as reverse proxy on "eddard". Setting the hudson.TcpSlaveAgentListener.hostName property to "j-jenkins" did not correct the problem. I receive the same error as before "IOException: https://eddard/jenkins/ provided port:41829 is not reachable".

          Dzianis Mazuronak added a comment - - edited

          I can only confirm the information from Chris. Unfortunately, using the system property "hudson.TcpSlaveAgentListener.hostName" did not bring any improvement.

          There is no such header "X-Jenkins-JNLP-Host" in http response.

          The property "hudson.TcpSlaveAgentListener.hostName" was set in the command line. In which configuration file can this property be set? I'm not sure that the command line was processed.

          Dzianis Mazuronak added a comment - - edited I can only confirm the information from Chris. Unfortunately, using the system property "hudson.TcpSlaveAgentListener.hostName" did not bring any improvement. There is no such header "X-Jenkins-JNLP-Host" in http response. The property "hudson.TcpSlaveAgentListener.hostName" was set in the command line. In which configuration file can this property be set? I'm not sure that the command line was processed.

          Oleg Nenashev added a comment -

          java -Dhudson.TcpSlaveAgentListener.hostName=fooBar -jar jenkins.war should do the job.
          Cannot suggest config file since I do not know Jenkins configuration

          Oleg Nenashev added a comment - java -Dhudson.TcpSlaveAgentListener.hostName=fooBar -jar jenkins.war should do the job. Cannot suggest config file since I do not know Jenkins configuration

          You can receive value of "hudson.TcpSlaveAgentListener.hostName". By Jenkins  script console (System.getProperty("hudson.TcpSlaveAgentListener.hostName") ).  Possible it is override by other  JAVA_OPTS/JENKINS_JAVA_OPTIONS)

          Andrei Laptsinski added a comment - You can receive value of "hudson.TcpSlaveAgentListener.hostName". By Jenkins  script console (System.getProperty("hudson.TcpSlaveAgentListener.hostName") ).  Possible it is override by other  JAVA_OPTS/JENKINS_JAVA_OPTIONS)

          No, the other options are empty, so no override here.

          If I type

          println(System.getProperty("hudson.TcpSlaveAgentListener.hostName") )
          

          in Jenkins script console, I get only

          null
          

          as result.

           

          There is an other interresting result:

          hudson.TcpSlaveAgentListener.hostName
          

          produces following exception:

          groovy.lang.MissingPropertyException: No such property: hostName for class: hudson.TcpSlaveAgentListener
          	at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1002)
          	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1857)
          	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1833)
          	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
          	at org.codehaus.groovy.runtime.callsite.ClassMetaClassGetPropertySite.getProperty(ClassMetaClassGetPropertySite.java:51)
          	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
          	at Script1.run(Script1.groovy:1)
          

          Dzianis Mazuronak added a comment - No, the other options are empty, so no override here. If I type println( System .getProperty( "hudson.TcpSlaveAgentListener.hostName" ) ) in Jenkins script console, I get only null as result.   There is an other interresting result: hudson.TcpSlaveAgentListener.hostName produces following exception: groovy.lang.MissingPropertyException: No such property: hostName for class: hudson.TcpSlaveAgentListener at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1002) at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1857) at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1833) at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758) at org.codehaus.groovy.runtime.callsite.ClassMetaClassGetPropertySite.getProperty(ClassMetaClassGetPropertySite.java:51) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296) at Script1.run(Script1.groovy:1)

          Oleg Nenashev added a comment - - edited

          > There is an other interresting result:

          As designed. System properties won't be visible in Groovy in such way.

          dmazuronak System properties are not passed to your Jenkins master due to whatever reason, so the proposed workaround will not work until the property is passed correctly.

          Oleg Nenashev added a comment - - edited > There is an other interresting result: As designed. System properties won't be visible in Groovy in such way. dmazuronak System properties are not passed to your Jenkins master due to whatever reason, so the proposed workaround will not work until the property is passed correctly.

          Oleg Nenashev added a comment -

          See my comment in https://github.com/jenkinsci/remoting/pull/279 . After the review, this issue happens IFF "-tunnel" is set. I believe that "-tunnel" flag should be deprecated and replaced by offering a custom port in TcpAgentListener response headers like it's already done for hosts

          So I propose to just disable checks in that case if denami agrees. We could implement a more advanced behavior and checks for tunnel options as a part of JENKINS-52246.

          Oleg Nenashev added a comment - See my comment in https://github.com/jenkinsci/remoting/pull/279 . After the review, this issue happens IFF "-tunnel" is set. I believe that "-tunnel" flag should be deprecated and replaced by offering a custom port in TcpAgentListener response headers like it's already done for hosts So I propose to just disable checks in that case if denami agrees. We could implement a more advanced behavior and checks for tunnel options as a part of JENKINS-52246 .

          You do it right, and it works: I found my error and placed the option in front of jenkins.jar. Now Windows slave is accessible again.

          Dzianis Mazuronak added a comment - You do it right, and it works: I found my error and placed the option in front of jenkins.jar. Now Windows slave is accessible again.

          Chris Rule added a comment -

          dmazuronak :

          Agreed. I had to modify the Jenkins startup script for the define (-D) to take effect. It appears you can't use the Jenkins console to make the change, even though it showed up in the Jenkins System Information page.

          Since I run on Linux Mint (Ubuntu derivative) I modified the file /etc/defaults/jenkins, JAVA_ARGS line, to add the -D as described above. My final line looks like (the first -D... was already there):

          JAVA_ARGS="-Djava.awt.headless=true -Dhudson.TcpSlaveAgentListener.hostName=j-jenkins"
          

          Chris Rule added a comment - dmazuronak : Agreed. I had to modify the Jenkins startup script for the define (-D) to take effect. It appears you can't use the Jenkins console to make the change, even though it showed up in the Jenkins System Information page. Since I run on Linux Mint (Ubuntu derivative) I modified the file /etc/defaults/jenkins, JAVA_ARGS line, to add the -D as described above. My final line looks like (the first -D... was already there): JAVA_ARGS= "-Djava.awt.headless= true -Dhudson.TcpSlaveAgentListener.hostName=j-jenkins"

          Oleg Nenashev added a comment -

          Remoting 3.23 is out. Created https://github.com/jenkinsci/jenkins/pull/3530

          Oleg Nenashev added a comment - Remoting 3.23 is out. Created https://github.com/jenkinsci/jenkins/pull/3530

          Oleg Nenashev added a comment -

          No need to backport the fix, it won't get into LTS

          Oleg Nenashev added a comment - No need to backport the fix, it won't get into LTS

            oleg_nenashev Oleg Nenashev
            dmazuronak Dzianis Mazuronak
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: