Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52204

Jenkins Windows Agent failed to start after upgrade to 2.1.129

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core, remoting
    • Labels:
      None
    • Environment:
      Jenkins 2.1.129
    • Similar Issues:
    • Released As:
      Jenkins 2.130, Remoting 3.23

      Description

      Jenkins Windows Agent failed to start after upgrade to version 2.1.129: Jenkins is running behind apache reverse proxy on different host that proxy itself. Parameter “tunnel connection through” is set correctly. From the error message we can see that remote agent was trying to connect to reverse proxy host instead of Jenkins master host.

      The version 2.1.128 works without issues.

        Attachments

          Issue Links

            Activity

            dmazuronak Dzianis Mazuronak created issue -
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-51818 [ JENKINS-51818 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Most likely it is related to JENKINS-51818, CC Andrei Laptsinski
            Dzianis Mazuronak have you updated Remoting on agents to 3.22?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Most likely it is related to JENKINS-51818 , CC Andrei Laptsinski Dzianis Mazuronak have you updated Remoting on agents to 3.22?
            Hide
            dmazuronak Dzianis Mazuronak added a comment -

            Remoting 3.22 has the problem. Remoting 3.21 works OK:

            INFORMATION: Setting up agent: MY-CLIENT
            Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener <init>
            INFORMATION: Jenkins agent is running in headless mode.
            Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main createEngine
            WARNUNG: Certificate validation for HTTPs endpoints is disabled
            Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine
            INFORMATION: Using Remoting version: 3.22
            Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine
            WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
            Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener status
            INFORMATION: Locating server among [https://tools.domain.local/devjenkins/]
            Jun 26, 2018 11:07:54 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
            WARNUNG: HTTPs certificate check is disabled for the endpoint.
            Jun 26, 2018 11:07:55 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
            INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
            Jun 26, 2018 11:07:56 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible
            WARNUNG: Connection refused: connect
            Jun 26, 2018 11:07:56 PM hudson.remoting.jnlp.Main$CuiListener error
            SCHWERWIEGEND: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable
            java.io.IOException: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable
            	at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:284)
            	at hudson.remoting.Engine.innerRun(Engine.java:518)
            	at hudson.remoting.Engine.run(Engine.java:469)
            
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine
            INFORMATION: Setting up agent: MY-CLIENT
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener <init>
            INFORMATION: Jenkins agent is running in headless mode.
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine
            WARNUNG: Certificate validation for HTTPs endpoints is disabled
            Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine
            INFORMATION: Using Remoting version: 3.21
            Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine
            WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFORMATION: Locating server among [https://tools.domain.local/devjenkins/]
            Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection
            WARNUNG: HTTPs certificate check is disabled for the endpoint.
            Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
            INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFORMATION: Agent discovery successful
              Agent address: 10.150.3.9
              Agent port:    44864
              Identity:      f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFORMATION: Handshaking
            Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status
            INFORMATION: Connecting to 10.150.3.9:44864
            

             Port 37367 is open on Jenkins server 10.150.3.9 and not on reverse proxy tools.domain.local.

            Show
            dmazuronak Dzianis Mazuronak added a comment - Remoting 3.22 has the problem. Remoting 3.21 works OK: INFORMATION: Setting up agent: MY-CLIENT Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener <init> INFORMATION: Jenkins agent is running in headless mode. Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main createEngine WARNUNG: Certificate validation for HTTPs endpoints is disabled Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine INFORMATION: Using Remoting version: 3.22 Jun 26, 2018 11:07:54 PM hudson.remoting.Engine startEngine WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars Jun 26, 2018 11:07:54 PM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Locating server among [https://tools.domain.local/devjenkins/] Jun 26, 2018 11:07:54 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection WARNUNG: HTTPs certificate check is disabled for the endpoint. Jun 26, 2018 11:07:55 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jun 26, 2018 11:07:56 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver isPortVisible WARNUNG: Connection refused: connect Jun 26, 2018 11:07:56 PM hudson.remoting.jnlp.Main$CuiListener error SCHWERWIEGEND: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable java.io.IOException: https://tools.domain.local/devjenkins/ provided port:37367 is not reachable at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:284) at hudson.remoting.Engine.innerRun(Engine.java:518) at hudson.remoting.Engine.run(Engine.java:469) Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine INFORMATION: Setting up agent: MY-CLIENT Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener <init> INFORMATION: Jenkins agent is running in headless mode. Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main createEngine WARNUNG: Certificate validation for HTTPs endpoints is disabled Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine INFORMATION: Using Remoting version: 3.21 Jun 27, 2018 8:46:17 AM hudson.remoting.Engine startEngine WARNUNG: No Working Directory. Using the legacy JAR Cache location: C:\Windows\system32\config\systemprofile\.jenkins\cache\jars Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Locating server among [https://tools.domain.local/devjenkins/] Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver openURLConnection WARNUNG: HTTPs certificate check is disabled for the endpoint. Jun 27, 2018 8:46:17 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFORMATION: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Agent discovery successful Agent address: 10.150.3.9 Agent port: 44864 Identity: f8:2a:3d:5a:d4:22:f4:a0:89:26:4a:a1:34:20:c3:11 Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Handshaking Jun 27, 2018 8:46:17 AM hudson.remoting.jnlp.Main$CuiListener status INFORMATION: Connecting to 10.150.3.9:44864  Port 37367 is open on Jenkins server 10.150.3.9 and not on reverse proxy tools.domain.local.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            From what I see in the https://github.com/jenkinsci/remoting/pull/275/files code, the logic should be using the resolved hostname for direct connection:

            host = defaultString(first(header(con, "X-Jenkins-JNLP-Host")), salURL.getHost());
            

            Are you sure that "hudson.TcpSlaveAgentListener.hostName" is set on your instance?
            If no, setting it up on the Master side would be a workaround until we figure out the tunneling issue

            Show
            oleg_nenashev Oleg Nenashev added a comment - From what I see in the https://github.com/jenkinsci/remoting/pull/275/files code, the logic should be using the resolved hostname for direct connection: host = defaultString(first(header(con, "X-Jenkins-JNLP-Host" )), salURL.getHost()); Are you sure that "hudson.TcpSlaveAgentListener.hostName" is set on your instance? If no, setting it up on the Master side would be a workaround until we figure out the tunneling issue
            Hide
            dmazuronak Dzianis Mazuronak added a comment -

            On master side this slave has the parameter "Tunnel connection through" set to 10.150.3.9.

            Show
            dmazuronak Dzianis Mazuronak added a comment - On master side this slave has the parameter "Tunnel connection through" set to 10.150.3.9.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Dzianis Mazuronak Could you please set the "-Dhudson.TcpSlaveAgentListener.hostName" system property on master as I suggested above?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Dzianis Mazuronak Could you please set the "-Dhudson.TcpSlaveAgentListener.hostName" system property on master as I suggested above?
            Hide
            crule Chris Rule added a comment - - edited

            I'm having the same issue. I have  jenkins running on host "j-jenkins" and the apache web-server running as reverse proxy on "eddard". Setting the hudson.TcpSlaveAgentListener.hostName property to "j-jenkins" did not correct the problem. I receive the same error as before "IOException: https://eddard/jenkins/ provided port:41829 is not reachable".

            Show
            crule Chris Rule added a comment - - edited I'm having the same issue. I have  jenkins running on host "j-jenkins" and the apache web-server running as reverse proxy on "eddard". Setting the hudson.TcpSlaveAgentListener.hostName property to "j-jenkins" did not correct the problem. I receive the same error as before "IOException: https://eddard/jenkins/ provided port:41829 is not reachable".
            Hide
            dmazuronak Dzianis Mazuronak added a comment - - edited

            I can only confirm the information from Chris. Unfortunately, using the system property "hudson.TcpSlaveAgentListener.hostName" did not bring any improvement.

            There is no such header "X-Jenkins-JNLP-Host" in http response.

            The property "hudson.TcpSlaveAgentListener.hostName" was set in the command line. In which configuration file can this property be set? I'm not sure that the command line was processed.

            Show
            dmazuronak Dzianis Mazuronak added a comment - - edited I can only confirm the information from Chris. Unfortunately, using the system property "hudson.TcpSlaveAgentListener.hostName" did not bring any improvement. There is no such header "X-Jenkins-JNLP-Host" in http response. The property "hudson.TcpSlaveAgentListener.hostName" was set in the command line. In which configuration file can this property be set? I'm not sure that the command line was processed.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            java -Dhudson.TcpSlaveAgentListener.hostName=fooBar -jar jenkins.war should do the job.
            Cannot suggest config file since I do not know Jenkins configuration

            Show
            oleg_nenashev Oleg Nenashev added a comment - java -Dhudson.TcpSlaveAgentListener.hostName=fooBar -jar jenkins.war should do the job. Cannot suggest config file since I do not know Jenkins configuration
            Hide
            denami Andrei Laptsinski added a comment -

            You can receive value of "hudson.TcpSlaveAgentListener.hostName". By Jenkins  script console (System.getProperty("hudson.TcpSlaveAgentListener.hostName") ).  Possible it is override by other  JAVA_OPTS/JENKINS_JAVA_OPTIONS)

            Show
            denami Andrei Laptsinski added a comment - You can receive value of "hudson.TcpSlaveAgentListener.hostName". By Jenkins  script console (System.getProperty("hudson.TcpSlaveAgentListener.hostName") ).  Possible it is override by other  JAVA_OPTS/JENKINS_JAVA_OPTIONS)
            Hide
            dmazuronak Dzianis Mazuronak added a comment -

            No, the other options are empty, so no override here.

            If I type

            println(System.getProperty("hudson.TcpSlaveAgentListener.hostName") )
            

            in Jenkins script console, I get only

            null
            

            as result.

             

            There is an other interresting result:

            hudson.TcpSlaveAgentListener.hostName
            

            produces following exception:

            groovy.lang.MissingPropertyException: No such property: hostName for class: hudson.TcpSlaveAgentListener
            	at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1002)
            	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1857)
            	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1833)
            	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758)
            	at org.codehaus.groovy.runtime.callsite.ClassMetaClassGetPropertySite.getProperty(ClassMetaClassGetPropertySite.java:51)
            	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
            	at Script1.run(Script1.groovy:1)
            
            Show
            dmazuronak Dzianis Mazuronak added a comment - No, the other options are empty, so no override here. If I type println( System .getProperty( "hudson.TcpSlaveAgentListener.hostName" ) ) in Jenkins script console, I get only null as result.   There is an other interresting result: hudson.TcpSlaveAgentListener.hostName produces following exception: groovy.lang.MissingPropertyException: No such property: hostName for class: hudson.TcpSlaveAgentListener at groovy.lang.MetaClassImpl.invokeStaticMissingProperty(MetaClassImpl.java:1002) at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1857) at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1833) at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3758) at org.codehaus.groovy.runtime.callsite.ClassMetaClassGetPropertySite.getProperty(ClassMetaClassGetPropertySite.java:51) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296) at Script1.run(Script1.groovy:1)
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            > There is an other interresting result:

            As designed. System properties won't be visible in Groovy in such way.

            Dzianis Mazuronak System properties are not passed to your Jenkins master due to whatever reason, so the proposed workaround will not work until the property is passed correctly.

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited > There is an other interresting result: As designed. System properties won't be visible in Groovy in such way. Dzianis Mazuronak System properties are not passed to your Jenkins master due to whatever reason, so the proposed workaround will not work until the property is passed correctly.
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Jeff Thompson [ jthompson ] Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue relates to JENKINS-52246 [ JENKINS-52246 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            See my comment in https://github.com/jenkinsci/remoting/pull/279 . After the review, this issue happens IFF "-tunnel" is set. I believe that "-tunnel" flag should be deprecated and replaced by offering a custom port in TcpAgentListener response headers like it's already done for hosts

            So I propose to just disable checks in that case if Andrei Laptsinski agrees. We could implement a more advanced behavior and checks for tunnel options as a part of JENKINS-52246.

            Show
            oleg_nenashev Oleg Nenashev added a comment - See my comment in https://github.com/jenkinsci/remoting/pull/279 . After the review, this issue happens IFF "-tunnel" is set. I believe that "-tunnel" flag should be deprecated and replaced by offering a custom port in TcpAgentListener response headers like it's already done for hosts So I propose to just disable checks in that case if Andrei Laptsinski agrees. We could implement a more advanced behavior and checks for tunnel options as a part of JENKINS-52246 .
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "https://github.com/jenkinsci/remoting/pull/279 (Web Link)" [ 21120 ]
            Hide
            dmazuronak Dzianis Mazuronak added a comment -

            You do it right, and it works: I found my error and placed the option in front of jenkins.jar. Now Windows slave is accessible again.

            Show
            dmazuronak Dzianis Mazuronak added a comment - You do it right, and it works: I found my error and placed the option in front of jenkins.jar. Now Windows slave is accessible again.
            Hide
            crule Chris Rule added a comment -

            Dzianis Mazuronak :

            Agreed. I had to modify the Jenkins startup script for the define (-D) to take effect. It appears you can't use the Jenkins console to make the change, even though it showed up in the Jenkins System Information page.

            Since I run on Linux Mint (Ubuntu derivative) I modified the file /etc/defaults/jenkins, JAVA_ARGS line, to add the -D as described above. My final line looks like (the first -D... was already there):

            JAVA_ARGS="-Djava.awt.headless=true -Dhudson.TcpSlaveAgentListener.hostName=j-jenkins"
            
            Show
            crule Chris Rule added a comment - Dzianis Mazuronak : Agreed. I had to modify the Jenkins startup script for the define (-D) to take effect. It appears you can't use the Jenkins console to make the change, even though it showed up in the Jenkins System Information page. Since I run on Linux Mint (Ubuntu derivative) I modified the file /etc/defaults/jenkins, JAVA_ARGS line, to add the -D as described above. My final line looks like (the first -D... was already there): JAVA_ARGS= "-Djava.awt.headless= true -Dhudson.TcpSlaveAgentListener.hostName=j-jenkins"
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Remoting 3.23 is out. Created https://github.com/jenkinsci/jenkins/pull/3530

            Show
            oleg_nenashev Oleg Nenashev added a comment - Remoting 3.23 is out. Created https://github.com/jenkinsci/jenkins/pull/3530
            oleg_nenashev Oleg Nenashev made changes -
            Remote Link This issue links to "https://github.com/jenkinsci/jenkins/pull/3530 (Web Link)" [ 21124 ]
            oleg_nenashev Oleg Nenashev made changes -
            Released As 2.130
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            No need to backport the fix, it won't get into LTS

            Show
            oleg_nenashev Oleg Nenashev added a comment - No need to backport the fix, it won't get into LTS
            oleg_nenashev Oleg Nenashev made changes -
            Released As 2.130 Jenkins 2.130
            oleg_nenashev Oleg Nenashev made changes -
            Released As Jenkins 2.130 Jenkins 2.130, Remoting 3.23
            oleg_nenashev Oleg Nenashev made changes -
            Component/s core [ 15593 ]
            Component/s windows-slaves-plugin [ 18327 ]

              People

              Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              dmazuronak Dzianis Mazuronak
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: