Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60199

jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9

      Description

      After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

       Workaround: For Apache: set "ProxyPreserveHost On" as documented in https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

      *Jenkins LTS Notice*: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU for the discussion and retrospective

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment -

            Olivier Lamy is this likely a result of the upgrade to Jetty 9.4.22?

            Show
            markewaite Mark Waite added a comment - Olivier Lamy is this likely a result of the upgrade to Jetty 9.4.22 ?
            Hide
            olamy Olivier Lamy added a comment - - edited

            olli hauer Is it possible to have more details about the reverse proxy? (version, configuration entry)

            Mark Waite yes maybe...

            Show
            olamy Olivier Lamy added a comment - - edited olli hauer Is it possible to have more details about the reverse proxy? (version, configuration entry) Mark Waite yes maybe...
            Hide
            markewaite Mark Waite added a comment -

            Olivier Lamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem.

            olli hauer can you provide configuration information that will allow another person to see the same issue?

            Show
            markewaite Mark Waite added a comment - Olivier Lamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem. olli hauer can you provide configuration information that will allow another person to see the same issue?
            Hide
            ohauer olli hauer added a comment -

            Sure, here are the parts from my Jenkins config:

            Jenkins parameters:
            --httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false

            Jenkins (tab) configure:
            Jenkins Location

            Relevant part from apache 2.4.41:

            ProxyRequests Off
            AllowEncodedSlashes NoDecode

            RewriteRule ^(/jenkins)$ $1/ [R=301,L]

            <Location /jenkins>
            ProxyPass http://127.0.0.1:8180/jenkins nocanon
            ProxyPassReverse http://127.0.0.1:8180/jenkins
            RequestHeader set X-Forwarded-Proto "https"
            RequestHeader set X-Forwarded-Port "443"
            RequestHeader set X-Forwarded-Host "fqdn.host.name"
            </Location>

             

            Show
            ohauer olli hauer added a comment - Sure, here are the parts from my Jenkins config: Jenkins parameters: --httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false Jenkins (tab) configure: Jenkins Location Jenkins URL: https://fqdn.host.name/jenkins Relevant part from apache 2.4.41: ProxyRequests Off AllowEncodedSlashes NoDecode RewriteRule ^(/jenkins)$ $1/ [R=301,L] <Location /jenkins> ProxyPass http://127.0.0.1:8180/jenkins nocanon ProxyPassReverse http://127.0.0.1:8180/jenkins RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" RequestHeader set X-Forwarded-Host "fqdn.host.name" </Location>  
            Hide
            ohauer olli hauer added a comment - - edited

            Perhaps a hint, the Apache is TLS only.

            I've tried to analyze what happens the time I press the sign in button with chrome see trace below.

             

            2.205 (broken)
            j_acegi_security_check:

            General:
            Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
            Request Method: POST
            Status Code: 302 Found
            Remote Address: IP.AD.DR.ES:443
            Referrer Policy: no-referrer-when-downgrade

            Response Header:
            Connection: Keep-Alive
            Content-Length: 0
            Date: Wed, 20 Nov 2019 09:33:02 GMT
            Expires: Thu, 01 Jan 1970 00:00:00 GMT
            Keep-Alive: timeout=5, max=100
            Location: https://127.0.0.1/jenkins/
            Server: Jetty(9.4.22.v20191022)
            Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly
            X-Content-Type-Options: nosniff

            Request Header:
            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
            Accept-Encoding: gzip, deflate, br
            Accept-Language: en-US,en;q=0.9,de;q=0.8
            Cache-Control: max-age=0
            Connection: keep-alive
            Content-Length: 71
            Content-Type: application/x-www-form-urlencoded
            Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200
            DNT: 1
            Host: fqdn.host.name
            Origin: https://fqdn.host.name
            Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
            Sec-Fetch-Mode: navigate
            Sec-Fetch-Site: same-origin
            Sec-Fetch-User: ?1
            Upgrade-Insecure-Requests: 1
            User-Agent: Mozilla/5.0 ...
            j_username: username
            j_password: password
            from: /jenkins/
            Submit: Sign in

             

            and with 2.204 (OK)
            j_acegi_security_check:

            General:
            Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
            Request Method: POST
            Status Code: 302 Found
            Remote Address: IP.AD.DR.ES:443
            Referrer Policy: no-referrer-when-downgrade

            Response Header:
            Connection: Keep-Alive
            Content-Length: 0
            Date: Wed, 20 Nov 2019 09:47:27 GMT
            Expires: Thu, 01 Jan 1970 00:00:00 GMT
            Keep-Alive: timeout=5, max=100
            Location: https://fqdn.host.name/jenkins/
            Server: Jetty(9.4.z-SNAPSHOT)
            Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly
            X-Content-Type-Options: nosniff

            Request Header:
            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
            Accept-Encoding: gzip, deflate, br
            Accept-Language: en-US,en;q=0.9,de;q=0.8
            Cache-Control: max-age=0
            Connection: keep-alive
            Content-Length: 71
            Content-Type: application/x-www-form-urlencoded
            Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200
            DNT: 1
            Host: fqdn.host.name
            Origin: https://fqdn.host.name
            Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
            Sec-Fetch-Mode: navigate
            Sec-Fetch-Site: same-origin
            Sec-Fetch-User: ?1
            Upgrade-Insecure-Requests: 1
            User-Agent: Mozilla/5.0 ...
            j_username: username
            j_password: password
            from: /jenkins/
            Submit: Sign in

             

            Show
            ohauer olli hauer added a comment - - edited Perhaps a hint, the Apache is TLS only. I've tried to analyze what happens the time I press the sign in button with chrome see trace below.   2.205 (broken) j_acegi_security_check: General: Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check Request Method: POST Status Code: 302 Found Remote Address: IP.AD.DR.ES:443 Referrer Policy: no-referrer-when-downgrade Response Header: Connection: Keep-Alive Content-Length: 0 Date: Wed, 20 Nov 2019 09:33:02 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Keep-Alive: timeout=5, max=100 Location: https://127.0.0.1/jenkins/ Server: Jetty(9.4.22.v20191022) Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly X-Content-Type-Options: nosniff Request Header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, / ;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,de;q=0.8 Cache-Control: max-age=0 Connection: keep-alive Content-Length: 71 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200 DNT: 1 Host: fqdn.host.name Origin: https://fqdn.host.name Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F Sec-Fetch-Mode: navigate Sec-Fetch-Site: same-origin Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 ... j_username: username j_password: password from: /jenkins/ Submit: Sign in   and with 2.204 (OK) j_acegi_security_check: General: Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check Request Method: POST Status Code: 302 Found Remote Address: IP.AD.DR.ES:443 Referrer Policy: no-referrer-when-downgrade Response Header: Connection: Keep-Alive Content-Length: 0 Date: Wed, 20 Nov 2019 09:47:27 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Keep-Alive: timeout=5, max=100 Location: https://fqdn.host.name/jenkins/ Server : Jetty(9.4.z-SNAPSHOT) Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly X-Content-Type-Options: nosniff Request Header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, / ;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,de;q=0.8 Cache-Control: max-age=0 Connection: keep-alive Content-Length: 71 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200 DNT: 1 Host: fqdn.host.name Origin: https://fqdn.host.name Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F Sec-Fetch-Mode: navigate Sec-Fetch-Site: same-origin Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 ... j_username: username j_password: password from: /jenkins/ Submit: Sign in  
            Hide
            muntmeister Rob Munten added a comment -

            Hi olli hauer,

            Add the following line to your apache conf.

            ProxyPreserveHost On

            This fixed the issue in my case.

            Show
            muntmeister Rob Munten added a comment - Hi olli hauer , Add the following line to your apache conf. ProxyPreserveHost On This fixed the issue in my case.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Mark Waite  olli hauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions

            Show
            oleg_nenashev Oleg Nenashev added a comment - Mark Waite   olli hauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions
            Hide
            danielbeck Daniel Beck added a comment -

            I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage.

            If so, wouldn't care that much.

            Show
            danielbeck Daniel Beck added a comment - I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage . If so, wouldn't care that much.
            Hide
            jglick Jesse Glick added a comment -

            Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch, for example using this?

            Show
            jglick Jesse Glick added a comment - Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch , for example using this ?
            Hide
            bcooksley Ben Cooksley added a comment -

            I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly:

            ```AllowEncodedSlashes NoDecode
            ProxyRequests Off
            ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
            ProxyPassReverse / https://127.0.0.1/```

            Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is:
            a) Forcing HTTPS to On
            b) Disregarding the port that Jenkins itself is listening on
            c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under

            Show
            bcooksley Ben Cooksley added a comment - I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly: ```AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http://127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / https://127.0.0.1/ ``` Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is: a) Forcing HTTPS to On b) Disregarding the port that Jenkins itself is listening on c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under
            Hide
            jglick Jesse Glick added a comment -

            I am guessing the issue only affects /j_acegi_security_check, the typical login form, and not other redirects in Jenkins?

            Show
            jglick Jesse Glick added a comment - I am guessing the issue only affects /j_acegi_security_check , the typical login form, and not other redirects in Jenkins?
            Hide
            bcooksley Ben Cooksley added a comment -

            Other endpoints, including those used to access build logs are also impacted.

            Anywhere where a redirect is used seems to be affected.

            Show
            bcooksley Ben Cooksley added a comment - Other endpoints, including those used to access build logs are also impacted. Anywhere where a redirect is used seems to be affected.
            Hide
            jglick Jesse Glick added a comment -

            So I tried

            • docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy
            • JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1
            • navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/
            • going through setup wizard (no plugins, confirming the above root URL, using default admin user)
            • logging out, logging in
            • running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12)

            I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https://whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s.

            The image mentioned above does set ProxyPreserveHost On, which has been our recommendation for Apache users for years.

            My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy.

            Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.

            Show
            jglick Jesse Glick added a comment - So I tried docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1 navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/ going through setup wizard (no plugins, confirming the above root URL, using default admin user) logging out, logging in running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12 ) I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https: //whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s. The image mentioned above does set ProxyPreserveHost On , which has been our recommendation for Apache users for years. My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy. Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.
            Hide
            jglick Jesse Glick added a comment -

            As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.

            Show
            jglick Jesse Glick added a comment - As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.
            Hide
            bcooksley Ben Cooksley added a comment -

            Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP.

            See https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

            In terms of a minimal reproduction usecase, the following should do the trick:

             

            <VirtualHost *:443>
                ServerAdmin  webmaster@example.org
                ServerName   jenkins.example.org
                DocumentRoot /srv/www/jenkins.example.org/
            
                ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log
                CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined
            
                AllowEncodedSlashes NoDecode
                ProxyRequests Off
                ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
                ProxyPassReverse  /  http://127.0.0.1:8200/
                
                RequestHeader set X-Forwarded-Proto "https"
                RequestHeader set X-Forwarded-Port "443"    # Enable SSL
            </VirtualHost>
            

             

            You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other):

            ProxyPassReverse / https://127.0.0.1/
            

            This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)

            Show
            bcooksley Ben Cooksley added a comment - Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP. See  https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache In terms of a minimal reproduction usecase, the following should do the trick:   <VirtualHost *:443> ServerAdmin webmaster@example.org ServerName jenkins.example.org DocumentRoot /srv/www/jenkins.example.org/ ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http: //127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / http: //127.0.0.1:8200/ RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" # Enable SSL </VirtualHost>   You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other): ProxyPassReverse / https: //127.0.0.1/ This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)
            Hide
            jglick Jesse Glick added a comment -

            setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS

            ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart.

            At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header (Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.

            Show
            jglick Jesse Glick added a comment - setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes ), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart. At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header ( Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.
            Hide
            bcooksley Ben Cooksley added a comment -

            Could you provide the ProxyPass / ProxyPassReverse statements from your test image?
            If they mirror the fix I mentioned above already, then this would explain why it works for you.

            Show
            bcooksley Ben Cooksley added a comment - Could you provide the ProxyPass / ProxyPassReverse statements from your test image? If they mirror the fix I mentioned above already, then this would explain why it works for you.
            Hide
            jglick Jesse Glick added a comment -

            These directives look like they are properly mirrored to me. I also tried the same without the context path (/jenkins/) and with nocanon replaced with retry=0 as in your example.

            Show
            jglick Jesse Glick added a comment - These directives look like they are properly mirrored to me. I also tried the same without the context path ( /jenkins → / ) and with nocanon replaced with retry=0 as in your example.
            Hide
            bcooksley Ben Cooksley added a comment -

            I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.

            Show
            bcooksley Ben Cooksley added a comment - I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.
            Hide
            jglick Jesse Glick added a comment -

            My second test was using local port 8090 so I doubt it.

            Show
            jglick Jesse Glick added a comment - My second test was using local port 8090 so I doubt it.
            Hide
            pyves Pierre-Yves Bigourdan added a comment -

            Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!

            Show
            pyves Pierre-Yves Bigourdan added a comment - Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!
            Hide
            katzdm Daniel Katz added a comment -

            I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).

            Show
            katzdm Daniel Katz added a comment - I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).
            Hide
            olamy Olivier Lamy added a comment -

            Daniel Katz you mean it doesn't work with?

            ProxyPreserveHost On 
            Show
            olamy Olivier Lamy added a comment - Daniel Katz you mean it doesn't work with? ProxyPreserveHost On
            Hide
            markewaite Mark Waite added a comment -

            Daniel Katz you'll need to provide more details about your configuration or even better, the greater precision of answers that Jesse Glick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.

            Show
            markewaite Mark Waite added a comment - Daniel Katz you'll need to provide more details about your configuration or even better, the greater precision of answers that Jesse Glick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.
            Hide
            danielbeck Daniel Beck added a comment -

            That Jetty backport

            Show
            danielbeck Daniel Beck added a comment - That Jetty backport
            Hide
            katzdm Daniel Katz added a comment - - edited

            Hi Mark Waite, I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers.

            1. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                  -H "Host: jenkins.example.com" \
                  -H "X-Forwarded-Port: 80" \
                  -H "X-Forwarded-Host: localhost" -o/dev/null \
                  2>/dev/null; echo
            2. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                  -H "Host: jenkins.example.com" \
                  -H "X-Forwarded-Host: localhost" \
                  -H "X-Forwarded-Port: 80" -o/dev/null \
                  2>/dev/null; echo

            I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:

            So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?

            To answer your previous question regarding "ProxyPreserveHost On", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso), so this isn't directly relevant to me.

            Show
            katzdm Daniel Katz added a comment - - edited Hi Mark Waite , I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Port: 80" \     -H "X-Forwarded-Host: localhost" -o/dev/null \     2>/dev/null; echo curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Host: localhost" \     -H "X-Forwarded-Port: 80" -o/dev/null \     2>/dev/null; echo I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors: jenkins/jenkins:2.204.2-slim : Command 1: http://localhost/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:2.204.3-slim : Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:slim : (2.223; latest) Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the  X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the  ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()? To answer your previous question regarding " ProxyPreserveHost On ", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso ), so this isn't directly relevant to me.
            Hide
            markewaite Mark Waite added a comment -

            Thanks very much Daniel Katz!

            Show
            markewaite Mark Waite added a comment - Thanks very much Daniel Katz !
            Hide
            katzdm Daniel Katz added a comment -

            I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27: https://github.com/eclipse/jetty.project/issues/4573

            Show
            katzdm Daniel Katz added a comment - I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27:  https://github.com/eclipse/jetty.project/issues/4573
            Hide
            katzdm Daniel Katz added a comment -

            There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release: https://github.com/jenkinsci/winstone/pull/94

            I've commented there with a reference to this bug, with hope that it can get merged + released.

            Show
            katzdm Daniel Katz added a comment - There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release:  https://github.com/jenkinsci/winstone/pull/94 I've commented there with a reference to this bug, with hope that it can get merged + released.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline.

            At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC Olivier Lamy who is also a Jetty maintainer

            Show
            oleg_nenashev Oleg Nenashev added a comment - Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline. At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC Olivier Lamy who is also a Jetty maintainer
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Winstone 5.9 with fixes was released: https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9

            Jenkins Core update pull request: https://github.com/jenkinsci/jenkins/pull/4542

            Show
            oleg_nenashev Oleg Nenashev added a comment - Winstone 5.9 with fixes was released:  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9 Jenkins Core update pull request:  https://github.com/jenkinsci/jenkins/pull/4542
            Hide
            markewaite Mark Waite added a comment -

            Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

            Show
            markewaite Mark Waite added a comment - Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Mark Waite it would be nice, I have not got to it yet due to LTS backports

            Show
            oleg_nenashev Oleg Nenashev added a comment - Mark Waite it would be nice, I have not got to it yet due to LTS backports
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions. 

            Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545

            Show
            oleg_nenashev Oleg Nenashev added a comment - I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1  but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions.  Pull request with the 2.204.x baseline update:  https://github.com/jenkinsci/jenkins/pull/4545
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - Jenkins LTS 2.204.5 with fixes is out:  https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.204.5  . Official Changelogs are coming soon. ETA for Jenkins weekly 2.224 is today

              People

              Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              ohauer olli hauer
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: