Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60199

jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

    • Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9

      After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

       Workaround: For Apache: set "ProxyPreserveHost On" as documented in https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

      *Jenkins LTS Notice*: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU for the discussion and retrospective

          [JENKINS-60199] jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

          Mark Waite added a comment -

          olamy is this likely a result of the upgrade to Jetty 9.4.22?

          Mark Waite added a comment - olamy is this likely a result of the upgrade to Jetty 9.4.22 ?

          Olivier Lamy added a comment - - edited

          ohauer Is it possible to have more details about the reverse proxy? (version, configuration entry)

          markewaite yes maybe...

          Olivier Lamy added a comment - - edited ohauer Is it possible to have more details about the reverse proxy? (version, configuration entry) markewaite yes maybe...

          Mark Waite added a comment -

          olamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem.

          ohauer can you provide configuration information that will allow another person to see the same issue?

          Mark Waite added a comment - olamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem. ohauer can you provide configuration information that will allow another person to see the same issue?

          olli hauer added a comment -

          Sure, here are the parts from my Jenkins config:

          Jenkins parameters:
          --httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false

          Jenkins (tab) configure:
          Jenkins Location

          Relevant part from apache 2.4.41:

          ProxyRequests Off
          AllowEncodedSlashes NoDecode

          RewriteRule ^(/jenkins)$ $1/ [R=301,L]

          <Location /jenkins>
          ProxyPass http://127.0.0.1:8180/jenkins nocanon
          ProxyPassReverse http://127.0.0.1:8180/jenkins
          RequestHeader set X-Forwarded-Proto "https"
          RequestHeader set X-Forwarded-Port "443"
          RequestHeader set X-Forwarded-Host "fqdn.host.name"
          </Location>

           

          olli hauer added a comment - Sure, here are the parts from my Jenkins config: Jenkins parameters: --httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false Jenkins (tab) configure: Jenkins Location Jenkins URL: https://fqdn.host.name/jenkins Relevant part from apache 2.4.41: ProxyRequests Off AllowEncodedSlashes NoDecode RewriteRule ^(/jenkins)$ $1/ [R=301,L] <Location /jenkins> ProxyPass http://127.0.0.1:8180/jenkins nocanon ProxyPassReverse http://127.0.0.1:8180/jenkins RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" RequestHeader set X-Forwarded-Host "fqdn.host.name" </Location>  

          olli hauer added a comment - - edited

          Perhaps a hint, the Apache is TLS only.

          I've tried to analyze what happens the time I press the sign in button with chrome see trace below.

           

          2.205 (broken)
          j_acegi_security_check:

          General:
          Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
          Request Method: POST
          Status Code: 302 Found
          Remote Address: IP.AD.DR.ES:443
          Referrer Policy: no-referrer-when-downgrade

          Response Header:
          Connection: Keep-Alive
          Content-Length: 0
          Date: Wed, 20 Nov 2019 09:33:02 GMT
          Expires: Thu, 01 Jan 1970 00:00:00 GMT
          Keep-Alive: timeout=5, max=100
          Location: https://127.0.0.1/jenkins/
          Server: Jetty(9.4.22.v20191022)
          Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly
          X-Content-Type-Options: nosniff

          Request Header:
          Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
          Accept-Encoding: gzip, deflate, br
          Accept-Language: en-US,en;q=0.9,de;q=0.8
          Cache-Control: max-age=0
          Connection: keep-alive
          Content-Length: 71
          Content-Type: application/x-www-form-urlencoded
          Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200
          DNT: 1
          Host: fqdn.host.name
          Origin: https://fqdn.host.name
          Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
          Sec-Fetch-Mode: navigate
          Sec-Fetch-Site: same-origin
          Sec-Fetch-User: ?1
          Upgrade-Insecure-Requests: 1
          User-Agent: Mozilla/5.0 ...
          j_username: username
          j_password: password
          from: /jenkins/
          Submit: Sign in

           

          and with 2.204 (OK)
          j_acegi_security_check:

          General:
          Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
          Request Method: POST
          Status Code: 302 Found
          Remote Address: IP.AD.DR.ES:443
          Referrer Policy: no-referrer-when-downgrade

          Response Header:
          Connection: Keep-Alive
          Content-Length: 0
          Date: Wed, 20 Nov 2019 09:47:27 GMT
          Expires: Thu, 01 Jan 1970 00:00:00 GMT
          Keep-Alive: timeout=5, max=100
          Location: https://fqdn.host.name/jenkins/
          Server: Jetty(9.4.z-SNAPSHOT)
          Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly
          X-Content-Type-Options: nosniff

          Request Header:
          Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
          Accept-Encoding: gzip, deflate, br
          Accept-Language: en-US,en;q=0.9,de;q=0.8
          Cache-Control: max-age=0
          Connection: keep-alive
          Content-Length: 71
          Content-Type: application/x-www-form-urlencoded
          Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200
          DNT: 1
          Host: fqdn.host.name
          Origin: https://fqdn.host.name
          Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
          Sec-Fetch-Mode: navigate
          Sec-Fetch-Site: same-origin
          Sec-Fetch-User: ?1
          Upgrade-Insecure-Requests: 1
          User-Agent: Mozilla/5.0 ...
          j_username: username
          j_password: password
          from: /jenkins/
          Submit: Sign in

           

          olli hauer added a comment - - edited Perhaps a hint, the Apache is TLS only. I've tried to analyze what happens the time I press the sign in button with chrome see trace below.   2.205 (broken) j_acegi_security_check: General: Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check Request Method: POST Status Code: 302 Found Remote Address: IP.AD.DR.ES:443 Referrer Policy: no-referrer-when-downgrade Response Header: Connection: Keep-Alive Content-Length: 0 Date: Wed, 20 Nov 2019 09:33:02 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Keep-Alive: timeout=5, max=100 Location: https://127.0.0.1/jenkins/ Server: Jetty(9.4.22.v20191022) Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly X-Content-Type-Options: nosniff Request Header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, / ;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,de;q=0.8 Cache-Control: max-age=0 Connection: keep-alive Content-Length: 71 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200 DNT: 1 Host: fqdn.host.name Origin: https://fqdn.host.name Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F Sec-Fetch-Mode: navigate Sec-Fetch-Site: same-origin Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 ... j_username: username j_password: password from: /jenkins/ Submit: Sign in   and with 2.204 (OK) j_acegi_security_check: General: Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check Request Method: POST Status Code: 302 Found Remote Address: IP.AD.DR.ES:443 Referrer Policy: no-referrer-when-downgrade Response Header: Connection: Keep-Alive Content-Length: 0 Date: Wed, 20 Nov 2019 09:47:27 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Keep-Alive: timeout=5, max=100 Location: https://fqdn.host.name/jenkins/ Server : Jetty(9.4.z-SNAPSHOT) Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly X-Content-Type-Options: nosniff Request Header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, / ;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,de;q=0.8 Cache-Control: max-age=0 Connection: keep-alive Content-Length: 71 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200 DNT: 1 Host: fqdn.host.name Origin: https://fqdn.host.name Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F Sec-Fetch-Mode: navigate Sec-Fetch-Site: same-origin Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 ... j_username: username j_password: password from: /jenkins/ Submit: Sign in  

          Rob Munten added a comment -

          Hi ohauer,

          Add the following line to your apache conf.

          ProxyPreserveHost On

          This fixed the issue in my case.

          Rob Munten added a comment - Hi ohauer , Add the following line to your apache conf. ProxyPreserveHost On This fixed the issue in my case.

          Oleg Nenashev added a comment -

          markewaite  ohauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions

          Oleg Nenashev added a comment - markewaite   ohauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions

          Daniel Beck added a comment -

          I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage.

          If so, wouldn't care that much.

          Daniel Beck added a comment - I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage . If so, wouldn't care that much.

          Jesse Glick added a comment -

          Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch, for example using this?

          Jesse Glick added a comment - Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch , for example using this ?

          Ben Cooksley added a comment -

          I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly:

          ```AllowEncodedSlashes NoDecode
          ProxyRequests Off
          ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
          ProxyPassReverse / https://127.0.0.1/```

          Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is:
          a) Forcing HTTPS to On
          b) Disregarding the port that Jenkins itself is listening on
          c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under

          Ben Cooksley added a comment - I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly: ```AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http://127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / https://127.0.0.1/ ``` Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is: a) Forcing HTTPS to On b) Disregarding the port that Jenkins itself is listening on c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under

          Jesse Glick added a comment -

          I am guessing the issue only affects /j_acegi_security_check, the typical login form, and not other redirects in Jenkins?

          Jesse Glick added a comment - I am guessing the issue only affects /j_acegi_security_check , the typical login form, and not other redirects in Jenkins?

          Ben Cooksley added a comment -

          Other endpoints, including those used to access build logs are also impacted.

          Anywhere where a redirect is used seems to be affected.

          Ben Cooksley added a comment - Other endpoints, including those used to access build logs are also impacted. Anywhere where a redirect is used seems to be affected.

          Jesse Glick added a comment -

          So I tried

          • docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy
          • JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1
          • navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/
          • going through setup wizard (no plugins, confirming the above root URL, using default admin user)
          • logging out, logging in
          • running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12)

          I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https://whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s.

          The image mentioned above does set ProxyPreserveHost On, which has been our recommendation for Apache users for years.

          My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy.

          Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.

          Jesse Glick added a comment - So I tried docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1 navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/ going through setup wizard (no plugins, confirming the above root URL, using default admin user) logging out, logging in running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12 ) I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https: //whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s. The image mentioned above does set ProxyPreserveHost On , which has been our recommendation for Apache users for years. My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy. Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.

          Jesse Glick added a comment -

          As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.

          Jesse Glick added a comment - As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.

          Ben Cooksley added a comment -

          Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP.

          See https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

          In terms of a minimal reproduction usecase, the following should do the trick:

           

          <VirtualHost *:443>
              ServerAdmin  webmaster@example.org
              ServerName   jenkins.example.org
              DocumentRoot /srv/www/jenkins.example.org/
          
              ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log
              CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined
          
              AllowEncodedSlashes NoDecode
              ProxyRequests Off
              ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
              ProxyPassReverse  /  http://127.0.0.1:8200/
              
              RequestHeader set X-Forwarded-Proto "https"
              RequestHeader set X-Forwarded-Port "443"    # Enable SSL
          </VirtualHost>
          

           

          You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other):

          ProxyPassReverse / https://127.0.0.1/
          

          This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)

          Ben Cooksley added a comment - Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP. See  https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache In terms of a minimal reproduction usecase, the following should do the trick:   <VirtualHost *:443> ServerAdmin webmaster@example.org ServerName jenkins.example.org DocumentRoot /srv/www/jenkins.example.org/ ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http: //127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / http: //127.0.0.1:8200/ RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" # Enable SSL </VirtualHost>   You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other): ProxyPassReverse / https: //127.0.0.1/ This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)

          Jesse Glick added a comment -

          setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS

          ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart.

          At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header (Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.

          Jesse Glick added a comment - setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes ), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart. At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header ( Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.

          Ben Cooksley added a comment -

          Could you provide the ProxyPass / ProxyPassReverse statements from your test image?
          If they mirror the fix I mentioned above already, then this would explain why it works for you.

          Ben Cooksley added a comment - Could you provide the ProxyPass / ProxyPassReverse statements from your test image? If they mirror the fix I mentioned above already, then this would explain why it works for you.

          Jesse Glick added a comment -

          These directives look like they are properly mirrored to me. I also tried the same without the context path (/jenkins → /) and with nocanon replaced with retry=0 as in your example.

          Jesse Glick added a comment - These directives look like they are properly mirrored to me. I also tried the same without the context path ( /jenkins → / ) and with nocanon replaced with retry=0 as in your example.

          Ben Cooksley added a comment -

          I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.

          Ben Cooksley added a comment - I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.

          Jesse Glick added a comment -

          My second test was using local port 8090 so I doubt it.

          Jesse Glick added a comment - My second test was using local port 8090 so I doubt it.

          Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!

          Pierre-Yves Bigourdan added a comment - Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!

          Daniel Katz added a comment -

          I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).

          Daniel Katz added a comment - I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).

          Olivier Lamy added a comment -

          katzdm you mean it doesn't work with?

          ProxyPreserveHost On 

          Olivier Lamy added a comment - katzdm you mean it doesn't work with? ProxyPreserveHost On

          Mark Waite added a comment -

          katzdm you'll need to provide more details about your configuration or even better, the greater precision of answers that jglick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.

          Mark Waite added a comment - katzdm you'll need to provide more details about your configuration or even better, the greater precision of answers that jglick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.

          Daniel Beck added a comment -

          That Jetty backport

          Daniel Beck added a comment - That Jetty backport

          Daniel Katz added a comment - - edited

          Hi markewaite, I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers.

          1. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                -H "Host: jenkins.example.com" \
                -H "X-Forwarded-Port: 80" \
                -H "X-Forwarded-Host: localhost" -o/dev/null \
                2>/dev/null; echo
          2. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                -H "Host: jenkins.example.com" \
                -H "X-Forwarded-Host: localhost" \
                -H "X-Forwarded-Port: 80" -o/dev/null \
                2>/dev/null; echo

          I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:

          So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?

          To answer your previous question regarding "ProxyPreserveHost On", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso), so this isn't directly relevant to me.

          Daniel Katz added a comment - - edited Hi markewaite , I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Port: 80" \     -H "X-Forwarded-Host: localhost" -o/dev/null \     2>/dev/null; echo curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Host: localhost" \     -H "X-Forwarded-Port: 80" -o/dev/null \     2>/dev/null; echo I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors: jenkins/jenkins:2.204.2-slim : Command 1: http://localhost/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:2.204.3-slim : Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:slim : (2.223; latest) Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the  X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the  ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()? To answer your previous question regarding " ProxyPreserveHost On ", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso ), so this isn't directly relevant to me.

          Mark Waite added a comment -

          Thanks very much katzdm!

          Mark Waite added a comment - Thanks very much katzdm !

          Daniel Katz added a comment -

          I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27: https://github.com/eclipse/jetty.project/issues/4573

          Daniel Katz added a comment - I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27:  https://github.com/eclipse/jetty.project/issues/4573

          Daniel Katz added a comment -

          There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release: https://github.com/jenkinsci/winstone/pull/94

          I've commented there with a reference to this bug, with hope that it can get merged + released.

          Daniel Katz added a comment - There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release:  https://github.com/jenkinsci/winstone/pull/94 I've commented there with a reference to this bug, with hope that it can get merged + released.

          Oleg Nenashev added a comment -

          Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline.

          At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer

          Oleg Nenashev added a comment - Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline. At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer

          Oleg Nenashev added a comment -

          Winstone 5.9 with fixes was released: https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9

          Jenkins Core update pull request: https://github.com/jenkinsci/jenkins/pull/4542

          Oleg Nenashev added a comment - Winstone 5.9 with fixes was released:  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9 Jenkins Core update pull request:  https://github.com/jenkinsci/jenkins/pull/4542

          Mark Waite added a comment -

          Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

          Mark Waite added a comment - Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

          Oleg Nenashev added a comment -

          markewaite it would be nice, I have not got to it yet due to LTS backports

          Oleg Nenashev added a comment - markewaite it would be nice, I have not got to it yet due to LTS backports

          Oleg Nenashev added a comment -

          I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions. 

          Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545

          Oleg Nenashev added a comment - I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1  but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions.  Pull request with the 2.204.x baseline update:  https://github.com/jenkinsci/jenkins/pull/4545

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - Jenkins LTS 2.204.5 with fixes is out:  https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.204.5  . Official Changelogs are coming soon. ETA for Jenkins weekly 2.224 is today

            oleg_nenashev Oleg Nenashev
            ohauer olli hauer
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: