Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60199

jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

    XMLWordPrintable

Details

    • Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9

    Description

      After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

       Workaround: For Apache: set "ProxyPreserveHost On" as documented in https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

      *Jenkins LTS Notice*: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU for the discussion and retrospective

      Attachments

        Issue Links

          Activity

            ohauer olli hauer created issue -
            markewaite Mark Waite added a comment -

            olamy is this likely a result of the upgrade to Jetty 9.4.22?

            markewaite Mark Waite added a comment - olamy is this likely a result of the upgrade to Jetty 9.4.22 ?
            olamy Olivier Lamy added a comment - - edited

            ohauer Is it possible to have more details about the reverse proxy? (version, configuration entry)

            markewaite yes maybe...

            olamy Olivier Lamy added a comment - - edited ohauer Is it possible to have more details about the reverse proxy? (version, configuration entry) markewaite yes maybe...
            markewaite Mark Waite added a comment -

            olamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem.

            ohauer can you provide configuration information that will allow another person to see the same issue?

            markewaite Mark Waite added a comment - olamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem. ohauer can you provide configuration information that will allow another person to see the same issue?
            ohauer olli hauer added a comment -

            Sure, here are the parts from my Jenkins config:

            Jenkins parameters:
            --httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false

            Jenkins (tab) configure:
            Jenkins Location

            Relevant part from apache 2.4.41:

            ProxyRequests Off
            AllowEncodedSlashes NoDecode

            RewriteRule ^(/jenkins)$ $1/ [R=301,L]

            <Location /jenkins>
            ProxyPass http://127.0.0.1:8180/jenkins nocanon
            ProxyPassReverse http://127.0.0.1:8180/jenkins
            RequestHeader set X-Forwarded-Proto "https"
            RequestHeader set X-Forwarded-Port "443"
            RequestHeader set X-Forwarded-Host "fqdn.host.name"
            </Location>

             

            ohauer olli hauer added a comment - Sure, here are the parts from my Jenkins config: Jenkins parameters: --httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false Jenkins (tab) configure: Jenkins Location Jenkins URL: https://fqdn.host.name/jenkins Relevant part from apache 2.4.41: ProxyRequests Off AllowEncodedSlashes NoDecode RewriteRule ^(/jenkins)$ $1/ [R=301,L] <Location /jenkins> ProxyPass http://127.0.0.1:8180/jenkins nocanon ProxyPassReverse http://127.0.0.1:8180/jenkins RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" RequestHeader set X-Forwarded-Host "fqdn.host.name" </Location>  
            ohauer olli hauer added a comment - - edited

            Perhaps a hint, the Apache is TLS only.

            I've tried to analyze what happens the time I press the sign in button with chrome see trace below.

             

            2.205 (broken)
            j_acegi_security_check:

            General:
            Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
            Request Method: POST
            Status Code: 302 Found
            Remote Address: IP.AD.DR.ES:443
            Referrer Policy: no-referrer-when-downgrade

            Response Header:
            Connection: Keep-Alive
            Content-Length: 0
            Date: Wed, 20 Nov 2019 09:33:02 GMT
            Expires: Thu, 01 Jan 1970 00:00:00 GMT
            Keep-Alive: timeout=5, max=100
            Location: https://127.0.0.1/jenkins/
            Server: Jetty(9.4.22.v20191022)
            Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly
            X-Content-Type-Options: nosniff

            Request Header:
            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
            Accept-Encoding: gzip, deflate, br
            Accept-Language: en-US,en;q=0.9,de;q=0.8
            Cache-Control: max-age=0
            Connection: keep-alive
            Content-Length: 71
            Content-Type: application/x-www-form-urlencoded
            Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200
            DNT: 1
            Host: fqdn.host.name
            Origin: https://fqdn.host.name
            Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
            Sec-Fetch-Mode: navigate
            Sec-Fetch-Site: same-origin
            Sec-Fetch-User: ?1
            Upgrade-Insecure-Requests: 1
            User-Agent: Mozilla/5.0 ...
            j_username: username
            j_password: password
            from: /jenkins/
            Submit: Sign in

             

            and with 2.204 (OK)
            j_acegi_security_check:

            General:
            Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
            Request Method: POST
            Status Code: 302 Found
            Remote Address: IP.AD.DR.ES:443
            Referrer Policy: no-referrer-when-downgrade

            Response Header:
            Connection: Keep-Alive
            Content-Length: 0
            Date: Wed, 20 Nov 2019 09:47:27 GMT
            Expires: Thu, 01 Jan 1970 00:00:00 GMT
            Keep-Alive: timeout=5, max=100
            Location: https://fqdn.host.name/jenkins/
            Server: Jetty(9.4.z-SNAPSHOT)
            Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly
            X-Content-Type-Options: nosniff

            Request Header:
            Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
            Accept-Encoding: gzip, deflate, br
            Accept-Language: en-US,en;q=0.9,de;q=0.8
            Cache-Control: max-age=0
            Connection: keep-alive
            Content-Length: 71
            Content-Type: application/x-www-form-urlencoded
            Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200
            DNT: 1
            Host: fqdn.host.name
            Origin: https://fqdn.host.name
            Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
            Sec-Fetch-Mode: navigate
            Sec-Fetch-Site: same-origin
            Sec-Fetch-User: ?1
            Upgrade-Insecure-Requests: 1
            User-Agent: Mozilla/5.0 ...
            j_username: username
            j_password: password
            from: /jenkins/
            Submit: Sign in

             

            ohauer olli hauer added a comment - - edited Perhaps a hint, the Apache is TLS only. I've tried to analyze what happens the time I press the sign in button with chrome see trace below.   2.205 (broken) j_acegi_security_check: General: Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check Request Method: POST Status Code: 302 Found Remote Address: IP.AD.DR.ES:443 Referrer Policy: no-referrer-when-downgrade Response Header: Connection: Keep-Alive Content-Length: 0 Date: Wed, 20 Nov 2019 09:33:02 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Keep-Alive: timeout=5, max=100 Location: https://127.0.0.1/jenkins/ Server: Jetty(9.4.22.v20191022) Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly X-Content-Type-Options: nosniff Request Header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, / ;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,de;q=0.8 Cache-Control: max-age=0 Connection: keep-alive Content-Length: 71 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200 DNT: 1 Host: fqdn.host.name Origin: https://fqdn.host.name Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F Sec-Fetch-Mode: navigate Sec-Fetch-Site: same-origin Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 ... j_username: username j_password: password from: /jenkins/ Submit: Sign in   and with 2.204 (OK) j_acegi_security_check: General: Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check Request Method: POST Status Code: 302 Found Remote Address: IP.AD.DR.ES:443 Referrer Policy: no-referrer-when-downgrade Response Header: Connection: Keep-Alive Content-Length: 0 Date: Wed, 20 Nov 2019 09:47:27 GMT Expires: Thu, 01 Jan 1970 00:00:00 GMT Keep-Alive: timeout=5, max=100 Location: https://fqdn.host.name/jenkins/ Server : Jetty(9.4.z-SNAPSHOT) Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly X-Content-Type-Options: nosniff Request Header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, / ;q=0.8,application/signed-exchange;v=b3 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9,de;q=0.8 Cache-Control: max-age=0 Connection: keep-alive Content-Length: 71 Content-Type: application/x-www-form-urlencoded Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200 DNT: 1 Host: fqdn.host.name Origin: https://fqdn.host.name Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F Sec-Fetch-Mode: navigate Sec-Fetch-Site: same-origin Sec-Fetch-User: ?1 Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 ... j_username: username j_password: password from: /jenkins/ Submit: Sign in  
            muntmeister Rob Munten added a comment -

            Hi ohauer,

            Add the following line to your apache conf.

            ProxyPreserveHost On

            This fixed the issue in my case.

            muntmeister Rob Munten added a comment - Hi ohauer , Add the following line to your apache conf. ProxyPreserveHost On This fixed the issue in my case.
            oleg_nenashev Oleg Nenashev added a comment -

            markewaite  ohauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions

            oleg_nenashev Oleg Nenashev added a comment - markewaite   ohauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions
            danielbeck Daniel Beck added a comment -

            I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage.

            If so, wouldn't care that much.

            danielbeck Daniel Beck added a comment - I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage . If so, wouldn't care that much.
            jglick Jesse Glick added a comment -

            Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch, for example using this?

            jglick Jesse Glick added a comment - Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch , for example using this ?
            bcooksley Ben Cooksley added a comment -

            I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly:

            ```AllowEncodedSlashes NoDecode
            ProxyRequests Off
            ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
            ProxyPassReverse / https://127.0.0.1/```

            Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is:
            a) Forcing HTTPS to On
            b) Disregarding the port that Jenkins itself is listening on
            c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under

            bcooksley Ben Cooksley added a comment - I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly: ```AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http://127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / https://127.0.0.1/ ``` Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is: a) Forcing HTTPS to On b) Disregarding the port that Jenkins itself is listening on c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under
            jglick Jesse Glick added a comment -

            I am guessing the issue only affects /j_acegi_security_check, the typical login form, and not other redirects in Jenkins?

            jglick Jesse Glick added a comment - I am guessing the issue only affects /j_acegi_security_check , the typical login form, and not other redirects in Jenkins?
            bcooksley Ben Cooksley added a comment -

            Other endpoints, including those used to access build logs are also impacted.

            Anywhere where a redirect is used seems to be affected.

            bcooksley Ben Cooksley added a comment - Other endpoints, including those used to access build logs are also impacted. Anywhere where a redirect is used seems to be affected.
            jglick Jesse Glick added a comment -

            So I tried

            • docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy
            • JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1
            • navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/
            • going through setup wizard (no plugins, confirming the above root URL, using default admin user)
            • logging out, logging in
            • running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12)

            I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https://whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s.

            The image mentioned above does set ProxyPreserveHost On, which has been our recommendation for Apache users for years.

            My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy.

            Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.

            jglick Jesse Glick added a comment - So I tried docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1 navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/ going through setup wizard (no plugins, confirming the above root URL, using default admin user) logging out, logging in running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12 ) I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https: //whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s. The image mentioned above does set ProxyPreserveHost On , which has been our recommendation for Apache users for years. My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy. Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.
            jglick Jesse Glick added a comment -

            As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.

            jglick Jesse Glick added a comment - As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.
            bcooksley Ben Cooksley added a comment -

            Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP.

            See https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

            In terms of a minimal reproduction usecase, the following should do the trick:

             

            <VirtualHost *:443>
                ServerAdmin  webmaster@example.org
                ServerName   jenkins.example.org
                DocumentRoot /srv/www/jenkins.example.org/
            
                ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log
                CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined
            
                AllowEncodedSlashes NoDecode
                ProxyRequests Off
                ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
                ProxyPassReverse  /  http://127.0.0.1:8200/
                
                RequestHeader set X-Forwarded-Proto "https"
                RequestHeader set X-Forwarded-Port "443"    # Enable SSL
            </VirtualHost>
            

             

            You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other):

            ProxyPassReverse / https://127.0.0.1/
            

            This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)

            bcooksley Ben Cooksley added a comment - Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP. See  https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache In terms of a minimal reproduction usecase, the following should do the trick:   <VirtualHost *:443> ServerAdmin webmaster@example.org ServerName jenkins.example.org DocumentRoot /srv/www/jenkins.example.org/ ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http: //127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / http: //127.0.0.1:8200/ RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" # Enable SSL </VirtualHost>   You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other): ProxyPassReverse / https: //127.0.0.1/ This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)
            jglick Jesse Glick added a comment -

            setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS

            ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart.

            At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header (Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.

            jglick Jesse Glick added a comment - setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes ), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart. At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header ( Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.
            bcooksley Ben Cooksley added a comment -

            Could you provide the ProxyPass / ProxyPassReverse statements from your test image?
            If they mirror the fix I mentioned above already, then this would explain why it works for you.

            bcooksley Ben Cooksley added a comment - Could you provide the ProxyPass / ProxyPassReverse statements from your test image? If they mirror the fix I mentioned above already, then this would explain why it works for you.
            jglick Jesse Glick added a comment -

            These directives look like they are properly mirrored to me. I also tried the same without the context path (/jenkins/) and with nocanon replaced with retry=0 as in your example.

            jglick Jesse Glick added a comment - These directives look like they are properly mirrored to me. I also tried the same without the context path ( /jenkins → / ) and with nocanon replaced with retry=0 as in your example.
            bcooksley Ben Cooksley added a comment -

            I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.

            bcooksley Ben Cooksley added a comment - I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.
            jglick Jesse Glick added a comment -

            My second test was using local port 8090 so I doubt it.

            jglick Jesse Glick added a comment - My second test was using local port 8090 so I doubt it.

            Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!

            pyves Pierre-Yves Bigourdan added a comment - Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!
            katzdm Daniel Katz added a comment -

            I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).

            katzdm Daniel Katz added a comment - I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).
            katzdm Daniel Katz made changes -
            Field Original Value New Value
            Environment Jenkins: 2.205
            Apache 2.4.41 as reverse proxy
            Jenkins: 2.205, 2.201.3, 2.201.4
            Apache 2.4.41 as reverse proxy
            olamy Olivier Lamy added a comment -

            katzdm you mean it doesn't work with?

            ProxyPreserveHost On 
            olamy Olivier Lamy added a comment - katzdm you mean it doesn't work with? ProxyPreserveHost On
            markewaite Mark Waite added a comment -

            katzdm you'll need to provide more details about your configuration or even better, the greater precision of answers that jglick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.

            markewaite Mark Waite added a comment - katzdm you'll need to provide more details about your configuration or even better, the greater precision of answers that jglick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.
            bonfy bon ch made changes -
            Comment [ I create example repo to this issue: https://github.com/bonfy/JENKINS-61317

            Jenkinsfile in master branch Last commit : 64dd9920
            library in jenkins-lib branch Last commit: a8c235b1

            and when echo GIT_COMMIT, it is overwitten by lib's commit

            [Pipeline] echo
            13:22:18 stage 1
            [Pipeline] echo
            13:22:18 GIT COMMIT : a8c235b10c55e2f065293b1c1ad9b3bffd45d9d9
            [Pipeline] echo
            13:22:18 Hello world
            [Pipeline] } ]
            danielbeck Daniel Beck added a comment -

            That Jetty backport

            danielbeck Daniel Beck added a comment - That Jetty backport
            katzdm Daniel Katz added a comment - - edited

            Hi markewaite, I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers.

            1. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                  -H "Host: jenkins.example.com" \
                  -H "X-Forwarded-Port: 80" \
                  -H "X-Forwarded-Host: localhost" -o/dev/null \
                  2>/dev/null; echo
            2. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                  -H "Host: jenkins.example.com" \
                  -H "X-Forwarded-Host: localhost" \
                  -H "X-Forwarded-Port: 80" -o/dev/null \
                  2>/dev/null; echo

            I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:

            So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?

            To answer your previous question regarding "ProxyPreserveHost On", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso), so this isn't directly relevant to me.

            katzdm Daniel Katz added a comment - - edited Hi markewaite , I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Port: 80" \     -H "X-Forwarded-Host: localhost" -o/dev/null \     2>/dev/null; echo curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Host: localhost" \     -H "X-Forwarded-Port: 80" -o/dev/null \     2>/dev/null; echo I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors: jenkins/jenkins:2.204.2-slim : Command 1: http://localhost/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:2.204.3-slim : Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:slim : (2.223; latest) Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the  X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the  ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()? To answer your previous question regarding " ProxyPreserveHost On ", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso ), so this isn't directly relevant to me.
            markewaite Mark Waite added a comment -

            Thanks very much katzdm!

            markewaite Mark Waite added a comment - Thanks very much katzdm !
            katzdm Daniel Katz made changes -
            Environment Jenkins: 2.205, 2.201.3, 2.201.4
            Apache 2.4.41 as reverse proxy
            Jenkins: 2.205, 2.204.3, 2.204.4
            Apache 2.4.41 as reverse proxy
            katzdm Daniel Katz added a comment -

            I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27: https://github.com/eclipse/jetty.project/issues/4573

            katzdm Daniel Katz added a comment - I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27:  https://github.com/eclipse/jetty.project/issues/4573
            katzdm Daniel Katz added a comment -

            There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release: https://github.com/jenkinsci/winstone/pull/94

            I've commented there with a reference to this bug, with hope that it can get merged + released.

            katzdm Daniel Katz added a comment - There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release:  https://github.com/jenkinsci/winstone/pull/94 I've commented there with a reference to this bug, with hope that it can get merged + released.
            oleg_nenashev Oleg Nenashev made changes -
            Assignee Oleg Nenashev [ oleg_nenashev ]
            oleg_nenashev Oleg Nenashev made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            oleg_nenashev Oleg Nenashev made changes -
            Component/s winstone-jetty [ 20645 ]
            oleg_nenashev Oleg Nenashev made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels lts-candidate regression
            oleg_nenashev Oleg Nenashev added a comment -

            Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline.

            At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer

            oleg_nenashev Oleg Nenashev added a comment - Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline. At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer
            olamy Olivier Lamy made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            oleg_nenashev Oleg Nenashev added a comment -

            Winstone 5.9 with fixes was released: https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9

            Jenkins Core update pull request: https://github.com/jenkinsci/jenkins/pull/4542

            oleg_nenashev Oleg Nenashev added a comment - Winstone 5.9 with fixes was released:  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9 Jenkins Core update pull request:  https://github.com/jenkinsci/jenkins/pull/4542
            oleg_nenashev Oleg Nenashev made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            oleg_nenashev Oleg Nenashev made changes -
            Environment Jenkins: 2.205, 2.204.3, 2.204.4
            Apache 2.4.41 as reverse proxy
            Jenkins: 2.205, Jenkins LTS 2.204.3 and 2.204.4
            Winstone 5.5 to 5.8
            Apache 2.4.41 as reverse proxy
            oleg_nenashev Oleg Nenashev made changes -
            Description After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login. After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

             

            **Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was backported as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective
            oleg_nenashev Oleg Nenashev made changes -
            Description After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

             

            **Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was backported as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective
            After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

             

            **Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective
            oleg_nenashev Oleg Nenashev made changes -
            Description After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

             

            **Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective
            After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

             *Workaround*: For Apache: set "ProxyPreserveHost On" as documented in [https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache]

            **Jenkins LTS Notice**: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see [https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU] for the discussion and retrospective
            markewaite Mark Waite added a comment -

            Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

            markewaite Mark Waite added a comment - Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?
            oleg_nenashev Oleg Nenashev added a comment -

            markewaite it would be nice, I have not got to it yet due to LTS backports

            oleg_nenashev Oleg Nenashev added a comment - markewaite it would be nice, I have not got to it yet due to LTS backports
            jglick Jesse Glick made changes -
            Status Reopened [ 4 ] Open [ 1 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            oleg_nenashev Oleg Nenashev added a comment -

            I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions. 

            Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545

            oleg_nenashev Oleg Nenashev added a comment - I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1  but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions.  Pull request with the 2.204.x baseline update:  https://github.com/jenkinsci/jenkins/pull/4545
            oleg_nenashev Oleg Nenashev made changes -
            Released As Winstone 5.4.1, Winstone 5.9
            oleg_nenashev Oleg Nenashev made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            danielbeck Daniel Beck made changes -
            Link This issue is duplicated by JENKINS-61328 [ JENKINS-61328 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Fixed but Unreleased [ 10203 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate regression 2.204.5-fixed lts-candidate regression
            oleg_nenashev Oleg Nenashev made changes -
            Released As Winstone 5.4.1, Winstone 5.9 Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9
            oleg_nenashev Oleg Nenashev added a comment -
            oleg_nenashev Oleg Nenashev added a comment - Jenkins LTS 2.204.5 with fixes is out:  https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.204.5  . Official Changelogs are coming soon. ETA for Jenkins weekly 2.224 is today
            oleg_nenashev Oleg Nenashev made changes -
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
            olivergondza Oliver Gondža made changes -
            Labels 2.204.5-fixed lts-candidate regression 2.204.5-fixed 2.222.1-fixed regression

            People

              oleg_nenashev Oleg Nenashev
              ohauer olli hauer
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: