Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60199

jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

    • Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9

      After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.

       Workaround: For Apache: set "ProxyPreserveHost On" as documented in https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache

      *Jenkins LTS Notice*: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU for the discussion and retrospective

          [JENKINS-60199] jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login

          Daniel Katz added a comment - - edited

          Hi markewaite, I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers.

          1. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                -H "Host: jenkins.example.com" \
                -H "X-Forwarded-Port: 80" \
                -H "X-Forwarded-Host: localhost" -o/dev/null \
                2>/dev/null; echo
          2. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
                -H "Host: jenkins.example.com" \
                -H "X-Forwarded-Host: localhost" \
                -H "X-Forwarded-Port: 80" -o/dev/null \
                2>/dev/null; echo

          I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:

          So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?

          To answer your previous question regarding "ProxyPreserveHost On", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso), so this isn't directly relevant to me.

          Daniel Katz added a comment - - edited Hi markewaite , I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers. curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Port: 80" \     -H "X-Forwarded-Host: localhost" -o/dev/null \     2>/dev/null; echo curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \     -H "Host: jenkins.example.com" \     -H "X-Forwarded-Host: localhost" \     -H "X-Forwarded-Port: 80" -o/dev/null \     2>/dev/null; echo I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors: jenkins/jenkins:2.204.2-slim : Command 1: http://localhost/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:2.204.3-slim : Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ jenkins/jenkins:slim : (2.223; latest) Command 1: http://jenkins.example.com/whoAmI/ Command 2: http://localhost/whoAmI/ So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the  X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the  ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()? To answer your previous question regarding " ProxyPreserveHost On ", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso ), so this isn't directly relevant to me.

          Mark Waite added a comment -

          Thanks very much katzdm!

          Mark Waite added a comment - Thanks very much katzdm !

          Daniel Katz added a comment -

          I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27: https://github.com/eclipse/jetty.project/issues/4573

          Daniel Katz added a comment - I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27:  https://github.com/eclipse/jetty.project/issues/4573

          Daniel Katz added a comment -

          There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release: https://github.com/jenkinsci/winstone/pull/94

          I've commented there with a reference to this bug, with hope that it can get merged + released.

          Daniel Katz added a comment - There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release:  https://github.com/jenkinsci/winstone/pull/94 I've commented there with a reference to this bug, with hope that it can get merged + released.

          Oleg Nenashev added a comment -

          Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline.

          At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer

          Oleg Nenashev added a comment - Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline. At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer

          Oleg Nenashev added a comment -

          Winstone 5.9 with fixes was released: https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9

          Jenkins Core update pull request: https://github.com/jenkinsci/jenkins/pull/4542

          Oleg Nenashev added a comment - Winstone 5.9 with fixes was released:  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9 Jenkins Core update pull request:  https://github.com/jenkinsci/jenkins/pull/4542

          Mark Waite added a comment -

          Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

          Mark Waite added a comment - Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?

          Oleg Nenashev added a comment -

          markewaite it would be nice, I have not got to it yet due to LTS backports

          Oleg Nenashev added a comment - markewaite it would be nice, I have not got to it yet due to LTS backports

          Oleg Nenashev added a comment -

          I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions. 

          Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545

          Oleg Nenashev added a comment - I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1  but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions.  Pull request with the 2.204.x baseline update:  https://github.com/jenkinsci/jenkins/pull/4545

          Oleg Nenashev added a comment -

          Oleg Nenashev added a comment - Jenkins LTS 2.204.5 with fixes is out:  https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.204.5  . Official Changelogs are coming soon. ETA for Jenkins weekly 2.224 is today

            oleg_nenashev Oleg Nenashev
            ohauer olli hauer
            Votes:
            4 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: