-
Bug
-
Resolution: Fixed
-
Major
-
Jenkins: 2.205, Jenkins LTS 2.204.3 and 2.204.4
Winstone 5.5 to 5.8
Apache 2.4.41 as reverse proxy
-
Powered by SuggestiMate -
Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9
After the upgrade from 2.204 to 2.205 jenkins redirects to http(s)://127.0.0.1/ after login.
Workaround: For Apache: set "ProxyPreserveHost On" as documented in https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache
*Jenkins LTS Notice*: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU for the discussion and retrospective
[JENKINS-60199] jenkins 2.205 behind reverse proxy redirects to 127.0.0.1 after login
ohauer Is it possible to have more details about the reverse proxy? (version, configuration entry)
markewaite yes maybe...
olamy I successfully created an Apache reverse proxy configuration on Debian 10 installing the apache package for Debian and the jenkins 2.205 package for Debian. Then I configured the Apache installation per the directions at https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache . I can't duplicate the problem.
ohauer can you provide configuration information that will allow another person to see the same issue?
Sure, here are the parts from my Jenkins config:
Jenkins parameters:
--httpPort=8180 --httpListenAddress=127.0.0.1 --prefix=/jenkins -Dhudson.DNSMultiCast.disabled=true -Djava.net.preferIPv6Addresses=false
Jenkins (tab) configure:
Jenkins Location
- Jenkins URL: https://fqdn.host.name/jenkins
Relevant part from apache 2.4.41:
ProxyRequests Off
AllowEncodedSlashes NoDecode
RewriteRule ^(/jenkins)$ $1/ [R=301,L]
<Location /jenkins>
ProxyPass http://127.0.0.1:8180/jenkins nocanon
ProxyPassReverse http://127.0.0.1:8180/jenkins
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
RequestHeader set X-Forwarded-Host "fqdn.host.name"
</Location>
Perhaps a hint, the Apache is TLS only.
I've tried to analyze what happens the time I press the sign in button with chrome see trace below.
2.205 (broken)
j_acegi_security_check:
General:
Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade
Response Header:
Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:33:02 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
Location: https://127.0.0.1/jenkins/
Server: Jetty(9.4.22.v20191022)
Set-Cookie: JSESSIONID.10db730a=node0173jclauyomxaxlnl8x11eypy2.node0; Path=/jenkins; Secure; HttpOnly
X-Content-Type-Options: nosniff
Request Header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID.c0515192=node09fh3q2bodtxf6wgyy85sirkm7.node0; JSESSIONID.10db730a=node01pf20r2evzrjn7h09d2m3bk201.node0; screenResolution=1920x1200
DNT: 1
Host: fqdn.host.name
Origin: https://fqdn.host.name
Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 ...
j_username: username
j_password: password
from: /jenkins/
Submit: Sign in
and with 2.204 (OK)
j_acegi_security_check:
General:
Request URL: https://fqdn.host.name/jenkins/j_acegi_security_check
Request Method: POST
Status Code: 302 Found
Remote Address: IP.AD.DR.ES:443
Referrer Policy: no-referrer-when-downgrade
Response Header:
Connection: Keep-Alive
Content-Length: 0
Date: Wed, 20 Nov 2019 09:47:27 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive: timeout=5, max=100
Location: https://fqdn.host.name/jenkins/
Server: Jetty(9.4.z-SNAPSHOT)
Set-Cookie: JSESSIONID.e3d4d209=node01eapoart737kjzv2n9r5jqf6n1.node0;Path=/jenkins;Secure;HttpOnly
X-Content-Type-Options: nosniff
Request Header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de;q=0.8
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 71
Content-Type: application/x-www-form-urlencoded
Cookie: JSESSIONID.e3d4d209=node01deci271w2led1hqp8smatfa2j0.node0; screenResolution=1920x1200
DNT: 1
Host: fqdn.host.name
Origin: https://fqdn.host.name
Referer: https://fqdn.host.name/jenkins/login?from=%2Fjenkins%2F
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 ...
j_username: username
j_password: password
from: /jenkins/
Submit: Sign in
Hi ohauer,
Add the following line to your apache conf.
ProxyPreserveHost On
This fixed the issue in my case.
markewaite ohauer looks like we need to add upgrade guidelines for this regression to the weekly changelog (summary banner for reverse proxy users?). AFAICT it also discourages selecting 2.205 s the new LTS baseline, needs more soak testing for other possible regressions
I wonder whether this would be an instance that previously showed an administrative warnings about a bad reverse proxy configuration on /manage.
If so, wouldn't care that much.
Would really need to see what headers were sent to/from Jenkins by Apache to see if Jetty if behaving improperly here. Is there a simple way to reproduce from scratch, for example using this?
I've found that for our instances (at build.kde.org and binary-factory.kde.org) that we've needed to use the following Apache snippets to make it work properly:
```AllowEncodedSlashes NoDecode
ProxyRequests Off
ProxyPass / http://127.0.0.1:8200/ nocanon retry=0
ProxyPassReverse / https://127.0.0.1/```
Given that ProxyPass and ProxyPassReverse should normally mirror each other, it would appear that whatever code is responsible for creating redirects in Jenkins/Jetty is:
a) Forcing HTTPS to On
b) Disregarding the port that Jenkins itself is listening on
c) Disregarding the Canonical URL preference set in Manage Jenkins > Configure in favour of the host it received the request under
I am guessing the issue only affects /j_acegi_security_check, the typical login form, and not other redirects in Jenkins?
Other endpoints, including those used to access build logs are also impacted.
Anywhere where a redirect is used seems to be affected.
So I tried
- docker run --rm -p 127.0.0.1:80:80 --network=host jglick/jenkins-demo-reverse-proxy
- JENKINS_HOME=… java -jar ~/.m2/repository/org/jenkins-ci/main/jenkins-war/2.205/jenkins-war-2.205.war --prefix=/jenkins --httpListenAddress=127.0.0.1
- navigating to http://whatever.127.0.0.1.beesdns.com/jenkins/
- going through setup wizard (no plugins, confirming the above root URL, using default admin user)
- logging out, logging in
- running builds, deleting builds, etc., stuff that definitely makes use of 302s (confirmed in the Network tab of Chromium’s F12)
I did not notice any issues; the host name part of URLs remained constant. Also tried the same with HTTPS, using https://whatever.127.0.0.1.beesdns.com/jenkins/configure to adjust the Jenkins URL (accepting the lack of a certificate), and again had no issues with 302s.
The image mentioned above does set ProxyPreserveHost On, which has been our recommendation for Apache users for years.
My provisional evaluation is that some behavioral change in the past couple years of Jetty updates introduced a new symptom for Jenkins users who had always had a somewhat misconfigured reverse proxy.
Again, for those experiencing an issue, please try to describe a minimal, self-contained test case which would allow developers to reproduce the problem, determine what exactly is at fault, and decide if code changes are warranted.
As an aside, I also checked CloudBees Core built from Jenkins 2.205 running in EKS using nginx-ingress and it was fine, so it is possible the issue is limited to Apache.
Sorry, but setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS, but there is no justification given for why it is there when it isn't for HTTP.
See https://wiki.jenkins.io/display/JENKINS/Running+Jenkins+behind+Apache
In terms of a minimal reproduction usecase, the following should do the trick:
<VirtualHost *:443> ServerAdmin webmaster@example.org ServerName jenkins.example.org DocumentRoot /srv/www/jenkins.example.org/ ErrorLog ${APACHE_LOG_DIR}/jenkins.example.org-error.log CustomLog ${APACHE_LOG_DIR}/jenkins.example.org.log combined AllowEncodedSlashes NoDecode ProxyRequests Off ProxyPass / http://127.0.0.1:8200/ nocanon retry=0 ProxyPassReverse / http://127.0.0.1:8200/ RequestHeader set X-Forwarded-Proto "https" RequestHeader set X-Forwarded-Port "443" # Enable SSL </VirtualHost>
You will need to change ProxyPassReverse to the very incorrect following line to make it work (as ProxyPass / ProxyPassReverse should be mirrors of each other):
ProxyPassReverse / https://127.0.0.1/
This indicates that Jenkins isn't respecting protocols and ports correctly - and setting ProxyPreserveHost just masks the issue (as most people don't run Jenkins without a web server in front of it, so users won't be connecting to it over a non-standard port)
setting ProxyPreserveHost to On doesn't appear to be a recommendation currently as it isn't mentioned in the default HTTP configuration in your own documentation for this type of setup. It appears under HTTPS
ProxyPreserveHost On is suggested in the initial section, before any mention of HTTPS. Beware however that wiki.jenkins.io is just a collection of random edits made over many years with little oversight, not formal documentation. The demo container is what I actually verified five years ago (tracking down some more subtle issues with AllowEncodedSlashes), but ultimately the Jenkins project does not document and test a specific reverse proxy configuration—this is left either to individual administrators, or to a third-party packaging such as a Helm chart.
At any rate, I retried my test with ProxyPreserveHost On commented out, using HTTPS, and redirects still worked. This is from a local development environment, which necessarily differs from a production network in various ways, so it may be more fruitful to discuss a particular HTTP response from Jenkins that could be reproduced without actually running a reverse proxy: given a specific minimal set of request headers (mixture of browser-provided and proxy-provided/amended), whether a specific response header (Location I suppose) has one value in Jenkins 2.204 and a different value in 2.205.
Could you provide the ProxyPass / ProxyPassReverse statements from your test image?
If they mirror the fix I mentioned above already, then this would explain why it works for you.
These directives look like they are properly mirrored to me. I also tried the same without the context path (/jenkins → /) and with nocanon replaced with retry=0 as in your example.
I wonder if this is due to the non-standard port for Jenkins itself - that configuration uses 8080, whereas my setup uses 8100/8200 (for each of the two impacted instances respectively), and Olli above uses 8180.
Adding ProxyPreserveHost On to our httpd configuration did solve the problem for us as well. Thanks for the tip!
I believe the 2.204.x LTS branch is now affected by this as well - I started seeing the issue yesterday, after upgrading to 2.204.3 (which lends support for the Jetty theory).
katzdm you'll need to provide more details about your configuration or even better, the greater precision of answers that jglick was seeking. He was unable to duplicate the problem in his attempts. If the issue is a result of the jetty upgrade, then it is likely that you'll need to apply the suggested configuration changes.
Hi markewaite, I've been able to isolate the problem. Consider the two following commands, which differ only in the ordering of the X-Forwarded-Port and X-Forwarded-Host headers.
- curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
-H "Host: jenkins.example.com" \
-H "X-Forwarded-Port: 80" \
-H "X-Forwarded-Host: localhost" -o/dev/null \
2>/dev/null; echo - curl -L -w "%{url_effective}" "http://localhost:33000/whoAmI" \
-H "Host: jenkins.example.com" \
-H "X-Forwarded-Host: localhost" \
-H "X-Forwarded-Port: 80" -o/dev/null \
2>/dev/null; echo
I ran these commands within docker containers based on various different Jenkins base-images, and observed the following behaviors:
- jenkins/jenkins:2.204.2-slim:
Command 1: http://localhost/whoAmI/
Command 2: http://localhost/whoAmI/ - jenkins/jenkins:2.204.3-slim:
Command 1: http://jenkins.example.com/whoAmI/
Command 2: http://localhost/whoAmI/ - jenkins/jenkins:slim: (2.223; latest)
Command 1: http://jenkins.example.com/whoAmI/
Command 2: http://localhost/whoAmI/
So if I'm interpreting this right, it seems that the host used for Jenkins redirections is dependent on the ordering of the X-Forwarded-Port and X-Forwarded-Host headers, beginning in Jenkins 2.205+ and Jenkins 2.204.3+ for LTS. If I had to guess, this is probably due to changes made to the ForwardedRequestCustomizer class in Jetty, which were picked up in the move to Winstone 5.7. Maybe a bug in handlePort()?
To answer your previous question regarding "ProxyPreserveHost On", I am not behind an Apache reverse proxy (we use https://github.com/buzzfeed/sso), so this isn't directly relevant to me.
I found the bug in Jetty, and it seems the root cause is corrected in 9.4.27: https://github.com/eclipse/jetty.project/issues/4573
There's currently a dependabot PR open to Winstone, which will pick up the latest Jetty release: https://github.com/jenkinsci/winstone/pull/94
I've commented there with a reference to this bug, with hope that it can get merged + released.
Assigned it to myself so that I do not forget about the merge/release towards the next weekly. We need it to be backported to the 2.220.x LTS baseline.
At the same time backporting Jetty dependency bump is risky, so needs risk assessment. CC olamy who is also a Jetty maintainer
Winstone 5.9 with fixes was released: https://github.com/jenkinsci/winstone/releases/tag/winstone-5.9
Jenkins Core update pull request: https://github.com/jenkinsci/jenkins/pull/4542
Do we also need a message applied to the 2.204.3 changelog and the 2.204.4 changelog to alert users that they may be affected by it?
markewaite it would be nice, I have not got to it yet due to LTS backports
I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions.
Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545
- Jenkins LTS 2.204.5 with fixes is out: https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.204.5 . Official Changelogs are coming soon.
- ETA for Jenkins weekly 2.224 is today
olamy is this likely a result of the upgrade to Jetty 9.4.22?