-
Bug
-
Resolution: Unresolved
-
Major
-
Jenkins 1.581, launched through the built in Jetty
-
Powered by SuggestiMate -
Jenkins 2.184
Each time Jenkins (re)starts, its session-cookie name changes (ie JSESSIONID.some_random_string).
After a while, the browser have a bunch of session cookies, each one having a different name, causing the "Cookie" request header to be very long. The server returns a HTTP 413 response and a blank page. The user must clean his cookies in order to access Jenkins again.
Workaround: Since Jenkins 2.66 there are custom options for managing Jetty session IDs: https://github.com/jenkinsci/extras-executable-war/#jetty-session-ids
- 413_on_requests.png
- 144 kB
- is duplicated by
-
JENKINS-36880 WARNING: badMessage: 413 for HttpChannelOverHttp and WARNING: Header is too large >8192
-
- Resolved
-
-
JENKINS-36993 Cookie header too long, causing a 413 HTTP error
-
- Resolved
-
-
JENKINS-43927 Unable to login due to too many jsessionids creating a too large header
-
- Resolved
-
-
JENKINS-73428 Unable to follow output of jenkins job
-
- Resolved
-
-
JENKINS-45449 "Bad Message 431" everywhere
-
- Closed
-
- is related to
-
JENKINS-44894 Make Jetty Session ID configurable via System properties
-
- Resolved
-
- links to
[JENKINS-25046] Cookie header too long, causing a 413 HTTP error
Nothing critical about this.
It's both rare (only affecting those restarting Jenkins like crazy) and trivial to fix (remove the cookie).
I restart Jenkins once a week – when you release a new version to the apt repository, or when an OS security update requires a reboot. Does that count as "restarting like crazy"? Seems to me it's not as much rare as "inevitable, once enough time passes.
Diagnosing the problem is not trivial: currently, with 156 session cookies, I get a completely blank page when I attempt to install or upgrade some plugins (because cookie size + form data size hits some limit). The error is only visible if I open up the dev tools and have sufficient web developer-fu to investigate.
It would be nice if Jenkins could automatically clear stale cookies when there are too many of them (say, over 50 – I imagine if you have two Jenkins instances available at the same domain, you may not want them interfering with each other's sessions).
Culprit (with explanation):
https://github.com/jenkinsci/extras-executable-war/blob/5a5f01362e7649da3b39a3a45940ba3ce1ab65e8/src/main/java/Main.java#L235
FWIW I think generating a hash based on JENKINS_HOME path, or maybe the instance ID would be sufficiently unique, and not get regenerated on every startup.
The easiest solution is probably to add an argument that allows overriding this behavior. Unfortunately if we make this the default in installers, it'll break users who reverse-proxy multiple instances from the same proxy host.
Unfortunately if we make this the default in installers
Seems the reasonable default in service scripts is to initialize it with a hash of hostname + actual port to be sufficiently unique. Not sure how to handle the Windows service, but that would at least take care of the Linuxes.
Considering that Jenkins is releasing a new version every ~5 days and that if you use 20-30 you will get daily updates, there is a high chance of you to reach this issue quite fast.
This is not "restarting jenkins like crazy" - is just trying to keep a Jenkins instance up-to-date! Quite a different approach.
This is a serious bug and cleaning up the cookies doesn't seem to help, I tried doing this on Chrome and they suddenly reappeared.
Can we fix this please? This bug is making Jenkins almost useless as the server is replying with 413 to some AJAX requests and the browsers do not display any error.
For example now we get some live console logs empty! The text-view returns the output but the live output fails because if this.
I don't want to reset cookies as I re-logging in to so many web sites is really painful as many of them do have 2FA!.
Same here with the LTS (2.7.1) and I see a (Windows/JNLP) slave also cumulating those cookies:
Aug 25 08:24:48.116 localhost httpd[ssl]: INFO <master> <slave> - - TLSv1.2 ECDHE-RSA-DES-CBC3-SHA \"GET /jenkins/tcpSlaveAgentListener/ HTTP/1.1\" 200 12 \"-\" \"Java/1.8.0_101\" \"JSESSIONID.fff7b3dd=11l7hv2dfznc3184d4is3wbu10; JSESSIONID.968f4893=1uwkng7acyzl76hadp68n8yqq; JSESSIONID.ca951ea0=wvur34byo02hakkwrq7n88gg; JSESSIONID.738879c3=ahclxy0pqfeh1ptgpsld0kut; JSESSIONID.e4031fa7=1fdr8gl8ywl2i6qqpntj0m9vf; JSESSIONID.2b52b2d7=nyrhdvz48z28begkc4vgkoo5; JSESSIONID.9d172e35=902aok32vjqcceaj80ndqvv7; JSESSIONID.9a44300d=1xeqmrzml9zmk1esvdtzyawbln; JSESSIONID.943f4575=17jml3nitbi352urb499w998z; JSESSIONID.38029651=10ndnhjicmvav1ldit0phkpmc6; JSESSIONID.1fb8e86c=1cott3tydzw6qijta32q6f4wn; JSESSIONID.c187efae=1cx6anjacypkw15hkka0en0sas; JSESSIONID.21c6cb9e=1hp54um5nxa5zvp76qwebr4ck; JSESSIONID.e3579812=1gfr59v4cr9sigobm2sc6k9ma; JSESSIONID.0ff2be94=1947yt88jearx142kr33bxcz18; JSESSIONID.d5d6f77a=ie8vrevmwvksk7l9rj2fjdcx; JSESSIONID.d927eaf9=1b0z10l3uhus5q03fxgjfsx26; JSESSIONID.eb158e24=bl6yte8nzkxq1cxfqj94d1lug; JSESSIONID.df59a26e=k5a356tpfguk1jfy88okv27h8; JSESSIONID.bf86d1fa=htu0skwixvyx1dz8rury4zy5h; JSESSIONID.e00c23d9=66tbk20lgawe1vgr42kfzuvy6; JSESSIONID.eca7f3ed=1fn4vunf6lb5p1e04lht1yylte; JSESSIONID.eea69353=1sufj78yvlg2k1jil96s51aw6t; JSESSIONID.c69bdaf8=1lrxn49sph8t0qcxzc8jxtlje; JSESSIONID.a9f71cc4=150jt1zaycb5m7vut4s819mv7; JSESSIONID.730c5a61=1ocsh9yy95rj99775cju7o14y; JSESSIONID.a3c1e801=1nuvij8odw28jnhsqlxt4lmq4; JSESSIONID.c4b45b1d=131jfczqa71c81ggev69tz0fit; JSESSIONID.8023cc01=j2x42se0fj6u1j4m8e291ozzo; JSESSIONID.d3f64b53=1uurqkjgwcrbk1kfjqp7o7v6gp; JSESSIONID.e3aabb49=nepfr2sdwpwb2pxfim51ghi3; JSESSIONID.86f1a901=1n01r80jookai1dqvylcsrghxk; JSESSIONID.ba8f9d35=1umbvucj9k2a1vbu0yerqt8k7; JSESSIONID.12989142=fz65vrkhy2cvonbxbt758vf; JSESSIONID.c738909a=11kysf3hezdbpfxkf6i1rm635; JSESSIONID.ecfdce60=yd6ufbwaku8n13yaoxb5jb5e8; JSESSIONID.d9040e39=1ffmb9o2zym5z1vqiaz9vpv5ze; JSESSIONID.25dc9165=1wmqe68gwsolnhxiv525ltmux; JSESSIONID.1041b5cc=ktg3c081xmzsh040wx39x7ve; JSESSIONID.4b267dcd=q037pjax8vhce9jf648q5xpl; JSESSIONID.b3613560=m105j0hpr319ag95lestlohx; JSESSIONID.d6140975=10vbk07hf3hi6u11tjxdc9cuv\"
I suspect this to slow down Jenkins in general (clients, slave and master), as the overhead per request goes up and the master may have to process all those cookies to find out which one is valid, right?
Due to the possible performance aspect, I don't think this issue is so Minor: promoting to Major...
I'm probably wrong about the master processing all the cookies though.
But I'm pretty confident there is difference in term of responsiveness when we get rid of those cookies for every client.
REM: I our case, cookie is changing every night because of a backup job that stop Jenkins for a few seconds in order to get consistent data.
bdonneaux ssbarnea There's no point in repeating the same things others wrote before you. You're just making the comments that e.g. explain how to go about fixing this issue more difficult to find. Try adding something original to the discussion instead.
Reading the previous comments once again, I don't see anyone referring the impact on slaves and the performance.
Sorry if my attempt to give more feedback is annoying some. Not coding too much my self, I'm trying to support community in other ways.
danielbeck, since you've already pointed at the cause and proposed a possible fix, do you expect us (the not very original users) to send a pull request with the solution you're proposing (hash port number)???
Meanwhile, I'd like to find workaround for our situation that does not require me patching and compiling Jenkins from source or restarting every browser and slave after each restart of the master...
Since danielbeck wants something original, here is my modest contribution that works for us for Apache (with mod_rewrite and mod_header):
# Detect multiple session cookies and extract the oldest (assuming it's the first set in the Cookie string) RewriteCond "%{HTTP_COOKIE}" ".*(JSESSIONID\.[^=]+)=.*(JSESSIONID\.[^=]+)=[^;]+.*" # Store the name of this session cookie in an environment variable RewriteRule /jenkins/ - [env=OLDJSESSIONID:%1] # Expire this obsolete session cookie if found Header add Set-Cookie "%{OLDJSESSIONID}e=;Expires=Thu, 01 Jan 1970 00:00:01 GMT;Path=/jenkins;Secure;HttpOnly" env=OLDJSESSIONID
I've tested this with Firefox and Chromium and it looks ok for now...
But, in my opinion, randomizing the cookie name is a bad idea from the first place.
I'm in deeply in favor of a static name, based on the context_path and/or the port number by default, or a custom property.
bdonneaux Please note that I am Daniel Beck, not Daniel Beckham.
since you've already pointed at the cause and proposed a possible fix, do you expect us … to send a pull request with the solution you're proposing (hash port number)???
No, I recorded my earlier investigation into this issue. While I'd really like to fix this, it doesn't rank highest on my (long and growing) to-do list. Meanwhile, anyone is free to tackle this, and my comments may even help them.
But, in my opinion, randomizing the cookie name is a bad idea from the first place.
Right – the issue here is multiple Jenkinses on the same host, possibly behind a reverse proxy. Then, you may have same Jenkins home directory (different machine), same port (different host / different context), same context (different port), same (front-end) host name, depending at what stage you check. And suddenly, they'd share cookies.
So while I'd like to keep some uniqueness to the cookie name, ideally it would be in a way that reuses the same key over multiple restarts, while still being unique per instance.
the not very original users
I apologize for the choice of words. It wasn't meant disparaging.
Definitely a problem for us. We run jenkins behind nginx, and for various reasons, we're required to have a fairly strict setting on cookie size, which means this problem hits our users every week or two. It's incredibly annoying.
I'd be happy to submit a PR for the behavior requested above if the Jenkins folks think it'd be accepted
I'd be happy to submit a PR for the behavior requested above if the Jenkins folks think it'd be accepted
If by that you're referring to my comments from Jul 19, at least this "Jenkins folk" thinks it's a reasonable approach No guarantees it'll pass the review gauntlet though.
Just adding another bit of evidence that this is now more common than you might think: you practically cannot get Chrome (the most popular browser) to end a Session scope. you can log out, close tabs, exit the browser, throw your machine in the ocean, buy a new one, and the Session scope will STILL be ocean.
Given that it takes only 150 odd cookies to swamp it or so - if you are foolishly restarting often (dev time: all the time) or with popular updates (once a week) it can happen more often than you think in reality these days. Moreover, even before it happens the amount of header noise increases slowing requests down etc.
(no solutions, just evidence that this would be a great one to solve one day).
I just came across this issue after wondering for weeks why the console log for my builds was no longer updating. The spinner would disappear after one loop, but reloading the page would show more logs. Interestingly, the Pipeline Step log view always worked fine, only the traditional console log broke. So I was getting a 413 on logText/progressiveHtml. Clearing cookies fixed it. I don't restart frequently, but I had 160 session cookies, I guess just accumulated over 2 years of administering the same Jenkins instance from this laptop.
Any update on this issue? The longer it is not fixed the more and more users are affected.
Thanks
Is there a workaround? Such as being able to tell Jenkins to use a stable cookie for its sessionid so it won't change on restarts?
I just ran into this after a year of running our Jenkins server and instead of a 413, I lose settings (e.g. iconSize) because it gets truncated by something (probably the nginx proxy in front?)
hotfix would be to make the Cookie configurable via a system property. Here: https://github.com/jenkinsci/extras-executable-war/blob/master/src/main/java/Main.java#L258 . It will retain the cookies fixed.
Would it work for you? Can create a PR then
Created a patch: https://github.com/jenkinsci/extras-executable-war/pull/9 . It won't fix the issue by default, but there will be a workaround at least
Code changed in jenkins
User: Oleg Nenashev
Path:
README.md
src/main/java/Main.java
http://jenkins-ci.org/commit/extras-executable-war/7afa0adbd7a6b0f1391e7ebd3213e4a79ae45e3f
Log:
JENKINS-44894 - Add options to disable random Cookie generation or to specify a custom one (workaround for JENKINS-25046) (#9)
JENKINS-44894 - Add options to disable random Cookie generation or to specify a custom one (workaround for JENKINS-25046)
I hope to get Extras Executable War 1.35 released and integrated into the next weekly release. Documentation regarding new system properties is available here: https://github.com/jenkinsci/extras-executable-war#jetty-session-ids
As we discussed in the pull request, it does not fully close this issue. Once the fix is released I am going to create a follow-up issue for change of the default approach as we discussed in https://github.com/jenkinsci/extras-executable-war/pull/9#issuecomment-307844186
Unassigning myself since I have provided a workaround. If somebody is interested to deliver a fix working out-of-the-box, please feel free to take it
The random session generation seems like a really bad design decision. Why not using Jenkins URL as base for a hash function. That's clearly unique per instance, doesn't change often and survives any number of restarts.
Why not using Jenkins URL as base for a hash function
The session ID is set in a component that is independent from Jenkins and knows nothing about it, when the configured URL is not yet even known, and the configured URL can change at any time (in fact, it will change for every initially configured Jenkins instance). There are so many problems here.
Although I suppose "All sessions are invalidated when I change the Jenkins URL" would make a fun bug report.
Well, in this case the solution is to save this generated ID in the working directory and always re-use them if found. I doubt anyone would be able to run multiple Jenkins instances from the same working-directory.
Bump this issue.
We use Jenkins enterprise and use the operation center to manage the masters.
With more masters, we run into this issue quite frequently.
https://github.com/jenkinsci/jenkins/pull/3939 was integrated and released in 2.184. I marked it as the LTS candidate, but I am not sure it will be considered for backporting by olivergondza talking the rebase issues in the pull request. Whomever is interested, please feel free to suggest a clean pull request against the LTS branch
This looks like it might be back in 2.303.1
Clearing cookies (of which there were many) fixed it.
I recognize what you are saying. We are having a loop in our SSO login procedure and if we remove the Cookie mentioned here it works again. This is with LTS 2.319.2.
File a new ticket. Provide a lot of information.
Please leave me out if it.
Josh Soref
Added 2022-01-24 09:49
File a new ticket. Provide a lot of information.
Please leave me out if it.
No new information. Issue is the same and it persists for years. It is definitely fixable, you need to simply store the ID somewhere in Jenkins config instead of generating a new one at each restart.
Stop reopening issues.
A code change was made and delivered.
File a new issue.
The issue is not about code change, it is about error that still needs to be fixed. And it is easy to fix. If you do not want to do this, unassign it, but stop closing it.
A code fix was delivered
If an app crashes once and a code fix is delivered to make it stop crashing and it later crashes again, that is not a good reason to reopen the bug. In that model you'd only need ~5 bugs and they'd constantly be reopened.
If the code fix didn't work and exactly the same issue is still there, then it 100% should be reopened.
If the bugs actually get fixed then they can stay closed.
I don't uderstand what fix you are talking about. The problem is that by default a new ID is generated each time Jenkins is restarted, and I don't see any changes to this behavior in git history. There's only a ugly workaround that nobody knows about until he face the issue and spend several hours as minimum to investigate it.
The code fix did work and was fine for more than enough time.
File a new ticket.
A change to that code would be a significant behavior change and would not have the same summary as this ticket.
> The code fix did work and was fine for more than enough time.
Again, it is not a fix, it is just a workaround that does not work with default settings.
> A change to that code would be a significant behavior change and would not have the same summary as this ticket.
It is not a significant change, just adding couple lines of code. Try to load a stored ID, if not successful generate a new one and store it.
bizdelnick as requested by jsoref, please open a new issue. Reopening this issue is not helping you get a fix and is not helping others.
I am acting in my role as a member of the Jenkins board by making this request. I believe that the request from Josh to open a new ticket is reasonable.
I won't stop reopening this issue. There were issues opened by other users about this bug that were closed as duplicates of this one, so I conclude filing a new issue does not work.
The bug, reported here, was not fixed. The provided workaround "if you don't want this happen to you, go change your system properties" is not acceptable.
I can agree with Dmitry. Since Cookies can be controlled on the server side, the server is responsible for deleting the obsolete ones.
Same problem seen with 2.5