-
Bug
-
Resolution: Fixed
-
Critical
-
Windows XP Professional, Version 2002, SP3
Intel(R)Xeon(R) CPU
X5670 @ 2.93 GHz
2,67 GHz, 2,00 GB of RAM
Physical Address Extension
Jenkins was today upgraded from 1.527 to 1.537.
-
Powered by SuggestiMate
I have created a new job (by copying an older job), and saved.
I want to integrate the new job into a Jenkins view, select it, press "Edit view", select the new job, press save
and get the following Java exception:
javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 241320>200000 at …
The issue is critical, because I have to implement a lot of new jobs for our testing environment, we are short before a new release of our software.
Thanks a lot for providing a solution for this problem.
Attached is the above stacktrace, as well as the output found in the error logging on the Jenkins master server, and the version of all installed plugins.
- is duplicated by
-
JENKINS-24804 javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 227340>200000
-
- Open
-
-
JENKINS-21627 Stack trace javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 238293>200000
-
- Resolved
-
-
JENKINS-30062 java.lang.IllegalStateException: Form too large 201117>200000
-
- Resolved
-
-
JENKINS-24421 Github webhook throwing "Form too large"
-
- Resolved
-
-
JENKINS-26933 Saving change in 'Configure Slicing Plugin' page causing Form too large exception
-
- Resolved
-
-
JENKINS-23221 Security Matrix too has has form too large
-
- Resolved
-
-
JENKINS-21059 Stack Trace: Form too large
-
- Closed
-
-
JENKINS-31412 Form too large
-
- Closed
-
-
JENKINS-31521 Private SSH keys: Form too large
-
- Closed
-
- is related to
-
JENKINS-24421 Github webhook throwing "Form too large"
-
- Resolved
-
- relates to
-
JENKINS-60409 Form Submission Length Cap
-
- Resolved
-
- links to
[JENKINS-20327] “Form too large” errors submitting view configurations with many jobs
In our Jenkins this problems starts in 1.536. Version used before was 1.528 or 1.526 and there was not this problem.
EDIT: New information:
I have this problem anymore. I don't be sure why, but no jenkins or plugins are updated. We had a problems to sending emails. Jenkins uses more than 4 minutes for each users to generate/send emails for developers. noe of plugin update help for this problem. Then I notice it takes long time only for those users who hasn't email configured to users config.xml file, but Jenkins resolve email address somehow. I add email address to each users config.xml and reload those. That helps to email problems and after couple of days I tried this view editing and it works also. For me there not seems to be any connect whit those two issue, but nothing else modifications are not done. Of cource amount of resets are done, but before that any reset not help this problem.
Hopley this may helps to solve this issue. Or if some one have problems to email sending takes too long, this may help them also.
Same here, we upgraded from 1.527 to 1.538 and the problem appeared.
Switching back is a bad thing since several critical issues for us were fixed > 1.527
I have the same issue in 1.538 and I found this document, is it help? Because I'm not sure where is the jetty.xml
https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=42469648
@Dennis
I found this: https://issues.jenkins-ci.org/browse/JENKINS-5210 ... but I did not find jetty.xml on our system ;-(
@Kari
could you please explain in more detail (which config.xml?) what you did to overcome the email problems? Thank you! I could not find any emails in <JENKINS_HOME>.config.xml (version 1.527).
@Angela
I mean those config.xml which are in users folder. E.g $JENKINS_HOME/users/<userid>/config.xml.
If you go to http://yourjenkins:8080/user/<userid>/configure you can configure this xml. If E-mail address on configure page are empty, it means it is not set in config.xml. If you then set email address in config page it comes to config.xml also. In some cases there may be something like <userid>@<yourdomain>.com in config page, but still nothing in config.xml. That comes from some plugin (ldap?)/ setting. But if there are not email in config.xml, it means Jenkins try to solve it everytime it send emails.
And one more issue here is that all users have not own config.xml in jenkins. I don't know why, but if you go to configure page and save your changes, config.xml file is generated.
We try to add this parameter is java -jar jenkins.war but it doesn't work. Is there any work around solution to extend the size of Jetty ?
-Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1
It seems -1 doesn't work, we change to use a number and it works now.
-Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000
I see this at Jenkins 1.542, Gerrit 2.5.2, Gerrit Trigger 2.10.1, when using the 'Query and Trigger Gerrit Patches' feature to manually trigger patchsets. Here's the stack trace:
java.lang.IllegalStateException: Form too large 216327>200000
at org.eclipse.jetty.server.Request.extractParameters(Request.java:352)
at org.eclipse.jetty.server.Request.getParameter(Request.java:790)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:184)
at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:69)
at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:62)
at org.kohsuke.stapler.AnnotationHandler.handle(AnnotationHandler.java:91)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:153)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Vivek - I also ran into the issue through 'Query and Trigger Gerrit Patches'.
- Jenkins 1.542
- Gerrit 2.8
- Gerrit Trigger 2.11.0-beta-2
What I found is that the error seems to be related to the size of the query result set, not the number of selections made for re-triggering. Essentially, the issue went away when triggering builds from a query like the following:
label:Verified=0 AND limit:10
Seems odd that it's the size of the query result set that causes the fault, not the size of the submitted 'trigger' set.
EDIT: Come to think of it, not so odd, as the whole form is likely passed back for evaluation, not just the selected elements.
@Vivek
please use the workaround as described by Dennys (thank you Dennys for providing it): -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000
it worked for me, I have upgraded last week to 1.542 and got the below error again.
I got this problem in 1.547 when I edited a large view that was constructed using the Sectioned View plugin.
I have a lot of jobs, and the way the edit form is constructed means that I hit up against the limit. This didn't use to happen - at what point was the limit introduced? I am about to try the -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 option to see if that works.
Adding -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 option to the Jenkins startup options does indeed fix this problem for me.
After adding this fix to the Jenkins start-up options, the "Restrict where this project can be run" job feature is broken. Jobs fail if this feature is used with the following error:
[EnvInject] - Loading node environment variables.
ERROR: SEVERE ERROR occurs
org.jenkinsci.lib.envinject.EnvInjectException: hudson.remoting.ChannelClosedException: channel is already closed
Is anyone else seeing the same behavior?
This is, perhaps unsurprisingly, and issue now present in the LTS. Given my setup with a large number of jobs, it seems I cannot create/configure a view anymore.
I too experienced this issue using the native windows jenkins package 1.653. However, I was not having this issue on another server running the same version of Jenkins but running on Tomcat. Once I switched to Tomcat, the issue has disappeared.
Reducing severity as there seems to be a workaround in the System property org.eclipse.jetty.server.Request.maxFormContentSize.
Given the version ranges mentioned, and mwebber's comments, it's safe to assume this was introduced in 1.535 with the switch from Winstone to Jetty.
Mine hangs at e.g. 64%, but no error in log... -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 doesn't work either...
For all those that the -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 doesn't work, add it straight after the java command and before the -jar
Example: java -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 -jar ${JENKINS_HOME}/jenkins.war
Commenters: please stop pasting long stack traces into comments, as it makes JIRA pages hard to load. Use file attachments. In this case the stack trace is of no interest anyway; what counts is the contents of the form being submitted. (Which would definitely have to go into a file attachment since it is huge.)
Restricting this issue to the problem encountered when submitting a view configuration with a large number of jobs displayed as options. There are apparently less common problems with the same symptom but different causes from submitting other big forms, which would best be filed separately and linked here.
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/resources/hudson/model/ListView/configure-entries.jelly
http://jenkins-ci.org/commit/jenkins/adfa9a30c5913563f0c7a9a6dbf826c6901ef043
Log:
[FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options.
This can be done easily by setting json="true"; otherwise the default is to pass =false for each unselected job.
Ironically, even with this fix we are sending 2× what is needed,
since currently ListView.submit does not even check the json parameter for job names:
it only pays attention to top-level request parameters that the browser defines for selected checkboxes.
Ideally we could ask buildFormTree to ignore a given form element entirely, so as to avoid the wasted data.
Compare: https://github.com/jenkinsci/jenkins/compare/3170a1bc153b...adfa9a30c591
Integrated in jenkins_main_trunk #3452
[FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options. (Revision adfa9a30c5913563f0c7a9a6dbf826c6901ef043)
Result = SUCCESS
Jesse Glick : adfa9a30c5913563f0c7a9a6dbf826c6901ef043
Files :
- core/src/main/resources/hudson/model/ListView/configure-entries.jelly
- changelog.html
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/resources/hudson/model/ListView/configure-entries.jelly
http://jenkins-ci.org/commit/jenkins/db26457e49ee5cb740b3fe2dd44477377dff7f7e
Log:
[FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options.
This can be done easily by setting json="true"; otherwise the default is to pass =false for each unselected job.
Ironically, even with this fix we are sending 2× what is needed,
since currently ListView.submit does not even check the json parameter for job names:
it only pays attention to top-level request parameters that the browser defines for selected checkboxes.
Ideally we could ask buildFormTree to ignore a given form element entirely, so as to avoid the wasted data.
(cherry picked from commit adfa9a30c5913563f0c7a9a6dbf826c6901ef043)
Compare: https://github.com/jenkinsci/jenkins/compare/4f89f983c154...db26457e49ee
Sorry, but the problem is still there...
I obtain the following error
javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 595864>200000
when I click Save to submit changes to even 1 job using Configuration Slicing plugin (http://<myjenkinshost>/slicing/jobdisabledbool/sliceconfigSubmit)
Jenkins version: 1.584
Configuration Slicing Plugin version: 1.39
@marco, you should open a separate issue for the job config slicing plugin (this issue is for the core problem).
Also, if you read the comments above, you will find a workaround.
to which xml file(name of the xml) should the below changes be done
-Dorg.eclipse.jetty.server.Request.maxFormContentSize=500000
we dont see jenkins.xml or jetty.xml in our installations
Jenkins running on Ubuntu and Centos
Updated:
The file to be updated is /etc/default/jenkins on Ubuntu, and /etc/sysconfig/jenkins on CentOS, This may vary in different deployments
I just noticed that this bug also affects the latest LTS edition as well - v1.596.3 at the current moment. I haven't compared the behavior with latest weekly builds, but assuming this defect has been successfully repaired on the master version it should probably be backported to the LTS branch as well.
I added the lts-candidate label for reference as well. Not sure if that will make any difference seeing as how the issue has already been resolved but at least it's there.
leedega Please be specific as to what you're seeing, and in which circumstances. Is this about submitting view configuration forms?
Yes, the problem is related to submitting changes to view configurations.
Was there some specific information you'd like me to provide that hasn't already been stated above?
Was there some specific information you'd like me to provide that hasn't already been stated above?
Is this a view with a very large number of items selected via checkbox?
In my latest case the view was a "Sectioned View" type, with maybe 10 sub-sections of type "List View". Each list view subsection has a regular expression for the job filter, and there are few if any jobs that are checked off individually.
I suspect I can reproduce the problem on other view types as well. If I get some time today to try that out I'll let you know the results.
I decided to just take some time right away to do some further tests and I can't seem to reproduce the problem on other view types. I have tried the following:
- nested views
- list views
- categorized jobs view
- build pipeline view
- my view
- status view
- dashboard view
Also, for some of our views of type "Sectioned View" with a smaller number of sub-sections on them, the error doesn't appear to occur. The combination of a view of type "Sectioned View" with a large number of sub-sections of type "List View" seems to be the problematic case. Also, our build farm has a large number of jobs on it (>1200 atm). If the Sectioned View were submitting information about each of these jobs for each of the list sub-views that could explain why the error only happens in this one use case.
Either way, the "hack" mentioned above for overriding the maxFormContentSize property for Jetty seems to prevent the problem, at least for us. This at least unblocks us for the time being. Still, it would be good to try and get a fix for this if possible.
In my latest case the view was a "Sectioned View" type, with maybe 10 sub-sections of type "List View". Each list view subsection has a regular expression for the job filter, and there are few if any jobs that are checked off individually.
Please file an issue against Sectioned View Plugin to investigate whether the form size can be reduced there.
the "hack" mentioned above for overriding the maxFormContentSize property for Jetty seems to prevent the problem
Not really a hack, just a legitimate option. It's reasonable to limit the size of submitted forms to a sensible value by default, but larger Jenkins instances may easily grow past that, especially given how Jenkins performs form submissions.
Please file an issue against Sectioned View Plugin to investigate whether the form size can be reduced there.
Done: JENKINS-28716
It's reasonable to limit the size of submitted forms to a sensible value by default...
Agreed. The only debatable point is what value is 'sensible' in this context. In our context / use case it is obviously not sensible.
Not really a hack, just a legitimate option.
I'm not sure I agree with your assessment. As a user of the Jenkins product I expect the tool to work right out of the box with minimal manual modification / tweaking. Unfortunately, in this case, this was not possible. To make matters worse the bug itself was not obvious at first glance, and the root cause and workaround to the problem were also not directly obvious from the error produced. In fact, the only reason I knew of the existence of this maxFormContentSize option was by searching through bug reports until I came across this one which made mention of it. Hardly a simple process.
Conversely, I suspect that if you were to select a sufficiently large form size limit such that large scale users such as us wouldn't encounter this problem in the first place it would have little to no effect on the other users of the tool. That would be a preferable solution than having your users manually modify configuration files to make the tool work. Hence my assessment that this is in fact a 'hack'.
PS: Sorry for the rant. I have been spending days / weeks testing the latest LTS edition of Jenkins in preparation for an upgrade to our build farm and I've been dealing with bug after bug, which are supposed to be 'resolved' by hack after hack and it has just been wearing on me. This experience is unfortunately reducing my confidence in the tool as a whole, and in the LTS maintenance stream in particular.
We chose to use the LTS edition instead of the head / master version to avoid problems just like this.
In some sense, it's similar to configuring Xmx/XX:MaxPermSize in that it really depends on the size of the Jenkins instance what a reasonable value is, and it's up to you to increase it if your instance grows large. It's different as it should be easier to default to a large value without negative effects, but it won't really solve the problem – large requests will still happen depending on plugin.
It's different as it should be easier to default to a large value without negative effects..
Yeah - that's what I suspected.
...large requests will still happen depending on plugin.
Yeah - this is I guess one of the unfortunate down-sides to the modular / plugin design of Jenkins. When something like this form size limit changes it takes time for that news to propagate out to all of the different maintainers of the different plugins, and even longer for them all to fix their plugins to compensate for the change.
Code changed in jenkins
User: Stephen Connolly
Path:
src/java/winstone/Launcher.java
src/java/winstone/cmdline/Option.java
http://jenkins-ci.org/commit/winstone/9ae4206d85863b1f478bf21aa1a80a9111c66722
Log:
[FIXED JENKINS-20327][FIXED JENKINS-23221][FIXED JENKINS-24804][FIXED JENKINS-21064] java.lang.IllegalStateException: Form too large
Code changed in jenkins
User: Stephen Connolly
Path:
src/java/winstone/Launcher.java
src/java/winstone/cmdline/Option.java
http://jenkins-ci.org/commit/winstone/4a97150de06b6288e021cc191ae4189e4a21b3a3
Log:
Merge pull request #20 from stephenc/master
[FIXED JENKINS-20327][FIXED JENKINS-23221][FIXED JENKINS-24804][FIXED JE...
Compare: https://github.com/jenkinsci/winstone/compare/f42497acd05d...4a97150de06b
I have marked a bunch of duplicates, but left them open, since it is typically still a crappy design in plugins (or core, as in this original issue) that they encourage forms with gigantic data. Jetty is now configured to meekly accept the whole submission, but your performance will still suffer.
What version of Jenkins picked up the jetty change to accept the whole submission? I see Stephen Connolly's patch was applied to winstone, but there hasn't been a deploy of winstone with that patch based on the tags I see, and core/war/pom.xml points at winstone-2.8, which doesn't have the patch.
Code changed in jenkins
User: Jesse Glick
Path:
war/pom.xml
http://jenkins-ci.org/commit/jenkins/8f012d63d771b22d74eec8396b2895f0e45f0e6f
Log:
[FIXED JENKINS-20327] Integrate Winstone 2.9.
Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
war/pom.xml
http://jenkins-ci.org/commit/jenkins/d76ee3e5179dd50e96e6c8e1d67ffbdec7f88a7b
Log:
JENKINS-20327 Noting merge of #1922.
Compare: https://github.com/jenkinsci/jenkins/compare/ab0198246c1c...d76ee3e5179d
Integrated in jenkins_main_trunk #4373
[FIXED JENKINS-20327] Integrate Winstone 2.9. (Revision 8f012d63d771b22d74eec8396b2895f0e45f0e6f)
Result = SUCCESS
jesse glick : 8f012d63d771b22d74eec8396b2895f0e45f0e6f
Files :
- war/pom.xml
Code changed in jenkins
User: Jesse Glick
Path:
war/pom.xml
http://jenkins-ci.org/commit/jenkins/1a91527767f6f4813835335506100848007623d6
Log:
[FIXED JENKINS-20327] Integrate Winstone 2.9.
(cherry picked from commit 8f012d63d771b22d74eec8396b2895f0e45f0e6f)
Code changed in jenkins
User: Jesse Glick
Path:
http://jenkins-ci.org/commit/jenkins/e3ccde03907f3464dec56f5dab18e849ce508548
Log:
[FIXED JENKINS-20327] Integrate Winstone 2.9.
(cherry picked from commit 8f012d63d771b22d74eec8396b2895f0e45f0e6f)
Compare: https://github.com/jenkinsci/jenkins/compare/f5f777f680b7^...e3ccde03907f
Integrated in jenkins_2.0 #5
[FIXED JENKINS-20327] Integrate Winstone 2.9. (Revision 8f012d63d771b22d74eec8396b2895f0e45f0e6f)
Result = SUCCESS
jesse glick : 8f012d63d771b22d74eec8396b2895f0e45f0e6f
Files :
- war/pom.xml
The versions after a release 1.527 must have some problems, because I realize the people are downgrading.
I have another issue (https://issues.jenkins-ci.org/browse/JENKINS-19248) that I'm waiting for the correction also.