Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20327

“Form too large” errors submitting view configurations with many jobs

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • Windows XP Professional, Version 2002, SP3
      Intel(R)Xeon(R) CPU
      X5670 @ 2.93 GHz
      2,67 GHz, 2,00 GB of RAM
      Physical Address Extension

      Jenkins was today upgraded from 1.527 to 1.537.

      I have created a new job (by copying an older job), and saved.
      I want to integrate the new job into a Jenkins view, select it, press "Edit view", select the new job, press save

      and get the following Java exception:

      javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 241320>200000
        at …
      

      The issue is critical, because I have to implement a lot of new jobs for our testing environment, we are short before a new release of our software.

      Thanks a lot for providing a solution for this problem.

      Attached is the above stacktrace, as well as the output found in the error logging on the Jenkins master server, and the version of all installed plugins.

          [JENKINS-20327] “Form too large” errors submitting view configurations with many jobs

          Daniel Baldo added a comment -

          The versions after a release 1.527 must have some problems, because I realize the people are downgrading.
          I have another issue (https://issues.jenkins-ci.org/browse/JENKINS-19248) that I'm waiting for the correction also.

          Daniel Baldo added a comment - The versions after a release 1.527 must have some problems, because I realize the people are downgrading. I have another issue ( https://issues.jenkins-ci.org/browse/JENKINS-19248 ) that I'm waiting for the correction also.

          Kari Sivonen added a comment - - edited

          In our Jenkins this problems starts in 1.536. Version used before was 1.528 or 1.526 and there was not this problem.

          EDIT: New information:
          I have this problem anymore. I don't be sure why, but no jenkins or plugins are updated. We had a problems to sending emails. Jenkins uses more than 4 minutes for each users to generate/send emails for developers. noe of plugin update help for this problem. Then I notice it takes long time only for those users who hasn't email configured to users config.xml file, but Jenkins resolve email address somehow. I add email address to each users config.xml and reload those. That helps to email problems and after couple of days I tried this view editing and it works also. For me there not seems to be any connect whit those two issue, but nothing else modifications are not done. Of cource amount of resets are done, but before that any reset not help this problem.

          Hopley this may helps to solve this issue. Or if some one have problems to email sending takes too long, this may help them also.

          Kari Sivonen added a comment - - edited In our Jenkins this problems starts in 1.536. Version used before was 1.528 or 1.526 and there was not this problem. EDIT: New information: I have this problem anymore. I don't be sure why, but no jenkins or plugins are updated. We had a problems to sending emails. Jenkins uses more than 4 minutes for each users to generate/send emails for developers. noe of plugin update help for this problem. Then I notice it takes long time only for those users who hasn't email configured to users config.xml file, but Jenkins resolve email address somehow. I add email address to each users config.xml and reload those. That helps to email problems and after couple of days I tried this view editing and it works also. For me there not seems to be any connect whit those two issue, but nothing else modifications are not done. Of cource amount of resets are done, but before that any reset not help this problem. Hopley this may helps to solve this issue. Or if some one have problems to email sending takes too long, this may help them also.

          wbauer added a comment -

          Same here, we upgraded from 1.527 to 1.538 and the problem appeared.
          Switching back is a bad thing since several critical issues for us were fixed > 1.527

          wbauer added a comment - Same here, we upgraded from 1.527 to 1.538 and the problem appeared. Switching back is a bad thing since several critical issues for us were fixed > 1.527

          Dennys Hsieh added a comment -

          I have the same issue in 1.538 and I found this document, is it help? Because I'm not sure where is the jetty.xml
          https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=42469648

          Dennys Hsieh added a comment - I have the same issue in 1.538 and I found this document, is it help? Because I'm not sure where is the jetty.xml https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=42469648

          Angela Bauer added a comment -

          @Dennis
          I found this: https://issues.jenkins-ci.org/browse/JENKINS-5210 ... but I did not find jetty.xml on our system ;-(

          @Kari
          could you please explain in more detail (which config.xml?) what you did to overcome the email problems? Thank you! I could not find any emails in <JENKINS_HOME>.config.xml (version 1.527).

          Angela Bauer added a comment - @Dennis I found this: https://issues.jenkins-ci.org/browse/JENKINS-5210 ... but I did not find jetty.xml on our system ;-( @Kari could you please explain in more detail (which config.xml?) what you did to overcome the email problems? Thank you! I could not find any emails in <JENKINS_HOME>.config.xml (version 1.527).

          Kari Sivonen added a comment -

          @Angela
          I mean those config.xml which are in users folder. E.g $JENKINS_HOME/users/<userid>/config.xml.
          If you go to http://yourjenkins:8080/user/<userid>/configure you can configure this xml. If E-mail address on configure page are empty, it means it is not set in config.xml. If you then set email address in config page it comes to config.xml also. In some cases there may be something like <userid>@<yourdomain>.com in config page, but still nothing in config.xml. That comes from some plugin (ldap?)/ setting. But if there are not email in config.xml, it means Jenkins try to solve it everytime it send emails.

          And one more issue here is that all users have not own config.xml in jenkins. I don't know why, but if you go to configure page and save your changes, config.xml file is generated.

          Kari Sivonen added a comment - @Angela I mean those config.xml which are in users folder. E.g $JENKINS_HOME/users/<userid>/config.xml. If you go to http://yourjenkins:8080/user/ <userid>/configure you can configure this xml. If E-mail address on configure page are empty, it means it is not set in config.xml. If you then set email address in config page it comes to config.xml also. In some cases there may be something like <userid>@<yourdomain>.com in config page, but still nothing in config.xml. That comes from some plugin (ldap?)/ setting. But if there are not email in config.xml, it means Jenkins try to solve it everytime it send emails. And one more issue here is that all users have not own config.xml in jenkins. I don't know why, but if you go to configure page and save your changes, config.xml file is generated.

          Kyle Kinkade added a comment -

          Submitting edit view form on a jenkins box that has 30912 jobs.

          Kyle Kinkade added a comment - Submitting edit view form on a jenkins box that has 30912 jobs.

          Dennys Hsieh added a comment -

          We try to add this parameter is java -jar jenkins.war but it doesn't work. Is there any work around solution to extend the size of Jetty ?
          -Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1

          Dennys Hsieh added a comment - We try to add this parameter is java -jar jenkins.war but it doesn't work. Is there any work around solution to extend the size of Jetty ? -Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1

          Dennys Hsieh added a comment -

          It seems -1 doesn't work, we change to use a number and it works now.
          -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000

          Dennys Hsieh added a comment - It seems -1 doesn't work, we change to use a number and it works now. -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000

          Vivek Ayer added a comment -

          I see this at Jenkins 1.542, Gerrit 2.5.2, Gerrit Trigger 2.10.1, when using the 'Query and Trigger Gerrit Patches' feature to manually trigger patchsets. Here's the stack trace:

          java.lang.IllegalStateException: Form too large 216327>200000
          at org.eclipse.jetty.server.Request.extractParameters(Request.java:352)
          at org.eclipse.jetty.server.Request.getParameter(Request.java:790)
          at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:184)
          at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:69)
          at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:62)
          at org.kohsuke.stapler.AnnotationHandler.handle(AnnotationHandler.java:91)
          at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:153)
          at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
          at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
          at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
          at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:225)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
          at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
          at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
          at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
          at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
          at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
          at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
          at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
          at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
          at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          at org.eclipse.jetty.server.Server.handle(Server.java:370)
          at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
          at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
          at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
          at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
          at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
          at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:701)

          Vivek Ayer added a comment - I see this at Jenkins 1.542, Gerrit 2.5.2, Gerrit Trigger 2.10.1, when using the 'Query and Trigger Gerrit Patches' feature to manually trigger patchsets. Here's the stack trace: java.lang.IllegalStateException: Form too large 216327>200000 at org.eclipse.jetty.server.Request.extractParameters(Request.java:352) at org.eclipse.jetty.server.Request.getParameter(Request.java:790) at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:184) at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:69) at org.kohsuke.stapler.QueryParameter$HandlerImpl.parse(QueryParameter.java:62) at org.kohsuke.stapler.AnnotationHandler.handle(AnnotationHandler.java:91) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:153) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631) at org.kohsuke.stapler.Stapler.service(Stapler.java:225) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701)

          Andrew Jeffery added a comment - - edited

          Vivek - I also ran into the issue through 'Query and Trigger Gerrit Patches'.

          • Jenkins 1.542
          • Gerrit 2.8
          • Gerrit Trigger 2.11.0-beta-2

          What I found is that the error seems to be related to the size of the query result set, not the number of selections made for re-triggering. Essentially, the issue went away when triggering builds from a query like the following:

          label:Verified=0 AND limit:10

          Seems odd that it's the size of the query result set that causes the fault, not the size of the submitted 'trigger' set.

          EDIT: Come to think of it, not so odd, as the whole form is likely passed back for evaluation, not just the selected elements.

          Andrew Jeffery added a comment - - edited Vivek - I also ran into the issue through 'Query and Trigger Gerrit Patches'. Jenkins 1.542 Gerrit 2.8 Gerrit Trigger 2.11.0-beta-2 What I found is that the error seems to be related to the size of the query result set, not the number of selections made for re-triggering. Essentially, the issue went away when triggering builds from a query like the following: label:Verified=0 AND limit:10 Seems odd that it's the size of the query result set that causes the fault, not the size of the submitted 'trigger' set. EDIT: Come to think of it, not so odd, as the whole form is likely passed back for evaluation, not just the selected elements.

          Angela Bauer added a comment -

          @Vivek
          please use the workaround as described by Dennys (thank you Dennys for providing it): -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000
          it worked for me, I have upgraded last week to 1.542 and got the below error again.

          Angela Bauer added a comment - @Vivek please use the workaround as described by Dennys (thank you Dennys for providing it): -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 it worked for me, I have upgraded last week to 1.542 and got the below error again.

          I got this problem in 1.547 when I edited a large view that was constructed using the Sectioned View plugin.

          I have a lot of jobs, and the way the edit form is constructed means that I hit up against the limit. This didn't use to happen - at what point was the limit introduced? I am about to try the -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 option to see if that works.

          Matthew Webber added a comment - I got this problem in 1.547 when I edited a large view that was constructed using the Sectioned View plugin. I have a lot of jobs, and the way the edit form is constructed means that I hit up against the limit. This didn't use to happen - at what point was the limit introduced? I am about to try the -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 option to see if that works.

          Adding -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 option to the Jenkins startup options does indeed fix this problem for me.

          Matthew Webber added a comment - Adding -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 option to the Jenkins startup options does indeed fix this problem for me.

          John Jackson added a comment -

          After adding this fix to the Jenkins start-up options, the "Restrict where this project can be run" job feature is broken. Jobs fail if this feature is used with the following error:
          [EnvInject] - Loading node environment variables.
          ERROR: SEVERE ERROR occurs
          org.jenkinsci.lib.envinject.EnvInjectException: hudson.remoting.ChannelClosedException: channel is already closed

          Is anyone else seeing the same behavior?

          John Jackson added a comment - After adding this fix to the Jenkins start-up options, the "Restrict where this project can be run" job feature is broken. Jobs fail if this feature is used with the following error: [EnvInject] - Loading node environment variables. ERROR: SEVERE ERROR occurs org.jenkinsci.lib.envinject.EnvInjectException: hudson.remoting.ChannelClosedException: channel is already closed Is anyone else seeing the same behavior?

          This is, perhaps unsurprisingly, and issue now present in the LTS. Given my setup with a large number of jobs, it seems I cannot create/configure a view anymore.

          Michael Tautschnig added a comment - This is, perhaps unsurprisingly, and issue now present in the LTS. Given my setup with a large number of jobs, it seems I cannot create/configure a view anymore.

          David Ruhmann added a comment -

          I too experienced this issue using the native windows jenkins package 1.653. However, I was not having this issue on another server running the same version of Jenkins but running on Tomcat. Once I switched to Tomcat, the issue has disappeared.

          David Ruhmann added a comment - I too experienced this issue using the native windows jenkins package 1.653. However, I was not having this issue on another server running the same version of Jenkins but running on Tomcat. Once I switched to Tomcat, the issue has disappeared.

          Daniel Beck added a comment -

          Reducing severity as there seems to be a workaround in the System property org.eclipse.jetty.server.Request.maxFormContentSize.

          Given the version ranges mentioned, and mwebber's comments, it's safe to assume this was introduced in 1.535 with the switch from Winstone to Jetty.

          Daniel Beck added a comment - Reducing severity as there seems to be a workaround in the System property org.eclipse.jetty.server.Request.maxFormContentSize . Given the version ranges mentioned, and mwebber 's comments, it's safe to assume this was introduced in 1.535 with the switch from Winstone to Jetty.

          Hirad Asadi added a comment -

          Mine hangs at e.g. 64%, but no error in log... -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 doesn't work either...

          Hirad Asadi added a comment - Mine hangs at e.g. 64%, but no error in log... -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 doesn't work either...

          Jason W added a comment -

          For all those that the -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 doesn't work, add it straight after the java command and before the -jar
          Example: java -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 -jar ${JENKINS_HOME}/jenkins.war

          Jason W added a comment - For all those that the -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 doesn't work, add it straight after the java command and before the -jar Example: java -Dorg.eclipse.jetty.server.Request.maxFormContentSize=5000000 -jar ${JENKINS_HOME}/jenkins.war

          Jesse Glick added a comment -

          Commenters: please stop pasting long stack traces into comments, as it makes JIRA pages hard to load. Use file attachments. In this case the stack trace is of no interest anyway; what counts is the contents of the form being submitted. (Which would definitely have to go into a file attachment since it is huge.)

          Restricting this issue to the problem encountered when submitting a view configuration with a large number of jobs displayed as options. There are apparently less common problems with the same symptom but different causes from submitting other big forms, which would best be filed separately and linked here.

          Jesse Glick added a comment - Commenters: please stop pasting long stack traces into comments, as it makes JIRA pages hard to load. Use file attachments. In this case the stack trace is of no interest anyway; what counts is the contents of the form being submitted. (Which would definitely have to go into a file attachment since it is huge.) Restricting this issue to the problem encountered when submitting a view configuration with a large number of jobs displayed as options. There are apparently less common problems with the same symptom but different causes from submitting other big forms, which would best be filed separately and linked here.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/resources/hudson/model/ListView/configure-entries.jelly
          http://jenkins-ci.org/commit/jenkins/adfa9a30c5913563f0c7a9a6dbf826c6901ef043
          Log:
          [FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options.
          This can be done easily by setting json="true"; otherwise the default is to pass =false for each unselected job.
          Ironically, even with this fix we are sending 2× what is needed,
          since currently ListView.submit does not even check the json parameter for job names:
          it only pays attention to top-level request parameters that the browser defines for selected checkboxes.
          Ideally we could ask buildFormTree to ignore a given form element entirely, so as to avoid the wasted data.

          Compare: https://github.com/jenkinsci/jenkins/compare/3170a1bc153b...adfa9a30c591

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/resources/hudson/model/ListView/configure-entries.jelly http://jenkins-ci.org/commit/jenkins/adfa9a30c5913563f0c7a9a6dbf826c6901ef043 Log: [FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options. This can be done easily by setting json="true"; otherwise the default is to pass =false for each unselected job. Ironically, even with this fix we are sending 2× what is needed, since currently ListView.submit does not even check the json parameter for job names: it only pays attention to top-level request parameters that the browser defines for selected checkboxes. Ideally we could ask buildFormTree to ignore a given form element entirely, so as to avoid the wasted data. Compare: https://github.com/jenkinsci/jenkins/compare/3170a1bc153b...adfa9a30c591

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3452
          [FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options. (Revision adfa9a30c5913563f0c7a9a6dbf826c6901ef043)

          Result = SUCCESS
          Jesse Glick : adfa9a30c5913563f0c7a9a6dbf826c6901ef043
          Files :

          • core/src/main/resources/hudson/model/ListView/configure-entries.jelly
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3452 [FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options. (Revision adfa9a30c5913563f0c7a9a6dbf826c6901ef043) Result = SUCCESS Jesse Glick : adfa9a30c5913563f0c7a9a6dbf826c6901ef043 Files : core/src/main/resources/hudson/model/ListView/configure-entries.jelly changelog.html

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/resources/hudson/model/ListView/configure-entries.jelly
          http://jenkins-ci.org/commit/jenkins/db26457e49ee5cb740b3fe2dd44477377dff7f7e
          Log:
          [FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options.
          This can be done easily by setting json="true"; otherwise the default is to pass =false for each unselected job.
          Ironically, even with this fix we are sending 2× what is needed,
          since currently ListView.submit does not even check the json parameter for job names:
          it only pays attention to top-level request parameters that the browser defines for selected checkboxes.
          Ideally we could ask buildFormTree to ignore a given form element entirely, so as to avoid the wasted data.
          (cherry picked from commit adfa9a30c5913563f0c7a9a6dbf826c6901ef043)

          Compare: https://github.com/jenkinsci/jenkins/compare/4f89f983c154...db26457e49ee

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/resources/hudson/model/ListView/configure-entries.jelly http://jenkins-ci.org/commit/jenkins/db26457e49ee5cb740b3fe2dd44477377dff7f7e Log: [FIXED JENKINS-20327] When saving a list view, submit POST data with size proportional to the # of jobs selected for inclusion, not the number of options. This can be done easily by setting json="true"; otherwise the default is to pass =false for each unselected job. Ironically, even with this fix we are sending 2× what is needed, since currently ListView.submit does not even check the json parameter for job names: it only pays attention to top-level request parameters that the browser defines for selected checkboxes. Ideally we could ask buildFormTree to ignore a given form element entirely, so as to avoid the wasted data. (cherry picked from commit adfa9a30c5913563f0c7a9a6dbf826c6901ef043) Compare: https://github.com/jenkinsci/jenkins/compare/4f89f983c154...db26457e49ee

          Sorry, but the problem is still there...
          I obtain the following error

          javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 595864>200000

          when I click Save to submit changes to even 1 job using Configuration Slicing plugin (http://<myjenkinshost>/slicing/jobdisabledbool/sliceconfigSubmit)

          Jenkins version: 1.584
          Configuration Slicing Plugin version: 1.39

          marco boschetti added a comment - Sorry, but the problem is still there... I obtain the following error javax.servlet.ServletException: java.lang.IllegalStateException: Form too large 595864>200000 when I click Save to submit changes to even 1 job using Configuration Slicing plugin (http://<myjenkinshost>/slicing/jobdisabledbool/sliceconfigSubmit) Jenkins version: 1.584 Configuration Slicing Plugin version: 1.39

          Daniel Beck added a comment -

          Clearly a different issue (not view config, but job config slicing).

          Daniel Beck added a comment - Clearly a different issue (not view config, but job config slicing).

          @marco, you should open a separate issue for the job config slicing plugin (this issue is for the core problem).
          Also, if you read the comments above, you will find a workaround.

          Matthew Webber added a comment - @marco, you should open a separate issue for the job config slicing plugin (this issue is for the core problem). Also, if you read the comments above, you will find a workaround.

          kishorerp added a comment - - edited

          to which xml file(name of the xml) should the below changes be done

          -Dorg.eclipse.jetty.server.Request.maxFormContentSize=500000
          we dont see jenkins.xml or jetty.xml in our installations
          Jenkins running on Ubuntu and Centos
          Updated:
          The file to be updated is /etc/default/jenkins on Ubuntu, and /etc/sysconfig/jenkins on CentOS, This may vary in different deployments

          kishorerp added a comment - - edited to which xml file(name of the xml) should the below changes be done -Dorg.eclipse.jetty.server.Request.maxFormContentSize=500000 we dont see jenkins.xml or jetty.xml in our installations Jenkins running on Ubuntu and Centos Updated: The file to be updated is /etc/default/jenkins on Ubuntu, and /etc/sysconfig/jenkins on CentOS, This may vary in different deployments

          I just noticed that this bug also affects the latest LTS edition as well - v1.596.3 at the current moment. I haven't compared the behavior with latest weekly builds, but assuming this defect has been successfully repaired on the master version it should probably be backported to the LTS branch as well.

          I added the lts-candidate label for reference as well. Not sure if that will make any difference seeing as how the issue has already been resolved but at least it's there.

          Kevin Phillips added a comment - I just noticed that this bug also affects the latest LTS edition as well - v1.596.3 at the current moment. I haven't compared the behavior with latest weekly builds, but assuming this defect has been successfully repaired on the master version it should probably be backported to the LTS branch as well. I added the lts-candidate label for reference as well. Not sure if that will make any difference seeing as how the issue has already been resolved but at least it's there.

          Daniel Beck added a comment -

          leedega Please be specific as to what you're seeing, and in which circumstances. Is this about submitting view configuration forms?

          Daniel Beck added a comment - leedega Please be specific as to what you're seeing, and in which circumstances. Is this about submitting view configuration forms?

          Yes, the problem is related to submitting changes to view configurations.

          Was there some specific information you'd like me to provide that hasn't already been stated above?

          Kevin Phillips added a comment - Yes, the problem is related to submitting changes to view configurations. Was there some specific information you'd like me to provide that hasn't already been stated above?

          Daniel Beck added a comment -

          Was there some specific information you'd like me to provide that hasn't already been stated above?

          Is this a view with a very large number of items selected via checkbox?

          Daniel Beck added a comment - Was there some specific information you'd like me to provide that hasn't already been stated above? Is this a view with a very large number of items selected via checkbox?

          In my latest case the view was a "Sectioned View" type, with maybe 10 sub-sections of type "List View". Each list view subsection has a regular expression for the job filter, and there are few if any jobs that are checked off individually.

          I suspect I can reproduce the problem on other view types as well. If I get some time today to try that out I'll let you know the results.

          Kevin Phillips added a comment - In my latest case the view was a "Sectioned View" type, with maybe 10 sub-sections of type "List View". Each list view subsection has a regular expression for the job filter, and there are few if any jobs that are checked off individually. I suspect I can reproduce the problem on other view types as well. If I get some time today to try that out I'll let you know the results.

          I decided to just take some time right away to do some further tests and I can't seem to reproduce the problem on other view types. I have tried the following:

          • nested views
          • list views
          • categorized jobs view
          • build pipeline view
          • my view
          • status view
          • dashboard view

          Also, for some of our views of type "Sectioned View" with a smaller number of sub-sections on them, the error doesn't appear to occur. The combination of a view of type "Sectioned View" with a large number of sub-sections of type "List View" seems to be the problematic case. Also, our build farm has a large number of jobs on it (>1200 atm). If the Sectioned View were submitting information about each of these jobs for each of the list sub-views that could explain why the error only happens in this one use case.

          Either way, the "hack" mentioned above for overriding the maxFormContentSize property for Jetty seems to prevent the problem, at least for us. This at least unblocks us for the time being. Still, it would be good to try and get a fix for this if possible.

          Kevin Phillips added a comment - I decided to just take some time right away to do some further tests and I can't seem to reproduce the problem on other view types. I have tried the following: nested views list views categorized jobs view build pipeline view my view status view dashboard view Also, for some of our views of type "Sectioned View" with a smaller number of sub-sections on them, the error doesn't appear to occur. The combination of a view of type "Sectioned View" with a large number of sub-sections of type "List View" seems to be the problematic case. Also, our build farm has a large number of jobs on it (>1200 atm). If the Sectioned View were submitting information about each of these jobs for each of the list sub-views that could explain why the error only happens in this one use case. Either way, the "hack" mentioned above for overriding the maxFormContentSize property for Jetty seems to prevent the problem, at least for us. This at least unblocks us for the time being. Still, it would be good to try and get a fix for this if possible.

          Daniel Beck added a comment -

          In my latest case the view was a "Sectioned View" type, with maybe 10 sub-sections of type "List View". Each list view subsection has a regular expression for the job filter, and there are few if any jobs that are checked off individually.

          Please file an issue against Sectioned View Plugin to investigate whether the form size can be reduced there.

          the "hack" mentioned above for overriding the maxFormContentSize property for Jetty seems to prevent the problem

          Not really a hack, just a legitimate option. It's reasonable to limit the size of submitted forms to a sensible value by default, but larger Jenkins instances may easily grow past that, especially given how Jenkins performs form submissions.

          Daniel Beck added a comment - In my latest case the view was a "Sectioned View" type, with maybe 10 sub-sections of type "List View". Each list view subsection has a regular expression for the job filter, and there are few if any jobs that are checked off individually. Please file an issue against Sectioned View Plugin to investigate whether the form size can be reduced there. the "hack" mentioned above for overriding the maxFormContentSize property for Jetty seems to prevent the problem Not really a hack, just a legitimate option. It's reasonable to limit the size of submitted forms to a sensible value by default, but larger Jenkins instances may easily grow past that, especially given how Jenkins performs form submissions.

          Please file an issue against Sectioned View Plugin to investigate whether the form size can be reduced there.

          Done: JENKINS-28716

          It's reasonable to limit the size of submitted forms to a sensible value by default...

          Agreed. The only debatable point is what value is 'sensible' in this context. In our context / use case it is obviously not sensible.

          Not really a hack, just a legitimate option.

          I'm not sure I agree with your assessment. As a user of the Jenkins product I expect the tool to work right out of the box with minimal manual modification / tweaking. Unfortunately, in this case, this was not possible. To make matters worse the bug itself was not obvious at first glance, and the root cause and workaround to the problem were also not directly obvious from the error produced. In fact, the only reason I knew of the existence of this maxFormContentSize option was by searching through bug reports until I came across this one which made mention of it. Hardly a simple process.

          Conversely, I suspect that if you were to select a sufficiently large form size limit such that large scale users such as us wouldn't encounter this problem in the first place it would have little to no effect on the other users of the tool. That would be a preferable solution than having your users manually modify configuration files to make the tool work. Hence my assessment that this is in fact a 'hack'.

          Kevin Phillips added a comment - Please file an issue against Sectioned View Plugin to investigate whether the form size can be reduced there. Done: JENKINS-28716 It's reasonable to limit the size of submitted forms to a sensible value by default... Agreed. The only debatable point is what value is 'sensible' in this context. In our context / use case it is obviously not sensible. Not really a hack, just a legitimate option. I'm not sure I agree with your assessment. As a user of the Jenkins product I expect the tool to work right out of the box with minimal manual modification / tweaking. Unfortunately, in this case, this was not possible. To make matters worse the bug itself was not obvious at first glance, and the root cause and workaround to the problem were also not directly obvious from the error produced. In fact, the only reason I knew of the existence of this maxFormContentSize option was by searching through bug reports until I came across this one which made mention of it. Hardly a simple process. Conversely, I suspect that if you were to select a sufficiently large form size limit such that large scale users such as us wouldn't encounter this problem in the first place it would have little to no effect on the other users of the tool. That would be a preferable solution than having your users manually modify configuration files to make the tool work. Hence my assessment that this is in fact a 'hack'.

          PS: Sorry for the rant. I have been spending days / weeks testing the latest LTS edition of Jenkins in preparation for an upgrade to our build farm and I've been dealing with bug after bug, which are supposed to be 'resolved' by hack after hack and it has just been wearing on me. This experience is unfortunately reducing my confidence in the tool as a whole, and in the LTS maintenance stream in particular.

          We chose to use the LTS edition instead of the head / master version to avoid problems just like this.

          Kevin Phillips added a comment - PS: Sorry for the rant. I have been spending days / weeks testing the latest LTS edition of Jenkins in preparation for an upgrade to our build farm and I've been dealing with bug after bug, which are supposed to be 'resolved' by hack after hack and it has just been wearing on me. This experience is unfortunately reducing my confidence in the tool as a whole, and in the LTS maintenance stream in particular. We chose to use the LTS edition instead of the head / master version to avoid problems just like this.

          Daniel Beck added a comment -

          In some sense, it's similar to configuring Xmx/XX:MaxPermSize in that it really depends on the size of the Jenkins instance what a reasonable value is, and it's up to you to increase it if your instance grows large. It's different as it should be easier to default to a large value without negative effects, but it won't really solve the problem – large requests will still happen depending on plugin.

          Daniel Beck added a comment - In some sense, it's similar to configuring Xmx/XX:MaxPermSize in that it really depends on the size of the Jenkins instance what a reasonable value is, and it's up to you to increase it if your instance grows large. It's different as it should be easier to default to a large value without negative effects, but it won't really solve the problem – large requests will still happen depending on plugin.

          It's different as it should be easier to default to a large value without negative effects..

          Yeah - that's what I suspected.

          ...large requests will still happen depending on plugin.

          Yeah - this is I guess one of the unfortunate down-sides to the modular / plugin design of Jenkins. When something like this form size limit changes it takes time for that news to propagate out to all of the different maintainers of the different plugins, and even longer for them all to fix their plugins to compensate for the change.

          Kevin Phillips added a comment - It's different as it should be easier to default to a large value without negative effects.. Yeah - that's what I suspected. ...large requests will still happen depending on plugin. Yeah - this is I guess one of the unfortunate down-sides to the modular / plugin design of Jenkins. When something like this form size limit changes it takes time for that news to propagate out to all of the different maintainers of the different plugins, and even longer for them all to fix their plugins to compensate for the change.

          This is not a valid LTS candidate - no code changes to backport.

          Oliver Gondža added a comment - This is not a valid LTS candidate - no code changes to backport.

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/java/winstone/Launcher.java
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/9ae4206d85863b1f478bf21aa1a80a9111c66722
          Log:
          [FIXED JENKINS-20327][FIXED JENKINS-23221][FIXED JENKINS-24804][FIXED JENKINS-21064] java.lang.IllegalStateException: Form too large

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/java/winstone/Launcher.java src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/9ae4206d85863b1f478bf21aa1a80a9111c66722 Log: [FIXED JENKINS-20327] [FIXED JENKINS-23221] [FIXED JENKINS-24804] [FIXED JENKINS-21064] java.lang.IllegalStateException: Form too large

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/java/winstone/Launcher.java
          src/java/winstone/cmdline/Option.java
          http://jenkins-ci.org/commit/winstone/4a97150de06b6288e021cc191ae4189e4a21b3a3
          Log:
          Merge pull request #20 from stephenc/master

          [FIXED JENKINS-20327][FIXED JENKINS-23221][FIXED JENKINS-24804][FIXED JE...

          Compare: https://github.com/jenkinsci/winstone/compare/f42497acd05d...4a97150de06b

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/java/winstone/Launcher.java src/java/winstone/cmdline/Option.java http://jenkins-ci.org/commit/winstone/4a97150de06b6288e021cc191ae4189e4a21b3a3 Log: Merge pull request #20 from stephenc/master [FIXED JENKINS-20327] [FIXED JENKINS-23221] [FIXED JENKINS-24804] [FIXED JE... Compare: https://github.com/jenkinsci/winstone/compare/f42497acd05d...4a97150de06b

          Jesse Glick added a comment -

          I have marked a bunch of duplicates, but left them open, since it is typically still a crappy design in plugins (or core, as in this original issue) that they encourage forms with gigantic data. Jetty is now configured to meekly accept the whole submission, but your performance will still suffer.

          Jesse Glick added a comment - I have marked a bunch of duplicates, but left them open, since it is typically still a crappy design in plugins (or core, as in this original issue) that they encourage forms with gigantic data. Jetty is now configured to meekly accept the whole submission, but your performance will still suffer.

          mdonohue added a comment -

          What version of Jenkins picked up the jetty change to accept the whole submission? I see Stephen Connolly's patch was applied to winstone, but there hasn't been a deploy of winstone with that patch based on the tags I see, and core/war/pom.xml points at winstone-2.8, which doesn't have the patch.

          mdonohue added a comment - What version of Jenkins picked up the jetty change to accept the whole submission? I see Stephen Connolly's patch was applied to winstone, but there hasn't been a deploy of winstone with that patch based on the tags I see, and core/war/pom.xml points at winstone-2.8, which doesn't have the patch.

          Jesse Glick added a comment -

          Indeed.

          Jesse Glick added a comment - Indeed.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/8f012d63d771b22d74eec8396b2895f0e45f0e6f
          Log:
          [FIXED JENKINS-20327] Integrate Winstone 2.9.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/8f012d63d771b22d74eec8396b2895f0e45f0e6f Log: [FIXED JENKINS-20327] Integrate Winstone 2.9.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/d76ee3e5179dd50e96e6c8e1d67ffbdec7f88a7b
          Log:
          JENKINS-20327 Noting merge of #1922.

          Compare: https://github.com/jenkinsci/jenkins/compare/ab0198246c1c...d76ee3e5179d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html war/pom.xml http://jenkins-ci.org/commit/jenkins/d76ee3e5179dd50e96e6c8e1d67ffbdec7f88a7b Log: JENKINS-20327 Noting merge of #1922. Compare: https://github.com/jenkinsci/jenkins/compare/ab0198246c1c...d76ee3e5179d

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4373
          [FIXED JENKINS-20327] Integrate Winstone 2.9. (Revision 8f012d63d771b22d74eec8396b2895f0e45f0e6f)

          Result = SUCCESS
          jesse glick : 8f012d63d771b22d74eec8396b2895f0e45f0e6f
          Files :

          • war/pom.xml

          dogfood added a comment - Integrated in jenkins_main_trunk #4373 [FIXED JENKINS-20327] Integrate Winstone 2.9. (Revision 8f012d63d771b22d74eec8396b2895f0e45f0e6f) Result = SUCCESS jesse glick : 8f012d63d771b22d74eec8396b2895f0e45f0e6f Files : war/pom.xml

          Code changed in jenkins
          User: Jesse Glick
          Path:
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/1a91527767f6f4813835335506100848007623d6
          Log:
          [FIXED JENKINS-20327] Integrate Winstone 2.9.

          (cherry picked from commit 8f012d63d771b22d74eec8396b2895f0e45f0e6f)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/1a91527767f6f4813835335506100848007623d6 Log: [FIXED JENKINS-20327] Integrate Winstone 2.9. (cherry picked from commit 8f012d63d771b22d74eec8396b2895f0e45f0e6f)

          Code changed in jenkins
          User: Jesse Glick
          Path:
          http://jenkins-ci.org/commit/jenkins/e3ccde03907f3464dec56f5dab18e849ce508548
          Log:
          [FIXED JENKINS-20327] Integrate Winstone 2.9.

          (cherry picked from commit 8f012d63d771b22d74eec8396b2895f0e45f0e6f)

          Compare: https://github.com/jenkinsci/jenkins/compare/f5f777f680b7^...e3ccde03907f

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: http://jenkins-ci.org/commit/jenkins/e3ccde03907f3464dec56f5dab18e849ce508548 Log: [FIXED JENKINS-20327] Integrate Winstone 2.9. (cherry picked from commit 8f012d63d771b22d74eec8396b2895f0e45f0e6f) Compare: https://github.com/jenkinsci/jenkins/compare/f5f777f680b7 ^...e3ccde03907f

          dogfood added a comment -

          Integrated in jenkins_2.0 #5
          [FIXED JENKINS-20327] Integrate Winstone 2.9. (Revision 8f012d63d771b22d74eec8396b2895f0e45f0e6f)

          Result = SUCCESS
          jesse glick : 8f012d63d771b22d74eec8396b2895f0e45f0e6f
          Files :

          • war/pom.xml

          dogfood added a comment - Integrated in jenkins_2.0 #5 [FIXED JENKINS-20327] Integrate Winstone 2.9. (Revision 8f012d63d771b22d74eec8396b2895f0e45f0e6f) Result = SUCCESS jesse glick : 8f012d63d771b22d74eec8396b2895f0e45f0e6f Files : war/pom.xml

          Not a candidate for 1.642 based LTS as it was fixed in 1.639.

          Oliver Gondža added a comment - Not a candidate for 1.642 based LTS as it was fixed in 1.639.

            stephenconnolly Stephen Connolly
            baue_ang Angela Bauer
            Votes:
            11 Vote for this issue
            Watchers:
            35 Start watching this issue

              Created:
              Updated:
              Resolved: