• Jenkins 2.204.5, Jenkins 2.224, Winstone 5.4.3, Winstone 5.9

      *Jenkins LTS Notice*: Jenkins LTS 2.204.3 and 2.204.4 are also affected due to the Winstone upgrade which was introduced as a part of the JENKINS-57888 fix backporting. Please see https://groups.google.com/forum/#!topic/jenkinsci-dev/M_RtDuDXtbU for the discussion and retrospective

      In Jenkins Version 2.205, PR #4339 moved the cloud configuration from Configure System into is own configuration form on the Manage Nodes page. There is a cap to the length of this form (200000) and prevents me from adding additional docker clouds into the settings.

      java.lang.IllegalStateException: Form is larger than max length 200000
      	at org.eclipse.jetty.server.Request.extractFormParameters(Request.java:562)
      	at org.eclipse.jetty.server.Request.extractContentParameters(Request.java:519)
      	at org.eclipse.jetty.server.Request.getParameters(Request.java:430)
      Caused: org.eclipse.jetty.http.BadMessageException: 400: Unable to parse form content
      	at org.eclipse.jetty.server.Request.getParameters(Request.java:434)
      	at org.eclipse.jetty.server.Request.getParameterNames(Request.java:1077)
      	at hudson.security.csrf.CrumbFilter.extractCrumbFromRequest(CrumbFilter.java:112)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:81)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:512)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1592)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1296)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1211)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.Server.handle(Server.java:500)
      	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386)
      	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
      	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
      	at java.lang.Thread.run(Thread.java:748)
      

          [JENKINS-60409] Form Submission Length Cap

          Oleg Nenashev added a comment -

          I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1 but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions. 

          Pull request with the 2.204.x baseline update: https://github.com/jenkinsci/jenkins/pull/4545

          Oleg Nenashev added a comment - I have just released an alternate Winstone 5.4.1 release for 2.204.x LTS. This patch reverts Jetty to older versions  https://github.com/jenkinsci/winstone/releases/tag/winstone-5.4.1  but keeps other regression fixes. It should be more stable than upgrade to Winstone 5.9 with just another Jetty upgrade and a risk of new regressions.  Pull request with the 2.204.x baseline update:  https://github.com/jenkinsci/jenkins/pull/4545

          Always Fail added a comment -

          Will this fix be included in 2.224?

          Always Fail added a comment - Will this fix be included in 2.224?

          Oleg Nenashev added a comment - - edited

          I believe so. https://github.com/jenkinsci/jenkins/pull/4542 is waiting for the 24hrs merge timeout which ends in 1 hour or so. Taking the approvals, I am pretty confident that the next weekly release will include the fix. LTS is a separate story, I am waiting for responses from olivergondza and ci_jenkinsci_org about out-of-order 2.204.5 LTS 

          Oleg Nenashev added a comment - - edited I believe so.  https://github.com/jenkinsci/jenkins/pull/4542  is waiting for the 24hrs merge timeout which ends in 1 hour or so. Taking the approvals, I am pretty confident that the next weekly release will include the fix. LTS is a separate story, I am waiting for responses from olivergondza and ci_jenkinsci_org about out-of-order 2.204.5 LTS 

          Baptiste Mathus added a comment - - edited

          Curious, did someone ever test

          -Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1

          to remove any limit instead of just bumping it higher?

           

          UPDATE: just tested it. This works. I think this should be the recommendation instead of any high number. FWIW, this is what Jenkins normally does internally.

          Baptiste Mathus added a comment - - edited Curious, did someone ever test -Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1 to remove any limit instead of just bumping it higher?   UPDATE: just tested it. This works. I think this should be the recommendation instead of any high number. FWIW, this is what Jenkins normally does internally.

          Jesse Glick added a comment -

          Minimal test in context to reproduce: run Jenkins with Winstone 5.8 on a fresh user dir. Go through setup wizard, installing no plugins. Create an API token for admin. Then run

          x=1; while :; do echo trying $x; (echo description=; seq -s. $x | tr -d '[:digit:]') > /tmp/$x-dots.txt; curl -f -u admin:YOURTOKEN -d @/tmp/$x-dots.txt http://localhost:8080/submitDescription || break; x=$((x * 3)); done
          

          You should see it fail after 200000:

          trying 1
          trying 3
          trying 9
          trying 27
          trying 81
          trying 243
          trying 729
          trying 2187
          trying 6561
          trying 19683
          trying 59049
          trying 177147
          trying 531441
          curl: (22) The requested URL returned error: 500 Server Error
          

          If you now start Jenkins with -Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1, or with Winstone 5.9, it keeps on going.

          Jesse Glick added a comment - Minimal test in context to reproduce: run Jenkins with Winstone 5.8 on a fresh user dir. Go through setup wizard, installing no plugins. Create an API token for admin . Then run x=1; while :; do echo trying $x ; (echo description=; seq -s. $x | tr -d '[:digit:]' ) > /tmp/ $x -dots.txt; curl -f -u admin:YOURTOKEN -d @/tmp/ $x -dots.txt http://localhost:8080/submitDescription || break; x=$((x * 3)); done You should see it fail after 200000: trying 1 trying 3 trying 9 trying 27 trying 81 trying 243 trying 729 trying 2187 trying 6561 trying 19683 trying 59049 trying 177147 trying 531441 curl: (22) The requested URL returned error: 500 Server Error If you now start Jenkins with -Dorg.eclipse.jetty.server.Request.maxFormContentSize=-1 , or with Winstone 5.9, it keeps on going.

          Byte Enable added a comment -

          I am experiencing this issue as well.  Source file is 200K in size.  I whittled it down to 177K and still experience the error.  Is there a work-around?

          Byte Enable added a comment - I am experiencing this issue as well.  Source file is 200K in size.  I whittled it down to 177K and still experience the error.  Is there a work-around?

          Oleg Nenashev added a comment -

           

          Oleg Nenashev added a comment - Jenkins LTS 2.204.5 with fixes is out:  https://github.com/jenkinsci/jenkins/releases/tag/jenkins-2.204.5  . Official Changelogs are coming soon. ETA for Jenkins weekly is today  

          Hi,

          I have the same issue on 2.222 (non-LTS) with websocket connection payload. I know the feature is still in beta, but I guess the 2.221. should fix the websocket layer as well.

          2020-03-24 09:47:21.616+0000 [id=219730] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-219730 for ************* terminated: java.nio.channels.ClosedChannelException
          2020-03-24 09:47:27.142+0000 [id=219727] WARNING j.agents.WebSocketAgents$Session#error
          org.eclipse.jetty.websocket.api.MessageTooLargeException: Binary message size [69632] exceeds maximum size [65536]
          at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidBinaryMessageSize(WebSocketPolicy.java:128)
          at org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.appendFrame(SimpleBinaryMessage.java:57)
          at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:61)
          at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.onContinuationFrame(AbstractEventDriver.java:183)
          at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onContinuationFrame(JettyListenerEventDriver.java:255)
          at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:155)
          at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:322)
          at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
          at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
          at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:460)
          at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
          at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
          at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
          at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
          at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
          at java.lang.Thread.run(Thread.java:748)
          
          

          Will try again when the 222.1 is released

          Thanks

          Valentin Delaye added a comment - Hi, I have the same issue on 2.222 (non-LTS) with websocket connection payload. I know the feature is still in beta, but I guess the 2.221. should fix the websocket layer as well. 2020-03-24 09:47:21.616+0000 [id=219730] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-219730 for ************* terminated: java.nio.channels.ClosedChannelException 2020-03-24 09:47:27.142+0000 [id=219727] WARNING j.agents.WebSocketAgents$Session#error org.eclipse.jetty.websocket.api.MessageTooLargeException: Binary message size [69632] exceeds maximum size [65536] at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidBinaryMessageSize(WebSocketPolicy.java:128) at org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.appendFrame(SimpleBinaryMessage.java:57) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:61) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.onContinuationFrame(AbstractEventDriver.java:183) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onContinuationFrame(JettyListenerEventDriver.java:255) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:155) at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:322) at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202) at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225) at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:460) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.lang. Thread .run( Thread .java:748) Will try again when the 222.1 is released Thanks

          Daniel Beck added a comment -

          jonesbusy

          same issue

          It's not. Note how the error message is completely different. You're looking for JENKINS-61409.

          Daniel Beck added a comment - jonesbusy same issue It's not. Note how the error message is completely different. You're looking for JENKINS-61409 .

          jglick Ok thanks

          Valentin Delaye added a comment - jglick Ok thanks

            jglick Jesse Glick
            mastershihochief Joshua Hunter
            Votes:
            12 Vote for this issue
            Watchers:
            22 Start watching this issue

              Created:
              Updated:
              Resolved: