Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61409

Websocket connections crash if message size is greater than 64Kb

    • Jenkins 2.229

      Websocket connections fail if the response from a pipeline step is greater than 64kb. 

      Steps to recreate the problem:

      1. Run a jenkins server using the docker container jenkins/jenkins:centos
      2. Create a New Node with webSocket enabled
      3. On another Linux VM/slave machine, run the agent.jar with -webSocket option to connect to the Jenkins Master
      4. Create a pipeline with a step that would output text >64kb. For example, create a text file with size > 64Kb. 'cat' the file as a pipeline step. 
      pipeline {
        agent {
          label 'test-channelclose'
        }  
        stages {
          stage ('Debug Issue') {
            steps {
              sh 'cat build_output.log'
              //archiveArtifacts 'build_output.log'
            }
          }
        }
      }
      

      5. Running the pipeline will break with the following error.

      Mar 07, 2020 2:31:28 PM jenkins.agents.WebSocketAgents$Session error
      WARNING: null
      org.eclipse.jetty.websocket.api.MessageTooLargeException: Binary message size [524494] exceeds maximum size [65536]
              at org.eclipse.jetty.websocket.api.WebSocketPolicy.assertValidBinaryMessageSize(WebSocketPolicy.java:128)
              at org.eclipse.jetty.websocket.common.Parser.assertSanePayloadLength(Parser.java:133)
              at org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:494)
              at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:253)
              at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:460)
              at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:441)
              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
              at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
              at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
              at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
              at java.lang.Thread.run(Thread.java:748)

      Note: One of the workaround is to write the output to a file and archive the file. However, not all pipeline steps support an easy way to write the output to file.

          [JENKINS-61409] Websocket connections crash if message size is greater than 64Kb

          Karthik Jayaraman created issue -

          Oleg Nenashev added a comment -

          CC jglick as a JEP-222 sponsor

          Oleg Nenashev added a comment - CC jglick as a JEP-222 sponsor
          Oleg Nenashev made changes -
          Assignee Original: Jeff Thompson [ jthompson ] New: Jesse Glick [ jglick ]

          Jesse Glick added a comment -

          Possibly also affects WebSocket-based CLI.

          Jesse Glick added a comment - Possibly also affects WebSocket-based CLI.
          Jesse Glick made changes -
          Remote Link New: This issue links to "CloudBees-internal issue (Web Link)" [ 24715 ]

          Jesse Glick added a comment -

          There seems to be a configurable maximum size for binary packets. (WebSocketPolicy.setMaxBinaryMessageSize) There is no particular reason to impose a limit in Jenkins that I know of, so it could just be set to some large value.

          Or Remoting may already have a maximum frame size; would have to dig into it. In this case the packet is coming via a RemoteOutputStream, I think, though the behavior may vary depending on whether or not org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING is set.

          Jesse Glick added a comment - There seems to be a configurable maximum size for binary packets. ( WebSocketPolicy.setMaxBinaryMessageSize ) There is no particular reason to impose a limit in Jenkins that I know of, so it could just be set to some large value. Or Remoting may already have a maximum frame size; would have to dig into it. In this case the packet is coming via a RemoteOutputStream , I think, though the behavior may vary depending on whether or not org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.USE_WATCHING is set.
          Daniel Beck made changes -
          Labels Original: websocket New: lts-candidate websocket

          Observation after enabling FINEST logging on the remoting component:

          The buffer size @ hudson.remoting.Engine#writeBlock() does not increase above 8192b. It seems that the remoting component is breaking it into multiple packets. However, the server errors out and closes the connection if there is more than 64kb of data received for a step.

          Also, note that archiving large files (>30MB) on the Jenkins master does not have an impact.

          Karthik Jayaraman added a comment - Observation after enabling FINEST logging on the remoting component: The buffer size @ hudson.remoting.Engine#writeBlock() does not increase above 8192b. It seems that the remoting component is breaking it into multiple packets. However, the server errors out and closes the connection if there is more than 64kb of data received for a step. Also, note that archiving large files (>30MB) on the Jenkins master does not have an impact.

          https://www.eclipse.org/jetty/javadoc/current/org/eclipse/jetty/websocket/api/WebSocketPolicy.html#getMaxBinaryMessageSize()

          Get the maximum size of a binary message during parsing.

          This is a memory conservation option, memory over this limit will not be allocated by Jetty for handling binary messages. This applies to individual frames, whole message handling, and partial message handling.

          Binary messages over this maximum will result in a close code 1009 StatusCode.MESSAGE_TOO_LARGE

          Using too big values could cause memory issues.

          Vincent Latombe added a comment - https://www.eclipse.org/jetty/javadoc/current/org/eclipse/jetty/websocket/api/WebSocketPolicy.html#getMaxBinaryMessageSize( ) Get the maximum size of a binary message during parsing. This is a memory conservation option, memory over this limit will not be allocated by Jetty for handling binary messages. This applies to individual frames, whole message handling, and partial message handling. Binary messages over this maximum will result in a close code 1009 StatusCode.MESSAGE_TOO_LARGE Using too big values could cause memory issues.

          Using a large value can potentially cause memory issues on the server. Note that the slave connection is crashing anyway.

          I think the size should be configurable so that we can change it based on the environment that we run our Jenkins servers on.

          Karthik Jayaraman added a comment - Using a large value can potentially cause memory issues on the server. Note that the slave connection is crashing anyway. I think the size should be configurable so that we can change it based on the environment that we run our Jenkins servers on.

            vlatombe Vincent Latombe
            kjayaraman Karthik Jayaraman
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: