The logstash plugin does not set a socket timeout on its http connections which means it will not detect dead TCP connections.   We send some large test result data (1-1.5meg) up to our logstash server via the logstash plugin.   If there is some issue with the networking between the Jenkins controller and the logstash server, these threads will hang around slowly consuming heap memory as the thread contains the large json test result data.

After several days or a couple of weeks, the Jenkins controller runs out of heap and stops responding.

Here is the thread for the http post to the logstash server:

java.lang.Thread.State: RUNNABLE
at sun.nio.ch.SocketDispatcher.read0(java.base@21.0.6/Native Method)
at sun.nio.ch.SocketDispatcher.read(java.base@21.0.6/Unknown Source)
at sun.nio.ch.NioSocketImpl.tryRead(java.base@21.0.6/Unknown Source)
at sun.nio.ch.NioSocketImpl.implRead(java.base@21.0.6/Unknown Source)
at sun.nio.ch.NioSocketImpl.read(java.base@21.0.6/Unknown Source)
at sun.nio.ch.NioSocketImpl$1.read(java.base@21.0.6/Unknown Source)
at java.net.Socket$SocketInputStream.read(java.base@21.0.6/Unknown Source)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at jenkins.plugins.logstash.persistence.ElasticSearchDao.push(ElasticSearchDao.java:215)
at jenkins.plugins.logstash.LogstashWriter.write(LogstashWriter.java:184)
at jenkins.plugins.logstash.LogstashWriter.writeBuildLog(LogstashWriter.java:149)
at jenkins.plugins.logstash.LogstashNotifier.perform(LogstashNotifier.java:104)
at jenkins.plugins.logstash.LogstashNotifier.perform(LogstashNotifier.java:83)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
at hudson.model.Build$BuildExecution.post2(Build.java:179)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
at hudson.model.Run.execute(Run.java:1856)
at hudson.matrix.MatrixRun.run(MatrixRun.java:153)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:446)

It is the push method here:

https://github.com/jenkinsci/logstash-plugin/blob/0aff8fefc12bc616a2555e80f01f036703a1312e/src/main/java/jenkins/plugins/logstash/persistence/ElasticSearchDao.java#L215

Loading Eclipse MAT and inspecting the attributes of the request, socketTimeout is not set so it will never detect a dead TCP connection and hang forever.   Maybe the other timeouts should have some sane defaults too.