-
Bug
-
Resolution: Fixed
-
Minor
-
CloudBees Jenkins Enterprise 2.138.2.2-rolling
Splunk Plugin for Jenkins 1.7.1
Filing on behalf of john.moss@viasat.com:
We are using the Splunk Plugin for Jenkins 1.7.1 (latest).
I am seeing some odd behavior that I would like to get some further feedback on please.
In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.
I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:
Here is the NGINX log of interest:
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
Here is the netstat output as well:
Before:
Mon Apr 29 16:46:47 MDT 2019 [root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088 tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java
After:
date;netstat -natp|grep java|grep :8088 Mon Apr 29 16:52:33 MDT 2019 tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.
