[JENKINS-57410] Splunk Plugin Causing threadlock When Testing Connections

Type: Bug
Resolution: Fixed
Priority: Minor
Component/s: splunk-devops-plugin
Labels:
- Splunk
Environment:
CloudBees Jenkins Enterprise 2.138.2.2-rolling
Splunk Plugin for Jenkins 1.7.1

Similar Issues:
Powered by SuggestiMate

Show

Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"

Here is the netstat output as well:

Before:

 Mon Apr 29 16:46:47 MDT 2019 
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088 
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java 
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java

After:

date;netstat -natp|grep java|grep :8088 
Mon Apr 29 16:52:33 MDT 2019 
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java 
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

threads_after.out
128 kB
2019-05-10 22:45
threads_before.out
120 kB
2019-05-10 22:45

Ryan Smith created issue - 2019-05-10 22:08

Ryan Smith made changes - 2019-05-10 22:09

Environment

Original: CloudBees Jenkins Enterprise 2.138.2.2-rollingSplunk Plugin for Jenkins 1.7.1

New: CloudBees Jenkins Enterprise 2.138.2.2-rolling
Splunk Plugin for Jenkins 1.7.1

Ryan Smith made changes - 2019-05-10 22:09

Description

Original: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code:java}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code:java}

Here is the netstat output as well:

before:
{code:java}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java{code}
{code:java}

{code:java}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code:java}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

New: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code:java}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code:java}

Here is the netstat output as well:

before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java{code}
{code}

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

Ryan Smith made changes - 2019-05-10 22:09

Description

Original: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code:java}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code:java}

Here is the netstat output as well:

before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java{code}
{code}

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

New: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code}

Here is the netstat output as well:

before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java{code}
{code}

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

Ryan Smith made changes - 2019-05-10 22:10

Description

Original: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code}

Here is the netstat output as well:

before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java{code}
{code}

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

New: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code}

Here is the netstat output as well:

before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java
{code}

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

Ryan Smith made changes - 2019-05-10 22:10

Description

Original: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code}

Here is the netstat output as well:

before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java
{code}

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

New: Filing on behalf of john.moss@viasat.com:

We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

I am seeing some odd behavior that I would like to get some further feedback on please.

In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

Here is the NGINX log of interest:

{code}
2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
{code}

Here is the netstat output as well:

Before:

{code}
Mon Apr 29 16:46:47 MDT 2019
[root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088
tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java
tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java
{code}

After:

{code}
date;netstat -natp|grep java|grep :8088
Mon Apr 29 16:52:33 MDT 2019
tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java
tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
{code}

Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

Ryan Smith made changes - 2019-05-10 22:45

Attachment		New: threads_before.out [ 47101 ]
Attachment		New: threads_after.out [ 47102 ]

Ted Xiao added a comment - 2019-05-20 16:50

fixed in 1.7.2

Ted Xiao added a comment - 2019-05-20 16:50 fixed in 1.7.2

Ted Xiao made changes - 2019-05-20 16:50

Resolution		New: Fixed [ 1 ]
Status	Original: Open [ 1 ]	New: Fixed but Unreleased [ 10203 ]

Ted Xiao made changes - 2019-05-20 16:50

Status

Original: Fixed but Unreleased [ 10203 ]

New: Resolved [ 5 ]

Assignee:: Ted Xiao

Reporter:: Ryan Smith

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2019-05-10 22:08

Updated:: 2019-05-20 16:50

Resolved:: 2019-05-20 16:50

Jenkins

Details

Description

Attachments

Attachments

Activity

Collapse comment: Ted Xiao added a comment - 2019-05-20 16:50

Expand comment: Ted Xiao added a comment - 2019-05-20 16:50

People

Dates