Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57410

Splunk Plugin Causing threadlock When Testing Connections

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • splunk-devops-plugin
    • CloudBees Jenkins Enterprise 2.138.2.2-rolling
      Splunk Plugin for Jenkins 1.7.1

      Filing on behalf of john.moss@viasat.com:

      We are using the Splunk Plugin for Jenkins 1.7.1 (latest).

      I am seeing some odd behavior that I would like to get some further feedback on please.

      In the configure system section there is a option to test the Splunk connection. If this is pressed several times it appears the the two splunkins-worker threads will become locked indefinitely until Jenkins is restarted (not sending any data to Splunk). Looking at netstat there are several sockets in CLOSE_WAIT to the Splunk HTTP input host.

      I am able to reproduce this at will within our test environment. It appears the test connection is using a call back to Jenkins for which we have NGINX in front. Once in this state additional attempts to test the connection fail due to a 504 timeout:

      Here is the NGINX log of interest:

      2019/04/29 16:01:36 [error] 9051#9051: *831865 upstream prematurely closed connection while reading response header from upstream, client: 10.68.153.30, server: wdc1jenkinst01.hq.corp.viasat.com, request: "POST /descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput HTTP/1.1", upstream: "http://127.0.0.1:8080/descriptorByName/com.splunk.splunkjenkins.SplunkJenkinsInstallation/testHttpInput", host: "jenkins.test.viasat.com", referrer: "https://jenkins.test.viasat.com/configure"
      

      Here is the netstat output as well:

      Before:
       

       Mon Apr 29 16:46:47 MDT 2019 
      [root@wdc1jenkinst01 ~]# netstat -ntap | grep java|grep :8088 
      tcp 0 0 10.68.154.134:35287 10.137.14.19:8088 ESTABLISHED 22706/java 
      tcp 0 0 10.68.154.134:35291 10.137.14.19:8088 ESTABLISHED 22706/java
      

      After:

      date;netstat -natp|grep java|grep :8088 
      Mon Apr 29 16:52:33 MDT 2019 
      tcp 102 0 10.68.154.134:11909 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36547 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:11895 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36207 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36569 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36581 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:11925 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36575 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:11885 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36529 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36531 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:11919 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36553 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:11915 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36519 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:11901 10.137.14.79:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36541 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36223 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36511 10.137.14.19:8088 CLOSE_WAIT 22706/java 
      tcp 102 0 10.68.154.134:36563 10.137.14.19:8088 CLOSE_WAIT 22706/java
      

      Attached is a thread dump taken before/after. Please let me know if anything else is needed to further investigate. I can reproduce at will and this is a test env so we can pretty much do whatever you need to get the relevant data.

        1. threads_after.out
          128 kB
          Ryan Smith
        2. threads_before.out
          120 kB
          Ryan Smith

            fengxx Ted Xiao
            rsmith Ryan Smith
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: