Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45914

Stash-pull-request-plugin-builder causing deadlock on SSLSocketImpl.readRecord

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Labels:
      None
    • Environment:
      Jenkins 2.66, stash-pullrequest-builder-plugin 1.7.1, openjdk8u131
    • Similar Issues:

      Description

      We are noticing that our Jenkins jvm hangs abruptly and it was impossible to get any thread dumps also. I just got lucky to get the threaddumps today and was able to identify stash-pullrequest-builder-plugin as the root cause.

      Our jenkins is heavily used by development teams and having this downtime disrupts our Pipeline Continuous integration.

      Here is the thread dump:

      "Thread-5682" #13574 daemon prio=5 os_prio=0 tid=0x00007fb88c0e5800 nid=0x263b runnable [0x00007fb834c7f000]
           java.lang.Thread.State: RUNNABLE
           at java.net.SocketInputStream.socketRead0(Native Method)
           at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
           at java.net.SocketInputStream.read(SocketInputStream.java:171)
           at java.net.SocketInputStream.read(SocketInputStream.java:141)
           at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
           at sun.security.ssl.InputRecord.read(InputRecord.java:503)
           at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
           - locked < 0x000000068be08ef0> (a java.lang.Object)
           at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
           at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
           - locked < 0x000000068be097f8> (a sun.security.ssl.AppInputStream)
           at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
           at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
           - locked < 0x000000068be09810> (a java.io.BufferedInputStream)
           at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
           at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
           at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
           at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
           at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
           at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
           at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
           at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
           at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
           at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
           at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient$1.call(StashApiClient.java:230)
           at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient$1.call(StashApiClient.java:221)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.lang.Thread.run(Thread.java:748)
      
      
      
      
      
      "jenkins.util.Timer [#6]" #65 daemon prio=5 os_prio=0 tid=0x00007fb850082000 nid=0x266f waiting for monitor entry [0x00007fb8417fb000]
           java.lang.Thread.State: BLOCKED (on object monitor)
           at sun.security.ssl.SSLSocketImpl.disposeCiphers(SSLSocketImpl.java:1802)
           - waiting to lock <0x000000068be08ef0> (a java.lang.Object)
           at sun.security.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1738)
           at sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:1612)
           at sun.security.ssl.AppOutputStream.close(AppOutputStream.java:146)
           at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
           at org.apache.commons.httpclient.HttpConnection.closeSocketAndStreams(HttpConnection.java:1226)
           at org.apache.commons.httpclient.HttpConnection.close(HttpConnection.java:1149)
           at org.apache.commons.httpclient.HttpMethodBase.abort(HttpMethodBase.java:1117)
           at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:258)
           at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestComments(StashApiClient.java:109)
           at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:280)
           at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75)
           at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
           at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
           at hudson.triggers.Trigger.checkTriggers(Trigger.java:278)
           at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)
           at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
           at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
           at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
           at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
           at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

          Hide
          jbochenski Jakub Bochenski added a comment -

          This issue has been automatically closed because of inactivity. Please reopen it if you think it's still valid

          Show
          jbochenski Jakub Bochenski added a comment - This issue has been automatically closed because of inactivity. Please reopen it if you think it's still valid

            People

            Assignee:
            nemccarthy nathan m
            Reporter:
            sumitkg Sumit Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: