-
Bug
-
Resolution: Unresolved
-
Major
-
None
On average once per day, our Jenkins server stops polling all SCMs and pull requests because the BitBucket Pull Request Builder Plugin hangs indefinitely on a socket connection. See the below excerpt for the stack trace of the hanging Timer thread. The only way to recover is to restart the Jenkins server. All Jenkins cron timers are blocked on this thread, so all other jobs are impacted when this happens, and we cannot even use Jenkins to schedule restarts for recovery.
We would love to see a socket timeout configuration added to ApiClient so that it would gracefully recover from the hanging socket read connection.
Stack trace:
"jenkins.util.Timer [#7]" Id=41 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) - locked java.lang.Object@7bc6ab76 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) - locked sun.security.ssl.AppInputStream@71767bcb at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked java.io.BufferedInputStream@77029758 at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78) at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106) at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116) at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973) at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send(ApiClient.java:239) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.get(ApiClient.java:213) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.hasBuildStatus(ApiClient.java:139) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.isBuildTarget(BitbucketRepository.java:224) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.getTargetPullRequests(BitbucketRepository.java:94) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketPullRequestsBuilder.run(BitbucketPullRequestsBuilder.java:37) at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketBuildTrigger.run(BitbucketBuildTrigger.java:187) at hudson.triggers.Trigger.checkTriggers(Trigger.java:274) at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2619b6f0
[JENKINS-37632] Bitbucket Pull Request plugin has no socket timeout and hangs indefinitely
Jeremy P
added a comment - Assigned over to you and thanks for looking into this issue. We don't run into it anymore over here for whatever reason (apparently mitigated over a core Jenkins upgrade at some point).
Jeremy P
added a comment - Assigned over to you and thanks for looking into this issue. We don't run into it anymore over here for whatever reason (apparently mitigated over a core Jenkins upgrade at some point). 
David Frascone
added a comment - So, the timeouts are set to 60000ms (60 seconds), in ApiClient.java here:
HttpClientParams params = client.getParams(); params.setConnectionManagerTimeout(DEFAULT_TIMEOUT); params.setSoTimeout(DEFAULT_TIMEOUT);
I then added debug to the `send(HttpMethodBase req)` function, and the client's parameters contain 60000 in params.getSoTimeout()
So, my question is: Is it being set correctly? And, if not in the apache commons Method Params (https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/params/HttpMethodParams.html), where should the timeout(s) be set, when using `client.executeMethod()`?
David Frascone
added a comment - So, the timeouts are set to 60000ms (60 seconds), in ApiClient.java here:
HttpClientParams params = client.getParams();
params.setConnectionManagerTimeout(DEFAULT_TIMEOUT);
params.setSoTimeout(DEFAULT_TIMEOUT);
I then added debug to the `send(HttpMethodBase req)` function, and the client's parameters contain 60000 in params.getSoTimeout()
So, my question is: Is it being set correctly? And, if not in the apache commons Method Params ( https://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/params/HttpMethodParams.html), where should the timeout(s) be set, when using `client.executeMethod()`?
David Frascone
added a comment - Relevant - this looks very similar to the plugin's implementation:
https://stackoverflow.com/questions/27968669/socket-timeout-not-working-inputstream-read-blocks
David Frascone
added a comment - Relevant - this looks very similar to the plugin's implementation:
https://stackoverflow.com/questions/27968669/socket-timeout-not-working-inputstream-read-blocks David Frascone
added a comment - David Frascone
added a comment - Shot in the dark: https://github.com/nishio-dens/bitbucket-pullrequest-builder-plugin/pull/183 David Frascone
added a comment - jpfund_sr: How did you find the hung thread and get that stack dump?
I'm going to test my fix above over time, and I'll need to be able to dump threads when our server starts working strangely. (I'm going to put that PR code up live, as a test, since I have low confidence in the fix)
Thanks in advance!
-Dave
David Frascone
added a comment - jpfund_sr : How did you find the hung thread and get that stack dump?
I'm going to test my fix above over time, and I'll need to be able to dump threads when our server starts working strangely. (I'm going to put that PR code up live, as a test, since I have low confidence in the fix)
Thanks in advance!
-Dave Jeremy P
added a comment - You can access `/threadDump` on a running Jenkins server to obtain the thread dumps. See https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump
Jeremy P
added a comment - You can access `/threadDump` on a running Jenkins server to obtain the thread dumps. See https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump 
Thank you for isolating this failure. We've been tracking it here: https://github.com/nishio-dens/bitbucket-pullrequest-builder-plugin/issues/180 but did not know exactly where the thread was hanging, since, once in state, we were down!
I'll try to get a patch testing over the weekend, and, if successful, expect a release in about a week. (Since we can't trigger the failure directly, I want to make sure we cause no harm for a week or so before pushing it to the masses.)
Feel free to assign this ticket to me, or whatever you do for bugs in plugins. I'll keep it updated, if you want, or you can follow on the github tracker.
-Dave