[JENKINS-37632] Bitbucket Pull Request plugin has no socket timeout and hangs indefinitely

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: bitbucket-pullrequest-builder-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

On average once per day, our Jenkins server stops polling all SCMs and pull requests because the BitBucket Pull Request Builder Plugin hangs indefinitely on a socket connection. See the below excerpt for the stack trace of the hanging Timer thread. The only way to recover is to restart the Jenkins server. All Jenkins cron timers are blocked on this thread, so all other jobs are impacted when this happens, and we cannot even use Jenkins to schedule restarts for recovery.

We would love to see a socket timeout configuration added to ApiClient so that it would gracefully recover from the hanging socket read connection.

Stack trace:

"jenkins.util.Timer [#7]" Id=41 Group=main RUNNABLE (in native)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	-  locked java.lang.Object@7bc6ab76
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	-  locked sun.security.ssl.AppInputStream@71767bcb
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	-  locked java.io.BufferedInputStream@77029758
	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send(ApiClient.java:239)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.get(ApiClient.java:213)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.hasBuildStatus(ApiClient.java:139)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.isBuildTarget(BitbucketRepository.java:224)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.getTargetPullRequests(BitbucketRepository.java:94)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketPullRequestsBuilder.run(BitbucketPullRequestsBuilder.java:37)
	at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketBuildTrigger.run(BitbucketBuildTrigger.java:187)
	at hudson.triggers.Trigger.checkTriggers(Trigger.java:274)
	at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@2619b6f0

Jeremy P created issue - 2016-08-23 17:10

Jeremy P made changes - 2016-08-23 17:13

Description

Original: On average once per day, our Jenkins server stops polling all SCMs and pull requests because the BitBucket Pull Request Builder Plugin hangs indefinitely on a socket connection. See the below excerpt for the stack trace of the hanging Timer thread. The only way to recover is to restart the Jenkins server. All Jenkins cron timers are blocked on this thread, so all other jobs are impacted when this happens, and we cannot even use Jenkins to schedule restarts for recovery.

We would love to see a socket timeout configuration added to ApiClient so that it would gracefully recover from the hanging socket read connection.

Stack trace:
"jenkins.util.Timer [#7]" Id=41 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked java.lang.Object@7bc6ab76
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked sun.security.ssl.AppInputStream@71767bcb
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked java.io.BufferedInputStream@77029758
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send(ApiClient.java:239)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.get(ApiClient.java:213)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.hasBuildStatus(ApiClient.java:139)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.isBuildTarget(BitbucketRepository.java:224)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.getTargetPullRequests(BitbucketRepository.java:94)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketPullRequestsBuilder.run(BitbucketPullRequestsBuilder.java:37)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketBuildTrigger.run(BitbucketBuildTrigger.java:187)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:274)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2619b6f0

New: On average once per day, our Jenkins server stops polling all SCMs and pull requests because the BitBucket Pull Request Builder Plugin hangs indefinitely on a socket connection. See the below excerpt for the stack trace of the hanging Timer thread. The only way to recover is to restart the Jenkins server. All Jenkins cron timers are blocked on this thread, so all other jobs are impacted when this happens, and we cannot even use Jenkins to schedule restarts for recovery.

We would love to see a socket timeout configuration added to ApiClient so that it would gracefully recover from the hanging socket read connection.

Stack trace:

{code:java}
"jenkins.util.Timer [#7]" Id=41 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked java.lang.Object@7bc6ab76
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
- locked sun.security.ssl.AppInputStream@71767bcb
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked java.io.BufferedInputStream@77029758
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.send(ApiClient.java:239)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.get(ApiClient.java:213)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.bitbucket.ApiClient.hasBuildStatus(ApiClient.java:139)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.isBuildTarget(BitbucketRepository.java:224)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketRepository.getTargetPullRequests(BitbucketRepository.java:94)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketPullRequestsBuilder.run(BitbucketPullRequestsBuilder.java:37)
at bitbucketpullrequestbuilder.bitbucketpullrequestbuilder.BitbucketBuildTrigger.run(BitbucketBuildTrigger.java:187)
at hudson.triggers.Trigger.checkTriggers(Trigger.java:274)
at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@2619b6f0
{code}

Jeremy P made changes - 2018-12-21 18:33

Assignee

New: David Frascone [ codemonky ]

Jenkins

Details

Description

Attachments

Activity

People

Dates