-
Task
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
Reproduced on:
* Jenkins 2.204.3
* Anka Build Plugin 2.2.0
If for any reason the Anka cloud is unresponsive, the plugin blocks the Jenkins queue meaning that no job can be triggered anymore.
In addition, the /configure page gets a really long time to load.
To reproduce, point to an unresponsive Anka cloud and take a thread dump:
java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) - locked <0x000000073361a498> (a java.lang.Object) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) - locked <0x000000073361a9c8> (a sun.security.ssl.AppInputStream) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:549) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:494) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.getNodes(AnkaMgmtCommunicator.java:469) at com.veertu.ankaMgmtSdk.AnkaAPI.getCloudCapacity(AnkaAPI.java:165) - locked <0x0000000578fa8d48> (a java.lang.Object) at com.veertu.plugin.anka.AnkaMgmtCloud.getCloudCapacity(AnkaMgmtCloud.java:210) at com.veertu.plugin.anka.AnkaMgmtCloud.canProvision(AnkaMgmtCloud.java:456) at hudson.model.Label.getClouds(Label.java:246) at hudson.model.Label.isEmpty(Label.java:452) at jenkins.model.Jenkins.trimLabels(Jenkins.java:2162) at jenkins.model.Jenkins$4.doRun(Jenkins.java:1004) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:70) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - <0x00000003c3965518> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Other stack in the logs:
2020-10-23 06:34:18.526+0000 [id=36] INFO c.v.plugin.anka.AnkaMgmtCloud#InternalLog: Got exception: com.veertu.ankaMgmtSdk.exceptions.AnkaMgmtException java.net.SocketTimeoutException: Read timed out java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:549) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:494) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.getNodes(AnkaMgmtCommunicator.java:469) at com.veertu.ankaMgmtSdk.AnkaAPI.getCloudCapacity(AnkaAPI.java:165) at com.veertu.plugin.anka.AnkaMgmtCloud.getCloudCapacity(AnkaMgmtCloud.java:210) at com.veertu.plugin.anka.AnkaMgmtCloud.canProvision(AnkaMgmtCloud.java:456) at hudson.model.Label.getClouds(Label.java:246) at hudson.model.Label.isEmpty(Label.java:452) at jenkins.model.Jenkins.trimLabels(Jenkins.java:2162) at jenkins.model.Nodes$9.run(Nodes.java:352) at hudson.model.Queue._withLock(Queue.java:1392) at hudson.model.Queue.withLock(Queue.java:1269) at jenkins.model.Nodes.load(Nodes.java:346) at jenkins.model.Jenkins$13.run(Jenkins.java:3198) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1122) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
[JENKINS-64018] Anka plugin blocks the whole instance in case the Anka cloud is unresponsive
Nathan
added a comment - Hi pierrebtz and kamalay can you please try and reproduce with 1.17.0 of the controller and the latest plugin? We were unable to reproduce this on the latest controller and plugin version.
Nathan
added a comment - Hi pierrebtz and kamalay can you please try and reproduce with 1.17.0 of the controller and the latest plugin? We were unable to reproduce this on the latest controller and plugin version. We were finally able to reproduce this and confirmed it's only an issue with cloudbees and not the LTS of Jenkins. We will be working on a solution soon.
Nathan
added a comment - - edited We were finally able to reproduce this and confirmed it's only an issue with cloudbees and not the LTS of Jenkins. We will be working on a solution soon. 
Lokesh Kamalay
added a comment - Cool - it is really great to know Nathan
added a comment - We still are unable to reproduce this issue as it was defined. We were able to reproduce something that seemed like it was the original issue but it turns out it was not the case. If someone has any details about how to reproduce we would appreciate it.
Nathan
added a comment - We still are unable to reproduce this issue as it was defined. We were able to reproduce something that seemed like it was the original issue but it turns out it was not the case. If someone has any details about how to reproduce we would appreciate it. 
CloudBees Jenkins Version: 2.204.3.1