Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64018

Anka plugin blocks the whole instance in case the Anka cloud is unresponsive

    XMLWordPrintable

    Details

    • Type: Task
    • Status: In Progress (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: anka-build-plugin
    • Labels:
      None
    • Environment:
      Reproduced on:

      * Jenkins 2.204.3
      * Anka Build Plugin 2.2.0
    • Similar Issues:

      Description

      If for any reason the Anka cloud is unresponsive, the plugin blocks the Jenkins queue meaning that no job can be triggered anymore.

      In addition, the /configure page gets a really long time to load.

       

      To reproduce, point to an unresponsive Anka cloud and take a thread dump: 

      java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) - locked <0x000000073361a498> (a java.lang.Object) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) - locked <0x000000073361a9c8> (a sun.security.ssl.AppInputStream) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:549) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:494) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.getNodes(AnkaMgmtCommunicator.java:469) at com.veertu.ankaMgmtSdk.AnkaAPI.getCloudCapacity(AnkaAPI.java:165) - locked <0x0000000578fa8d48> (a java.lang.Object) at com.veertu.plugin.anka.AnkaMgmtCloud.getCloudCapacity(AnkaMgmtCloud.java:210) at com.veertu.plugin.anka.AnkaMgmtCloud.canProvision(AnkaMgmtCloud.java:456) at hudson.model.Label.getClouds(Label.java:246) at hudson.model.Label.isEmpty(Label.java:452) at jenkins.model.Jenkins.trimLabels(Jenkins.java:2162) at jenkins.model.Jenkins$4.doRun(Jenkins.java:1004) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:70) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - <0x00000003c3965518> (a java.util.concurrent.ThreadPoolExecutor$Worker)

       

       

      Other stack in the logs:

       
      2020-10-23 06:34:18.526+0000 [id=36] INFO c.v.plugin.anka.AnkaMgmtCloud#InternalLog: Got exception: com.veertu.ankaMgmtSdk.exceptions.AnkaMgmtException java.net.SocketTimeoutException: Read timed out java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:933) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:549) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.doRequest(AnkaMgmtCommunicator.java:494) at com.veertu.ankaMgmtSdk.AnkaMgmtCommunicator.getNodes(AnkaMgmtCommunicator.java:469) at com.veertu.ankaMgmtSdk.AnkaAPI.getCloudCapacity(AnkaAPI.java:165) at com.veertu.plugin.anka.AnkaMgmtCloud.getCloudCapacity(AnkaMgmtCloud.java:210) at com.veertu.plugin.anka.AnkaMgmtCloud.canProvision(AnkaMgmtCloud.java:456) at hudson.model.Label.getClouds(Label.java:246) at hudson.model.Label.isEmpty(Label.java:452) at jenkins.model.Jenkins.trimLabels(Jenkins.java:2162) at jenkins.model.Nodes$9.run(Nodes.java:352) at hudson.model.Queue._withLock(Queue.java:1392) at hudson.model.Queue.withLock(Queue.java:1269) at jenkins.model.Nodes.load(Nodes.java:346) at jenkins.model.Jenkins$13.run(Jenkins.java:3198) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1122) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
      

        

        Attachments

          Activity

          Hide
          kamalay Lokesh Kamalay added a comment -

          CloudBees Jenkins Version: 2.204.3.1

          Show
          kamalay Lokesh Kamalay added a comment - CloudBees Jenkins Version: 2.204.3.1
          Hide
          veertunathan Nathan added a comment -

          Hi Pierre Beitz and Lokesh Kamalay can you please try and reproduce with 1.17.0 of the controller and the latest plugin? We were unable to reproduce this on the latest controller and plugin version.

          Show
          veertunathan Nathan added a comment - Hi Pierre Beitz and Lokesh Kamalay  can you please try and reproduce with 1.17.0 of the controller and the latest plugin? We were unable to reproduce this on the latest controller and plugin version.
          Hide
          veertunathan Nathan added a comment -

          We were finally able to reproduce this and confirmed it's only an issue with cloudbees and not the LTS of Jenkins. We will be working on a solution soon.

          Show
          veertunathan Nathan added a comment - We were finally able to reproduce this and confirmed it's only an issue with cloudbees and not the LTS of Jenkins. We will be working on a solution soon.
          Hide
          lokeshkamalay Lokesh Kamalay added a comment -

          Cool - it is really great to know

          Show
          lokeshkamalay Lokesh Kamalay added a comment - Cool - it is really great to know

            People

            Assignee:
            veertunathan Nathan
            Reporter:
            pierrebtz Pierre Beitz
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: