Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56273

Jenkins leaves nodes running in GCE if the termination failed

XMLWordPrintable

      I have seen nodes being left running in GCE after it has been removed from Jenkins because the termination failed. From the log:

      WARNING: Failed to terminate <node>
      java.net.SocketTimeoutException: connect timed out
              at java.net.PlainSocketImpl.socketConnect(Native Method)
              at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
              at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
              at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
              at java.net.Socket.connect(Socket.java:589)
              at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666)
              at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
              at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
              at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
              at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
              at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
              at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
              at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
              at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
              at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
              at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
              at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
              at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
              at com.google.jenkins.plugins.computeengine.client.ComputeClient.getInstance(ComputeClient.java:370)
              at com.google.jenkins.plugins.computeengine.client.ComputeClient.terminateInstanceWithStatus(ComputeClient.java:361)
              at com.google.jenkins.plugins.computeengine.ComputeEngineInstance._terminate(ComputeEngineInstance.java:75)
              at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67)
              at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:59)
              at hudson.slaves.CloudRetentionStrategy.check(CloudRetentionStrategy.java:43)
              at hudson.slaves.ComputerRetentionWork$1.run(ComputerRetentionWork.java:72)
              at hudson.model.Queue._withLock(Queue.java:1381)
              at hudson.model.Queue.withLock(Queue.java:1258)
              at hudson.slaves.ComputerRetentionWork.doRun(ComputerRetentionWork.java:63)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)

      If a node can't be terminated the plugin cannot terminate a node it should retry rather than just forgetting about the node.

            zombiemoose Rachel Yen
            organised_chaos James Robson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: