Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: google-compute-engine-plugin
Labels:
None

Similar Issues:

Show

Last Thursday (3rd July, 2025) we've observed that new agents in region us-west2 are not connecting to Jenkins.
New instances were spawned, but not used as Jenkins agents.
This problem was present only in us-west2 region, switching to a different zone/machine type helped, but only temporarily.

Plugin returned the following error:

2025-07-03 01:44:48.950+0000 [id=75]    WARNING c.g.j.p.c.ComputeEngineCloud#provision: Error provisioning node
java.net.SocketTimeoutException: Read timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedRead(Unknown Source)
        at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source)
        at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source)
        at java.base/java.net.Socket$SocketInputStream.read(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketInputRecord.read(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
        at java.base/java.io.BufferedInputStream.fill(Unknown Source)
        at java.base/java.io.BufferedInputStream.read1(Unknown Source)
        at java.base/java.io.BufferedInputStream.implRead(Unknown Source)
        at java.base/java.io.BufferedInputStream.read(Unknown Source)
        at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
        at java.base/sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:525)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:466)
        at PluginClassLoader for google-oauth-plugin//com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:576)
        at PluginClassLoader for google-compute-engine//com.google.cloud.graphite.platforms.plugin.client.ComputeWrapper.insertInstanceWithTemplate(ComputeWrapper.java:112)
        at PluginClassLoader for google-compute-engine//com.google.cloud.graphite.platforms.plugin.client.ComputeClient.insertInstance(ComputeClient.java:303)
        at PluginClassLoader for google-compute-engine//com.google.jenkins.plugins.computeengine.InstanceConfiguration.provision(InstanceConfiguration.java:334)
        at PluginClassLoader for google-compute-engine//com.google.jenkins.plugins.computeengine.ComputeEngineCloud.provision(ComputeEngineCloud.java:276)
        at PluginClassLoader for google-compute-engine//com.google.jenkins.plugins.computeengine.NoDelayProvisionerStrategy.apply(NoDelayProvisionerStrategy.java:78)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
        at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

We've determined that the API call response from Google Compute API is taking long enough for the plugin to time out. The VMs were still getting created, but since the process was interrupted by a timeout, they were abandoned and never used as Jenkins agents. This also led to the capacity of the cloud being reported in negative numbers:

2025-07-03 09:21:03.204+0000 [id=86]    INFO    c.g.j.p.c.ComputeEngineCloud#availableNodeCapacity: Found capacity for -54 nodes in cloud somecloud

Creating VMs manually with gcloud CLI tool was indeed taking longer than usual in this region, however, not resulting in any errors.

We believe the timeout for this API call response should be increased or configurable.

The launch Timeout for the agent is already much higher (300 seconds) than the required time for spawning the VM through gcloud (about 40 seconds).

Details

Description

Attachments

Activity

People

Dates