-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
ec2-plugin 2.0.5
Jenkins 2.375.2
when below exception occurs, we are unable to provision new EC2 instances:
java.lang.IllegalStateException: Connection pool shut down
we run 5 Jenkins instances in different environments, all managed by CASC plugin, this bug occurred in all of them at a certain point in time, never simultaneously.
when we try to provision a new EC2 instance, we get this error [attached screenshot]:
A problem occurred while processing the request.
in the error page this is the URI: JENKINS_URL/cloud/ec2-AWS/provision
this is a longstanding bug we are experiencing, for at least 6 months, through several LTS Jenkins core versions, and ec2-plugin and aws-java-sdk-* updates, it keeps failing without a pattern we recognized.
to resolve this temporarily, we restart our jenkins docker container, and the error does not reoccur for some time, but it's an aggressive solution.
so, we were lucky enough to find a workaround which does not mandate down time.
the workaround is to click the Test connection button under:
Manage Jenkins > Manage nodes and clouds > Configure Clouds > Amazon EC2 cloud > Test connection button > we get "Success" message > click Apply > click Save.
Then, we try to provision again, and the EC2 instance works as expected, and the error does not reoccur for some time.
Log output examples are all in UTC time.
SEVERE error example #1 [ec2-plugin 2.0.5 ; Jenkins 2.375.2]:
2023-01-17 08:09:27.249+0000 [id=26] SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.plugins.ec2.EC2Cloud$EC2ConnectionUpdater@dec8358 failed java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy68.requestConnection(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:15804) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:15772) at hudson.plugins.ec2.EC2Cloud$EC2ConnectionUpdater.doRun(EC2Cloud.java:1242) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
older SEVERE error example #2 [ec2-plugin 1.68 ; Jenkins 2.346.2]:
2022-08-14 06:23:58.199+0000 [id=62] SEVERE hudson.triggers.SafeTimerTask#run: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@44f9ca8c failed java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) at jdk.internal.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy60.requestConnection(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:33873) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:33840) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:33829) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:15437) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:15405) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:494) at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:677) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:695) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:740) at com.cloudbees.jenkins.plugins.amazonecs.ECSProvisioningStrategy.apply(ECSProvisioningStrategy.java:65) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:326) at hudson.slaves.NodeProvisioner.access$1000(NodeProvisioner.java:71) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:824) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Here's the log output [from today with ec2-plugin 2.0.5 ; Jenkins 2.375.2] we get after we click the Test connection button, and the EC2 instance starts provisioning: [Note: replaced potential sensitive data with "REDUCTED" keyword]:
2023-01-17 08:38:45.146+0000 [id=43736] SEVERE h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (EC2 alive agents monitor thread/43736) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) at jdk.internal.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy68.requestConnection(Unknown Source) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654) at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeInstances(AmazonEC2Client.java:15804) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:15772) at hudson.plugins.ec2.CloudHelper.getInstance(CloudHelper.java:54) at hudson.plugins.ec2.CloudHelper.getInstanceWithRetry(CloudHelper.java:32) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:631) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:597) at hudson.plugins.ec2.EC2SlaveMonitor.removeDeadNodes(EC2SlaveMonitor.java:52) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:43) at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:102) at java.base/java.lang.Thread.run(Thread.java:834) 2023-01-17 08:39:21.756+0000 [id=43622] INFO h.p.s.ScmSyncConfigurationBusiness#queueChangeSet: Queue of changeset A config.xml aborted (scm manipulator not settled !) 2023-01-17 08:39:23.968+0000 [id=43621] INFO h.p.s.ScmSyncConfigurationBusiness#queueChangeSet: Queue of changeset A config.xml aborted (scm manipulator not settled !) 2023-01-17 08:40:15.502+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#getImage: Getting image for request {ExecutableUsers: [],Filters: [],ImageIds: [ami-REDUCTED],Owners: [],} 2023-01-17 08:40:15.725+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description='REDUCTED', labels='REDUCTED'}. Considering launching 2023-01-17 08:40:15.728+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#setupRootDevice: AMI had /dev/REDUCTED 2023-01-17 08:40:15.729+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#setupRootDevice: {DeleteOnTermination: true,SnapshotId: snap-REDUCTED,VolumeSize: 100,VolumeType: gp2,Encrypted: false} 2023-01-17 08:40:15.729+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description='REDUCTED', labels='REDUCTED'}. EBS default encryption value set to: Based on AMI (null) 2023-01-17 08:40:15.730+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{description='REDUCTED', labels='REDUCTED'}. Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate 2023-01-17 08:40:17.493+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{REDUCTED} 2023-01-17 08:40:19.258+0000 [id=43621] INFO hudson.plugins.ec2.SlaveTemplate#logProvisionInfo: SlaveTemplate{REDUCTED} 2023-01-17 08:40:19.259+0000 [id=43621] INFO h.p.ec2.EC2RetentionStrategy#start: Start requested for EC2 (AWS) - REDUCTED (i-REDUCTED) 2023-01-17 08:40:19.260+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Launching instance: i-REDUCTED 2023-01-17 08:40:19.261+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: bootstrap() 2023-01-17 08:40:19.262+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Getting keypair... 2023-01-17 08:40:19.659+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Using private key REDUCTED (SHA-1 fingerprint REDUCTED) 2023-01-17 08:40:19.659+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Authenticating as REDUCTED 2023-01-17 08:40:19.974+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to REDUCTED on port REDUCTED, with timeout 10000. 2023-01-17 08:40:29.982+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: The kexTimeout (10000 ms) expired. 2023-01-17 08:40:29.983+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5. 2023-01-17 08:40:35.234+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to REDUCTED on port 22, with timeout 10000. 2023-01-17 08:40:45.237+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: The kexTimeout (10000 ms) expired. 2023-01-17 08:40:45.237+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5. 2023-01-17 08:40:50.433+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to REDUCTED on port REDUCTED, with timeout 10000. 2023-01-17 08:41:00.061+0000 [id=62] INFO h.p.ec2.EC2RetentionStrategy#internalCheck: Idle timeout of EC2 (AWS) - REDUCTED (i-REDUCTED) after 8140 idle minutes, instance statusRUNNING 2023-01-17 08:41:00.061+0000 [id=62] INFO h.plugins.ec2.EC2AbstractSlave#idleTimeout: EC2 instance idle time expired: i-REDUCTED 2023-01-17 08:41:00.435+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: The kexTimeout (10000 ms) expired. 2023-01-17 08:41:00.436+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5. 2023-01-17 08:41:00.574+0000 [id=43756] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Terminated EC2 instance (terminated): i-REDUCTED 2023-01-17 08:41:00.591+0000 [id=43756] INFO h.plugins.ec2.EC2OndemandSlave#lambda$terminate$0: Removed EC2 instance from jenkins controller: i-REDUCTED 2023-01-17 08:41:05.685+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to REDUCTED on port REDUCTED, with timeout 10000. 2023-01-17 08:41:05.839+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to REDUCTED:REDUCTED 2023-01-17 08:41:05.840+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5. 2023-01-17 08:41:11.033+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to REDUCTED on port REDUCTED, with timeout 10000. 2023-01-17 08:41:11.887+0000 [id=43760] INFO hudson.plugins.ec2.EC2Cloud#log: No SSH key verification (ssh-REDUCTED REDUCTED) for connections to EC2 (AWS) - REDUCTED (i-REDUCTED) 2023-01-17 08:41:11.905+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH. 2023-01-17 08:41:12.683+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: connect fresh as root 2023-01-17 08:41:12.879+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to REDUCTED on port REDUCTED, with timeout 10000. 2023-01-17 08:41:13.674+0000 [id=43761] INFO hudson.plugins.ec2.EC2Cloud#log: No SSH key verification (ssh-REDUCTED REDUCTED) for connections to EC2 (AWS) - REDUCTED (i-REDUCTED) 2023-01-17 08:41:13.682+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Connected via SSH. 2023-01-17 08:41:14.377+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Creating tmp directory (/tmp) if it does not exist 2023-01-17 08:41:15.541+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Executing init script 2023-01-17 08:46:24.047+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Creating ~/.hudson-run-init 2023-01-17 08:46:24.700+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: java -fullversion 2023-01-17 08:46:25.138+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Verifying: which scp 2023-01-17 08:46:25.647+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Copying remoting.jar to: /tmp 2023-01-17 08:46:33.154+0000 [id=43753] INFO hudson.plugins.ec2.EC2Cloud#log: Launching remoting agent (via SSH client process): ssh -o StrictHostKeyChecking=no -i /tmp/REDUCTED.pem ec2-user@REDUCTED -p REDUCTED java -jar /tmp/remoting.jar -workDir /home/ec2-user 2023-01-17 08:46:55.435+0000 [id=43753] INFO hudson.slaves.CommandLauncher#launch: agent launched for EC2 (AWS) - REDUCTED (i-REDUCTED)
Please assist, as this error occurs every couple of days in a different Jenkins controller.
In the meantime, we will be grateful if someone knows of a way to trigger the 'Test connection' programmatically, as we didn't find a way to do so, in order to automate the workaround, instead of waking up an OnCall in the middle of the night.
Thanks!
