I am opening this after spending most of the day researching this issue and combing through log files.

It's my first Jenkins issue, should I do something wrong please let me know and I will try to fix it.

We have a Jenkins instance running in Kubernetes (kOps on AWS), versions will be provided at the bottom.

We have noticed that EC2 instances, created on demand using the EC2 plugin are becoming orphaned and remain running in AWS even if they no longer exist in Jenkins.

This has caused at least in one case, the issue of preventing the EC2 plugin from creating new EC2 workers because the capping had been reached for that type of worker.

We have noticed this especially after restarts, but we suspect that the EC2 is becoming orphan earlier, since these "orphaned" EC2 are long lived despite low usage overnight and an idle time of 30m.

One older case seems to match our situation https://issues.jenkins.io/browse/JENKINS-57795 in a particularly accurate way, however it's closed and we are running versions which are much newer.

Details:

Kubernetes v1.23.13

 Jenkins 2.346.3

EC2 plugin: Amazon EC2Version1.68

Amazon Web Services SDK :: EC2Version1.12.287-357.vf82d85a_6eefd

We are using the Jenkins operator (https://github.com/jenkinsci/kubernetes-operator).

We have combed through the following logs:

• jenkins-master container inside the jenkins-jenkins Pod
• our log collection tool 
• cloudtrail

We have not been able to find anything useful.

Any help?

Thanks