Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29964

EC2 plugin floods AWS with API calls if it references a deleted AMI

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • ec2-plugin
    • Jenkins 1.609.1
      Amazon EC2 plugin 1.28

      If an AMI referenced in a cloud is deleted but a reference to it is retained in the cloud config, following behavior is observed

      1. A build requesting the orphaned cloud's label will wait in the queue
      2. Jenkins will flood the associated amazon account with a sufficient number of API calls to trigger rate limiting on the account, effectively taking it out of service
      3. The frequency of API calls never backs off.

      This is a very dangerous bug, and can take down your jenkins server and every server on the same AWS account that requires activity on the AWS API. Some sort of exponential backoff would be a far better behavior.

          [JENKINS-29964] EC2 plugin floods AWS with API calls if it references a deleted AMI

          Luis Pollo added a comment - - edited

          If it helps anyone that lands here, I wanted to report that I ran across a very similar issue with the plugin, however upon further investigation determined that, in my case, all AMIs referenced in cloud profiles were still valid, but the plugin seems to be recurrently and indefinitely checking for EC2 instances that had either been stopped or deleted, and that had previously been configured as nodes in Jenkins.

          Luis Pollo added a comment - - edited If it helps anyone that lands here, I wanted to report that I ran across a very similar issue with the plugin, however upon further investigation determined that, in my case, all AMIs referenced in cloud profiles were still valid, but the plugin seems to be recurrently and indefinitely checking for EC2 instances that had either been stopped or deleted, and that had previously been configured as nodes in Jenkins.

          Scott Marlow added a comment -

          I heard feedback on https://github.com/jenkinsci/openstack-cloud-plugin/pull/365 that adding exponential backoff support to another part of Jenkins was not correct and learned that a deeper approach would be better.  I thought I would echo that here in case it helps those still working on this issue. 

          Also, https://www.jenkins.io/projects/gsoc/2023/project-ideas/agent_reconnections_exponential_backoff/ may also be related in that I think that seeks to unify changing how retry occurs in Jenkins.  At least that is what I think the intention is.  Note that they also identify using jitter to reduce spikes in CPU usage that otherwise might occur when several clients try to connect at the same time.

          Scott Marlow added a comment - I heard feedback on https://github.com/jenkinsci/openstack-cloud-plugin/pull/365 that adding exponential backoff support to another part of Jenkins was not correct and learned that a deeper approach would be better.  I thought I would echo that here in case it helps those still working on this issue.  Also, https://www.jenkins.io/projects/gsoc/2023/project-ideas/agent_reconnections_exponential_backoff/ may also be related in that I think that seeks to unify changing how retry occurs in Jenkins.  At least that is what I think the intention is.  Note that they also identify using jitter to reduce spikes in CPU usage that otherwise might occur when several clients try to connect at the same time.

            francisu Francis Upton
            csanders Carter Sanders
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: