Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-75942

Orphan node cleanup terminates instances for another Jenkins

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • ec2-plugin
    • None
    • Jenkins 2.516.1, EC2 Plugin 1982.vd5c4f4008e4c
    • 2007.v9053121e9e07

      The EC2 plugin recently added an "orphan node clean-up" option (https://github.com/jenkinsci/ec2-plugin/pull/1115).

      We run several different Jenkins within the same AWS account. On one of those, the Clean Up Orphan Nodes option had been enabled.
      We observed a case where one Jenkins Controller terminated an EC2 instance for an Agent that belonged to a different Jenkins. We realised that something had gone wrong, because there was a job active on the agent at the time, and so it was killed.

      This was identified from AWS CloudTrail, where we saw the Controller call TerminateInstances for the Agent.

      The "orphan node clean-up" should check the value of the Tag jenkins_server_url on the instances it is terminating, to make sure that it only terminates its own instances, but that doesn't appear to be the case.
      Looking at the code, it appears that it checks for the presence of that tag, but not its value (disclaimer: I am not a Java programmer).

      ===

      There's one other related issue with the way that the "orphan node clean-up" identifies orphan instances. The EC2 plugin allows you to define multiple clouds (in the same Jenkins Controller). The Clean Up Orphan Nodes option can be selected on a cloud-by-cloud basis (i.e one cloud could enable the option, another one not). However, there's no way I am aware of for the Controller to know which Cloud an Agent belongs to. It looks like if you enable that option in one cloud definition, it will apply to all the clouds.

       

            nsunny Nevin
            mwebber Matthew Webber
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: