• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • ec2-plugin
    • None

      I am opening this after spending most of the day researching this issue and combing through log files.

      It's my first Jenkins issue, should I do something wrong please let me know and I will try to fix it.

      We have a Jenkins instance running in Kubernetes (kOps on AWS), versions will be provided at the bottom.

      We have noticed that EC2 instances, created on demand using the EC2 plugin are becoming orphaned and remain running in AWS even if they no longer exist in Jenkins.

      This has caused at least in one case, the issue of preventing the EC2 plugin from creating new EC2 workers because the capping had been reached for that type of worker.

      We have noticed this especially after restarts, but we suspect that the EC2 is becoming orphan earlier, since these "orphaned" EC2 are long lived despite low usage overnight and an idle time of 30m.

      One older case seems to match our situation https://issues.jenkins.io/browse/JENKINS-57795 in a particularly accurate way, however it's closed and we are running versions which are much newer.

      Details:

      Kubernetes v1.23.13

       Jenkins 2.346.3

      EC2 plugin: Amazon EC2Version1.68

      Amazon Web Services SDK :: EC2Version1.12.287-357.vf82d85a_6eefd

      We are using the Jenkins operator (https://github.com/jenkinsci/kubernetes-operator).

      We have combed through the following logs:

      • jenkins-master container inside the jenkins-jenkins Pod
      • our log collection tool 
      • cloudtrail

      We have not been able to find anything useful.

      Any help?

      Thanks

       

          [JENKINS-70151] EC2s workers become orphaned

          Rocco Giovinazzo added a comment - - edited

          Hi thoulen are there any news on this?

          We are experiencing the same issue.

          After Jenkins updates and required restart the ec2 plugin is not able to reconnect the previously ondemand created nodes configured in the section configureClouds.

          Checking the AWS EC2 console the EC2 instances are all running as they were before the jenkins restart.
          The file `remoting.jar` is present in the EC2 instances but there is not java process running anymore

          I increased the priority of this task to critical because it is really impacting our CI experience

          Jenkins version 2.401.2
          ec2 plugin version 2.0.7

          Rocco Giovinazzo added a comment - - edited Hi thoulen are there any news on this? We are experiencing the same issue. After Jenkins updates and required restart the ec2 plugin is not able to reconnect the previously ondemand created nodes configured in the section configureClouds. Checking the AWS EC2 console the EC2 instances are all running as they were before the jenkins restart. The file `remoting.jar` is present in the EC2 instances but there is not java process running anymore I increased the priority of this task to critical because it is really impacting our CI experience Jenkins version 2.401.2 ec2 plugin version 2.0.7

            thoulen FABRIZIO MANFREDI
            logube benedetto lo giudice
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: