Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46505

Jenkins slaves are being stopped multiple times with ec2 plugin

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ec2-plugin
    • Jenkins Version: 2.60.3
      Amazon EC2 plugins: 1.36
      Idle termination time: -5

      I am seeing cases where Jenkins slaves are being stopped two or three times when using the ec2 plugin.  I was able to go to the AWS console and confirm the slave was successfully stopped after the first request. 

      Update: We changed our local branch to only go through the logic if the agent isOnline() which seemed to resolve the issue.

      From /var/log/jenkins/jenkins.log:

      Aug 26, 2017 8:02:05 AM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
      INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 24 idle minutes, with 3 minutes remaining in billing period
      
      Aug 26, 2017 8:02:05 AM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
      
      Aug 26, 2017 8:02:06 AM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx
      
      
      Aug 26, 2017 8:03:59 AM hudson.plugins.ec2.EC2RetentionStrategy internalCheck
      INFO: Idle timeout of jenkins-slave-name (i-xxxxxxxxxxxxxxxx) after 26 idle minutes, with 2 minutes remaining in billing period
      
      Aug 26, 2017 8:03:59 AM hudson.plugins.ec2.EC2AbstractSlave idleTimeout
      INFO: EC2 instance idle time expired: i-xxxxxxxxxxxxxxxx
      
      Aug 26, 2017 8:04:00 AM hudson.plugins.ec2.EC2AbstractSlave stop
      INFO: EC2 instance stop request sent for i-xxxxxxxxxxxxxxxx

       

      From AWS console:

      State transition reason:
      User initiated (2017-08-26 08:02:05 GMT)
          
      State transition reason message:
      Client.UserInitiatedShutdown: User initiated shutdown

          [JENKINS-46505] Jenkins slaves are being stopped multiple times with ec2 plugin

          I'm having the same issue but in my case any build taking over 30 min is broken because the node is shut down no matter what... Any ideas?

          George Cimpoies added a comment - I'm having the same issue but in my case any build taking over 30 min is broken because the node is shut down no matter what... Any ideas?

            francisu Francis Upton
            mkozell Mike Kozell
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: