-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Jenkins 2.144 and ec2-plugin 1.40 or 1.40.1. Amazon region eu-west-1.
The slave stops after idle time correctly. But then on a build starting, instead of starting the stopped slave, it creates a new slave instead. If the number of stopped slaves is the instance cap, then it can't start or create any slaves and the build hangs forever.
Reverting to 1.39 everything works as expected.
This are the logs on 1.39:
No existing instance found - created new instance: {AmiLaunchIndex: 0,ImageId: ami-123123213123,InstanceId: i-087a3e1ca8c5fbbce,InstanceType: t3.medium,KeyName: developer,LaunchTime: Fri Oct 05 10:12:00 UTC 2018,Monitoring: {State: disabled},Placement: {AvailabilityZone: eu-west-1a,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,ProductCodes: [],PublicDnsName: ,State: {Code: 0,Name: pending},StateTransitionReason: ,SubnetId: subnet-111111111,VpcId: vpc-22222222,Architecture: x86_64,BlockDeviceMappings: [],ClientToken: 123,EbsOptimized: true,Hypervisor: xen,ElasticGpuAssociations: [],NetworkInterfaces: [{Attachment: {AttachTime: Fri Oct 05 10:12:00 UTC 2018,AttachmentId: eni-attach-123,DeleteOnTermination: true,DeviceIndex: 0,Status: attaching},Description: ,Groups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],Ipv6Addresses: [],MacAddress: 12:21:12:12:12,NetworkInterfaceId: eni-123,OwnerId: 123123,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,PrivateIpAddresses: [{Primary: true,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-111111111,VpcId: vpc-222222}],RootDeviceName: /dev/xvda,RootDeviceType: ebs,SecurityGroups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],SourceDestCheck: true,StateReason: {Code: pending,Message: pending},Tags: [{Key: jenkins_slave_type,Value: demand_Slave}],VirtualizationType: hvm,CpuOptions: {CoreCount: 1,ThreadsPerCore: 2}} ---- Found stopped instance - starting it: {AmiLaunchIndex: 0,ImageId: ami-123123213123,InstanceId: i-087a3e1ca8c5fbbce,InstanceType: t3.medium,KeyName: developer,LaunchTime: Fri Oct 05 10:12:00 UTC 2018,Monitoring: {State: disabled},Placement: {AvailabilityZone: eu-west-1a,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,ProductCodes: [],PublicDnsName: ,State: {Code: 80,Name: stopped},StateTransitionReason: User initiated (2018-10-05 10:23:12 GMT),SubnetId: subnet-111111111,VpcId: vpc-222222,Architecture: x86_64,BlockDeviceMappings: [{DeviceName: /dev/xvda,Ebs: {AttachTime: Fri Oct 05 10:12:01 UTC 2018,DeleteOnTermination: true,Status: attached,VolumeId: vol-0c6543588e4595d61}}],ClientToken: 123,EbsOptimized: true,EnaSupport: true,Hypervisor: xen,ElasticGpuAssociations: [],NetworkInterfaces: [{Attachment: {AttachTime: Fri Oct 05 10:12:00 UTC 2018,AttachmentId: eni-attach-123,DeleteOnTermination: true,DeviceIndex: 0,Status: attached},Description: ,Groups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],Ipv6Addresses: [],MacAddress: 12:21:12:12:12,NetworkInterfaceId: eni-123,OwnerId: 123123,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,PrivateIpAddresses: [{Primary: true,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-111111111,VpcId: vpc-222222}],RootDeviceName: /dev/xvda,RootDeviceType: ebs,SecurityGroups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],SourceDestCheck: true,StateReason: {Code: Client.UserInitiatedShutdown,Message: Client.UserInitiatedShutdown: User initiated shutdown},Tags: [{Key: jenkins_slave_type,Value: demand_Slave}],VirtualizationType: hvm,CpuOptions: {CoreCount: 1,ThreadsPerCore: 2}} result:{StartingInstances: [{CurrentState: {Code: 0,Name: pending},InstanceId: i-087a3e1ca8c5fbbce,PreviousState: {Code: 80,Name: stopped}}]}
As you can see 1.39 prints "Found stopped instance".
On 1.40 I get this:
SlaveTemplate{ami='ami-123123213123', labels='ec2'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-123123213123]}, {Name: instance-type,Values: [t3.medium]}, {Name: key-name,Values: [developer]}, {Name: availability-zone,Values: [eu-west-1a]}, {Name: subnet-id,Values: [subnet-111111111]}, {Name: instance.group-id,Values: [sg-111111111]}, {Name: tag:jenkins_slave_type,Values: [demand_Slave]}],InstanceIds: [],} SlaveTemplate{ami='ami-123123213123', labels='ec2'}. Return instance: {AmiLaunchIndex: 0,ImageId: ami-123123213123,InstanceId: i-0adb33f8b01edbeb9,InstanceType: t3.medium,KeyName: developer,LaunchTime: Fri Oct 05 11:33:38 UTC 2018,Monitoring: {State: disabled},Placement: {AvailabilityZone: eu-west-1a,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,ProductCodes: [],PublicDnsName: ,State: {Code: 0,Name: pending},StateTransitionReason: ,SubnetId: subnet-111111111,VpcId: vpc-222222,Architecture: x86_64,BlockDeviceMappings: [],ClientToken: 123,EbsOptimized: true,Hypervisor: xen,ElasticGpuAssociations: [],NetworkInterfaces: [{Attachment: {AttachTime: Fri Oct 05 11:33:38 UTC 2018,AttachmentId: eni-attach-03d008e758569bfb0,DeleteOnTermination: true,DeviceIndex: 0,Status: attaching},Description: ,Groups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],Ipv6Addresses: [],MacAddress: 12:21:12:12:12,NetworkInterfaceId: eni-123,OwnerId: 123123,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,PrivateIpAddresses: [{Primary: true,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-111111111,VpcId: vpc-222222}],RootDeviceName: /dev/xvda,RootDeviceType: ebs,SecurityGroups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],SourceDestCheck: true,StateReason: {Code: pending,Message: pending},Tags: [{Key: jenkins_slave_type,Value: demand_Slave}],VirtualizationType: hvm,CpuOptions: {CoreCount: 1,ThreadsPerCore: 2}} --- SlaveTemplate{ami='ami-123123213123', labels='ec2'}. Looking for existing instances with describe-instance: {Filters: [{Name: image-id,Values: [ami-123123213123]}, {Name: instance-type,Values: [t3.medium]}, {Name: key-name,Values: [developer]}, {Name: availability-zone,Values: [eu-west-1a]}, {Name: subnet-id,Values: [subnet-111111111]}, {Name: instance.group-id,Values: [sg-111111111]}, {Name: tag:jenkins_slave_type,Values: [demand_Slave]}],InstanceIds: [],} SlaveTemplate{ami='ami-123123213123', labels='ec2'}. checkInstance: i-0adb33f8b01edbeb9.. false - found existing corresponding Jenkins slave: i-0adb33f8b01edbeb9 SlaveTemplate{ami='ami-123123213123', labels='ec2'}. Return instance: {AmiLaunchIndex: 0,ImageId: ami-123123213123,InstanceId: i-0640e006ce9d0f32f,InstanceType: t3.medium,KeyName: developer,LaunchTime: Fri Oct 05 11:45:34 UTC 2018,Monitoring: {State: disabled},Placement: {AvailabilityZone: eu-west-1a,GroupName: ,Tenancy: default,},PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,ProductCodes: [],PublicDnsName: ,State: {Code: 0,Name: pending},StateTransitionReason: ,SubnetId: subnet-111111111,VpcId: vpc-222222,Architecture: x86_64,BlockDeviceMappings: [],ClientToken: 123,EbsOptimized: true,Hypervisor: xen,ElasticGpuAssociations: [],NetworkInterfaces: [{Attachment: {AttachTime: Fri Oct 05 11:45:34 UTC 2018,AttachmentId: eni-attach-123,DeleteOnTermination: true,DeviceIndex: 0,Status: attaching},Description: ,Groups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],Ipv6Addresses: [],MacAddress: 12:21:12:12:12,NetworkInterfaceId: eni-123,OwnerId: 123123,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0,PrivateIpAddresses: [{Primary: true,PrivateDnsName: ip-10-0-0-0.eu-west-1.compute.internal,PrivateIpAddress: 10.0.0.0}],SourceDestCheck: true,Status: in-use,SubnetId: subnet-111111111,VpcId: vpc-222222}],RootDeviceName: /dev/xvda,RootDeviceType: ebs,SecurityGroups: [{GroupName: system-dev-default-instances,GroupId: sg-111111111}],SourceDestCheck: true,StateReason: {Code: pending,Message: pending},Tags: [{Key: jenkins_slave_type,Value: demand_Slave}],VirtualizationType: hvm,CpuOptions: {CoreCount: 1,ThreadsPerCore: 2}}
When it says "Return instance" above, it returns a new instance, instead of the one stopped.
I am still going to go through the changes on Github and see if I can understand where this regression was introduced.
PS: I changed IDs on logs to hide amazon details
- is duplicated by
-
JENKINS-54041 EC2 Plugin no longer able to start build slaves
- Resolved
-
JENKINS-54392 EC2 Plugin Does Not Start Stopped Nodes
- Closed
-
JENKINS-54071 EC2-plugin not spooling up stopped nodes
- Closed