Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39283

Unexpected number of reservations reported, fails to launch instance

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • ec2-plugin
    • None

      When launching instances, the EC2 plugin sometimes fails to launch the instance, though it doesn't throw any error.

      The following was in the Jenkins logs. The final line seems to be printing out all reservations in the entire AWS account, not just those controlled by Jenkins. In our case, this is hundreds of pages in a single line, which I've trimmed.

      INFO: Launching ami-a78ec5b0 for template jenkins-builder-12.04
      Spot instance id in provision: sir-5cvg5szg
      Oct 20, 2016 6:48:42 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
      INFO: Spot instance id in provision: sir-5cvg5szg
      Oct 20, 2016 6:48:43 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: We have now 14 computers
      Oct 20, 2016 6:48:43 PM hudson.plugins.ec2.EC2RetentionStrategy start
      INFO: Start requested for jenkins-builder-12.04 (sir-5cvg5szg)
      Oct 20, 2016 6:48:44 PM hudson.plugins.ec2.EC2Computer _describeInstanceOnce
      INFO: Unexpected number of reservations reported by EC2 for instance id 'null', expected 1 result, found [{ReservationId: r-2e8ac590,OwnerId: 074332678689,Groups: [],Instances: [{InstanceId: i-28bbf919,ImageId: ami-2060e937,State: {Code: 16,Name: running},PrivateDnsName
      : ip-10-60-5-254.ec2.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: vagrant-insecure,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m4.4xlarge,LaunchTime: Wed Oct 12 20:18:55 UTC 2016,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default,},M
      onitoring: {State: disabled},SubnetId: subnet-2b874e01,VpcId: vpc-fc762198,PrivateIpAddress: 10.60.5.254,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-5dce4288,Status: attached,AttachT
      ime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/sdb,Ebs: {VolumeId: vol-9fce424a,Status: attached,AttachTime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/sdf,Ebs: {VolumeId: vol-d4ce4201,Status: attached,A
      ttachTime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/xvdg,Ebs: {VolumeId: vol-90c1ce45,Status: attached,AttachTime: Wed Oct 12 20:10:15 UTC 2016,DeleteOnTermination: false}}],VirtualizationType: hvm,ClientToken: ,Tags: [{Key: Hostname,V
      alue: app01}, {Key: Roles,Value: application}, {Key: Name,Value: vagrant-fds-release}, {Key: Owner,Value: jhuang}, {Key: Launched,Value: 2016-09-19T17:10:55Z}, {Key: Jira,Value: }],SecurityGroups: [{GroupName: fdsautomation_sg,GroupId: sg-62dc5619}, {GroupName: basic_ac
      cess_sg,GroupId: sg-d39070ab}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-e99b7bfb,SubnetId: subnet-2b874e01,VpcId: vpc-fc762198,Description: ,OwnerId: 074332678689,Status: in-use,MacAddress: 12:a2:36:c3:64:95,PrivateIpAddress: 10
      .60.5.254,PrivateDnsName: ip-10-60-5-254.ec2.internal,SourceDestCheck: true,Groups: [{GroupName: fdsautomation_sg,GroupId: sg-62dc5619}, {GroupName: basic_access_sg,GroupId: sg-d39070ab}],Attachment: {AttachmentId: eni-attach-1e77d9b5,DeviceIndex: 0,Status: attached,Att
      achTime: Mon Sep 19 17:10:59 UTC 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.60.5.254,PrivateDnsName:
      

      It looks like it is querying AWS for instances matching a non-existant ID. No instance is created, and there are no errors in the log.

      This went on for about an hour for us, with it retrying the launch every few seconds. It eventually was able to launch nodes, after all our existing nodes destroyed themselves (idle time).

          [JENKINS-39283] Unexpected number of reservations reported, fails to launch instance

          Alex Hunt created issue -
          Alex Hunt made changes -
          Description Original: When launching instances, the EC2 plugin sometimes fails to launch the instance, though it doesn't throw any error.

          The following in in the Jenkins logs. The final line seems to be printing out all reservations in the entire AWS account, not just those controlled by Jenkins. In our case, this is hundreds of pages in a single line, which I've trimmed.
          {noformat}
          INFO: Launching ami-a78ec5b0 for template jenkins-builder-12.04
          Spot instance id in provision: sir-5cvg5szg
          Oct 20, 2016 6:48:42 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
          INFO: Spot instance id in provision: sir-5cvg5szg
          Oct 20, 2016 6:48:43 PM hudson.plugins.ec2.EC2Cloud provision
          INFO: We have now 14 computers
          Oct 20, 2016 6:48:43 PM hudson.plugins.ec2.EC2RetentionStrategy start
          INFO: Start requested for jenkins-builder-12.04 (sir-5cvg5szg)
          Oct 20, 2016 6:48:44 PM hudson.plugins.ec2.EC2Computer _describeInstanceOnce
          INFO: Unexpected number of reservations reported by EC2 for instance id 'null', expected 1 result, found [{ReservationId: r-2e8ac590,OwnerId: 074332678689,Groups: [],Instances: [{InstanceId: i-28bbf919,ImageId: ami-2060e937,State: {Code: 16,Name: running},PrivateDnsName
          : ip-10-60-5-254.ec2.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: vagrant-insecure,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m4.4xlarge,LaunchTime: Wed Oct 12 20:18:55 UTC 2016,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default,},M
          onitoring: {State: disabled},SubnetId: subnet-2b874e01,VpcId: vpc-fc762198,PrivateIpAddress: 10.60.5.254,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-5dce4288,Status: attached,AttachT
          ime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/sdb,Ebs: {VolumeId: vol-9fce424a,Status: attached,AttachTime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/sdf,Ebs: {VolumeId: vol-d4ce4201,Status: attached,A
          ttachTime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/xvdg,Ebs: {VolumeId: vol-90c1ce45,Status: attached,AttachTime: Wed Oct 12 20:10:15 UTC 2016,DeleteOnTermination: false}}],VirtualizationType: hvm,ClientToken: ,Tags: [{Key: Hostname,V
          alue: app01}, {Key: Roles,Value: application}, {Key: Name,Value: vagrant-fds-release}, {Key: Owner,Value: jhuang}, {Key: Launched,Value: 2016-09-19T17:10:55Z}, {Key: Jira,Value: }],SecurityGroups: [{GroupName: fdsautomation_sg,GroupId: sg-62dc5619}, {GroupName: basic_ac
          cess_sg,GroupId: sg-d39070ab}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-e99b7bfb,SubnetId: subnet-2b874e01,VpcId: vpc-fc762198,Description: ,OwnerId: 074332678689,Status: in-use,MacAddress: 12:a2:36:c3:64:95,PrivateIpAddress: 10
          .60.5.254,PrivateDnsName: ip-10-60-5-254.ec2.internal,SourceDestCheck: true,Groups: [{GroupName: fdsautomation_sg,GroupId: sg-62dc5619}, {GroupName: basic_access_sg,GroupId: sg-d39070ab}],Attachment: {AttachmentId: eni-attach-1e77d9b5,DeviceIndex: 0,Status: attached,Att
          achTime: Mon Sep 19 17:10:59 UTC 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.60.5.254,PrivateDnsName:
          {noformat}

          It looks like it is querying AWS for instances matching a non-existant ID. No instance is created, and there are no errors in the log.

          This went on for about an hour for us, with it retrying the launch every few seconds. It eventually was able to launch nodes, after all our existing nodes destroyed themselves (idle time).
          New: When launching instances, the EC2 plugin sometimes fails to launch the instance, though it doesn't throw any error.

          The following was in the Jenkins logs. The final line seems to be printing out all reservations in the entire AWS account, not just those controlled by Jenkins. In our case, this is hundreds of pages in a single line, which I've trimmed.
          {noformat}
          INFO: Launching ami-a78ec5b0 for template jenkins-builder-12.04
          Spot instance id in provision: sir-5cvg5szg
          Oct 20, 2016 6:48:42 PM hudson.plugins.ec2.SlaveTemplate provisionSpot
          INFO: Spot instance id in provision: sir-5cvg5szg
          Oct 20, 2016 6:48:43 PM hudson.plugins.ec2.EC2Cloud provision
          INFO: We have now 14 computers
          Oct 20, 2016 6:48:43 PM hudson.plugins.ec2.EC2RetentionStrategy start
          INFO: Start requested for jenkins-builder-12.04 (sir-5cvg5szg)
          Oct 20, 2016 6:48:44 PM hudson.plugins.ec2.EC2Computer _describeInstanceOnce
          INFO: Unexpected number of reservations reported by EC2 for instance id 'null', expected 1 result, found [{ReservationId: r-2e8ac590,OwnerId: 074332678689,Groups: [],Instances: [{InstanceId: i-28bbf919,ImageId: ami-2060e937,State: {Code: 16,Name: running},PrivateDnsName
          : ip-10-60-5-254.ec2.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: vagrant-insecure,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: m4.4xlarge,LaunchTime: Wed Oct 12 20:18:55 UTC 2016,Placement: {AvailabilityZone: us-east-1a,GroupName: ,Tenancy: default,},M
          onitoring: {State: disabled},SubnetId: subnet-2b874e01,VpcId: vpc-fc762198,PrivateIpAddress: 10.60.5.254,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-5dce4288,Status: attached,AttachT
          ime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/sdb,Ebs: {VolumeId: vol-9fce424a,Status: attached,AttachTime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/sdf,Ebs: {VolumeId: vol-d4ce4201,Status: attached,A
          ttachTime: Mon Sep 19 17:11:00 UTC 2016,DeleteOnTermination: true}}, {DeviceName: /dev/xvdg,Ebs: {VolumeId: vol-90c1ce45,Status: attached,AttachTime: Wed Oct 12 20:10:15 UTC 2016,DeleteOnTermination: false}}],VirtualizationType: hvm,ClientToken: ,Tags: [{Key: Hostname,V
          alue: app01}, {Key: Roles,Value: application}, {Key: Name,Value: vagrant-fds-release}, {Key: Owner,Value: jhuang}, {Key: Launched,Value: 2016-09-19T17:10:55Z}, {Key: Jira,Value: }],SecurityGroups: [{GroupName: fdsautomation_sg,GroupId: sg-62dc5619}, {GroupName: basic_ac
          cess_sg,GroupId: sg-d39070ab}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-e99b7bfb,SubnetId: subnet-2b874e01,VpcId: vpc-fc762198,Description: ,OwnerId: 074332678689,Status: in-use,MacAddress: 12:a2:36:c3:64:95,PrivateIpAddress: 10
          .60.5.254,PrivateDnsName: ip-10-60-5-254.ec2.internal,SourceDestCheck: true,Groups: [{GroupName: fdsautomation_sg,GroupId: sg-62dc5619}, {GroupName: basic_access_sg,GroupId: sg-d39070ab}],Attachment: {AttachmentId: eni-attach-1e77d9b5,DeviceIndex: 0,Status: attached,Att
          achTime: Mon Sep 19 17:10:59 UTC 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: 10.60.5.254,PrivateDnsName:
          {noformat}

          It looks like it is querying AWS for instances matching a non-existant ID. No instance is created, and there are no errors in the log.

          This went on for about an hour for us, with it retrying the launch every few seconds. It eventually was able to launch nodes, after all our existing nodes destroyed themselves (idle time).

          This is happening to us using Jenkins 2.121.1 and Amazon EC2 plugin 1.39.

           

          Gustavo Chaves added a comment - This is happening to us using Jenkins 2.121.1 and Amazon EC2 plugin 1.39.  

            francisu Francis Upton
            ahuntpindrop Alex Hunt
            Votes:
            4 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: