Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36516

EC2 plugin: Plugin fails to automatically launch slaves to meet demand

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • ec2-plugin
    • None
    • Jenkins v2.12 / Ubuntu 14.04.4 x86_64 / Oracle Java 1.80_91-b14

      Hi,

      We've recently started seeing a problem auto-launching EC2 build slaves. The plugin detects that it needs to launch a slave to meet demand and that no current instances are runnign - it just doesn't launch anything.

      Manual provisioning works.

      Log (some info sanitised):

      Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@6a756eca needed by excess workload of 1 units of label 'null'
      Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
      WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors
      Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine
      Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
      INFO: Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine
      Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
      Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
      INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
      Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-6c14310f]}, {Name: subnet-id,Values: [subnet-XXXXX]}, {Name: instance.group-id,Values: [sg-XXXXX]}, {Name: key-name,Values: [XXXXX]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],}
      No existing instance found - but cannot create new instance
      Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision
      INFO: Attempting provision - finished, excess workload: 1
      

          [JENKINS-36516] EC2 plugin: Plugin fails to automatically launch slaves to meet demand

          Francis Upton added a comment -

          Which version of the ec2 plugin are you using?

          Francis Upton added a comment - Which version of the ec2 plugin are you using?

          Chin Godawita added a comment -

          It was happening on 1.33 but upgrading to 1.35 didn't fix it.

          Chin Godawita added a comment - It was happening on 1.33 but upgrading to 1.35 didn't fix it.

          karthic jayaraman added a comment - - edited

          I've observed the same issue in the latest Jenkins Version (2.10).

          Manual provisioning works.

          No existing instance found - but cannot create new instance
          Jul 13, 2016 6:05:33 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 6:05:41 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 6:05:43 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          

          karthic jayaraman added a comment - - edited I've observed the same issue in the latest Jenkins Version (2.10). Manual provisioning works. No existing instance found - but cannot create new instance Jul 13, 2016 6:05:33 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 6:05:41 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 6:05:43 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1

          Francis Upton added a comment -

          Can you try increasing the log level so we get the debug/fine level of logging?

          Francis Upton added a comment - Can you try increasing the log level so we get the debug/fine level of logging?

          Here you go Francis Upton

          INFO: Finished EC2 alive slaves monitor. 402 ms
          Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 229 ms
          Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 184 ms
          Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 162 ms
          Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 200 ms
          Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 175 ms
          Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 234 ms
          Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 182 ms
          Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 185 ms
          Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 155 ms
          Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 138 ms
          Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 163 ms
          Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 167 ms
          Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 166 ms
          Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 169 ms
          Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 165 ms
          Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 161 ms
          Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 163 ms
          Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 167 ms
          Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 163 ms
          Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 157 ms
          Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 172 ms
          Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 171 ms
          Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 175 ms
          Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 160 ms
          Jul 13, 2016 5:39:14 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started Download metadata
          Jul 13, 2016 5:39:16 AM hudson.model.UpdateSite updateData
          INFO: Obtained the latest update center data file for UpdateSource default
          Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
          Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller
          Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller
          Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load
          INFO: Obtained the updated data file for hudson.tools.JDKInstaller
          Jul 13, 2016 5:39:16 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished Download metadata. 1,668 ms
          Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 185 ms
          Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate
          INFO: Terminated EC2 instance (terminated): i-d81e5a5e
          Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate
          INFO: Removed EC2 instance from jenkins master: i-d81e5a5e
          Jul 13, 2016 5:52:24 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:52:34 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:52:44 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:52:54 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:53:04 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:53:14 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:53:24 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:53:34 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:53:44 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:53:54 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:53:56 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:54:02 AM jenkins.model.Jenkins$26 run
          INFO: Restart in 10 seconds
          Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$26 run
          SEVERE: Restarting VM as requested by anonymous
          Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins cleanUp
          INFO: Stopping Jenkins
          Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained
          INFO: Started termination
          Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained
          INFO: Completed termination
          Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins _cleanUpDisconnectComputers
          INFO: Starting node disconnection
          Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager
          INFO: Stopping plugin manager
          Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpPersistQueue
          INFO: Persisting build queue
          Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects
          INFO: Waiting for node disconnection completion
          Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins cleanUp
          INFO: Jenkins stopped
          Running from: /usr/share/jenkins/jenkins.war
          Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info
          INFO: Logging initialized @380ms
          Jul 13, 2016 5:54:13 AM winstone.Logger logInternal
          INFO: Beginning extraction from war file
          Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog warn
          WARNING: Empty contextPath
          Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info
          INFO: jetty-9.2.z-SNAPSHOT
          Jul 13, 2016 5:54:14 AM org.eclipse.jetty.util.log.JavaUtilLog info
          INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
          Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
          Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info
          INFO: Started w.@1e125710{/,file:/var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
          Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info
          INFO: Started ServerConnector@42eded9{HTTP/1.1}{0.0.0.0:8080}
          Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info
          INFO: Started @1689ms
          Jul 13, 2016 5:54:15 AM winstone.Logger logInternal
          INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled
          Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Started initialization
          Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Listed all plugins
          Jul 13, 2016 5:54:15 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider
          INFO: Initializing Bouncy Castle security provider.
          Jul 13, 2016 5:54:16 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider
          INFO: Bouncy Castle security provider initialized.
          Jul 13, 2016 5:54:20 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Prepared all plugins
          Jul 13, 2016 5:54:20 AM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error
          INFO: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
          Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Started all plugins
          Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Augmented all extensions
          Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect
          INFO: Attempting to reconnect gscqascheduler113.fourvinternal.com
          Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect
          INFO: Attempting to reconnect gscqascheduler120.fourvinternal.com
          Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect
          INFO: Attempting to reconnect qa04scheduler.fourvinternal.com
          Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Loaded all jobs
          Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started Download metadata
          Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished Download metadata. 1 ms
          Jul 13, 2016 5:54:22 AM org.jenkinsci.main.modules.sshd.SSHD start
          INFO: Started SSHD at port 37928
          Jul 13, 2016 5:54:22 AM jenkins.InitReactorRunner$1 onAttained
          INFO: Completed initialization
          Jul 13, 2016 5:54:22 AM hudson.WebAppMain$3 run
          INFO: Jenkins is fully up and running
          [07/13/16 05:54:24] SSH Launch of gscqascheduler120.fourvinternal.com on 10.187.27.222 completed in 2,567 ms
          [07/13/16 05:54:26] SSH Launch of gscqascheduler113.fourvinternal.com on 10.187.29.0 completed in 4,673 ms
          [07/13/16 05:54:26] SSH Launch of qa04scheduler.fourvinternal.com on 10.187.29.100 completed in 4,840 ms
          Jul 13, 2016 5:56:01 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:56:11 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:56:21 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:56:31 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:56:41 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:56:43 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:56:51 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Jul 13, 2016 5:57:01 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          No existing instance found - but cannot create new instance
          Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          Launching ami-e8e952ff for template GSC Cloud Slave
          Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Launching ami-e8e952ff for template GSC Cloud Slave
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
          
          

          karthic jayaraman added a comment - Here you go Francis Upton INFO: Finished EC2 alive slaves monitor. 402 ms Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 229 ms Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 184 ms Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 162 ms Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 200 ms Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 175 ms Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 234 ms Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 182 ms Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 185 ms Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 155 ms Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 138 ms Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 163 ms Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 167 ms Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 166 ms Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 169 ms Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 165 ms Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 161 ms Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 163 ms Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 167 ms Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 163 ms Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 157 ms Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 172 ms Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 171 ms Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 175 ms Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 160 ms Jul 13, 2016 5:39:14 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started Download metadata Jul 13, 2016 5:39:16 AM hudson.model.UpdateSite updateData INFO: Obtained the latest update center data file for UpdateSource default Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tools.JDKInstaller Jul 13, 2016 5:39:16 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Download metadata. 1,668 ms Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 185 ms Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Terminated EC2 instance (terminated): i-d81e5a5e Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-d81e5a5e Jul 13, 2016 5:52:24 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:52:34 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:52:44 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:52:54 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:04 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:14 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:24 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:34 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:44 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:54 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:56 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:54:02 AM jenkins.model.Jenkins$26 run INFO: Restart in 10 seconds Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$26 run SEVERE: Restarting VM as requested by anonymous Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins cleanUp INFO: Stopping Jenkins Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained INFO: Started termination Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained INFO: Completed termination Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins _cleanUpDisconnectComputers INFO: Starting node disconnection Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager INFO: Stopping plugin manager Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpPersistQueue INFO: Persisting build queue Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects INFO: Waiting for node disconnection completion Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins cleanUp INFO: Jenkins stopped Running from: /usr/share/jenkins/jenkins.war Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Logging initialized @380ms Jul 13, 2016 5:54:13 AM winstone.Logger logInternal INFO: Beginning extraction from war file Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog warn WARNING: Empty contextPath Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: jetty-9.2.z-SNAPSHOT Jul 13, 2016 5:54:14 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME") Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started w.@1e125710{/,file:/var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war} Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started ServerConnector@42eded9{HTTP/1.1}{0.0.0.0:8080} Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started @1689ms Jul 13, 2016 5:54:15 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins Jul 13, 2016 5:54:15 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider INFO: Initializing Bouncy Castle security provider. Jul 13, 2016 5:54:16 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider INFO: Bouncy Castle security provider initialized. Jul 13, 2016 5:54:20 AM jenkins.InitReactorRunner$1 onAttained INFO: Prepared all plugins Jul 13, 2016 5:54:20 AM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error INFO: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Started all plugins Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Augmented all extensions Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect gscqascheduler113.fourvinternal.com Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect gscqascheduler120.fourvinternal.com Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect qa04scheduler.fourvinternal.com Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started Download metadata Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Download metadata. 1 ms Jul 13, 2016 5:54:22 AM org.jenkinsci.main.modules.sshd.SSHD start INFO: Started SSHD at port 37928 Jul 13, 2016 5:54:22 AM jenkins.InitReactorRunner$1 onAttained INFO: Completed initialization Jul 13, 2016 5:54:22 AM hudson.WebAppMain$3 run INFO: Jenkins is fully up and running [07/13/16 05:54:24] SSH Launch of gscqascheduler120.fourvinternal.com on 10.187.27.222 completed in 2,567 ms [07/13/16 05:54:26] SSH Launch of gscqascheduler113.fourvinternal.com on 10.187.29.0 completed in 4,673 ms [07/13/16 05:54:26] SSH Launch of qa04scheduler.fourvinternal.com on 10.187.29.100 completed in 4,840 ms Jul 13, 2016 5:56:01 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:11 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:21 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:31 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:41 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:43 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:51 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:57:01 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}

          Francis Upton added a comment -

          Thanks for the additional tracing (it's good to get it from the beginning), but I also need the logging level changed so we get the "fine" tracing as well (not just "info")

          Francis Upton added a comment - Thanks for the additional tracing (it's good to get it from the beginning), but I also need the logging level changed so we get the "fine" tracing as well (not just "info")

          Chin Godawita added a comment -

          I realise this is not much help but strangely enough, it started working for us again. No Jenkins/plugin upgrades have been done since the issue happened.

          The immediate set of actions I did before it started working was:

          • To get around this issue, we'd provisioned (through the EC2 plugin) an always-running instance (with no idle timeout)
          • Today, because we'd run out of space, I deleted it
          • In the interim before I'd had a chance to , we had a build get triggered via a commit
          • This automatically brought up a new instance as it used to

          I'm also trying to get a debug log across but it's a little difficult as I need to lock in a period of time where our master build server is inaccessible.

          In the meantime, here's the log from when it started working:

          Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate
          INFO: Terminated EC2 instance (terminated): i-xxxxxx
          Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate
          INFO: Removed EC2 instance from jenkins master: i-xxxxxx
          Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@xxx needed by excess workload of 1 units of label 'null'
          Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision
          WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors
          Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine
          Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-xxxxxx]}, {Name: subnet-id,Values: [subnet-xxxxxx]}, {Name: instance.group-id,Values: [sg-xxxxxx]}, {Name: key-name,Values: [jenkins-slave]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],}
          No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,}
          Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,}
          Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: We have now 1 computers
          Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2RetentionStrategy start
          INFO: Start requested for Remote Ubuntu 14.04 Docker Machine (i-xxxxxx)
          Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Added node named: Remote Ubuntu 14.04 Docker Machine (i-xxxxxx), We have now 2 computers
          Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: -1
          Jul 28, 2016 11:44:26 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
          INFO: Started provisioning Remote Ubuntu 14.04 Docker Machine (ami-xxxxxx) from ec2-jenkins-user with 2 executors. Remaining excess workload: -1
          Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Started EC2 alive slaves monitor
          Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished EC2 alive slaves monitor. 55 ms
          

          Chin Godawita added a comment - I realise this is not much help but strangely enough, it started working for us again. No Jenkins/plugin upgrades have been done since the issue happened. The immediate set of actions I did before it started working was: To get around this issue, we'd provisioned (through the EC2 plugin) an always-running instance (with no idle timeout) Today, because we'd run out of space, I deleted it In the interim before I'd had a chance to , we had a build get triggered via a commit This automatically brought up a new instance as it used to I'm also trying to get a debug log across but it's a little difficult as I need to lock in a period of time where our master build server is inaccessible. In the meantime, here's the log from when it started working: Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Terminated EC2 instance (terminated): i-xxxxxx Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-xxxxxx Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@xxx needed by excess workload of 1 units of label 'null' Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-xxxxxx]}, {Name: subnet-id,Values: [subnet-xxxxxx]}, {Name: instance.group-id,Values: [sg-xxxxxx]}, {Name: key-name,Values: [jenkins-slave]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],} No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,} Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,} Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: We have now 1 computers Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2RetentionStrategy start INFO: Start requested for Remote Ubuntu 14.04 Docker Machine (i-xxxxxx) Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: Added node named: Remote Ubuntu 14.04 Docker Machine (i-xxxxxx), We have now 2 computers Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: -1 Jul 28, 2016 11:44:26 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Remote Ubuntu 14.04 Docker Machine (ami-xxxxxx) from ec2-jenkins-user with 2 executors. Remaining excess workload: -1 Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 55 ms

          Joel McGinnis added a comment -

          I am seeing what appear to me to be the same issue on EC2 1.36. After adding an additional EC2 node to a previously working system, auto-re-launch of timed out instances doesn't occur. Manually launching will work. My logs look very similar.

          Joel McGinnis added a comment - I am seeing what appear to me to be the same issue on EC2 1.36. After adding an additional EC2 node to a previously working system, auto-re-launch of timed out instances doesn't occur. Manually launching will work. My logs look very similar.

          same issue here with 1.36

          Vincent Brouillet added a comment - same issue here with 1.36

          Dan Stark added a comment -

          I'm also experiencing this issue with 1.36. I'm using spot instances if that matters. Here's the exception I'm seeing:

          Dec 12, 2016 7:02:19 PM hudson.plugins.ec2.EC2Cloud provision
          WARNING: Exception during provisioning
          com.amazonaws.SdkClientException: Unable to unmarshall response (Read timed out). Response Code: 200, Response Text: OK
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1468)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583)
                  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447)
                  at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:11396)
                  at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11372)
                  at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5542)
                  at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5554)
                  at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363)
                  at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:499)
                  at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:518)
                  at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:547)
                  at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701)
                  at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307)
                  at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60)
                  at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798)
                  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:745)
          Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out
                  at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708)
                  at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086)
                  at com.ctc.wstx.evt.WstxEventReader.peek(WstxEventReader.java:306)
                  at com.amazonaws.transform.StaxUnmarshallerContext.nextEvent(StaxUnmarshallerContext.java:220)
                  at com.amazonaws.services.ec2.model.transform.InstanceStaxUnmarshaller.unmarshall(InstanceStaxUnmarshaller.java:42)
                  at com.amazonaws.services.ec2.model.transform.ReservationStaxUnmarshaller.unmarshall(ReservationStaxUnmarshaller.java:69)
                  at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:49)
                  at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:31)
                  at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:101)
                  at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:43)
                  at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
                  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444)
                  ... 28 more
          Caused by: java.net.SocketTimeoutException: Read timed out
                  at java.net.SocketInputStream.socketRead0(Native Method)
                  at java.net.SocketInputStream.read(SocketInputStream.java:152)
                  at java.net.SocketInputStream.read(SocketInputStream.java:122)
                  at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
                  at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
                  at sun.security.ssl.InputRecord.read(InputRecord.java:509)
                  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:944)
                  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:901)
                  at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
                  at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87)
                  at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
                  at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200)
                  at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:191)
                  at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
                  at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
                  at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181)
                  at java.io.FilterInputStream.read(FilterInputStream.java:107)
                  at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365)
                  at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110)
                  at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84)
                  at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57)
                  at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992)
                  at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628)
                  at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126)
                  at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701)
                  at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1023)
                  ... 38 more
          

          Dan Stark added a comment - I'm also experiencing this issue with 1.36. I'm using spot instances if that matters. Here's the exception I'm seeing: Dec 12, 2016 7:02:19 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Exception during provisioning com.amazonaws.SdkClientException: Unable to unmarshall response (Read timed out). Response Code: 200, Response Text: OK at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1468) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:11396) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11372) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5542) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5554) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363) at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:499) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:518) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:547) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086) at com.ctc.wstx.evt.WstxEventReader.peek(WstxEventReader.java:306) at com.amazonaws.transform.StaxUnmarshallerContext.nextEvent(StaxUnmarshallerContext.java:220) at com.amazonaws.services.ec2.model.transform.InstanceStaxUnmarshaller.unmarshall(InstanceStaxUnmarshaller.java:42) at com.amazonaws.services.ec2.model.transform.ReservationStaxUnmarshaller.unmarshall(ReservationStaxUnmarshaller.java:69) at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:49) at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:31) at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:101) at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:43) at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444) ... 28 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) at sun.security.ssl.InputRecord.read(InputRecord.java:509) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:944) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:901) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200) at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:191) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1023) ... 38 more

          Vlad Korolev added a comment -

          Seeing same problem with 1.36

          Using regular on demand instances

           

           

          INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@73450f26 needed by excess workload of 1 units of label 'std'
          Considering launching ami-39f5e340 for template Standard AMI
          Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Considering launching ami-39f5e340 for template Standard AMI
          Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo
          INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate
          Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-39f5e340]}, {Name: availability-zone,Values: [us-west-2b]}, , {Name: instance-type,Values: [m4.xlarge]}, {Name: tag:Name,Values: [jenkins-build-slave-std]}, {Name: tag:Environment,Values: [Build]}],}
          No existing instance found - but cannot create new instance
          Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.EC2Cloud provision
          INFO: Attempting provision - finished, excess workload: 1
          

          Vlad Korolev added a comment - Seeing same problem with 1.36 Using regular on demand instances     INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@73450f26 needed by excess workload of 1 units of label 'std' Considering launching ami-39f5e340 for template Standard AMI Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Considering launching ami-39f5e340 for template Standard AMI Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-39f5e340]}, {Name: availability-zone,Values: [us-west-2b]}, , {Name: instance-type,Values: [m4.xlarge]}, {Name: tag:Name,Values: [jenkins-build-slave-std]}, {Name: tag:Environment,Values: [Build]}],} No existing instance found - but cannot create new instance Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1

          eddy kim added a comment - - edited

          I'm seeing the similar behavior:

          it actually launches the first ec2 slave automatically, but no more after that, no matter how long or how many jobs are in the queue.   I can launch them manually from the /computer/ page.  The teardowns based on idle timeouts seem to work.

          I've upped the global instance cap and the ami instance cap to 255, still the same.

          jenkins 2.66, openjdk version "1.8.0_31"

          Any suggestions on how to debug this?  This might be relevant:

          Jul 02, 2017 12:08:49 AM org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener publish
          WARNING: Error publishing Run queued event.
          org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.
              at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)
          

           

          WARNING: Error publishing Run queued event.
          org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.
              at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)
              at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178)
              at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.access$300(SyncQueueListener.java:57)
              at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener$1.run(SyncQueueListener.java:106)
          Caused by: java.lang.NullPointerException

          eddy kim added a comment - - edited I'm seeing the similar behavior: it actually launches the first ec2 slave automatically, but no more after that, no matter how long or how many jobs are in the queue.   I can launch them manually from the /computer/ page.  The teardowns based on idle timeouts seem to work. I've upped the global instance cap and the ami instance cap to 255, still the same. jenkins 2.66, openjdk version "1.8.0_31" Any suggestions on how to debug this?  This might be relevant: Jul 02, 2017 12:08:49 AM org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener publish WARNING: Error publishing Run queued event. org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.     at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)   WARNING: Error publishing Run queued event. org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.     at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)     at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178)     at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.access$300(SyncQueueListener.java:57)     at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener$1.run(SyncQueueListener.java:106) Caused by: java.lang.NullPointerException

          Vlad Korolev added a comment -

          More information.  This problem seems to be very intermittent we had a bunch of builds today and all instances got launched correctly

          Vlad Korolev added a comment - More information.  This problem seems to be very intermittent we had a bunch of builds today and all instances got launched correctly

          Steve Talbot added a comment -

          Plugin v1.36 will launch a slave and successfully run a build for us. Plugin v1.37 launches the EC2 instance, but fails to recognise that it can use it to build.

          Steve Talbot added a comment - Plugin v1.36 will launch a slave and successfully run a build for us. Plugin v1.37 launches the EC2 instance, but fails to recognise that it can use it to build.

          si Joey added a comment -

          I am also having similar  issues with v1.37 of the plugin.  Builds run fine the first time, but after the agent timeout from inactivity, no new builds will start, however, it will work again if I change something in the plugins' profile.  Moreover, the instance remains on in the AWS console.  It's supposed to terminate after 30 minutes of inactivity.

          I see the following errors in the log:

          Nov 27, 2017 9:43:58 PM hudson.plugins.ec2.EC2Cloud provision
          WARNING: Exception during provisioning
          com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 309aa7e7-89e0-42c4-9df5-5b4985222ba5)

          si Joey added a comment - I am also having similar  issues with v1.37 of the plugin.  Builds run fine the first time, but after the agent timeout from inactivity, no new builds will start, however, it will work again if I change something in the plugins' profile.  Moreover, the instance remains on in the AWS console.  It's supposed to terminate after 30 minutes of inactivity. I see the following errors in the log: Nov 27, 2017 9:43:58 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Exception during provisioning com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 309aa7e7-89e0-42c4-9df5-5b4985222ba5)

          I've had this issue after upgrading Jenkins from LTS to Weekly.

          But it was "accidentally" fixed by:

           - Increased instance cap from 1 to 2 -> then changed back to 1

           - Checked AWS console - there was a "zombie" instance which was created by Jenkins, but no longer visible on Nodes page

          Oleksandr Tereshchuk added a comment - I've had this issue after upgrading Jenkins from LTS to Weekly. But it was "accidentally" fixed by:  - Increased instance cap from 1 to 2 -> then changed back to 1  - Checked AWS console - there was a "zombie" instance which was created by Jenkins, but no longer visible on Nodes page

          si Joey added a comment -

          I was able to correct my issue with the plugin by checking the box for, "Use EC2 instance profile to obtain credentials." With that check, the ec2 agent was able to get new credentials to launch.

          si Joey added a comment - I was able to correct my issue with the plugin by checking the box for, "Use EC2 instance profile to obtain credentials." With that check, the ec2 agent was able to get new credentials to launch.

          We experienced the same issue, the "com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired" error started appearing randomly after launched new nodes ok with the EC2 plugin for a while.

          Updating the server configuration would solve the issue, but only temporarily.

          After upgrading to Jenkins LTS 2.89.3 with 1.38 EC2 plugin the issue does not happen anymore. Not 100% sure though if the bug is really fixed.

          Stefan Verhoeff added a comment - We experienced the same issue, the " com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired" error started appearing randomly after launched new nodes ok with the EC2 plugin for a while. Updating the server configuration would solve the issue, but only temporarily. After upgrading to Jenkins LTS 2.89.3 with 1.38 EC2 plugin the issue does not happen anymore. Not 100% sure though if the bug is really fixed.

          Francis Upton added a comment -

          Please try 1.39 of the plugin, particularly if you are upgrading from 1.36 or earlier.

          Francis Upton added a comment - Please try 1.39 of the plugin, particularly if you are upgrading from 1.36 or earlier.

          Ben Copeland added a comment -

          Still an issue in 1.42

          Ben Copeland added a comment - Still an issue in 1.42

          bithead is this still an inssue in 1.45?

          Raihaan Shouhell added a comment - bithead is this still an inssue in 1.45?

          ovi craciun added a comment - - edited

          still an issue with 1.48
          I can see this log but the spotinst mentioned is never spawned, no other error:

          ```
          Spot instance id in provision: sir-px3sj14h
          SlaveTemplate

          {ami='ami-17de70f54e3bad981', labels='windows'}

          . Attempting provision finished, excess workload: 0
          We have now 5 computers, waiting for 1 more
          ```

          and then
          ```
          Started EC2 alive slaves monitor
          EC2 alive slaves monitor. 112 ms
          ```

          and
          ```
          Available capacity=1, currentDemand=1
          Provisioning completedFinished
          ```

          however capacity is 0 cause there is a build in progress already and one in the waiting queue.

          and thus, the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned.

          Later: even when there's no build in progress, I see this message
          `Available capacity=1, currentDemand=1`
          and a new slave is not provisioned when there's a build waiting in the build queue.

          How can I get around this mess?

          ovi craciun added a comment - - edited still an issue with 1.48 I can see this log but the spotinst mentioned is never spawned, no other error: ``` Spot instance id in provision: sir-px3sj14h SlaveTemplate {ami='ami-17de70f54e3bad981', labels='windows'} . Attempting provision finished, excess workload: 0 We have now 5 computers, waiting for 1 more ``` and then ``` Started EC2 alive slaves monitor EC2 alive slaves monitor. 112 ms ``` and ``` Available capacity=1, currentDemand=1 Provisioning completedFinished ``` however capacity is 0 cause there is a build in progress already and one in the waiting queue. and thus, the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned. Later: even when there's no build in progress, I see this message `Available capacity=1, currentDemand=1` and a new slave is not provisioned when there's a build waiting in the build queue. How can I get around this mess?

            francisu Francis Upton
            torpy Chin Godawita
            Votes:
            10 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated: