-
Bug
-
Resolution: Unresolved
-
Blocker
-
None
-
Jenkins v2.12 / Ubuntu 14.04.4 x86_64 / Oracle Java 1.80_91-b14
-
Powered by SuggestiMate
Hi,
We've recently started seeing a problem auto-launching EC2 build slaves. The plugin detects that it needs to launch a slave to meet demand and that no current instances are runnign - it just doesn't launch anything.
Manual provisioning works.
Log (some info sanitised):
Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@6a756eca needed by excess workload of 1 units of label 'null' Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Considering launching ami-6c14310f for template Remote Ubuntu 14.04 Docker Machine Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-6c14310f]}, {Name: subnet-id,Values: [subnet-XXXXX]}, {Name: instance.group-id,Values: [sg-XXXXX]}, {Name: key-name,Values: [XXXXX]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],} No existing instance found - but cannot create new instance Jul 08, 2016 3:47:34 PM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1
[JENKINS-36516] EC2 plugin: Plugin fails to automatically launch slaves to meet demand
I've observed the same issue in the latest Jenkins Version (2.10).
Manual provisioning works.
No existing instance found - but cannot create new instance Jul 13, 2016 6:05:33 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 6:05:41 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 6:05:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 6:05:43 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1
Can you try increasing the log level so we get the debug/fine level of logging?
Here you go Francis Upton
INFO: Finished EC2 alive slaves monitor. 402 ms Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 1:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 229 ms Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 1:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 184 ms Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 162 ms Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 200 ms Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 175 ms Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 234 ms Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 182 ms Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 2:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 185 ms Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 155 ms Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 138 ms Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 163 ms Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 167 ms Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 166 ms Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 3:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 169 ms Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 165 ms Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 161 ms Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 163 ms Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 167 ms Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 163 ms Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 4:55:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 157 ms Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:05:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 172 ms Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:15:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 171 ms Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:25:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 175 ms Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:35:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 160 ms Jul 13, 2016 5:39:14 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started Download metadata Jul 13, 2016 5:39:16 AM hudson.model.UpdateSite updateData INFO: Obtained the latest update center data file for UpdateSource default Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tasks.Ant.AntInstaller Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.plugins.gradle.GradleInstaller Jul 13, 2016 5:39:16 AM hudson.model.DownloadService$Downloadable load INFO: Obtained the updated data file for hudson.tools.JDKInstaller Jul 13, 2016 5:39:16 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Download metadata. 1,668 ms Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 13, 2016 5:45:50 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 185 ms Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Terminated EC2 instance (terminated): i-d81e5a5e Jul 13, 2016 5:49:08 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-d81e5a5e Jul 13, 2016 5:52:24 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:52:34 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:36 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:52:44 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:46 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:52:54 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:52:56 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:04 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:06 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:14 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:16 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:24 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:25 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:34 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:35 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:44 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:45 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:53:54 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:53:55 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:53:56 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:54:02 AM jenkins.model.Jenkins$26 run INFO: Restart in 10 seconds Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$26 run SEVERE: Restarting VM as requested by anonymous Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins cleanUp INFO: Stopping Jenkins Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained INFO: Started termination Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins$21 onAttained INFO: Completed termination Jul 13, 2016 5:54:12 AM jenkins.model.Jenkins _cleanUpDisconnectComputers INFO: Starting node disconnection Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpShutdownPluginManager INFO: Stopping plugin manager Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpPersistQueue INFO: Persisting build queue Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins _cleanUpAwaitDisconnects INFO: Waiting for node disconnection completion Jul 13, 2016 5:54:13 AM jenkins.model.Jenkins cleanUp INFO: Jenkins stopped Running from: /usr/share/jenkins/jenkins.war Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Logging initialized @380ms Jul 13, 2016 5:54:13 AM winstone.Logger logInternal INFO: Beginning extraction from war file Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog warn WARNING: Empty contextPath Jul 13, 2016 5:54:13 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: jetty-9.2.z-SNAPSHOT Jul 13, 2016 5:54:14 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet Jenkins home directory: /var/lib/jenkins found at: EnvVars.masterEnvVars.get("JENKINS_HOME") Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started w.@1e125710{/,file:/var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war} Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started ServerConnector@42eded9{HTTP/1.1}{0.0.0.0:8080} Jul 13, 2016 5:54:15 AM org.eclipse.jetty.util.log.JavaUtilLog info INFO: Started @1689ms Jul 13, 2016 5:54:15 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v2.0 running: controlPort=disabled Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization Jul 13, 2016 5:54:15 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins Jul 13, 2016 5:54:15 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider INFO: Initializing Bouncy Castle security provider. Jul 13, 2016 5:54:16 AM jenkins.bouncycastle.api.SecurityProviderInitializer addSecurityProvider INFO: Bouncy Castle security provider initialized. Jul 13, 2016 5:54:20 AM jenkins.InitReactorRunner$1 onAttained INFO: Prepared all plugins Jul 13, 2016 5:54:20 AM hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1 error INFO: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Started all plugins Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Augmented all extensions Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect gscqascheduler113.fourvinternal.com Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect gscqascheduler120.fourvinternal.com Jul 13, 2016 5:54:21 AM hudson.slaves.SlaveComputer tryReconnect INFO: Attempting to reconnect qa04scheduler.fourvinternal.com Jul 13, 2016 5:54:21 AM jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started Download metadata Jul 13, 2016 5:54:22 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Download metadata. 1 ms Jul 13, 2016 5:54:22 AM org.jenkinsci.main.modules.sshd.SSHD start INFO: Started SSHD at port 37928 Jul 13, 2016 5:54:22 AM jenkins.InitReactorRunner$1 onAttained INFO: Completed initialization Jul 13, 2016 5:54:22 AM hudson.WebAppMain$3 run INFO: Jenkins is fully up and running [07/13/16 05:54:24] SSH Launch of gscqascheduler120.fourvinternal.com on 10.187.27.222 completed in 2,567 ms [07/13/16 05:54:26] SSH Launch of gscqascheduler113.fourvinternal.com on 10.187.29.0 completed in 4,673 ms [07/13/16 05:54:26] SSH Launch of qa04scheduler.fourvinternal.com on 10.187.29.100 completed in 4,840 ms Jul 13, 2016 5:56:01 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:04 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:11 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:12 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:21 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:22 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:31 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:32 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:41 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:42 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:43 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:56:51 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:56:52 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Jul 13, 2016 5:57:01 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],} No existing instance found - but cannot create new instance Jul 13, 2016 5:57:03 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1 Launching ami-e8e952ff for template GSC Cloud Slave Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Launching ami-e8e952ff for template GSC Cloud Slave Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 13, 2016 5:57:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-e8e952ff]}, {Name: availability-zone,Values: [us-east-1a]}, {Name: subnet-id,Values: [subnet-6aa49c1d]}, {Name: instance.group-id,Values: [sg-71d9bf16]}, {Name: key-name,Values: [VPC-GreySpark]}, {Name: instance-type,Values: [t2.medium]}, {Name: tag:ChargeLine,Values: [Internal-QA]}, {Name: tag:Lifecycle,Values: [Permanent]}, {Name: tag:Owner,Values: [rsamson@fourv.com"]}],}
Thanks for the additional tracing (it's good to get it from the beginning), but I also need the logging level changed so we get the "fine" tracing as well (not just "info")
I realise this is not much help but strangely enough, it started working for us again. No Jenkins/plugin upgrades have been done since the issue happened.
The immediate set of actions I did before it started working was:
- To get around this issue, we'd provisioned (through the EC2 plugin) an always-running instance (with no idle timeout)
- Today, because we'd run out of space, I deleted it
- In the interim before I'd had a chance to , we had a build get triggered via a commit
- This automatically brought up a new instance as it used to
I'm also trying to get a debug log across but it's a little difficult as I need to lock in a period of time where our master build server is inaccessible.
In the meantime, here's the log from when it started working:
Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Terminated EC2 instance (terminated): i-xxxxxx Jul 28, 2016 11:43:26 AM hudson.plugins.ec2.EC2OndemandSlave terminate INFO: Removed EC2 instance from jenkins master: i-xxxxxx Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@xxx needed by excess workload of 1 units of label 'null' Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.EC2Cloud provision WARNING: Label is null - can't calculate how many executors slave will have. Using 2 number of executors Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Considering launching ami-xxxxxx for template Remote Ubuntu 14.04 Docker Machine Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jul 28, 2016 11:44:24 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-xxxxxx]}, {Name: subnet-id,Values: [subnet-xxxxxx]}, {Name: instance.group-id,Values: [sg-xxxxxx]}, {Name: key-name,Values: [jenkins-slave]}, {Name: instance-type,Values: [t2.small]}, {Name: tag:Name,Values: [Jenkins Build Slave]}],} No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,} Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: No existing instance found - created new instance: {InstanceId: i-xxxxxx,ImageId: ami-xxxxxx,State: {Code: 0,Name: pending},PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,PublicDnsName: ,StateTransitionReason: ,KeyName: jenkins-slave,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: t2.small,LaunchTime: Thu Jul 28 11:44:03 AEST 2016,Placement: {AvailabilityZone: ap-southeast-2a,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,PrivateIpAddress: x.x.x.x,StateReason: {Code: pending,Message: pending},Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [],VirtualizationType: hvm,ClientToken: x-x-x-x-x,Tags: [{Key: jenkins_slave_type,Value: demand_Remote Ubuntu 14.04 Docker Machine}, {Key: Name,Value: Jenkins Build Slave}],SecurityGroups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-xxxxxx,SubnetId: subnet-xxxxxx,VpcId: vpc-xxxxxx,Description: ,OwnerId: xxxxxx,Status: in-use,MacAddress: xxx,PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: jenkins,GroupId: sg-xxxxxx}],Attachment: {AttachmentId: eni-attach-xxxxxx,DeviceIndex: 0,Status: attaching,AttachTime: Thu Jul 28 11:44:03 AEST 2016,DeleteOnTermination: true},PrivateIpAddresses: [{PrivateIpAddress: x.x.x.x,PrivateDnsName: ip-x-x-x-x.ap-southeast-2.compute.internal,Primary: true,}]}],EbsOptimized: false,} Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: We have now 1 computers Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2RetentionStrategy start INFO: Start requested for Remote Ubuntu 14.04 Docker Machine (i-xxxxxx) Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: Added node named: Remote Ubuntu 14.04 Docker Machine (i-xxxxxx), We have now 2 computers Jul 28, 2016 11:44:26 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: -1 Jul 28, 2016 11:44:26 AM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning Remote Ubuntu 14.04 Docker Machine (ami-xxxxxx) from ec2-jenkins-user with 2 executors. Remaining excess workload: -1 Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run INFO: Started EC2 alive slaves monitor Jul 28, 2016 11:44:38 AM hudson.model.AsyncPeriodicWork$1 run INFO: Finished EC2 alive slaves monitor. 55 ms
I am seeing what appear to me to be the same issue on EC2 1.36. After adding an additional EC2 node to a previously working system, auto-re-launch of timed out instances doesn't occur. Manually launching will work. My logs look very similar.
I'm also experiencing this issue with 1.36. I'm using spot instances if that matters. Here's the exception I'm seeing:
Dec 12, 2016 7:02:19 PM hudson.plugins.ec2.EC2Cloud provision WARNING: Exception during provisioning com.amazonaws.SdkClientException: Unable to unmarshall response (Read timed out). Response Code: 200, Response Text: OK at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1468) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1151) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:964) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:676) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:650) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:633) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:601) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:583) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:447) at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:11396) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:11372) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5542) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:5554) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:363) at hudson.plugins.ec2.EC2Cloud.getPossibleNewSlavesCount(EC2Cloud.java:499) at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:518) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:547) at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:701) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:307) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:60) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:798) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.ctc.wstx.exc.WstxIOException: Read timed out at com.ctc.wstx.sr.StreamScanner.throwFromIOE(StreamScanner.java:708) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1086) at com.ctc.wstx.evt.WstxEventReader.peek(WstxEventReader.java:306) at com.amazonaws.transform.StaxUnmarshallerContext.nextEvent(StaxUnmarshallerContext.java:220) at com.amazonaws.services.ec2.model.transform.InstanceStaxUnmarshaller.unmarshall(InstanceStaxUnmarshaller.java:42) at com.amazonaws.services.ec2.model.transform.ReservationStaxUnmarshaller.unmarshall(ReservationStaxUnmarshaller.java:69) at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:49) at com.amazonaws.services.ec2.model.transform.DescribeInstancesResultStaxUnmarshaller.unmarshall(DescribeInstancesResultStaxUnmarshaller.java:31) at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:101) at com.amazonaws.http.StaxResponseHandler.handle(StaxResponseHandler.java:43) at com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1444) ... 28 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) at sun.security.ssl.InputRecord.read(InputRecord.java:509) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:944) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:901) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.conn.LoggingInputStream.read(LoggingInputStream.java:87) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200) at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:191) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181) at java.io.FilterInputStream.read(FilterInputStream.java:107) at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:365) at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:110) at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1023) ... 38 more
Seeing same problem with 1.36
Using regular on demand instances
INFO: Attempting to provision slave from template hudson.plugins.ec2.SlaveTemplate@73450f26 needed by excess workload of 1 units of label 'std' Considering launching ami-39f5e340 for template Standard AMI Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Considering launching ami-39f5e340 for template Standard AMI Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.SlaveTemplate logProvisionInfo INFO: Setting Instance Initiated Shutdown Behavior : ShutdownBehavior.Terminate Looking for existing instances with describe-instance: {InstanceIds: [],Filters: [{Name: image-id,Values: [ami-39f5e340]}, {Name: availability-zone,Values: [us-west-2b]}, , {Name: instance-type,Values: [m4.xlarge]}, {Name: tag:Name,Values: [jenkins-build-slave-std]}, {Name: tag:Environment,Values: [Build]}],} No existing instance found - but cannot create new instance Jun 30, 2017 12:59:33 AM hudson.plugins.ec2.EC2Cloud provision INFO: Attempting provision - finished, excess workload: 1
I'm seeing the similar behavior:
it actually launches the first ec2 slave automatically, but no more after that, no matter how long or how many jobs are in the queue. I can launch them manually from the /computer/ page. The teardowns based on idle timeouts seem to work.
I've upped the global instance cap and the ami instance cap to 255, still the same.
jenkins 2.66, openjdk version "1.8.0_31"
Any suggestions on how to debug this? This might be relevant:
Jul 02, 2017 12:08:49 AM org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener publish WARNING: Error publishing Run queued event. org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher. at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)
WARNING: Error publishing Run queued event. org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher. at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135) at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178) at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.access$300(SyncQueueListener.java:57) at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener$1.run(SyncQueueListener.java:106) Caused by: java.lang.NullPointerException
More information. This problem seems to be very intermittent we had a bunch of builds today and all instances got launched correctly
Plugin v1.36 will launch a slave and successfully run a build for us. Plugin v1.37 launches the EC2 instance, but fails to recognise that it can use it to build.
I am also having similar issues with v1.37 of the plugin. Builds run fine the first time, but after the agent timeout from inactivity, no new builds will start, however, it will work again if I change something in the plugins' profile. Moreover, the instance remains on in the AWS console. It's supposed to terminate after 30 minutes of inactivity.
I see the following errors in the log:
Nov 27, 2017 9:43:58 PM hudson.plugins.ec2.EC2Cloud provision
WARNING: Exception during provisioning
com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 309aa7e7-89e0-42c4-9df5-5b4985222ba5)
I've had this issue after upgrading Jenkins from LTS to Weekly.
But it was "accidentally" fixed by:
- Increased instance cap from 1 to 2 -> then changed back to 1
- Checked AWS console - there was a "zombie" instance which was created by Jenkins, but no longer visible on Nodes page
I was able to correct my issue with the plugin by checking the box for, "Use EC2 instance profile to obtain credentials." With that check, the ec2 agent was able to get new credentials to launch.
We experienced the same issue, the "com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired" error started appearing randomly after launched new nodes ok with the EC2 plugin for a while.
Updating the server configuration would solve the issue, but only temporarily.
After upgrading to Jenkins LTS 2.89.3 with 1.38 EC2 plugin the issue does not happen anymore. Not 100% sure though if the bug is really fixed.
Please try 1.39 of the plugin, particularly if you are upgrading from 1.36 or earlier.
still an issue with 1.48
I can see this log but the spotinst mentioned is never spawned, no other error:
```
Spot instance id in provision: sir-px3sj14h
SlaveTemplate
. Attempting provision finished, excess workload: 0
We have now 5 computers, waiting for 1 more
```
and then
```
Started EC2 alive slaves monitor
EC2 alive slaves monitor. 112 ms
```
and
```
Available capacity=1, currentDemand=1
Provisioning completedFinished
```
however capacity is 0 cause there is a build in progress already and one in the waiting queue.
and thus, the build queue keeps having that one waiting build to pe pushed to a provisioned slave but the slave is never provisioned.
Later: even when there's no build in progress, I see this message
`Available capacity=1, currentDemand=1`
and a new slave is not provisioned when there's a build waiting in the build queue.
How can I get around this mess?
Which version of the ec2 plugin are you using?