-
Bug
-
Resolution: Fixed
-
Minor
-
Jenkins 2.73.1, EC2 Plugin 1.37
With the EC2 plugin version 1.37, existing stopped instances are not started again during provisioning with the following message in the Jenkins log when the instance cap is reached:
Cannot provision - no capacity for instances: -1
reverting to version 1.36 fixes the issue.
- is related to
-
-
- Resolved
-
-
-
- Closed
-
[JENKINS-47130] EC2 plugin 1.37 fails to provision previously defined slaves
Francis Upton
added a comment - vineus thanks for the helpful comment. This is clearly a regression and we are looking into the cause so that you don't have to delete the configuration. We will make a new release as soon as we have the fix.
Francis Upton
added a comment - vineus thanks for the helpful comment. This is clearly a regression and we are looking into the cause so that you don't have to delete the configuration. We will make a new release as soon as we have the fix. 
Vincent de Lagabbe
added a comment - ok perfect! In the meantime re creating the configuration just does the job, so it's not that critical
Keep up the fantastic work!
Vincent de Lagabbe
added a comment - ok perfect! In the meantime re creating the configuration just does the job, so it's not that critical
Keep up the fantastic work! vineus I tried to reproduce this issue but I couldn't. Until then, the workaround seems to be what you suggest at least for now.
Vasco Pinho
added a comment - Also having this problem, after upgrading the plugin, stopped AWS instances are started in AWS but the agents do not report as "online" unless someone clicks the "launch agent" button in the node.
Here's some logs:
Vasco Pinho
added a comment - Also having this problem, after upgrading the plugin, stopped AWS instances are started in AWS but the agents do not report as "online" unless someone clicks the "launch agent" button in the node.
Here's some logs:
https://pastebin.com/hbDarKNC Im pretty sure it was this pull request : https://github.com/jenkinsci/ec2-plugin/pull/238/commits/23fb932a7e9d8ec5590ad9061e398fb5622c9a39 which is causing instances to not start. Im not sure what would need that logic to be inverted, but there wasnt any tests added or updated.
Not that at all.
Ryan Aslett
added a comment - - edited Im pretty sure it was this pull request : https://github.com/jenkinsci/ec2-plugin/pull/238/commits/23fb932a7e9d8ec5590ad9061e398fb5622c9a39 which is causing instances to not start. Im not sure what would need that logic to be inverted, but there wasnt any tests added or updated.
Not that at all. Encountered the same issue as described on JENKINS-46869. Not just for stopped instances but any newly launched instances are not registered. The nodes are visible on the AWS EC2 console but do not show up on the Jenkins "Nodes" overview.
Did not see the "Cannot provision - no capacity for instances: -1" error in jenkins.log.
The issue was fixed by deleting the EC2 Cloud configuration and re-entering it.
Plugin version: 1.37
Jenkins version: 2.73.2
marc young
added a comment - For the time being Ive just not upgraded to 1.37, JENKINS-46869 requires more effort than is gained to recreate all my configuration
marc young
added a comment - For the time being Ive just not upgraded to 1.37, JENKINS-46869 requires more effort than is gained to recreate all my configuration I just ran into this issue myself after weeks of avoiding upgrading to 1.37 (I forgot today). I went to recreate my cloud and things got messy real fast so I just changed one item (in my case switching from T2Large to T2Micro) and saved the configuration. This was enough to allow the plugin to start my agents up successfully.
Vishal K
added a comment - krachynski , thanks for info. That trick worked for me as well. Francis Upton
added a comment - It's a bug in 1.37 that things don't work when there is an upgrade. I need to reproduce this so I can see what went into 1.37 to break this.
Francis Upton
added a comment - It's a bug in 1.37 that things don't work when there is an upgrade. I need to reproduce this so I can see what went into 1.37 to break this. I failed to record my configuration before I made that change so I'm not sure what the plugin writes on save that fixes it.
Further comment. I have a failing agent right now because the 'root command prefix' which I attempted to set as 'sudo' is not being saved in the configuration. It rather looks like I need to uninstall the entire thing and start over now.
Ryan Smith
added a comment - I upgraded from 1.36 to 1.38, and nodes stopped automatically launching. I was able to manually trigger node launches by going to Nodes and using the "Provision via" drop down.
Reading the note above from Ken Rachynski on 2017-11-24 12:38, I changed the EC2 ami configs from m3.large to m3.xlarge, saved, and then changed them back to m3.large, saved. And immediately following that instances started auto launching and taking work.
Ryan Smith
added a comment - I upgraded from 1.36 to 1.38, and nodes stopped automatically launching. I was able to manually trigger node launches by going to Nodes and using the "Provision via" drop down.
Reading the note above from Ken Rachynski on 2017-11-24 12:38, I changed the EC2 ami configs from m3.large to m3.xlarge, saved, and then changed them back to m3.large, saved. And immediately following that instances started auto launching and taking work. Francis Upton
added a comment - I see the problem here. This was caused by the "node" field that was added to the SlaveTemplate https://github.com/jenkinsci/ec2-plugin/pull/232. The initial value of "node" was set to "true", allowing it to be provisioned normally. However, any serialized SlaveTemplates would (incorrectly) have the value of "false" which caused them to be ignored during the normal provisioning. The work of the PR did not consider this case.
Francis Upton
added a comment - I see the problem here. This was caused by the "node" field that was added to the SlaveTemplate https://github.com/jenkinsci/ec2-plugin/pull/232 . The initial value of "node" was set to "true", allowing it to be provisioned normally. However, any serialized SlaveTemplates would (incorrectly) have the value of "false" which caused them to be ignored during the normal provisioning. The work of the PR did not consider this case.
Code changed in jenkins
User: Francis Upton IV
Path:
src/main/java/hudson/plugins/ec2/EC2Cloud.java
src/main/java/hudson/plugins/ec2/EC2Step.java
src/main/java/hudson/plugins/ec2/SlaveTemplate.java
src/test/java/hudson/plugins/ec2/EC2StepTest.java
http://jenkins-ci.org/commit/ec2-plugin/a80c336bfc12ee28828981bdceae88842daabb07
Log:
JENKINS-50105 EC2 Step provisioning incorrectly specifies a label
JENKINS-47130 EC2 plugin 1.37 fails to provision previously defined slaves
Code changed in jenkins
User: Francis Upton IV
Path:
src/main/java/hudson/plugins/ec2/EC2Cloud.java
src/main/java/hudson/plugins/ec2/EC2Step.java
src/main/java/hudson/plugins/ec2/SlaveTemplate.java
src/test/java/hudson/plugins/ec2/EC2StepTest.java
http://jenkins-ci.org/commit/ec2-plugin/affb7f407cd024accdf4e9093b07efada117814d
Log:
JENKINS-50105 EC2 Step provisioning incorrectly specifies a label (#268)
JENKINS-47130 EC2 plugin 1.37 fails to provision previously defined slaves
Deleting the existing Jenkins EC2 plugin configuration (instances setup and cloud config) and re creating it seems to fix the problem