-
Bug
-
Resolution: Unresolved
-
Major
-
None
It looks as if there is a scenario where vSphereCloud.java Hashtable currentOnline does not get updated as it should.
The nodes will work as they should, then it will fail to launch new nodes.
From the looks of it the method vSphereCloud.canMarkVMOnline() return FALSE on 2nd or 3rd if statement.
I had a quick look and I think the problem is in the vSphereCloudLauncher.postDisconnectVSphereActions() method.
The vSphereCloud vsC = findOurVsInstance(); throws an exception just before the markVMOffline() is called.
It will then never mark the VM as offline and you can never launch a node on that VM again until you restart Jenkins.
[JENKINS-24479] The vSphere cloud will not allow this slave to start at this time.
Since we started running into this we've been restarting Jenkins once or twice per week to clear this state, but I haven't found an explicit trigger condition
D Pasto
added a comment - - edited Since we started running into this we've been restarting Jenkins once or twice per week to clear this state, but I haven't found an explicit trigger condition 
pjdarton
added a comment - antoniek, wgc123 The code that monitors what (cloud) slaves are up/down/starting/stopped etc was significantly re-written a while ago - is this still an issue for you, or is it fixed now?
pjdarton
added a comment - antoniek , wgc123 The code that monitors what (cloud) slaves are up/down/starting/stopped etc was significantly re-written a while ago - is this still an issue for you, or is it fixed now? 
Antonie Kriek
added a comment - We are in the process of upgrading to a new environment and the latest version of Jenkins. We wont be using the vSphere cloud plugin on our new environment. You can close the issue.
Antonie Kriek
added a comment - We are in the process of upgrading to a new environment and the latest version of Jenkins. We wont be using the vSphere cloud plugin on our new environment. You can close the issue. Issue occuring again. Jenkins is latest LTS version, with latest plugins installed.
The node in question was relabeled to fix a typo, and then this error prevents restart.
pjdarton
added a comment - So, just to clarify: You edited the node's configuration and changed just the labels, and then it all went wrong?
As a workaround, have you tried configuring "cloud" templates (manage jenkins -> configure system, scroll down to clouds, select the vSphere cloud and then add template definitions there)? That's what we use at work and I'm fairly sure I've ironed out all the bumps on that functionality, whereas I've never used manually defined/edited nodes (that functionality pre-dates my involvement with this plugin).
pjdarton
added a comment - So, just to clarify: You edited the node's configuration and changed just the labels, and then it all went wrong?
As a workaround, have you tried configuring "cloud" templates (manage jenkins -> configure system, scroll down to clouds, select the vSphere cloud and then add template definitions there)? That's what we use at work and I'm fairly sure I've ironed out all the bumps on that functionality, whereas I've never used manually defined/edited nodes (that functionality pre-dates my involvement with this plugin). 
Artur Szostak
added a comment - We are using Jenkins 2.121.3 and vsphere-cloud-plugin 2.18.
Getting "ERROR: The vSphere cloud will not allow this slave to start at this time." randomly for our Jenkins + vSphere cloud setup. Have a single vSphere cloud configured with about 10 different templates (configured automatically with a Groovy script), each template is limited to a max of 50 instances. The total instance count for the cloud is limited to 100. We see between 5% - 30% VMs do not start properly with the above error message in the log.
I would say this shows classic symptoms of a race condition in the plugin / Jenkins code.
Artur Szostak
added a comment - We are using Jenkins 2.121.3 and vsphere-cloud-plugin 2.18.
Getting "ERROR: The vSphere cloud will not allow this slave to start at this time." randomly for our Jenkins + vSphere cloud setup. Have a single vSphere cloud configured with about 10 different templates (configured automatically with a Groovy script), each template is limited to a max of 50 instances. The total instance count for the cloud is limited to 100. We see between 5% - 30% VMs do not start properly with the above error message in the log.
I would say this shows classic symptoms of a race condition in the plugin / Jenkins code. 
Did this ever go anywhere? We may have just run into this issue, possibly triggered by a storage issue in vSphere. We got some failed startups due to the storage issue, but after clearing that out, slaves would still not start. The only thing in the slave log was "ERROR: The vSphere cloud will not allow this slave to start at this time.". As in this bug report, restarting Jenkins cleared it up