-
Improvement
-
Resolution: Unresolved
-
Minor
-
None
-
CentOS 6.8,
Jenkins 2.141
Docker-Plugin 1.1.5
Changing Cloud configuration while there are existing containers seems to really screw things up. I changed the docker host URI (to a different host) and changed the Container Cap.
As a result, on the new host, I had a disagreement between the Container Cap and the actual container count. For example, I had the following combination at one point:
- Containers running: 1
- Container Cap: 7 (I kept increasing it, but it only helped temporarily)
- Log shows:
Sep 17, 2018 7:16:35 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Not provisioning additional slaves for container&¢os&&mem32; we have 3 executors being started already
This did not change in 3 days (Jenkins just kept printing this in the log), so I hacked my way into clearing the count of containers in progress in a jenkins script console: (I have only one Docker cloud)
Jenkins.getInstance().clouds.get(com.nirima.jenkins.plugins.docker.DockerCloud).CONTAINERS_IN_PROGRESS.clear()
And now I'm getting the right count of provisioned containers!
Incidentally, the content of "CONTAINERS_IN_PROGRESS" was a map with two keys (old and new Docker host), new one was set to 3 and the old one was at 7...
I'd seen anecdotal reports of this kind of issue before ... but none mentioning "Changing Cloud configuration while there are existing containers seems to really screw things up" - I think you've found the underlying cause there
That information makes this kind of bug a lot easier to fix...