-
New Feature
-
Resolution: Won't Fix
-
Major
when using kubernetes and jenkins docker workflow, its possible to use docker-in-docker (dind) in a slave or to try share the local docker daemon.
Though ideally it'd be great if using jenkins and kubernetes together (e.g. with Atomic / OpenShift / OpenStack / Google GKE / vanilla kubernetes) that we let kubernetes takes care of provisioning all the docker containers; pulling images and restarting any failed pods if the machine thats running a jenkins workflow has issues (or the pod dies).
To do that nicely on kubernetes we'd need to turn each Docker Workflow script into a Pod; with a docker container to run the main groovy workflow process; then for each container in the
docker.image("foo") {}
block we'd add a container to the pod.
e.g. this workflow
docker.image("maven") { // some stuff } docker.image("nodejs") { // some stuff }
would be turned into a Pod with these containers:
- workflow
- maven
- nodejs
Then the workflow container could then talk directly to the other docker containers using localhost as the Pod would know all the ports of each docker container - and it'd be easy to share the build volume between each container nicely.
One thing to be careful of is that right now in Kubernetes; a Pod definition is static; so rather than imperatively iterating through the Groovy DSL for the workflow; we'd have to have a kind of 'compile' stage where we evaluate all the `docker.image` blocks; then once we know them, we can generate a Pod which has the docker images baked into it which we can then start. Once the pod starts; all the containers would be provisioned together on the same host (and atomically destroyed at the end of the build)
[JENKINS-29342] to use docker workflow nicely on kubernetes we should turn a docker workflow into a Pod/RC
1 container to run the workflow
Does not make sense, Workflow scripts always run on the Jenkins master. Perhaps you are talking about using one container for the Jenkins slave agent corresponding to node.
another container for each docker.image() statement
docker.image(…).inside I suppose you mean.
analyse the jenkins workflow DSL to figure out all the docker images that are going to be required
Impossible. The only way this could work is if Kubernetes supports dynamic additions to a pod.
Yeah - to really use Pods properly on Kubernetes we'd need the master running the workflow to analyse the flow; figure out the pod/containers then create the 'slave' pod and wait for it to finish (and maybe restart it if it dies). If you kinda squint - this is almost like using "docker workflows pods" as a kinda slave in Jenkins (conceptually at least).
It looks like Kubernetes isn't gonna support dynamic additions to a pod any time soon 
FWIW we've just had some success using Docker Workflow with DIND inside Swarm slaves using Jenkernetes on Kubernetes; which at least looks like it might work.
https://github.com/iocanel/jenkins-poc/tree/master/swarm
Its just a shame that all the docker containers inside the Swarm slave pod are kinda invisible from a Kubernetes tooling perspective (web UI / CLI) since they are all inside a docker-in-docker
James Strachan
added a comment - - edited Yeah - to really use Pods properly on Kubernetes we'd need the master running the workflow to analyse the flow; figure out the pod/containers then create the 'slave' pod and wait for it to finish (and maybe restart it if it dies). If you kinda squint - this is almost like using "docker workflows pods" as a kinda slave in Jenkins (conceptually at least).
It looks like Kubernetes isn't gonna support dynamic additions to a pod any time soon
FWIW we've just had some success using Docker Workflow with DIND inside Swarm slaves using Jenkernetes on Kubernetes; which at least looks like it might work.
https://github.com/iocanel/jenkins-poc/tree/master/swarm
Its just a shame that all the docker containers inside the Swarm slave pod are kinda invisible from a Kubernetes tooling perspective (web UI / CLI) since they are all inside a docker-in-docker James Strachan
added a comment - OK - I guess DIND it is then! Thanks for listening James Strachan
added a comment - doesn't seem possible It looks like Kubernetes isn't gonna support dynamic additions to a pod any time soon
Then you cannot use Kubernetes for this purpose, unless you introduce some non-pod-based network filesystem, which is probably unwanted.
Swarm (without Kubernetes) should work, in principle; requires changes in Jenkins plugins (docker + docker-workflow) so that --link-from is specified when running containers for Image.inside to ensure that the slave workspace is shared. Not currently planning this work but I have given it some thought.
James Strachan
added a comment - We've just got Jenkernetes working; which uses a pod per slave with Swarm
https://github.com/GoogleCloudPlatform/jenkernetes/
then using Docker in Docker (DIND) in each pod container so that each slave pod can create other docker containers (all inside the same single slave pod - these containers are invisible to kubernetes). So far its working quite well https://github.com/iocanel/jenkins-poc/tree/master/swarm
After much head scratching trying to figure out how to get docker workflow, kubernetes, jenkins kubernetes plugin and/or jenkins docker plugin; we might have found a nice permutation!
Many thanks!
James Strachan
added a comment - We've just got Jenkernetes working; which uses a pod per slave with Swarm
https://github.com/GoogleCloudPlatform/jenkernetes/
then using Docker in Docker (DIND) in each pod container so that each slave pod can create other docker containers (all inside the same single slave pod - these containers are invisible to kubernetes). So far its working quite well https://github.com/iocanel/jenkins-poc/tree/master/swarm
After much head scratching trying to figure out how to get docker workflow, kubernetes, jenkins kubernetes plugin and/or jenkins docker plugin; we might have found a nice permutation!
Many thanks! Yeah, DIND is one approach to this kind of problem. Does not seem like the right long-term approach, but good enough for now.
James Strachan
added a comment - Agreed 
A bit more background on this issue
https://github.com/fabric8io/fabric8/issues/4340