-
New Feature
-
Resolution: Unresolved
-
Minor
-
None
The idleMinutes feature is nice since it will help jobs sometimes avoid pod creation delays but other strategies would also be very beneficial.
Pod creation delays can be time consuming, approx 30-60 seconds depending on pod image size. The creation delay is often several minutes when using Kubernetes horizontal node auto-scaling. Nodes typically take several minutes to come online. The combined delay can often mean that it takes 5 minutes for a job to be started.
idleMinutes set to a long value can help retain pods but you might retain too much capacity and prevent k8 cluster scale down.
idleMinutes set to a short value increases the chance to a job may encounter pod creation delays.
Web servers such as Apache's HTTPD have a minimum spares concept where they will maintain a number of pre-created processes so that incoming requests can be handled without delay.
If implemented for the Jenkins Kubernetes plugin the minimumSpare would need to play nicely with both idleMinutes and cluster concurrency limits. Worker pods are would still be retained for idleMinutes and after that time they would be released if doing so would not decrease the spare pods below minimumSpare. When starting up the kubernetes plugin the minimumSpare numver of worker pods would be proactively brought online.
The Jenkins Amazon EC2 plugin supports this concept using a configuration option called minimumNumberOfSpareInstances
In the meantime what I'm doing is to set the "Time in minutes to retain agent when idle" to a short value such as 5 minutes and have a Job scheduled to run a small number multiple concurrent instances to make sure a couple pods always hang around.
I'm using the Parameterized Scheduler plugin so that I can cause multiple executions of the same job to run at the same time. I also had to add a bash script with a "sleep 1" in it to make the job executions last long enough that they get scheduled across different pods. I'm running the job at the same frequency as the idle timer will make sure that my desired number of pods always hang around. Its not perfect but its better than having the idle time set to a huge value.