Search

Context: I use Jenkins to run test automation, we have multiple jobs that will run concurrently on limited resources (VM's). We use Freestyle projects to get this done. There are 16 jobs that must be run nightly but we have only 6 VMs. I am trying to use resource locking to ensure these 16 jobs don't clobber each other, however I am not having much success.

Problem: Even though I have all resources defined and in the lock pool using a label, when it comes to executing these Jenkins will execute a job on a resource it did not request a lock for. Example: I have VM A, B, and C. When The jobs get queued (we'll say 6 jobs for this example) they will request to lock a resource. For the first three jobs, they lock and execute correctly on VM's ABC. However, when one of the three VM's finishes running the job, the first job in queue will be executed on it regardless of what resource it locked when it originally queued. So let's say VM A finished first, but the next queued job had already locked VM C, the queued job will execute on VM A regardless of the lock it reserved for VM C.

This is playing havoc with our test automation, as the resources will execute on the incorrect VM, taking up an critical executor slot on the VM, and causes a deadlock for resources.

Steps to Repro: 

1. Create a project that contains 3 lockable Resources that are VM's
2. Create 6 Jobs, each of the jobs we use have 3 basic build steps:
   1. Launches a job called Automation Setup that sets up our test VM env
   2. Runs TestExecute (the actual automation suite we use) on the test VM
   3. Launches a job called Automation Teardown that pushes artifacts around and cleans the test VM env for the next run
3. Queue all jobs and watch as they execute

Actual Result:

Queued jobs launch on different resource than what it locked\reserved when queued

Expected Result:

The queued job should wait for the resource that it requested a lock for and be served that resource in chronological order based on when the lock was requested. i.e. if two jobs requested the lock on the same resource, then the one that requested first gets it and the second one will wait for the resource to free up.