Details
-
Type:
Bug
-
Status: Open (View Workflow)
-
Priority:
Major
-
Resolution: Unresolved
-
Component/s: external-resource-dispatcher-plugin
-
Labels:None
-
Similar Issues:
Description
I frequently see two jobs using the same resource, whereas only one job has actually locked the resource. These jobs all have a SCM configured.
I suspect that the locking occurs too late (only after the check out). So, while the first job is updating the workspace the resource is not locked yet and the other job thinks he can use that same resource.
Attachments
Activity
Running Core Jenkins 1.651
Externeral Dispatcher: 1.1.0
Just started hitting this issue just recently as we are expanding our pipelines and resources. From reviewing the code (very briefly), I don't see anywhere that would prevent multiple threads obtaining the resource lock at the same time. Following the code from https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/ExternalResourceQueueTaskDispatcher.java#L113 to https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/utils/resourcemanagers/ResourceMonitorExternalResourceManager.java#L203 there appear to be multiple places that could have mutual exclusion problems. Perhaps the job dispatcher in Jenkins runs in a single thread; but I would have to review the architecture to be certain and from what is being observed this may not be the case.
The really unfortunate part of this and any mutual exclusion problem is I turned on full logging to observe the behavior and it went away. I'm going to fork this off and see if we can propose a possible fix.
Unless, the Queue handler in Jenkins is itself running in a single thread.
Jonathan Strickland
added a comment - - edited Running Core Jenkins 1.651
Externeral Dispatcher: 1.1.0
Just started hitting this issue just recently as we are expanding our pipelines and resources. From reviewing the code (very briefly), I don't see anywhere that would prevent multiple threads obtaining the resource lock at the same time. Following the code from https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/ExternalResourceQueueTaskDispatcher.java#L113 to https://github.com/jenkinsci/external-resource-dispatcher-plugin/blob/master/src/main/java/com/sonyericsson/jenkins/plugins/externalresource/dispatcher/utils/resourcemanagers/ResourceMonitorExternalResourceManager.java#L203 there appear to be multiple places that could have mutual exclusion problems. Perhaps the job dispatcher in Jenkins runs in a single thread; but I would have to review the architecture to be certain and from what is being observed this may not be the case.
The really unfortunate part of this and any mutual exclusion problem is I turned on full logging to observe the behavior and it went away. I'm going to fork this off and see if we can propose a possible fix.
Unless, the Queue handler in Jenkins is itself running in a single thread.
I solved this by increasing the reservation time in the External Resource Dispatcher Configuration and limit the checkout time (shorter than the reservation time of the jobs that ). But this is only workaround of course.
I guess that the actually locking of the resources happens too late: now it happens in the prebuild method of JobProperty whereas it should happen earlier (in the onStarted method of RunListener).