-
Bug
-
Resolution: Unresolved
-
Major
-
CloudBees Jenkins Enterprise 2.89.4.2-rolling
Pipeline: Groovy (workflow-cps): 2.47
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
P4 Plugin (p4): 1.7.3-DRE1.25
Operating System: CentOS release 6.7 (Final)
Java version: java version "1.8.0_131" , Java(TM) SE Runtime Environment (build 1.8.0_131-b11)CloudBees Jenkins Enterprise 2.89.4.2-rolling Pipeline: Groovy (workflow-cps): 2.47 Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9 P4 Plugin (p4): 1.7.3-DRE1.25 Operating System: CentOS release 6.7 (Final) Java version: java version "1.8.0_131" , Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
We're encountering an issue where multiple jobs syncing a groovy shared library concurrently results in the wrong sync root being used for some of the jobs. For example, job1 and job2 are triggered at the same time and use the p4 plugin for syncing the groovy shared library. job1 will have the correct workspace root (e.g. /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME) for the groovy shared library sync, but job2 will sync the groovy shared library to the same workspace root as job1 (e.g. /var/lib/jenkins/jobs/job1/workspace%40libs/LIBRARYNAME). When job2 loads the groovy shared library, it references what the workspace root should be for job2 (e.g. /var/lib/jenkins/jobs/job2/workspace%40libs/LIBRARYNAME), which was not sync'd to, and causes job2 to run based off of an old groovy shared library sync.
[JENKINS-50975] Concurrent Groovy Shared Library syncs on different jobs use same workspace root
Paul Allen
added a comment - Resolved in 1.8.12 Paul Allen
made changes -
| Assignee | New: Paul Allen [ p4paul ] | |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |
Sven Schubert
made changes -
| Attachment | New: error_logs.txt [ 44723 ] |
Sven Schubert
made changes -
| Comment |
[ This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5: We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ([^error_logs.txt]). If the jobs are not started at the same time, everything works fine. Environment: * Pipeline: 2.5 * Pipeline: Groovy (workflow-cps): 2.54 * Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9 ] |
Sven Schubert
added a comment - This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5: We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace (error_logs.txt
).
If the jobs are not started at the same time, everything works fine.
Environment:
- Pipeline: 2.5
- Pipeline: Groovy (workflow-cps): 2.54
- Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
Sven Schubert
added a comment - This bug still exists with version 1.8.14 of the P4 plugin, and Pipeline 2.5: We have one shared library configured in Jenkins. If the Jenkins tries to start multiple jobs (A, B, C) at the same time (e.g. because an upstream project has been built and multiple jobs depend on this project), only the first job (A) will be able to load the shared library. The other jobs (B, C) get confused about the workspace when trying to load the shared library, e.g. job B tries to find the shared library (pipeline-commons) within the workspace of another job and not within his own workspace ( error_logs.txt ).
If the jobs are not started at the same time, everything works fine.
Environment:
Pipeline: 2.5
Pipeline: Groovy (workflow-cps): 2.54
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.9
Sven Schubert
made changes -
| Resolution | Original: Fixed [ 1 ] | |
| Status | Original: Closed [ 6 ] | New: Reopened [ 4 ] |
I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922
https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/
I may need to add ${EXECUTOR_NUMBER} to the Global Library Workspace name formatter:
In file GlobalLibraryScmScurce.java:43
setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id);
setFormat("jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id);
Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?
(Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER)
Paul Allen
added a comment - - edited I have recently made changes to Global Library workspaces for multiple instances JENKINS-53922
https://ci.jenkins.io/job/Plugins/job/p4-plugin/job/master/281/
I may need to add ${EXECUTOR_NUMBER } to the Global Library Workspace name formatter:
In file GlobalLibraryScmScurce.java:43
setFormat( "jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id);
setFormat( "jenkins-lib-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}" + id);
Did you want to try this out and build it yourself or I can submit a change for you to try on the nightly build?
(Just re-reading your comment and logs - Build 281 might be enough without the EXECUTOR_NUMBER) 
Prem Gangana
added a comment - Hi p4paul,
I have tested this with the build 281. It is using
"jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id
as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there.
However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.
Prem Gangana
added a comment - Hi p4paul ,
I have tested this with the build 281. It is using
"jenkins-lib-${NODE_NAME}-${JOB_NAME}" + id
as client name now, but i would suggest replacing / with _ in id instead of . (dot). So this is unique across different jobs due to JOB_NAME in there.
However, the issue will surface if a given job has two builds triggered at / almost at the same time, since the jenkins appends @<number> to the build workspace, and so the client root dirs will be different. I would suggest adding WORKSPACE base name some where in there.
stuartrowe mentioned it could be due to https://github.com/jenkinsci/workflow-cps-global-lib-plugin/blob/e080b501af942c15325fd17f974bec9cc04142c2/src/main/java/org/jenkinsci/plugins/workflow/libs/SCMSourceRetriever.java , specifically the doRetrive method. It could also be a knock-on from https://issues.jenkins-ci.org/browse/JENKINS-40408