-
Bug
-
Resolution: Fixed
-
Minor
-
None
-
Coverity Plugin 1.1.3
Coverity Version 5.5.3 (though this happens on the previous version as well)
Jenkins 1.447.1 (LTS Jenkins Enterprise)
10.04 Ubuntu LTS 1 64 bit
I'm encountering a recurring issue when running Jenkins builds that utilize the Coverity Plugin. I have 8 Jobs that use Coverity and 4 executors to run them on. Occasionally 2 Jobs execute on the same executor and I get an error in Coverity:
[send_data] $ /usr/local/coverity-integrity-center/static-analysis/bin/cov-build --dir /var/lib/jenkins/coverity/temp-8397632612055929334.tmp --config /var/lib/jenkins/coverity/gcc-config.xml /bin/sh -xe /tmp/hudson9177997551113715078.sh
/tmp/.covlk: File exists
Error creating or opening lock file for: /var/lib/jenkins/coverity/temp-8397632612055929334.tmp/c/emit
Cannot protect emit DB creation.
Coverity seems to create some kind of lock when cov-build is running to prevent multiple instances running at a time. I suppose there could be several good reasons for doing this (licensing, any kind of common generated data, etc.). I'm wondering if maybe the plugin should have it's own lock when cov-build is being executed to prevent multiple Jobs on an executor from running cov-build at the same time.
It seems like a pretty trivial change, so I'll try it out, but I don't know if it's the right kind of fix. Perhaps this problem is unique to our coverity install. Thought I'd see if anyone else has encountered this as well.
Thanks!
Andy
[JENKINS-15630] Build using Coverity Plugin fails because /tmp/.covlk file exists (Multiple jobs on executor)
Code changed in jenkins
User: Andrew Tolbert
Path:
src/main/java/jenkins/plugins/coverity/CoverityLauncherDecorator.java
http://jenkins-ci.org/commit/coverity-plugin/93b65fe7f0536abd435af745bafc08ffe5c82464
Log:
JENKINS-15630 Build using Coverity Plugin fails because /tmp/.covlk file exists (Multiple jobs on executor)
Code changed in jenkins
User: Josh Vinson
Path:
src/main/java/jenkins/plugins/coverity/CoverityLauncherDecorator.java
http://jenkins-ci.org/commit/coverity-plugin/017076d80ae5702bf5c1658a400afc7628a9c72c
Log:
Merge pull request #2 from weak/JENKINS-15630
JENKINS-15630 Build using Coverity Plugin fails because /tmp/.covlk file exists (Multiple jobs on executor)
Compare: https://github.com/jenkinsci/coverity-plugin/compare/de84e772a195...017076d80ae5
Andrew Tolbert
added a comment - Unfortunately it looks like even after my attempted fix this can still possibly happen. There's a small window between when the code makes the .covlk check and when cov-build checks this file that it could be created again. I think this fix is still good, but perhaps there needs to be a Executor-level lock that prevents multiple builds using Coverity from executing at the same time.
Does anyone have any experience with running multiple Jobs with Coverity at the same time on the same Executor?
Andrew Tolbert
added a comment - Unfortunately it looks like even after my attempted fix this can still possibly happen. There's a small window between when the code makes the .covlk check and when cov-build checks this file that it could be created again. I think this fix is still good, but perhaps there needs to be a Executor-level lock that prevents multiple builds using Coverity from executing at the same time.
Does anyone have any experience with running multiple Jobs with Coverity at the same time on the same Executor? Code changed in jenkins
User: Josh Vinson
Path:
src/main/java/jenkins/plugins/coverity/CoverityLauncherDecorator.java
http://jenkins-ci.org/commit/coverity-plugin/64e49f4c29ac773ef926ee54683d133272805aaf
Log:
Revert "JENKINS-15630 Build using Coverity Plugin fails because /tmp/.covlk file exists (Multiple jobs on executor)"
This reverts commit 93b65fe7f0536abd435af745bafc08ffe5c82464.
Josh Vinson
added a comment - I reverted your fix because while it looked good initially, I ran into some issues running the same build twice. Checking /tmp/.covlk is a good start, but it's not fine-grained enough to avoid just preventing all builds on a machine after the first one (which is what I encountered). This is because covlk stores all locks for coverity, and individual file locks are contained inside. Waiting for the whole directory to be gone fails even when the specific file lock relevant to this bug is removed.
The issue you encountered looks like it's due to the same intermediate directory being used by two builds, which is something else to work out.
Josh Vinson
added a comment - I reverted your fix because while it looked good initially, I ran into some issues running the same build twice. Checking /tmp/.covlk is a good start, but it's not fine-grained enough to avoid just preventing all builds on a machine after the first one (which is what I encountered). This is because covlk stores all locks for coverity, and individual file locks are contained inside. Waiting for the whole directory to be gone fails even when the specific file lock relevant to this bug is removed.
The issue you encountered looks like it's due to the same intermediate directory being used by two builds, which is something else to work out. 
I submitted a pull request here: https://github.com/jenkinsci/coverity-plugin/pull/2. I'm not sure if this is an ideal solution, but it solves my problem.