Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
Windows XP, Windows 7 using MSBuild or devenv.exe to build MS Visual Studio Projects
Description
I run into errors when using a customized build system which uses Visual Studio's devenv.exe under the hood to compile VisualStudio 2005 projects (with VC++ compiler). When starting two parallel builds with Jenkins (on different code base) the second job will always fail with "Fatal error C1090: PDB API call failed, error code '23' : '(" in exactly the same second the first job finishes processing. Running both jobs outside Jenkins does not produce the error.
This has also been reported for builds executed by MSBuild on the Jenkins user mailing list [1].
I analysed this issue thoroughly and can track the problem down to the usage of mspdbsrv.exe. This program is automatically spawned when building a VisualStudio project. All Visual Studio instances normally share one common pdb-server which shutdown itself after a idle period (standard is 10 minutes). "It ensures access to .pdb files is properly serialized in parallel builds when multiple instances of the compiler try to access the same .pdb file" [2].
I assume that Jenkins does a clean up of its build environment when a automatically started job finishes (like as described at http://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build). I checked mspbsrv.exe with ProcessExplorer and the process indeed has a variable JENKINS_COOKIE/HUDSON_COOKIE set in its environment if started through Jenkins. Killing mspdbsrv.exe while projects are still connected will break compilation.
Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time.
–
[1] http://jenkins.361315.n4.nabble.com/MSBuild-fatal-errors-when-build-triggered-by-timer-td385181.html
[2] http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/b1d1bceb-06b6-47ef-a0ea-23ea752e0c4f/
Attachments
Issue Links
- is duplicated by
-
-
- Resolved
-
- is related to
-
-
- Open
-
-
JENKINS-3105 Configuration UI to disable process tree killer selectively
-
- Resolved
-
Activity
John Doe
added a comment - billhoo, thanks for the tip! I was running VS 2017 (v141 toolset), but there were indeed two simultaneous jobs! So the workaround is to limit this agent to one job at a time. Pity, as it's a pretty powerful multicore server, but it's better than flaky builds.
John Doe
added a comment - billhoo , thanks for the tip! I was running VS 2017 (v141 toolset), but there were indeed two simultaneous jobs! So the workaround is to limit this agent to one job at a time. Pity, as it's a pretty powerful multicore server, but it's better than flaky builds. vuiletgiraffe, totaly the same, we have many different jobs which use MSVC14 as toolchain, but now we can only perform one build at a time, its a huge waste of mashine resources ;(
Hope it can be truly solved.
Solution is still the same, before invoking `msbuild`, set the following environment variables to something unique:
_MSPDBSRV_ENDPOINT_=<UUID> TMP=<Unique Tempdir> TEMP=$TMP TMPDIR=$TMP
Once you have done that, you can launch as many parallel MSBuild instances as you like, even mixing different msbuild versions or whatever. They will not interfere in any way. Doing that on a regular base with mixed MSVC12, MSVC14 and MSVC15 toolchains on the same machine, and didn't have any issues since.
The "official" fix for this problem (trying not to kill the job scheduler) is plain wrong, and causes massive issues. Mostly because MSBuild itself isn't exactly stable either when using the same job server for multiple parallel builds. And if the builds are using different toolchains, a crash is ensured.
Andreas Ringlstetter
added a comment - - edited Solution is still the same, before invoking `msbuild`, set the following environment variables to something unique:
_MSPDBSRV_ENDPOINT_=<UUID>
TMP=<Unique Tempdir>
TEMP=$TMP
TMPDIR=$TMP
Once you have done that, you can launch as many parallel MSBuild instances as you like, even mixing different msbuild versions or whatever. They will not interfere in any way. Doing that on a regular base with mixed MSVC12, MSVC14 and MSVC15 toolchains on the same machine, and didn't have any issues since.
The "official" fix for this problem (trying not to kill the job scheduler) is plain wrong, and causes massive issues. Mostly because MSBuild itself isn't exactly stable either when using the same job server for multiple parallel builds. And if the builds are using different toolchains, a crash is ensured. I used ext3h's solution:
we solved it like this in a jenkins github multi-branch setup with jenkinsfiles:
bat """
mkdir tmp
set _MSPDBSRV_ENDPOINT_= ${BUILD_TAG}
set TMP=${Workspace}\\tmp
set TEMP=${Workspace}\\tmp
set TMPDIR=${Workspace}\\tmp
build.bat
"""
Roman Pickl
added a comment - - edited I used ext3h 's solution:
https://issues.jenkins-ci.org/browse/JENKINS-9104?focusedCommentId=360603&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-360603
we solved it like this in a jenkins github multi-branch setup with jenkinsfiles:
bat """
mkdir tmp
set _MSPDBSRV_ENDPOINT_= ${BUILD_TAG}
set TMP=${Workspace}\\tmp
set TEMP=${Workspace}\\tmp
set TMPDIR=${Workspace}\\tmp
build.bat
"""
Same error with latest Jenkins ver. 2.150.3
The error is aways occured when running two jobs concurrently on the same agent with VS2015:
fatal error C1090: PDB API