Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9104

Visual studio builds started by Jenkins fail with "Fatal error C1090" because mspdbsrv.exe gets killed

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • Windows XP, Windows 7 using MSBuild or devenv.exe to build MS Visual Studio Projects

      I run into errors when using a customized build system which uses Visual Studio's devenv.exe under the hood to compile VisualStudio 2005 projects (with VC++ compiler). When starting two parallel builds with Jenkins (on different code base) the second job will always fail with "Fatal error C1090: PDB API call failed, error code '23' : '(" in exactly the same second the first job finishes processing. Running both jobs outside Jenkins does not produce the error.
      This has also been reported for builds executed by MSBuild on the Jenkins user mailing list [1].

      I analysed this issue thoroughly and can track the problem down to the usage of mspdbsrv.exe. This program is automatically spawned when building a VisualStudio project. All Visual Studio instances normally share one common pdb-server which shutdown itself after a idle period (standard is 10 minutes). "It ensures access to .pdb files is properly serialized in parallel builds when multiple instances of the compiler try to access the same .pdb file" [2].
      I assume that Jenkins does a clean up of its build environment when a automatically started job finishes (like as described at http://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build). I checked mspbsrv.exe with ProcessExplorer and the process indeed has a variable JENKINS_COOKIE/HUDSON_COOKIE set in its environment if started through Jenkins. Killing mspdbsrv.exe while projects are still connected will break compilation.

      Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time.


      [1] http://jenkins.361315.n4.nabble.com/MSBuild-fatal-errors-when-build-triggered-by-timer-td385181.html
      [2] http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/b1d1bceb-06b6-47ef-a0ea-23ea752e0c4f/

          [JENKINS-9104] Visual studio builds started by Jenkins fail with "Fatal error C1090" because mspdbsrv.exe gets killed

          Christoph Vogtländer created issue -

          Mark Waite added a comment - - edited

          You might refer to JENKINS-3105 for a work around or alternative technique. I assume the process tree killer is what is killing the mspdbsrv.exe process, and disabling process tree killer for these jobs may avoid the problem.

          Mark Waite added a comment - - edited You might refer to JENKINS-3105 for a work around or alternative technique. I assume the process tree killer is what is killing the mspdbsrv.exe process, and disabling process tree killer for these jobs may avoid the problem.

          Christoph Vogtländer added a comment - - edited

          Yes, you are right. Thanks a lot for pointing this out. I set BUILD_ID=dontKillMe globally in the Jenkins configuration and can confirm that mspdbsrv.exe is not killed any more which works around the problem.

          Christoph Vogtländer added a comment - - edited Yes, you are right. Thanks a lot for pointing this out. I set BUILD_ID=dontKillMe globally in the Jenkins configuration and can confirm that mspdbsrv.exe is not killed any more which works around the problem.

          changed priority to Minor because an easy work around is available

          Christoph Vogtländer added a comment - changed priority to Minor because an easy work around is available
          Christoph Vogtländer made changes -
          Priority Original: Critical [ 2 ] New: Minor [ 4 ]

          chanti vlad added a comment -

          "Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time."

          Although i will try the indicated workaround, i think this would be good to have it as a feature somewhere, maybe in the node configuration?

          chanti vlad added a comment - "Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time." Although i will try the indicated workaround, i think this would be good to have it as a feature somewhere, maybe in the node configuration?

          Daniel Weber added a comment -

          How about adding a configurable list of process names which shall not be killed by the "process tree killer"?
          The workaround described above leaves mspdbsrv.exe running, but also any other "dangling" processes a build
          might have left behind. I'd still like to use the process tree killer in general, it should just not kill
          mspdbsrv.exe.

          Daniel Weber added a comment - How about adding a configurable list of process names which shall not be killed by the "process tree killer"? The workaround described above leaves mspdbsrv.exe running, but also any other "dangling" processes a build might have left behind. I'd still like to use the process tree killer in general, it should just not kill mspdbsrv.exe.

          Neil Bird added a comment -

          Visual Studio is so ubiquitous that I personally think this could warrant a special case built into Jenkins, but anyway this is what we have just started doing (before calling devenv but after the MSC setup.bat has been called) to work around this (for all Windows builds):

          :: PITA to keep MSPDBSRV alive
          set ORIG_BUILD_ID=%BUILD_ID%
          set BUILD_ID=DoNotKillMe
          start mspdbsrv -start -spawn
          set BUILD_ID=%ORIG_BUILD_ID%
          set ORIG_BUILD_ID=

          It seems, from reading around, that the Jenkins process “tree” killer rummages through the whole process tree looking for processes with the environment variable BUILD_ID set to what Jenkins originally set it to, and killing any it finds. The above temporarily changes that to something else (anything, you could even just blank it), launches mspdbsrv manually (so it has the altered env. var.) and then puts it back (to restore the usual no-resource-leak fix).

          When you run mspdbsrv, it seems to immediately exit if there's already an appropriate version running, so it doesn't matter having multiple jobs trying to kick it off.

          Neil Bird added a comment - Visual Studio is so ubiquitous that I personally think this could warrant a special case built into Jenkins, but anyway this is what we have just started doing (before calling devenv but after the MSC setup.bat has been called) to work around this (for all Windows builds): :: PITA to keep MSPDBSRV alive set ORIG_BUILD_ID=%BUILD_ID% set BUILD_ID=DoNotKillMe start mspdbsrv -start -spawn set BUILD_ID=%ORIG_BUILD_ID% set ORIG_BUILD_ID= It seems, from reading around, that the Jenkins process “tree” killer rummages through the whole process tree looking for processes with the environment variable BUILD_ID set to what Jenkins originally set it to, and killing any it finds. The above temporarily changes that to something else (anything, you could even just blank it), launches mspdbsrv manually (so it has the altered env. var.) and then puts it back (to restore the usual no-resource-leak fix). When you run mspdbsrv, it seems to immediately exit if there's already an appropriate version running, so it doesn't matter having multiple jobs trying to kick it off.

          Daniel Weber added a comment -

          We adopted Neil's workaround, but found a small issue. The problem is that checking if
          mspdbsrv is running does not reset its timeout. Consider this scenario where the compiler
          is first used 5 min after build start:

          00:00 job A starts -> mspdbsrv not yet running -> mspdbsrv is started with the changed BUILD_ID
          01:00 job A finishes (mspdbsrv default timeout of 10 min starts here)
          01:07 job B starts -> mspdbsrv still running -> mspdbsrv is not restarted
          01:10 mspdbsrv timeout elapses, shuts down
          01:12 job B tries to use mspdbsrv for the first time. As mspdbsrv is not running, it starts a new
          instance with unchanged BUILD_ID, which is what we wanted to avoid in the first place.

          I suggest to change the build script as follows

          start mspdbsrv -start -spawn -shutdowntime 2147483647

          This sets the shutdown time to the max value of 2^31-1. This timeout (~68 years) should be long enough

          Daniel Weber added a comment - We adopted Neil's workaround, but found a small issue. The problem is that checking if mspdbsrv is running does not reset its timeout. Consider this scenario where the compiler is first used 5 min after build start: 00:00 job A starts -> mspdbsrv not yet running -> mspdbsrv is started with the changed BUILD_ID 01:00 job A finishes (mspdbsrv default timeout of 10 min starts here) 01:07 job B starts -> mspdbsrv still running -> mspdbsrv is not restarted 01:10 mspdbsrv timeout elapses, shuts down 01:12 job B tries to use mspdbsrv for the first time. As mspdbsrv is not running, it starts a new instance with unchanged BUILD_ID, which is what we wanted to avoid in the first place. I suggest to change the build script as follows start mspdbsrv -start -spawn -shutdowntime 2147483647 This sets the shutdown time to the max value of 2^31-1. This timeout (~68 years) should be long enough

          I would not recommend setting the shutdown time to such a high value. mspdbsrv.exe tends to hang after some time because of handle leaks (at least the version shipped with Visual Studio 2005) causing the compiler to exit with error when trying to access pdb files. mspdbsrv process must be killed manually, then.

          Christoph Vogtländer added a comment - I would not recommend setting the shutdown time to such a high value. mspdbsrv.exe tends to hang after some time because of handle leaks (at least the version shipped with Visual Studio 2005) causing the compiler to exit with error when trying to access pdb files. mspdbsrv process must be killed manually, then.

            danielweber Daniel Weber
            gordin Christoph Vogtländer
            Votes:
            71 Vote for this issue
            Watchers:
            94 Start watching this issue

              Created:
              Updated:
              Resolved: