Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9104

Visual studio builds started by Jenkins fail with "Fatal error C1090" because mspdbsrv.exe gets killed

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • Windows XP, Windows 7 using MSBuild or devenv.exe to build MS Visual Studio Projects

      I run into errors when using a customized build system which uses Visual Studio's devenv.exe under the hood to compile VisualStudio 2005 projects (with VC++ compiler). When starting two parallel builds with Jenkins (on different code base) the second job will always fail with "Fatal error C1090: PDB API call failed, error code '23' : '(" in exactly the same second the first job finishes processing. Running both jobs outside Jenkins does not produce the error.
      This has also been reported for builds executed by MSBuild on the Jenkins user mailing list [1].

      I analysed this issue thoroughly and can track the problem down to the usage of mspdbsrv.exe. This program is automatically spawned when building a VisualStudio project. All Visual Studio instances normally share one common pdb-server which shutdown itself after a idle period (standard is 10 minutes). "It ensures access to .pdb files is properly serialized in parallel builds when multiple instances of the compiler try to access the same .pdb file" [2].
      I assume that Jenkins does a clean up of its build environment when a automatically started job finishes (like as described at http://wiki.jenkins-ci.org/display/JENKINS/Aborting+a+build). I checked mspbsrv.exe with ProcessExplorer and the process indeed has a variable JENKINS_COOKIE/HUDSON_COOKIE set in its environment if started through Jenkins. Killing mspdbsrv.exe while projects are still connected will break compilation.

      Jenkins mustn't kill mspdbsrv.exe to be able to build more than one Visual Studio project at the same time.

      –
      [1] http://jenkins.361315.n4.nabble.com/MSBuild-fatal-errors-when-build-triggered-by-timer-td385181.html
      [2] http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/b1d1bceb-06b6-47ef-a0ea-23ea752e0c4f/

          [JENKINS-9104] Visual studio builds started by Jenkins fail with "Fatal error C1090" because mspdbsrv.exe gets killed

          Code changed in jenkins
          User: Daniel Beck
          Path:
          content/_data/changelogs/weekly.yml
          http://jenkins-ci.org/commit/jenkins.io/0391fcb9b4c957e9e41fde03409de330a3de571d
          Log:
          Remove JENKINS-9104 fix from release to unblock it

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: content/_data/changelogs/weekly.yml http://jenkins-ci.org/commit/jenkins.io/0391fcb9b4c957e9e41fde03409de330a3de571d Log: Remove JENKINS-9104 fix from release to unblock it

          Code changed in jenkins
          User: Daniel Beck
          Path:
          content/_data/changelogs/weekly.yml
          http://jenkins-ci.org/commit/jenkins.io/62409d42a5769cac66337cbd4b5df5754f0e2384
          Log:
          Merge pull request #1522 from daniel-beck/changelog-2.119-amended

          Remove JENKINS-9104 fix from release to unblock it

          Compare: https://github.com/jenkins-infra/jenkins.io/compare/58f029c79331...62409d42a576

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: content/_data/changelogs/weekly.yml http://jenkins-ci.org/commit/jenkins.io/62409d42a5769cac66337cbd4b5df5754f0e2384 Log: Merge pull request #1522 from daniel-beck/changelog-2.119-amended Remove JENKINS-9104 fix from release to unblock it Compare: https://github.com/jenkins-infra/jenkins.io/compare/58f029c79331...62409d42a576

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/hudson/util/ProcessTree.java
          test/src/test/java/hudson/util/ProcessTreeKillerTest.java
          http://jenkins-ci.org/commit/jenkins/3465da4764c322baf4fb5b90651ef6b9bcd409fb
          Log:
          Merge pull request #3419 from dwnusbaum/JENKINS-9104-test-fix

          Fix test failure by cleaning up static state after tests

          Compare: https://github.com/jenkinsci/jenkins/compare/ddbc4bbce7d3...3465da4764c3
          *NOTE:* This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/

          Functionality will be removed from GitHub.com on January 31st, 2019.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/util/ProcessTree.java test/src/test/java/hudson/util/ProcessTreeKillerTest.java http://jenkins-ci.org/commit/jenkins/3465da4764c322baf4fb5b90651ef6b9bcd409fb Log: Merge pull request #3419 from dwnusbaum/ JENKINS-9104 -test-fix Fix test failure by cleaning up static state after tests Compare: https://github.com/jenkinsci/jenkins/compare/ddbc4bbce7d3...3465da4764c3 * NOTE: * This service been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/ Functionality will be removed from GitHub.com on January 31st, 2019.

          Daniel Beck added a comment -

          Jenkins 2.120 contains a fix for the previous problem of the ProcessKillingVeto extension point not working on agents.

          Daniel Beck added a comment - Jenkins 2.120 contains a fix for the previous problem of the ProcessKillingVeto extension point not working on agents.

          John Doe added a comment -

          I'm occasionally getting this error with the latest versions of Jenkins and all the plugins. It started in the recent months, haven't been a problem for a year before that. The problem seems to have NOT been resolved, or possibly re-emerged.

          What can I do, is there a workaround? Sporadic build failures for no reason are super annoying.

          John Doe added a comment - I'm occasionally getting this error with the latest versions of Jenkins and all the plugins. It started in the recent months, haven't been a problem for a year before that. The problem seems to have NOT been resolved, or possibly re-emerged. What can I do, is there a workaround? Sporadic build failures for no reason are super annoying.

          Bill Hoo added a comment -

          Same error with latest Jenkins ver. 2.150.3

          The error is aways occured when running two jobs concurrently on the same agent with VS2015:
          fatal error C1090: PDB API

          Bill Hoo added a comment - Same error with latest Jenkins ver. 2.150.3 The error is aways occured when running two jobs concurrently on the same agent with VS2015: fatal error C1090: PDB API

          John Doe added a comment -

          billhoo, thanks for the tip! I was running VS 2017 (v141 toolset), but there were indeed two simultaneous jobs! So the workaround is to limit this agent to one job at a time. Pity, as it's a pretty powerful multicore server, but it's better than flaky builds.

          John Doe added a comment - billhoo , thanks for the tip! I was running VS 2017 (v141 toolset), but there were indeed two simultaneous jobs! So the workaround is to limit this agent to one job at a time. Pity, as it's a pretty powerful multicore server, but it's better than flaky builds.

          Bill Hoo added a comment -

          vuiletgiraffe, totaly the same, we have many different jobs which use MSVC14 as toolchain, but now we can only perform one build at a time, its a huge waste of mashine resources ;(

          Hope it can be truly solved.

          Bill Hoo added a comment - vuiletgiraffe , totaly the same, we have many different jobs which use MSVC14 as toolchain, but now we can only perform one build at a time, its a huge waste of mashine resources ;( Hope it can be truly solved.

          Andreas Ringlstetter added a comment - - edited

          Solution is still the same, before invoking `msbuild`, set the following environment variables to something unique:

          _MSPDBSRV_ENDPOINT_=<UUID>
          TMP=<Unique Tempdir>
          TEMP=$TMP
          TMPDIR=$TMP

          Once you have done that, you can launch as many parallel MSBuild instances as you like, even mixing different msbuild versions or whatever. They will not interfere in any way. Doing that on a regular base with mixed MSVC12, MSVC14 and MSVC15 toolchains on the same machine, and didn't have any issues since.

          The "official" fix for this problem (trying not to kill the job scheduler) is plain wrong, and causes massive issues. Mostly because MSBuild itself isn't exactly stable either when using the same job server for multiple parallel builds. And if the builds are using different toolchains, a crash is ensured.

          Andreas Ringlstetter added a comment - - edited Solution is still the same, before invoking `msbuild`, set the following environment variables to something unique: _MSPDBSRV_ENDPOINT_=<UUID> TMP=<Unique Tempdir> TEMP=$TMP TMPDIR=$TMP Once you have done that, you can launch as many parallel MSBuild instances as you like, even mixing different msbuild versions or whatever. They will not interfere in any way. Doing that on a regular base with mixed MSVC12, MSVC14 and MSVC15 toolchains on the same machine, and didn't have any issues since. The "official" fix for this problem (trying not to kill the job scheduler) is plain wrong, and causes massive issues. Mostly because MSBuild itself isn't exactly stable either when using the same job server for multiple parallel builds. And if the builds are using different toolchains, a crash is ensured.

          Roman Pickl added a comment - - edited

          I used ext3h's solution:

          https://issues.jenkins-ci.org/browse/JENKINS-9104?focusedCommentId=360603&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-360603

          we solved it like this in a jenkins github multi-branch setup with jenkinsfiles:

                          bat """
                              mkdir tmp
                              set _MSPDBSRV_ENDPOINT_= ${BUILD_TAG}
                              set TMP=${Workspace}\\tmp
                              set TEMP=${Workspace}\\tmp
                              set TMPDIR=${Workspace}\\tmp
                              build.bat
                          """ 

           

          Roman Pickl added a comment - - edited I used ext3h 's solution: https://issues.jenkins-ci.org/browse/JENKINS-9104?focusedCommentId=360603&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-360603 we solved it like this in a jenkins github multi-branch setup with jenkinsfiles: bat """ mkdir tmp set _MSPDBSRV_ENDPOINT_= ${BUILD_TAG} set TMP=${Workspace}\\tmp set TEMP=${Workspace}\\tmp set TMPDIR=${Workspace}\\tmp build.bat """  

            danielweber Daniel Weber
            gordin Christoph VogtlÃĪnder
            Votes:
            71 Vote for this issue
            Watchers:
            94 Start watching this issue

              Created:
              Updated:
              Resolved: