Jenkins Server get blue screen to death crash about twice a day. After sending the dump to microsoft, it turns out there The Java related binary D:\Jenkins_CI\war\WEB-INF\lib\winp.x64.35A3F35D2ED629A6CEC5B41DD1D280C3.dll has called to Terminate the csrss.exe and csrss.exe is a critical process since it was called to Terminate this process the machine Bug Checked.
      What is CSRSS.EXE?
      This is the user-mode portion of the Win32 subsystem (with Win32.sys being the kernel-mode portion). Csrss stands for client/server run-time subsystem and is an essential subsystem that must be running at all times. Csrss is responsible for console windows, creating and/or deleting threads, and some parts of the 16-bit virtual MS-DOS environment

          [JENKINS-24453] Jenkins server got blue screen to death

          sharon xia created issue -

          sharon xia added a comment -

          System Properties
          Name ↓ Value
          awt.toolkit sun.awt.windows.WToolkit
          executable-war D:\Jenkins_CI\jenkins.war
          file.encoding Cp1252
          file.encoding.pkg sun.io
          file.separator \
          hudson.diyChunking true
          hudson.lifecycle hudson.lifecycle.WindowsServiceLifecycle
          java.awt.graphicsenv sun.awt.Win32GraphicsEnvironment
          java.awt.headless true
          java.awt.printerjob sun.awt.windows.WPrinterJob
          java.class.path D:\Jenkins_CI\jenkins.war
          java.class.version 51.0
          java.endorsed.dirs C:\Program Files\Java\jre7\lib\endorsed
          java.ext.dirs C:\Program Files\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext
          java.home C:\Program Files\Java\jre7
          java.io.tmpdir C:\Users\ADMINI~1\AppData\Local\Temp\
          java.library.path C:\Program Files\Java\jre7\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jre7\bin;C:\Python27;C:\Program Files (x86)\Git\bin;.
          java.runtime.name Java(TM) SE Runtime Environment
          java.runtime.version 1.7.0_60-b19
          java.specification.name Java Platform API Specification
          java.specification.vendor Oracle Corporation
          java.specification.version 1.7
          java.vendor Oracle Corporation
          java.vendor.url http://java.oracle.com/
          java.vendor.url.bug http://bugreport.sun.com/bugreport/
          java.version 1.7.0_60
          java.vm.info mixed mode
          java.vm.name Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.name Java Virtual Machine Specification
          java.vm.specification.vendor Oracle Corporation
          java.vm.specification.version 1.7
          java.vm.vendor Oracle Corporation
          java.vm.version 24.60-b09
          javamelody.analytics-id UA-1335263-7
          javamelody.gzip-compression-disabled true
          javamelody.http-transform-pattern /\d+/|/site/.|avadoc/.|/ws/.|obertura/.|estReport/.|iolations/file/.|/user/.|/static/\w/
          javamelody.no-database true
          javamelody.storage-directory /D:\Jenkins_CI\monitoring
          javamelody.system-actions-enabled true
          line.separator
          mail.smtp.sendpartial true
          mail.smtps.sendpartial true
          os.arch amd64
          os.name Windows Server 2008 R2
          os.version 6.1
          path.separator ;
          pid 1340
          sun.arch.data.model 64
          sun.awt.enableExtraMouseButtons true
          sun.boot.class.path C:\Program Files\Java\jre7\lib\resources.jar;C:\Program Files\Java\jre7\lib\rt.jar;C:\Program Files\Java\jre7\lib\sunrsasign.jar;C:\Program Files\Java\jre7\lib\jsse.jar;C:\Program Files\Java\jre7\lib\jce.jar;C:\Program Files\Java\jre7\lib\charsets.jar;C:\Program Files\Java\jre7\lib\jfr.jar;C:\Program Files\Java\jre7\classes
          sun.boot.library.path C:\Program Files\Java\jre7\bin
          sun.cpu.endian little
          sun.cpu.isalist amd64
          sun.desktop windows
          sun.io.unicode.encoding UnicodeLittle
          sun.java.command D:\Jenkins_CI\jenkins.war --httpPort=8080 --httpsPort=443 --httpsKeyStore=D:\Jenkins_CI\jre\bin\jenkins.jks --httpsKeyStorePassword=******
          sun.java.launcher SUN_STANDARD
          sun.jnu.encoding Cp1252
          sun.management.compiler HotSpot 64-Bit Tiered Compilers
          sun.os.patch.level Service Pack 1
          svnkit.http.methods Digest,Basic,NTLM,Negotiate
          svnkit.ssh2.persistent false
          svnkit.symlinks false
          user.country US
          user.dir D:\Jenkins_CI
          user.home C:\Users\Administrator
          user.language en
          user.name Administrator
          user.script
          user.timezone America/New_York
          user.variant
          Environment Variables
          Name ↓ Value
          ALLUSERSPROFILE C:\ProgramData
          APPDATA C:\Users\Administrator\AppData\Roaming
          BASE D:\Jenkins_CI
          CommonProgramFiles C:\Program Files\Common Files
          CommonProgramFiles(x86) C:\Program Files (x86)\Common Files
          CommonProgramW6432 C:\Program Files\Common Files
          COMPUTERNAME WIN2K8-R2-0
          ComSpec C:\Windows\system32\cmd.exe
          FP_NO_HOST_CHECK NO
          JAVA_HOME C:\Program Files (x86)\Java\jre7\bin
          JENKINS_HOME D:\Jenkins_CI
          LOCALAPPDATA C:\Users\Administrator\AppData\Local
          NUMBER_OF_PROCESSORS 12
          OS Windows_NT
          Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jre7\bin;C:\Python27;C:\Program Files (x86)\Git\bin
          PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
          PROCESSOR_ARCHITECTURE AMD64
          PROCESSOR_IDENTIFIER Intel64 Family 6 Model 44 Stepping 2, GenuineIntel
          PROCESSOR_LEVEL 6
          PROCESSOR_REVISION 2c02
          ProgramData C:\ProgramData
          ProgramFiles C:\Program Files
          ProgramFiles(x86) C:\Program Files (x86)
          ProgramW6432 C:\Program Files
          PSModulePath C:\Windows\system32\WindowsPowerShell\v1.0\Modules\
          PUBLIC C:\Users\Public
          SystemDrive C:
          SystemRoot C:\Windows
          TEMP C:\Users\ADMINI~1\AppData\Local\Temp
          TMP C:\Users\ADMINI~1\AppData\Local\Temp
          USERDOMAIN WIN2K8-R2-0
          USERNAME Administrator
          USERPROFILE C:\Users\Administrator
          windir C:\Windows
          windows_tracing_flags 3
          windows_tracing_logfile C:\BVTBin\Tests\installpackage\csilogfile.log
          Plugins
          Name ↓ Version Enabled Pinned
          active-directory 1.22 true false
          ant 1.2 true false
          antisamy-markup-formatter 1.0 true false
          artifactory 1.1.0 false false
          async-http-client 1.7.8 true false
          backup 1.6 true false
          build-name-setter 1.0 true false
          build-timeout 1.12.2 true false
          buildgraph-view 1.1 true false
          cloudbees-credentials 3.2 false false
          cloudbees-deployer-plugin 4.3 false false
          cloudbees-plugin-gateway 4.1 false false
          cloudbees-registration 3.11 false false
          cloudbees-wasted-minutes-tracker 3.5 false false
          clover 3.0.2 false false
          cobertura 1.9.3 true false
          conditional-buildstep 1.3.3 true false
          copy-to-slave 1.2.1 true false
          copyartifact 1.14 true false
          coverity 1.2.7 true false
          cppcheck 1.14 true false
          credentials 1.14 true true
          cron_column 1.003 true false
          cvs 2.11 true false
          dashboard-view 2.0 true false
          dependencyanalyzer 0.6 true false
          deployed-on-column 1.7 true false
          disk-usage 0.21 true false
          downstream-buildview 1.8 true false
          downstream-ext 1.6 true false
          doxygen 0.6 true false
          email-ext 2.38.1 true false
          emotional-hudson 1.2 false false
          envinject 1.89 true false
          extended-choice-parameter 0.4 true false
          external-monitor-job 1.2 true false
          files-found-trigger 1.1.3 true false
          flow 1.2 true false
          gallio 1.6 true false
          gcal 0.3 false false
          git 2.2.1 true false
          git-client 1.9.1 true false
          greenballs 1.10 true false
          groovy 1.4 true false
          htmlpublisher 1.3 true false
          javadoc 1.1 true false
          jenkins-multijob-plugin 1.10 true false
          jenkinswalldisplay 0.5.9 true false
          job-dsl 1.16 true false
          jobConfigHistory 1.5 true false
          jobtype-column 1.0 false false
          join 1.9 true false
          ldap 1.10.2 true true
          locks-and-latches 0.6 true false
          log-parser 1.0.4 false false
          mailer 1.6 true false
          matrix-auth 1.1 true false
          maven-plugin 2.1 true false
          monitoring 1.43.0 true false
          multiple-scms 0.3 true false
          nodenamecolumn 1.1 true false
          nunit 0.15 true false
          pam-auth 1.1 true false
          parameterized-trigger 2.12 true false
          platformlabeler 1.1 true false
          plot 1.7 true false
          postbuild-task 1.8 true false
          PrioritySorter 1.1 true false
          rebuild 1.14 true false
          regexemail 0.3 true false
          regression-report-plugin 1.1 true false
          run-condition 1.0 true false
          scm-api 0.2 true false
          screenshot 1.1 true false
          sectioned-view 1.12 true false
          sidebar-link 1.5 true false
          sonar 2.1 true false
          sonargraph-plugin 1.2 true false
          sounds 0.2 false false
          speaks 0.1.1 false false
          ssh-credentials 1.7.1 true true
          ssh-slaves 0.14 true true
          status-view 1.0 true false
          statusmonitor 1.2 true false
          subversion 1.23 true true
          svn-release-mgr 1.2 true false
          svn-tag 1.14 true false
          test-stability 1.0 true false
          testInProgress 1.1 true false
          testlink 2.3.1 true false
          timestamper 1.2.2 true false
          token-macro 1.10 true false
          trac 1.10 true false
          translation 1.10 true false
          valgrind 0.20 true false
          violations 0.7.15-SNAPSHOT (private-01/22/2014 20:25-tanakd1) true false
          vsphere-cloud 1.1.10 false false
          windows-slaves 1.0 true false
          xunit 1.80 false false
          Thread Dumps

          Visit this page for master and slave thread dumps.
          Help us localize this page

          sharon xia added a comment - System Properties Name ↓ Value awt.toolkit sun.awt.windows.WToolkit executable-war D:\Jenkins_CI\jenkins.war file.encoding Cp1252 file.encoding.pkg sun.io file.separator \ hudson.diyChunking true hudson.lifecycle hudson.lifecycle.WindowsServiceLifecycle java.awt.graphicsenv sun.awt.Win32GraphicsEnvironment java.awt.headless true java.awt.printerjob sun.awt.windows.WPrinterJob java.class.path D:\Jenkins_CI\jenkins.war java.class.version 51.0 java.endorsed.dirs C:\Program Files\Java\jre7\lib\endorsed java.ext.dirs C:\Program Files\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext java.home C:\Program Files\Java\jre7 java.io.tmpdir C:\Users\ADMINI~1\AppData\Local\Temp\ java.library.path C:\Program Files\Java\jre7\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jre7\bin;C:\Python27;C:\Program Files (x86)\Git\bin;. java.runtime.name Java(TM) SE Runtime Environment java.runtime.version 1.7.0_60-b19 java.specification.name Java Platform API Specification java.specification.vendor Oracle Corporation java.specification.version 1.7 java.vendor Oracle Corporation java.vendor.url http://java.oracle.com/ java.vendor.url.bug http://bugreport.sun.com/bugreport/ java.version 1.7.0_60 java.vm.info mixed mode java.vm.name Java HotSpot(TM) 64-Bit Server VM java.vm.specification.name Java Virtual Machine Specification java.vm.specification.vendor Oracle Corporation java.vm.specification.version 1.7 java.vm.vendor Oracle Corporation java.vm.version 24.60-b09 javamelody.analytics-id UA-1335263-7 javamelody.gzip-compression-disabled true javamelody.http-transform-pattern /\d+/|/site/. |avadoc/. |/ws/. |obertura/. |estReport/. |iolations/file/. |/user/. |/static/\w / javamelody.no-database true javamelody.storage-directory /D:\Jenkins_CI\monitoring javamelody.system-actions-enabled true line.separator mail.smtp.sendpartial true mail.smtps.sendpartial true os.arch amd64 os.name Windows Server 2008 R2 os.version 6.1 path.separator ; pid 1340 sun.arch.data.model 64 sun.awt.enableExtraMouseButtons true sun.boot.class.path C:\Program Files\Java\jre7\lib\resources.jar;C:\Program Files\Java\jre7\lib\rt.jar;C:\Program Files\Java\jre7\lib\sunrsasign.jar;C:\Program Files\Java\jre7\lib\jsse.jar;C:\Program Files\Java\jre7\lib\jce.jar;C:\Program Files\Java\jre7\lib\charsets.jar;C:\Program Files\Java\jre7\lib\jfr.jar;C:\Program Files\Java\jre7\classes sun.boot.library.path C:\Program Files\Java\jre7\bin sun.cpu.endian little sun.cpu.isalist amd64 sun.desktop windows sun.io.unicode.encoding UnicodeLittle sun.java.command D:\Jenkins_CI\jenkins.war --httpPort=8080 --httpsPort=443 --httpsKeyStore=D:\Jenkins_CI\jre\bin\jenkins.jks --httpsKeyStorePassword=****** sun.java.launcher SUN_STANDARD sun.jnu.encoding Cp1252 sun.management.compiler HotSpot 64-Bit Tiered Compilers sun.os.patch.level Service Pack 1 svnkit.http.methods Digest,Basic,NTLM,Negotiate svnkit.ssh2.persistent false svnkit.symlinks false user.country US user.dir D:\Jenkins_CI user.home C:\Users\Administrator user.language en user.name Administrator user.script user.timezone America/New_York user.variant Environment Variables Name ↓ Value ALLUSERSPROFILE C:\ProgramData APPDATA C:\Users\Administrator\AppData\Roaming BASE D:\Jenkins_CI CommonProgramFiles C:\Program Files\Common Files CommonProgramFiles(x86) C:\Program Files (x86)\Common Files CommonProgramW6432 C:\Program Files\Common Files COMPUTERNAME WIN2K8-R2-0 ComSpec C:\Windows\system32\cmd.exe FP_NO_HOST_CHECK NO JAVA_HOME C:\Program Files (x86)\Java\jre7\bin JENKINS_HOME D:\Jenkins_CI LOCALAPPDATA C:\Users\Administrator\AppData\Local NUMBER_OF_PROCESSORS 12 OS Windows_NT Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Java\jre7\bin;C:\Python27;C:\Program Files (x86)\Git\bin PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC PROCESSOR_ARCHITECTURE AMD64 PROCESSOR_IDENTIFIER Intel64 Family 6 Model 44 Stepping 2, GenuineIntel PROCESSOR_LEVEL 6 PROCESSOR_REVISION 2c02 ProgramData C:\ProgramData ProgramFiles C:\Program Files ProgramFiles(x86) C:\Program Files (x86) ProgramW6432 C:\Program Files PSModulePath C:\Windows\system32\WindowsPowerShell\v1.0\Modules\ PUBLIC C:\Users\Public SystemDrive C: SystemRoot C:\Windows TEMP C:\Users\ADMINI~1\AppData\Local\Temp TMP C:\Users\ADMINI~1\AppData\Local\Temp USERDOMAIN WIN2K8-R2-0 USERNAME Administrator USERPROFILE C:\Users\Administrator windir C:\Windows windows_tracing_flags 3 windows_tracing_logfile C:\BVTBin\Tests\installpackage\csilogfile.log Plugins Name ↓ Version Enabled Pinned active-directory 1.22 true false ant 1.2 true false antisamy-markup-formatter 1.0 true false artifactory 1.1.0 false false async-http-client 1.7.8 true false backup 1.6 true false build-name-setter 1.0 true false build-timeout 1.12.2 true false buildgraph-view 1.1 true false cloudbees-credentials 3.2 false false cloudbees-deployer-plugin 4.3 false false cloudbees-plugin-gateway 4.1 false false cloudbees-registration 3.11 false false cloudbees-wasted-minutes-tracker 3.5 false false clover 3.0.2 false false cobertura 1.9.3 true false conditional-buildstep 1.3.3 true false copy-to-slave 1.2.1 true false copyartifact 1.14 true false coverity 1.2.7 true false cppcheck 1.14 true false credentials 1.14 true true cron_column 1.003 true false cvs 2.11 true false dashboard-view 2.0 true false dependencyanalyzer 0.6 true false deployed-on-column 1.7 true false disk-usage 0.21 true false downstream-buildview 1.8 true false downstream-ext 1.6 true false doxygen 0.6 true false email-ext 2.38.1 true false emotional-hudson 1.2 false false envinject 1.89 true false extended-choice-parameter 0.4 true false external-monitor-job 1.2 true false files-found-trigger 1.1.3 true false flow 1.2 true false gallio 1.6 true false gcal 0.3 false false git 2.2.1 true false git-client 1.9.1 true false greenballs 1.10 true false groovy 1.4 true false htmlpublisher 1.3 true false javadoc 1.1 true false jenkins-multijob-plugin 1.10 true false jenkinswalldisplay 0.5.9 true false job-dsl 1.16 true false jobConfigHistory 1.5 true false jobtype-column 1.0 false false join 1.9 true false ldap 1.10.2 true true locks-and-latches 0.6 true false log-parser 1.0.4 false false mailer 1.6 true false matrix-auth 1.1 true false maven-plugin 2.1 true false monitoring 1.43.0 true false multiple-scms 0.3 true false nodenamecolumn 1.1 true false nunit 0.15 true false pam-auth 1.1 true false parameterized-trigger 2.12 true false platformlabeler 1.1 true false plot 1.7 true false postbuild-task 1.8 true false PrioritySorter 1.1 true false rebuild 1.14 true false regexemail 0.3 true false regression-report-plugin 1.1 true false run-condition 1.0 true false scm-api 0.2 true false screenshot 1.1 true false sectioned-view 1.12 true false sidebar-link 1.5 true false sonar 2.1 true false sonargraph-plugin 1.2 true false sounds 0.2 false false speaks 0.1.1 false false ssh-credentials 1.7.1 true true ssh-slaves 0.14 true true status-view 1.0 true false statusmonitor 1.2 true false subversion 1.23 true true svn-release-mgr 1.2 true false svn-tag 1.14 true false test-stability 1.0 true false testInProgress 1.1 true false testlink 2.3.1 true false timestamper 1.2.2 true false token-macro 1.10 true false trac 1.10 true false translation 1.10 true false valgrind 0.20 true false violations 0.7.15-SNAPSHOT (private-01/22/2014 20:25-tanakd1) true false vsphere-cloud 1.1.10 false false windows-slaves 1.0 true false xunit 1.80 false false Thread Dumps Visit this page for master and slave thread dumps. Help us localize this page

          Daniel Beck added a comment -

          Please specify your Jenkins version.

          Daniel Beck added a comment - Please specify your Jenkins version.

          Daniel Beck added a comment -

          Also, it helps if you're not running Jenkins as Local System or a similarly privileged user.

          Daniel Beck added a comment - Also, it helps if you're not running Jenkins as Local System or a similarly privileged user.

          sharon xia added a comment -

          Jenkins Version: Jenkins ver. 1.554.3

          sharon xia added a comment - Jenkins Version: Jenkins ver. 1.554.3

          sharon xia added a comment -

          Actually we are running Jenkins as Local System account "Administrator", does it matter? Will this trigger this issue?

          sharon xia added a comment - Actually we are running Jenkins as Local System account "Administrator", does it matter? Will this trigger this issue?

          sharon xia added a comment -

          Do you mean if we do not run it as Local System then we can get more debugging information or it will trigger this issue? For now I have to move the winp.x64.35A3F35D2ED629A6CEC5B41DD1D280C3.dll out of the WEN-INF/lib folder, however I am not sure what plugin or what function of jenkins will rely on it

          sharon xia added a comment - Do you mean if we do not run it as Local System then we can get more debugging information or it will trigger this issue? For now I have to move the winp.x64.35A3F35D2ED629A6CEC5B41DD1D280C3.dll out of the WEN-INF/lib folder, however I am not sure what plugin or what function of jenkins will rely on it

          Daniel Beck added a comment -

          If you run Jenkins as an unprivileged user it doesn't get to kill csrss, so yes, that should be a workaround.

          winp is used as a process killer, usually however only for processes that are started in your builds. This is designed as regular "clean up" after a build finishes.

          Daniel Beck added a comment - If you run Jenkins as an unprivileged user it doesn't get to kill csrss, so yes, that should be a workaround. winp is used as a process killer, usually however only for processes that are started in your builds. This is designed as regular "clean up" after a build finishes.

          sharon xia added a comment -

          However when I installed the newest LTS version of jenkins it does not have this dll in it. Do you know which plugin/version might use it?
          Does it work if I upgrade my jenkins to the latest one 1.556.1?

          Also, By default the jenkins will run as Local System and we didn't have this issue before for several years. Why it turns out only recently?

          sharon xia added a comment - However when I installed the newest LTS version of jenkins it does not have this dll in it. Do you know which plugin/version might use it? Does it work if I upgrade my jenkins to the latest one 1.556.1? Also, By default the jenkins will run as Local System and we didn't have this issue before for several years. Why it turns out only recently?

          Daniel Beck added a comment -

          winp is used by Jenkins core, only indirectly by plugins.

          There have been a few changes to winp/process killing recently. As I don't use Windows I cannot say for sure what the current status is, generally, killing processes has always been a bit brittle/prone to error. In this case, the fix for JENKINS-22685 comes to mind as a potential candidate.

          Daniel Beck added a comment - winp is used by Jenkins core, only indirectly by plugins. There have been a few changes to winp/process killing recently. As I don't use Windows I cannot say for sure what the current status is, generally, killing processes has always been a bit brittle/prone to error. In this case, the fix for JENKINS-22685 comes to mind as a potential candidate.

            Unassigned Unassigned
            sharon_xia sharon xia
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: