Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24453

Jenkins server got blue screen to death

    XMLWordPrintable

Details

    Description

      Jenkins Server get blue screen to death crash about twice a day. After sending the dump to microsoft, it turns out there The Java related binary D:\Jenkins_CI\war\WEB-INF\lib\winp.x64.35A3F35D2ED629A6CEC5B41DD1D280C3.dll has called to Terminate the csrss.exe and csrss.exe is a critical process since it was called to Terminate this process the machine Bug Checked.
      What is CSRSS.EXE?
      This is the user-mode portion of the Win32 subsystem (with Win32.sys being the kernel-mode portion). Csrss stands for client/server run-time subsystem and is an essential subsystem that must be running at all times. Csrss is responsible for console windows, creating and/or deleting threads, and some parts of the 16-bit virtual MS-DOS environment

      Attachments

        Issue Links

          Activity

            p55p Peter p55p added a comment - - edited

            Hello, we are facing similar issue as described in this thread. Based on comments I thought the issue was solved in version 1.580.1 but we are running version 2.19.1 and we got the strange behavior also there.
            One of our Windows boxes resulted in BSOD. When we inspected memory dump we found following details.

            Debugging Details:
            ------------------

            DUMP_CLASS: 1

            DUMP_QUALIFIER: 401

            BUILD_VERSION_STRING: 7601.23572.amd64fre.win7sp1_ldr.161011-0600

            SYSTEM_MANUFACTURER: VMware, Inc.

            VIRTUAL_MACHINE: VMware

            SYSTEM_PRODUCT_NAME: VMware Virtual Platform

            SYSTEM_VERSION: None

            BIOS_VENDOR: Phoenix Technologies LTD

            BIOS_VERSION: 6.00

            BIOS_DATE: 09/17/2015

            BASEBOARD_MANUFACTURER: Intel Corporation

            BASEBOARD_PRODUCT: 440BX Desktop Reference Platform

            BASEBOARD_VERSION: None

            DUMP_TYPE: 1

            BUGCHECK_P1: 3

            BUGCHECK_P2: fffffa8182e4c2a0

            BUGCHECK_P3: fffffa8182e4c580

            BUGCHECK_P4: fffff80001994b70

            PROCESS_NAME: csrss.exe

            CRITICAL_PROCESS: csrss.exe

            IMAGE_NAME: csrss.exe

            DEBUG_FLR_IMAGE_TIMESTAMP: 0

            MODULE_NAME: csrss

            FAULTING_MODULE: 0000000000000000

            EXCEPTION_CODE: (HRESULT) 0x855ee060 (2237587552) - <Unable to get error code text>

            ERROR_CODE: (NTSTATUS) 0x855ee060 - <Unable to get error code text>

            CPU_COUNT: 4

            CPU_MHZ: 960

            CPU_VENDOR: GenuineIntel

            CPU_FAMILY: 6

            CPU_MODEL: f

            CPU_STEPPING: 1

            CPU_MICROCODE: 6,f,1,0 (F,M,S,R) SIG: 428'00000000 (cache) 428'00000000 (init)

            DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

            BUGCHECK_STR: 0xF4

            CURRENT_IRQL: 0

            ANALYSIS_SESSION_HOST: NVPC100

            ANALYSIS_SESSION_TIME: 03-01-2017 12:25:49.0246

            ANALYSIS_VERSION: 10.0.14321.1024 amd64fre

            STACK_TEXT:
            fffff880`069abb18 fffff800`01a1e852 : 00000000`000000f4 00000000`00000003 fffffa81`82e4c2a0 fffffa81`82e4c580 : nt!KeBugCheckEx
            fffff880`069abb20 fffff800`019dc09b : 00000000`00000001 fffffa81`855ee060 fffffa81`82e4c2a0 fffffa81`83d68b01 : nt!PspCatchCriticalBreak+0x92
            fffff880`069abb60 fffff800`01945454 : 00000000`00000001 00000000`00000328 fffffa81`82e4c2a0 fffffa81`00000008 : nt! ?? ::NNGAKEGL::`string'+0x27296
            fffff880`069abbb0 fffff800`01689693 : 00000000`00000328 fffffa81`855ee060 fffffa81`82e4c2a0 00000000`00000328 : nt!NtTerminateProcess+0x284
            fffff880`069abc20 00000000`776cbffa : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
            00000000`1ac8e008 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x776cbffa

            STACK_COMMAND: kb

            FOLLOWUP_NAME: MachineOwner

            FAILURE_BUCKET_ID: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_jenkins-slave.exe_855ee060

            BUCKET_ID: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_jenkins-slave.exe_855ee060

            PRIMARY_PROBLEM_CLASS: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_jenkins-slave.exe_855ee060

            TARGET_TIME: 2017-03-01T09:52:59.000Z

            OSBUILD: 7601

            OSSERVICEPACK: 1000

            SERVICEPACK_NUMBER: 0

            OS_REVISION: 0

            SUITE_MASK: 400

            PRODUCT_TYPE: 3

            OSPLATFORM_TYPE: x64

            OSNAME: Windows 7

            OSEDITION: Windows 7 Server (Service Pack 1) TerminalServer DataCenter SingleUserTS

            OS_LOCALE:

            USER_LCID: 0

            OSBUILD_TIMESTAMP: 2016-10-11 16:57:55

            BUILDDATESTAMP_STR: 161011-0600

            BUILDLAB_STR: win7sp1_ldr

            BUILDOSVER_STR: 6.1.7601.23572.amd64fre.win7sp1_ldr.161011-0600

            ANALYSIS_SESSION_ELAPSED_TIME: 9e0

            ANALYSIS_SOURCE: KM

            FAILURE_ID_HASH_STRING: km:x64_0xf4_csrss.exe_bugcheck_critical_process_terminated_by_jenkins-slave.exe_855ee060

            FAILURE_ID_HASH:

            {795c17b0-7869-c608-f0a3-7708c7ee022c}

            Followup: MachineOwner

            p55p Peter p55p added a comment - - edited Hello, we are facing similar issue as described in this thread. Based on comments I thought the issue was solved in version 1.580.1 but we are running version 2.19.1 and we got the strange behavior also there. One of our Windows boxes resulted in BSOD. When we inspected memory dump we found following details. Debugging Details: ------------------ DUMP_CLASS: 1 DUMP_QUALIFIER: 401 BUILD_VERSION_STRING: 7601.23572.amd64fre.win7sp1_ldr.161011-0600 SYSTEM_MANUFACTURER: VMware, Inc. VIRTUAL_MACHINE: VMware SYSTEM_PRODUCT_NAME: VMware Virtual Platform SYSTEM_VERSION: None BIOS_VENDOR: Phoenix Technologies LTD BIOS_VERSION: 6.00 BIOS_DATE: 09/17/2015 BASEBOARD_MANUFACTURER: Intel Corporation BASEBOARD_PRODUCT: 440BX Desktop Reference Platform BASEBOARD_VERSION: None DUMP_TYPE: 1 BUGCHECK_P1: 3 BUGCHECK_P2: fffffa8182e4c2a0 BUGCHECK_P3: fffffa8182e4c580 BUGCHECK_P4: fffff80001994b70 PROCESS_NAME: csrss.exe CRITICAL_PROCESS: csrss.exe IMAGE_NAME: csrss.exe DEBUG_FLR_IMAGE_TIMESTAMP: 0 MODULE_NAME: csrss FAULTING_MODULE: 0000000000000000 EXCEPTION_CODE: (HRESULT) 0x855ee060 (2237587552) - <Unable to get error code text> ERROR_CODE: (NTSTATUS) 0x855ee060 - <Unable to get error code text> CPU_COUNT: 4 CPU_MHZ: 960 CPU_VENDOR: GenuineIntel CPU_FAMILY: 6 CPU_MODEL: f CPU_STEPPING: 1 CPU_MICROCODE: 6,f,1,0 (F,M,S,R) SIG: 428'00000000 (cache) 428'00000000 (init) DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT BUGCHECK_STR: 0xF4 CURRENT_IRQL: 0 ANALYSIS_SESSION_HOST: NVPC100 ANALYSIS_SESSION_TIME: 03-01-2017 12:25:49.0246 ANALYSIS_VERSION: 10.0.14321.1024 amd64fre STACK_TEXT: fffff880`069abb18 fffff800`01a1e852 : 00000000`000000f4 00000000`00000003 fffffa81`82e4c2a0 fffffa81`82e4c580 : nt!KeBugCheckEx fffff880`069abb20 fffff800`019dc09b : 00000000`00000001 fffffa81`855ee060 fffffa81`82e4c2a0 fffffa81`83d68b01 : nt!PspCatchCriticalBreak+0x92 fffff880`069abb60 fffff800`01945454 : 00000000`00000001 00000000`00000328 fffffa81`82e4c2a0 fffffa81`00000008 : nt! ?? ::NNGAKEGL::`string'+0x27296 fffff880`069abbb0 fffff800`01689693 : 00000000`00000328 fffffa81`855ee060 fffffa81`82e4c2a0 00000000`00000328 : nt!NtTerminateProcess+0x284 fffff880`069abc20 00000000`776cbffa : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 00000000`1ac8e008 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x776cbffa STACK_COMMAND: kb FOLLOWUP_NAME: MachineOwner FAILURE_BUCKET_ID: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_jenkins-slave.exe_855ee060 BUCKET_ID: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_jenkins-slave.exe_855ee060 PRIMARY_PROBLEM_CLASS: X64_0xF4_csrss.exe_BUGCHECK_CRITICAL_PROCESS_TERMINATED_BY_jenkins-slave.exe_855ee060 TARGET_TIME: 2017-03-01T09:52:59.000Z OSBUILD: 7601 OSSERVICEPACK: 1000 SERVICEPACK_NUMBER: 0 OS_REVISION: 0 SUITE_MASK: 400 PRODUCT_TYPE: 3 OSPLATFORM_TYPE: x64 OSNAME: Windows 7 OSEDITION: Windows 7 Server (Service Pack 1) TerminalServer DataCenter SingleUserTS OS_LOCALE: USER_LCID: 0 OSBUILD_TIMESTAMP: 2016-10-11 16:57:55 BUILDDATESTAMP_STR: 161011-0600 BUILDLAB_STR: win7sp1_ldr BUILDOSVER_STR: 6.1.7601.23572.amd64fre.win7sp1_ldr.161011-0600 ANALYSIS_SESSION_ELAPSED_TIME: 9e0 ANALYSIS_SOURCE: KM FAILURE_ID_HASH_STRING: km:x64_0xf4_csrss.exe_bugcheck_critical_process_terminated_by_jenkins-slave.exe_855ee060 FAILURE_ID_HASH: {795c17b0-7869-c608-f0a3-7708c7ee022c} Followup: MachineOwner
            oleg_nenashev Oleg Nenashev added a comment -

            Yes, there is a known issue with WinP: https://github.com/kohsuke/winp/issues/22 . This fix has been released in winp-1.24 and then integrated into Jenkins 2.34: https://github.com/jenkinsci/jenkins/commit/63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385.

            According to the code the fix has been also backported to Jenkins 2.32.1 though changelogs do not say it explicitly. I will make sure that changelogs get updated.

            The recommendation is to update the core to the latest LTS

            oleg_nenashev Oleg Nenashev added a comment - Yes, there is a known issue with WinP: https://github.com/kohsuke/winp/issues/22 . This fix has been released in winp-1.24 and then integrated into Jenkins 2.34: https://github.com/jenkinsci/jenkins/commit/63c2f6c5d7d154a3a0f58c54f04f9b1a25ea5385 . According to the code the fix has been also backported to Jenkins 2.32.1 though changelogs do not say it explicitly. I will make sure that changelogs get updated. The recommendation is to update the core to the latest LTS
            p55p Peter p55p added a comment -

            Thank you Oleg for your quick response. We will do the upgrade and will let you know if the issue reoccur.

            p55p Peter p55p added a comment - Thank you Oleg for your quick response. We will do the upgrade and will let you know if the issue reoccur.

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            content/_data/changelogs/lts.yml
            content/_partials/changelog-weekly.html
            http://jenkins-ci.org/commit/jenkins.io/1e74ef665f37621c5d7fda4c0f5a42e3e4462a59
            Log:
            JENKINS-24453 - Noting fix in Jenkins 2.34 and 2.32.1

            It was a missing fix, since I didn't notice there is a Jenkins JIRA issue for it.
            This PR updates the Weekly release entry and also adds the missing entry for the Stable release

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: content/_data/changelogs/lts.yml content/_partials/changelog-weekly.html http://jenkins-ci.org/commit/jenkins.io/1e74ef665f37621c5d7fda4c0f5a42e3e4462a59 Log: JENKINS-24453 - Noting fix in Jenkins 2.34 and 2.32.1 It was a missing fix, since I didn't notice there is a Jenkins JIRA issue for it. This PR updates the Weekly release entry and also adds the missing entry for the Stable release

            Code changed in jenkins
            User: R. Tyler Croy
            Path:
            content/_data/changelogs/lts.yml
            content/_partials/changelog-weekly.html
            http://jenkins-ci.org/commit/jenkins.io/89b2c800c392195aca397baac4ceb91af08ad7f4
            Log:
            Merge pull request #723 from oleg-nenashev/changelog/JENKINS-24453

            JENKINS-24453 - Noting fix in Jenkins 2.34 and 2.32.1

            Compare: https://github.com/jenkins-infra/jenkins.io/compare/c49bb48b33bc...89b2c800c392

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/_data/changelogs/lts.yml content/_partials/changelog-weekly.html http://jenkins-ci.org/commit/jenkins.io/89b2c800c392195aca397baac4ceb91af08ad7f4 Log: Merge pull request #723 from oleg-nenashev/changelog/ JENKINS-24453 JENKINS-24453 - Noting fix in Jenkins 2.34 and 2.32.1 Compare: https://github.com/jenkins-infra/jenkins.io/compare/c49bb48b33bc...89b2c800c392

            People

              Unassigned Unassigned
              sharon_xia sharon xia
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: