{"componentChunkName":"component---src-templates-post-template-js","path":"/magical-windbg-vol1-05-en","result":{"data":{"markdownRemark":{"id":"02aa0c80-fd51-5b1c-9eb6-4eb74e7f3554","html":"
\n

This page has been machine-translated from the original page.

\n
\n

In Chapter 4, we analyzed a simple application crash dump.

\n

In Chapter 5, which follows, we will analyze a full memory dump collected when a simple system crash occurred.

\n\n

Table of Contents

\n\n

Triggering a system crash and creating a full memory dump

\n

As we saw in Chapter 4, when an exception occurs in the processing of an application running in user mode, the application crashes and an application crash dump is generated.

\n

Similarly, if an exception occurs inside a system process running in kernel mode, a system crash called a BSOD (Blue Screen of Death) occurs, and a system crash dump is generated.

\n

\n \n \n \n \n \n \n \n \n

\n

There are several kinds of system crash dumps, but the full memory dump we will capture this time contains all page information from all physical memory accessible to the Windows system running on that machine.

\n

A full memory dump is not generated under Windows default settings, but if your environment has been configured to collect dump files according to the procedure in Chapter 1, the setting to capture a full memory dump is already in place.

\n

Therefore, we will use D4C.exe to obtain a full memory dump for analysis.

\n

First, run the D4C.exe downloaded in Chapter 1, enter 3 in the menu shown at the prompt, and press Enter.

\n

\n \n \n \n \n \n \n \n \n

\n

When you execute option 3 in D4C.exe, a system crash occurs and the machine automatically reboots.

\n

After the system restarts, if a file named FULL_MEMORY.DMP has been created directly under the C:\\Windows folder, the full memory dump has been successfully captured.

\n

Loading the full memory dump in WinDbg

\n

When analyzing a full memory dump, just as with an application crash dump, launch the 64-bit version of WinDbg as administrator and use the [Ctrl + D] shortcut to load the dump file you captured from that folder.

\n

Once loading finishes and the message For analysis of this file, run !analyze -v appears, the dump is ready. (Depending on the full memory dump file size, loading may take a little time.)

\n

\n \n \n \n \n \n \n \n \n

\n

Analyzing the crash dump with the !analyze extension

\n

As with application crash dump investigations, the !analyze -v command is extremely useful when investigating a system crash.

\n

So let’s go through the output you get by running !analyze -v in the Command window, section by section.

\n

The first section shows the analysis result of the bug check data1 that can be obtained with the .bugcheck command, as shown below.

\n
CRITICAL_PROCESS_DIED (ef)\n        A critical system process died\nArguments:\nArg1: ffffc18f36303080, Process object or thread object\nArg2: 0000000000000000, If this is 0, a process died. If this is 1, a thread died.\nArg3: 0000000000000000, The process object that initiated the termination.\nArg4: 0000000000000000
\n

By referring to the output above, we can see that the cause of the system crash is CRITICAL_PROCESS_DIED.

\n

If you actually run the .bugcheck command in WinDbg, you get the following output.

\n
6: kd> .bugcheck\nBugcheck code 000000EF\nArguments ffffc18f`36303080 00000000`00000000 00000000`00000000 00000000`00000000
\n

Windows bug check codes are published in the official documentation below.

\n

If you look up the value 000000EF in that reference, you can see that it matches the bug check code for CRITICAL_PROCESS_DIED shown by !analyze -v.

\n

Also, the output of !analyze -v is equivalent to analyzing the bug check with the !analyze -show <bug check code> <Arg1> command.

\n
\n

Bug check codes:

\n

https://learn.microsoft.com/ja-jp/windows-hardware/drivers/debugger/bug-check-code-reference2#bug-check-codes

\n
\n

From the output of these commands, we can see that the system crash was CRITICAL_PROCESS_DIED (0xef), caused by the termination of a critical Windows system process.

\n

So what exactly is meant by a critical Windows system process?

\n

Critical Windows system processes include built-in system processes such as csrss.exe, wininit.exe, logonui.exe, smss.exe, services.exe, conhost.exe, and winlogon.exe.

\n
\n

Bug Check 0xEF: CRITICAL_PROCESS_DIED

\n

https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/bug-check-0xef—critical-process-died

\n
\n

In addition, because the value of Arg2 is 0, we can tell that the cause was the termination of a process rather than a thread, and that the actual stopped process object exists at 0xffffc18f36303080.

\n

If we inspect the information managed by the EPROCESS structure at that address, we can identify the stopped system process as svchost.exe with PID 0x3bc.

\n
6: kd> !process ffffc18f`36303080 0\nPROCESS ffffc18f36303080\n    SessionId: 0  Cid: 02dc    Peb: 27dedd6000  ParentCid: 03bc\n    DirBase: 7fbd5002  ObjectTable: ffffe50cc9052800  HandleCount: 1465.\n    Image: svchost.exe\n\n6: kd> dt nt!_EPROCESS 0xffffc18f`36303080\n   {{ omitted }}\n   +0x440 UniqueProcessId  : 0x00000000`000002dc Void\n   +0x5a8 ImageFileName    : [15]  \"svchost.exe\"\n   {{ omitted }}
\n

As you can see, just by analyzing the bug check based on the initial output of !analyze -v, we were already able to get very close to the cause of the system crash.

\n

Let’s continue to the next section.

\n

The section after the bug check analysis also contains very interesting information. From CriticalProcessDied.Process, we can again see that the crashed process was svchost.exe.

\n
Debugging Details:\n------------------\n\nKEY_VALUES_STRING: 1\n\n    Key  : Analysis.CPU.mSec\n    Value: 3765\n\n    Key  : Analysis.DebugAnalysisManager\n    Value: Create\n\n    Key  : Analysis.Elapsed.mSec\n    Value: 4092\n\n    Key  : Analysis.Init.CPU.mSec\n    Value: 78343\n\n    Key  : Analysis.Init.Elapsed.mSec\n    Value: 78474127\n\n    Key  : Analysis.Memory.CommitPeak.Mb\n    Value: 128\n\n    Key  : CriticalProcessDied.ExceptionCode\n    Value: 42bd7080\n\n    Key  : CriticalProcessDied.Process\n    Value: svchost.exe\n\n    Key  : WER.OS.Branch\n    Value: vb_release\n\n    Key  : WER.OS.Timestamp\n    Value: 2019-12-06T14:06:00Z\n\n    Key  : WER.OS.Version\n    Value: 10.0.19041.1
\n

The following section outputs the information below.

\n
FILE_IN_CAB:  FULL_MEMORY.DMP\n\nBUGCHECK_CODE:  ef\n\nBUGCHECK_P1: ffffc18f36303080\n\nBUGCHECK_P2: 0\n\nBUGCHECK_P3: 0\n\nBUGCHECK_P4: 0\n\nPROCESS_NAME:  svchost.exe\n\nCRITICAL_PROCESS:  svchost.exe\n\nERROR_CODE: (NTSTATUS) 0x42bd7080 - <Unable to get error code text>\n\nBLACKBOXBSD: 1 (!blackboxbsd)\n\nBLACKBOXNTFS: 1 (!blackboxntfs)\n\nBLACKBOXPNP: 1 (!blackboxpnp)\n\nBLACKBOXWINLOGON: 1
\n

The information from BUGCHECK_CODE through BUGCHECK_P4 matches the bug check information we saw in the first section.

\n

Also, CRITICAL_PROCESS tells us that the stopped process that caused the system crash was svchost.exe.

\n

The next section is the stack backtrace.

\n

Here, the output is equivalent to running the .cxr; .ecxr ; kb command.

\n
STACK_TEXT:  \nffffa209`4982f838 fffff801`2d70e592 : {{ omitted }} : nt!KeBugCheckEx\nffffa209`4982f840 fffff801`2d616045 : {{ omitted }} : nt!PspCatchCriticalBreak+0x10e\nffffa209`4982f8e0 fffff801`2d4819b0 : {{ omitted }} : nt!PspTerminateAllThreads+0x15e655\nffffa209`4982f950 fffff801`2d4817ac : {{ omitted }} : nt!PspTerminateProcess+0xe0\nffffa209`4982f990 fffff801`2d2105f5 : {{ omitted }} : nt!NtTerminateProcess+0x9c\nffffa209`4982fa00 00007ff8`7accd3d4 : {{ omitted }} : nt!KiSystemServiceCopyEnd+0x25\n00000087`d59ef568 00000000`00000000 : {{ omitted }} : ntdll!NtTerminateProcess+0x14
\n

The KeBugCheckEx function at the top of the stack backtrace is the function that directly triggers a system crash.2

\n

The KeBugCheckEx function receives a stop code and four parameters whose meanings depend on that stop code.

\n

That corresponds to the bug check information we just examined with the .bugcheck command and similar output.

\n

Conversely, the NtTerminateProcess function at the bottom of the stack backtrace is, as documented below, a function normally used when a user-mode application calls an API to terminate a process.

\n
\n

ZwTerminateProcess function (ntddk.h):

\n

https://learn.microsoft.com/ja-jp/windows-hardware/drivers/ddi/ntddk/nf-ntddk-zwterminateprocess

\n
\n

Considering the stack backtrace output together with the bug check information, we can conclude that it is highly likely that some user-mode application terminated svchost.exe, which is a critical system process, and that this caused the system crash.

\n

In the following output, which appears in the last section of !analyze -v, FAILURE_BUCKET_ID is also shown as 0xEF_svchost.exe_BUGCHECK_CRITICAL_PROCESS_42bd7080_ntdll!NtTerminateProcess, so we can judge that this system crash occurred because svchost.exe was terminated by the NtTerminateProcess function.

\n
SYMBOL_NAME:  ntdll!NtTerminateProcess+14\n\nMODULE_NAME: ntdll\n\nIMAGE_NAME:  ntdll.dll\n\nSTACK_COMMAND:  .cxr; .ecxr ; kb\n\nBUCKET_ID_FUNC_OFFSET:  14\n\nFAILURE_BUCKET_ID:  0xEF_svchost.exe_BUGCHECK_CRITICAL_PROCESS_42bd7080_ntdll!NtTerminateProcess\n\nOS_VERSION:  10.0.19041.1\n\nBUILDLAB_STR:  vb_release\n\nOSPLATFORM_TYPE:  x64\n\nOSNAME:  Windows 10\n\nFAILURE_ID_HASH:  {f6ece2b4-3d35-e4e4-9739-fdbc46a086b0}\n\nFollowup:     MachineOwner
\n

Identifying the process that caused the crash

\n

From the output of !analyze -v, we were able to identify the exception that directly caused the crash.

\n

However, the stack backtrace shown by !analyze -v did not include any information earlier than ntdll!NtTerminateProcess+0x14, so we still could not determine exactly what caused the exception.

\n

This is because the context associated with this exception is the process context of the terminated svchost.exe.

\n

You can check the current process context associated with the exception by running .ecxr; !peb. (The !peb extension3 displays the PEB (Process Environment Block) corresponding to the current process context.)

\n

Because it was difficult to investigate further from the exception-related context selected by .ecxr, we need to switch the context to the process that may have caused the crash in order to proceed with a more detailed investigation.

\n

In this case, that process is D4C.exe.

\n

If you display thread information for the exception-related context with the .ecxr; !thread command, you can see that Owning Process is D4C.exe, and the address of the process object is shown as 0xffffc18f45284080.

\n

\n \n \n \n \n \n \n \n \n

\n

In this way, rather than analyzing only with WinDbg, there are cases where analysis proceeds more efficiently by using multiple powerful tools such as decompilers and approaching the problem from different angles.

\n

This completes the identification, through dump file analysis, that the system crash was caused by svchost.exe being terminated by the TerminateProcess API called from D4C.exe.

\n

Chapter 5 summary

\n

That concludes the analysis of this simple system crash dump.

\n

As with the application crash dump analyzed in Chapter 4, the direct cause (exception) of the BSOD itself can be identified relatively easily from the dump file.

\n

Of course, if you want to investigate the program behavior that caused that exception, it will rarely be identified as easily as it was in this chapter.

\n

In such cases, not only analyzing the dump file with WinDbg but also tracing the environment while reproducing the problem with tools such as Process Monitor or packet capture, using source code or decompiled results for offline debugging, and even live debugging may help you investigate the cause more efficiently.

\n

Chapter links

\n\n
\n
\n
    \n
  1. \n

    Blue Screen Data https://learn.microsoft.com/ja-jp/windows-hardware/drivers/debugger/blue-screen-data

    \n↩\n
    2. \n
  2. \n

    Windows Internals, 6th Edition, Vol. 2, p.606 (Mark E. Russinovich・David A. Solomon・Alex Ionescu / translated by 株式会社クイープ / 日経 BP / 2013)

    \n↩\n
    3. \n
  3. \n

    !peb extension https://learn.microsoft.com/ja-jp/windows-hardware/drivers/debugger/-peb

    \n↩\n
    4. \n
  4. \n

    !dh https://learn.microsoft.com/ja-jp/windows-hardware/drivers/debugger/-dh

    \n↩\n
    5. \n
  5. \n

    dps https://learn.microsoft.com/ja-jp/windows-hardware/drivers/debugger/dds—dps—dqs—display-words-and-symbols-

    \n↩\n
    6. \n
\n
","fields":{"slug":"/magical-windbg-vol1-05-en","tagSlugs":["/tag/magical-win-dbg/","/tag/windows/","/tag/win-dbg/","/tag/english/"]},"frontmatter":{"date":"2023-11-15","description":"This is the web edition of Magical WinDbg - Enjoying Windows Dump Analysis and Troubleshooting by Feel - VOL.1, distributed at Tech Book Fest 15.","tags":["Magical WinDbg","Windows","WinDbg","English"],"title":"Magical WinDbg VOL.1 [Chapter 5: Analyzing a Full Memory Dump from a System Crash]","socialImage":{"publicURL":"/static/2dbf3e09d59db889dc9dc41adcc8e827/magical-windbg-vol1.png"}}}},"pageContext":{"slug":"/magical-windbg-vol1-05-en"}},"staticQueryHashes":["251939775","401334301","825871152"]}