来自 Complete Memory Dump 的 CLR 堆栈跟踪和 CLR 异常答案

【问题标题】：CLR stack trace & CLR exceptions from Complete Memory Dump来自 Complete Memory Dump 的 CLR 堆栈跟踪和 CLR 异常
【发布时间】：2015-10-21 23:39:02
【问题描述】：

拥有 Windows Server 2008 x64 Server 的完整内存转储（VMWare 内存快照使用https://labs.vmware.com/flings/vmss2core 转换为内存 dmp）

是否可以从 .NET x64 进程中获取 CLR 堆栈

!eestack 什么都不显示，!clrstack 表示线程不受管理。

~Ns 切换线程在内核调试器中不起作用，它会切换 CPU。为了识别 CLR 线程，我使用 !threads 输出中的 OSID 列，然后在 !process <process> 1f 的输出中搜索匹配项，寻找 CID 第二部分的匹配数

目前我已尝试使用 !process、.context、.process、.thread、!clrstack、pe 进行 SOS 扩展

0: kd> .load C:\windows\Microsoft.NET\framework64\v4.0.30319\SOS.dll
0: kd> .cordll -ve -u -l
Automatically loaded SOS Extension
CLRDLL: Loaded DLL C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscordacwks.dll
CLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscordacwks.dll
0: kd> !process fffffa809b797890 1f
PROCESS fffffa809b797890
    SessionId: 0  Cid: 1908    Peb: 7fffffdf000  ParentCid: 0258
    DirBase: 1f1127000  ObjectTable: fffff8a008d37150  HandleCount: 500.
    Image: MSExchangeMailSubmission.exe
    VadRoot fffffa80992a6090 Vads 253 Clone 0 Private 8608. Modified 157967. Locked 0.
    DeviceMap fffff8a0000087d0
    Token                             fffff8a02e910c50
    ElapsedTime                       12 Days 17:47:43.597
    UserTime                          00:00:18.158
    KernelTime                        00:00:07.472
    QuotaPoolUsage[PagedPool]         331528
    QuotaPoolUsage[NonPagedPool]      33852
    Working Set Sizes (now,min,max)  (8066, 50, 345) (32264KB, 200KB, 1380KB)
    PeakWorkingSetSize                15907
    VirtualSize                       594 Mb
    PeakVirtualSize                   600 Mb
    PageFaultCount                    264942
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      17695

        THREAD fffffa809bad8ad0  Cid 1908.1238  Teb: 000007fffffdd000 Win32Thread: fffff900c1ccd8c0 WAIT: (UserRequest) UserMode Non-Alertable
            fffffa809b700f50  SynchronizationEvent
        Not impersonating
        DeviceMap                 fffff8a0000087d0
        Owning Process            fffffa809b797890       Image:         MSExchangeMailSubmission.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      73292676       Ticks: 657119 (0:02:50:51.122)
        Context Switch Count      253            IdealProcessor: 2                 LargeStack
        UserTime                  00:00:00.015
        KernelTime                00:00:00.093
*** ERROR: Module load completed but symbols could not be loaded for MSExchangeMailSubmission.exe
        Win32 Start Address MSExchangeMailSubmission (0x000000000111ab2e)
        Stack Init fffff8800850ec70 Current fffff8800850e7c0
        Base fffff8800850f000 Limit fffff88008509000 Call 0
        Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.
        Child-SP          RetAddr           Call Site
        fffff880`0850e800 fffff800`01881802 nt!KiSwapContext+0x7a
        fffff880`0850e940 fffff800`0188401f nt!KiCommitThreadWait+0x1d2
        fffff880`0850e9d0 fffff800`01b726de nt!KeWaitForSingleObject+0x19f
        fffff880`0850ea70 fffff800`0187b853 nt!NtWaitForSingleObject+0xde
        fffff880`0850eae0 00000000`7713d9fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0850eae0)
        00000000`0022e348 000007fe`fd6010dc ntdll!ZwWaitForSingleObject+0xa
        00000000`0022e350 000007fe`fec1affb KERNELBASE!WaitForSingleObjectEx+0x79
        00000000`0022e3f0 000007fe`fec19d61 sechost!ScSendResponseReceiveControls+0x13b
        00000000`0022e4e0 000007fe`fec19c16 sechost!ScDispatcherLoop+0x121
        00000000`0022e5f0 000007fe`f2e9bec7 sechost!StartServiceCtrlDispatcherW+0x14e
        00000000`0022e640 000007fe`ec9df0a8 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
        00000000`0022e6e0 000007fe`ec9e1478 System_ServiceProcess_ni+0x2f0a8
        00000000`0022e7a0 000007fe`dc6eeecb System_ServiceProcess_ni+0x31478
        00000000`0022e840 00000000`01545f78 MSExchangeMailSubmission_ni+0x1eecb
        00000000`0022e848 00000000`01545e38 0x1545f78
        00000000`0022e850 00000000`01545e38 0x1545e38
        00000000`0022e858 000007fe`f2137680 0x1545e38
        00000000`0022e860 00000000`0000001d mscorlib_ni+0x437680
        00000000`0022e868 00000000`00000000 0x1d

        THREAD fffffa809bf0fa00  Cid 1908.1290  Teb: 000007fffffdb000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
            fffffa809bfd2d60  SynchronizationEvent
            fffffa809be67730  SynchronizationEvent
            fffffa809ba327e0  SynchronizationEvent
        Not impersonating
        DeviceMap                 fffff8a0000087d0
        Owning Process            fffffa809b797890       Image:         MSExchangeMailSubmission.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      3385133        Ticks: 70564662 (12:17:46:55.782)
        Context Switch Count      4              IdealProcessor: 3             
        UserTime                  00:00:00.000
        KernelTime                00:00:00.000
        Win32 Start Address mscorwks!DebuggerRCThread::ThreadProcStatic (0x000007fef2cdfe98)
        Stack Init fffff88009341c70 Current fffff88009340e80
        Base fffff88009342000 Limit fffff8800933c000 Call 0
        Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Kernel stack not resident.

  etc ...

0: kd> .context 1f1127000
0: kd> .process /r /p fffffa809b797890
0: kd> !threads
ThreadCount: 16
UnstartedThread: 0
BackgroundThread: 8
PendingThread: 0
DeadThread: 7
Hosted Runtime: no
                                              PreEmptive                                                Lock
       ID OSID        ThreadOBJ     State   GC     GC Alloc Context                  Domain           Count APT Exception
       1 1238 00000000003221b0      a020 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 MTA
       2 1670 000000000032c2a0      b220 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 MTA (Finalizer)
       3 1600 000000000104ca40   880b220 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 MTA (Threadpool Completion Port)
       6  3f8 000000000107cfe0    80a220 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 MTA (Threadpool Completion Port)
       8 124c 000000001c5090e0   200b220 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 Ukn
      1b    0 000000001d865360   1801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 Ukn (Threadpool Worker)
      17    0 000000001d8641f0   1801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 MTA (Threadpool Worker)
      1d 21bc 000000001d866aa0   180b220 Enabled  0000000001ca45e8:0000000001ca5b10 00000000003196d0     0 MTA (Threadpool Worker)
      1c 2568 000000001d865f00   180b220 Enabled  0000000001c96270:0000000001c96480 00000000003196d0     0 MTA (Threadpool Worker)
      1a    0 000000001d8664d0   1801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 MTA (Threadpool Worker)
      18    0 000000001d863c20   1801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 Ukn (Threadpool Worker)
       d    0 000000001d865930   1801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 Ukn (Threadpool Worker)
      19    0 000000001d8647c0   1801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 Ukn (Threadpool Worker)
      11    0 000000001d864d90  21801820 Enabled  0000000000000000:0000000000000000 00000000003196d0     0 Ukn (Threadpool Worker)
      16  868 000000001d867070   180b220 Enabled  0000000001c99630:0000000001c9a480 00000000003196d0     0 MTA (Threadpool Worker)
       a 1be4 000000001c5a0f00   180b220 Enabled  0000000001ca3368:0000000001ca3b10 00000000003196d0     0 MTA (Threadpool Worker)
0: kd> .thread
Implicit thread is now fffffa80`937c2b50
0: kd> .thread /r /p fffffa809b6330f0
Implicit thread is now fffffa80`9b6330f0
Implicit process is now fffffa80`9b797890
Loading User Symbols
.......................Unable to read NT module Base Name string at 00000000`00305ff8 - Win32 error 0n30
.Unable to read NT module Base Name string at 00000000`00306098 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00306138 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`01016b5e - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
........................Unable to read NT module Base Name string at 00000000`0101eb5e - NTSTATUS 0xC0000147
..Unable to read NT module Base Name string at 00000000`0101ed5e - NTSTATUS 0xC0000147
............
...Unable to read NT module Base Name string at 00000000`0101f65e - NTSTATUS 0xC0000147
.........Unable to read NT module Base Name string at 00000000`1c510eb8 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
............Unable to read NT module Base Name string at 00000000`1c526858 - NTSTATUS 0xC0000147
...

************* Symbol Loading Error Summary **************
Module name            Error
pvscsi                 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0: kd> !clrstack
OS Thread Id: 0x0 (0)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
0: kd> !pe
The current thread is unmanaged
0: kd> kv
  *** Stack trace for last set context - .thread/.cxr resets it
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`09a6e800 fffff800`01881802 : fffffa80`9b6330f0 fffffa80`9b6330f0 fffff8a0`00000000 00000000`00000001 : nt!KiSwapContext+0x7a
fffff880`09a6e940 fffff800`0188401f : 00000000`00000254 00000000`1c541c58 00000000`0000005e 00000000`1d5dd998 : nt!KiCommitThreadWait+0x1d2
fffff880`09a6e9d0 fffff800`01b726de : fffffa80`9b633000 fffff880`00000006 00000000`00000001 00000000`00000000 : nt!KeWaitForSingleObject+0x19f
fffff880`09a6ea70 fffff800`0187b853 : fffffa80`9b6330f0 00000000`00009c40 fffff880`09a6eab8 fffffa80`9bf77920 : nt!NtWaitForSingleObject+0xde
fffff880`09a6eae0 00000000`7713d9fa : 000007fe`fd6010dc 00000000`1d866aa0 ffffffff`ffffffff 00000000`00000001 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`09a6eae0)
00000000`1d5df3d8 000007fe`fd6010dc : 00000000`1d866aa0 ffffffff`ffffffff 00000000`00000001 000007fe`f2d23a6f : ntdll!ZwWaitForSingleObject+0xa
00000000`1d5df3e0 000007fe`f2dde540 : 00000000`ffffffff 00000000`00009c40 00000000`00000000 00000000`00000254 : KERNELBASE!WaitForSingleObjectEx+0x79
00000000`1d5df480 000007fe`f2dde42f : 00000000`00000000 00000000`1d866aa0 00000000`00000000 00000000`00009c40 : mscorwks!CLREvent::WaitEx+0x174
00000000`1d5df4d0 000007fe`f2dde18b : 000007fe`f2cd22bc 00000000`00000001 00000000`1d866aa0 000007fe`f2de5f93 : mscorwks!CLREvent::WaitEx+0x63
00000000`1d5df580 000007fe`f2dd162a : 00000000`00000001 00000000`1d866aa0 00000000`00000001 00000000`00000000 : mscorwks!ThreadpoolMgr::SafeWait+0x7b
00000000`1d5df640 000007fe`f2e20134 : 00000000`00000000 00000000`00000000 00000000`1d5dfd80 00000000`00000000 : mscorwks!ThreadpoolMgr::WorkerThreadStart+0x11a
00000000`1d5df6e0 00000000`76dc5a4d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : mscorwks!Thread::intermediateThreadProc+0x78
00000000`1d5dfdb0 00000000`7711b831 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0xd
00000000`1d5dfde0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d

也试过了：

0: kd> .thread /r /p fffffa809bad8ad0
Implicit thread is now fffffa80`9bad8ad0
Implicit process is now fffffa80`9b797890
Loading User Symbols
.......................Unable to read NT module Base Name string at 00000000`00305ff8 - Win32 error 0n30
.Unable to read NT module Base Name string at 00000000`00306098 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00306138 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`01016b5e - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
........................Unable to read NT module Base Name string at 00000000`0101eb5e - NTSTATUS 0xC0000147
..Unable to read NT module Base Name string at 00000000`0101ed5e - NTSTATUS 0xC0000147
............
...Unable to read NT module Base Name string at 00000000`0101f65e - NTSTATUS 0xC0000147
.........Unable to read NT module Base Name string at 00000000`1c510eb8 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
............Unable to read NT module Base Name string at 00000000`1c526858 - NTSTATUS 0xC0000147
...
0: kd> kn
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 fffff880`0850e800 fffff800`01881802 nt!KiSwapContext+0x7a
01 fffff880`0850e940 fffff800`0188401f nt!KiCommitThreadWait+0x1d2
02 fffff880`0850e9d0 fffff800`01b726de nt!KeWaitForSingleObject+0x19f
03 fffff880`0850ea70 fffff800`0187b853 nt!NtWaitForSingleObject+0xde
04 fffff880`0850eae0 00000000`7713d9fa nt!KiSystemServiceCopyEnd+0x13
05 00000000`0022e348 000007fe`fd6010dc ntdll!ZwWaitForSingleObject+0xa
06 00000000`0022e350 000007fe`fec1affb KERNELBASE!WaitForSingleObjectEx+0x79
07 00000000`0022e3f0 000007fe`fec19d61 sechost!ScSendResponseReceiveControls+0x13b
08 00000000`0022e4e0 000007fe`fec19c16 sechost!ScDispatcherLoop+0x121
09 00000000`0022e5f0 000007fe`f2e9bec7 sechost!StartServiceCtrlDispatcherW+0x14e
*** WARNING: Unable to verify checksum for System.ServiceProcess.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.ServiceProcess.ni.dll
0a 00000000`0022e640 000007fe`ec9df0a8 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
0b 00000000`0022e6e0 000007fe`ec9e1478 System_ServiceProcess_ni+0x2f0a8
*** WARNING: Unable to verify checksum for MSExchangeMailSubmission.ni.exe
*** ERROR: Module load completed but symbols could not be loaded for MSExchangeMailSubmission.ni.exe
0c 00000000`0022e7a0 000007fe`dc6eeecb System_ServiceProcess_ni+0x31478
0d 00000000`0022e840 00000000`01545f78 MSExchangeMailSubmission_ni+0x1eecb
0e 00000000`0022e848 00000000`01545e38 0x1545f78
0f 00000000`0022e850 00000000`01545e38 0x1545e38
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
10 00000000`0022e858 000007fe`f2137680 0x1545e38
11 00000000`0022e860 00000000`0000001d mscorlib_ni+0x437680
12 00000000`0022e868 00000000`00000000 0x1d
0: kd> !clrstack
OS Thread Id: 0x0 (0)
Unable to walk the managed stack. The current thread is likely not a 
managed thread. You can run !threads to get a list of managed threads in
the process
0: kd> !pe
The current thread is unmanaged

我也尝试过使用 sosex 扩展 (http://stevestechspot.com/SOSEXV40NowAvailable.aspx)，它会抛出错误“如果没有完整的内存信息，此命令可能无法正常工作。”

正如所指出的，堆栈跟踪是 .NET 2/3 而不是 .NET 4，所以我尝试了 .NET 2 SOS.dll

0: kd> .load C:\windows\microsoft.net\framework64\v2.0.50727\sos.dll
0: kd> .thread /r /p  fffffa809d19d980
Implicit thread is now fffffa80`9d19d980
Implicit process is now fffffa80`9b797890
Loading User Symbols
.......................Unable to read NT module Base Name string at 00000000`00305ff8 - Win32 error 0n30
.Unable to read NT module Base Name string at 00000000`00306098 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`00306138 - NTSTATUS 0xC0000147
.Unable to read NT module Base Name string at 00000000`01016b5e - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
........................Unable to read NT module Base Name string at 00000000`0101eb5e - NTSTATUS 0xC0000147
..Unable to read NT module Base Name string at 00000000`0101ed5e - NTSTATUS 0xC0000147
............
...Unable to read NT module Base Name string at 00000000`0101f65e - NTSTATUS 0xC0000147
.........Unable to read NT module Base Name string at 00000000`1c510eb8 - NTSTATUS 0xC0000147
Missing image name, possible paged-out or corrupt data.
............Unable to read NT module Base Name string at 00000000`1c526858 - NTSTATUS 0xC0000147
...
0: kd> .context 2ed797000
0: kd> .cordll -ve -u -l
CLRDLL: ERROR: Unable to get version information for mscorwks, NTSTATUS 0xC0000147
CLR DLL status: ERROR: Unable to get version information for mscorwks, NTSTATUS 0xC0000147
0: kd> lmvm mscorwks
start             end                 module name
000007fe`f2be0000 000007fe`f357e000   mscorwks   (deferred)             
    Image path: C:\Windows\Microsoft.NET\Framework64\v2.0.50727\mscorwks.dll
    Image name: mscorwks.dll
    Timestamp:        Wed Jun 18 15:02:36 2014 (53A11D6C)
    CheckSum:         0098CBB7
    ImageSize:        0099E000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Unable to enumerate user-mode unloaded modules, NTSTATUS 0xC0000147
0: kd> .cordll -ve -se -u -I 000007fe`f2be0000 -N
CLRDLL: ERROR: Unable to get version information for mscorwks, NTSTATUS 0xC0000147
NOTE: This code is not supported and may not work.
Do not report any problems you have.
CLR DLL status: ERROR: Unable to get version information for mscorwks, NTSTATUS 0xC0000147
  Test new CLR interfaces

【问题讨论】：

当这个应用似乎是基于 .NET 2/3.* 的时候，你为什么要使用 .NET 4 的 SOS？重新启动 WinDbg，并使用“.loadby sos mscorwks”加载 SOS。然后“！threads”应该让您更深入地了解哪些是托管线程。只有在托管线程上，“！CLRStack”才能给你一些东西。
我也尝试了 .NET 2 中的 SOS.dll，!threads 仍然有效。但我收到错误 CLRDLL: ERROR: Unable to get version information for mscorwks,
可能是因为它在创建转储时被分页了。
这些“无法读取...”消息是否可能是由于Known issue with Windows 7 kernel symbols 而引起的？
这个问题刚刚在 2015 年 10 月 13 日补丁星期二在内核 7601.19018.x86fre.win7sp1_gdr.150928-1507 中得到了修补，尽管我会仔细检查这是运行修补过的内核，虽然我没有遇到问题倾销 peb 等，我认为它不适用于损坏的符号

标签： .net windbg

【解决方案1】：

SOS 和其他 .NET 扩展在虚拟内存上运行，而不是在物理内存上运行，因此它们在内核模式下无法正常工作。内核模式dump的部分内存可能被page out，所以相关信息丢失。

拥有user mode dump with full memory 会好很多。

【讨论】：

托马斯是正确的。 .NET 调试根本无法在内核模式下工作，除非您非常幸运地在转储时调入了所有必要的 .NET 运行时结构。我从未见过在内核模式下工作的 .NET 调试示例，尽管理论上是可行的。

【解决方案2】：

您似乎在错误的线程上。堆栈显示 mscorwks!Thread::intermediateThreadProc 的事实表明了这一点。请改用 .thread fffffa809bad8ad0。

【讨论】：

这不起作用，我已经尝试过了，在输出中包含更多详细信息。