SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion
Posted by Karthick P.K on October 16, 2012
I have got few request’s from SQL Server DBA’s in past to blog about analyzing SQL Server exceptions and assertions . After seeing lot of DBA’s getting stuck when they get EXCEPTION_ACCESS_VIOLATION (or) Assertion in SQL ServersI decided to write this blog.
This blog is published with intention to make DBA’s analyze and resolve EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion before contacting Microsoft support. Exception and assertion are two different things. SQL handles both assertions and exceptions by writing the current thread’s stack to the Error log and generating a dump. In simple An exception is an event that occurs during the execution of a program, and requires the execution of code outside the normal flow of control and assertion is the check that the programmer inserted into the code to make sure that some condition is true, If it returns false an assert is raised. SQL handles both assertions and exceptions by writing the current thread’s stack to the Error log and generating a dump, so trouble shooting steps are similar.
You will find messages similar to one below in SQL Serve error logs when you get Exception or EXCEPTION_ACCESS_VIOLATION .
{
Error
External dump process returned no errors.
Using ‘dbghelp.dll’ version ’4.0.5′
SqlDumpExceptionHandler: Process 510 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.
* *******************************************************************************
* BEGIN STACK DUMP:
* Exception Address = 000000007752485C Module(ntdll+000000000002285C)
* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
* Access Violation occurred reading address 0000041EA9AE2EF0
* Input Buffer 510 bytes –
ex_terminator – Last chance exception handling
}
You will find messages similar to one below in SQL Server error logs when you get an Assertion.
{
Error
spid323 Error: 17065, Severity: 16, State: 1.
spid323 SQL Server Assertion: File: < .cpp>, line = 2576 Failed Assertion = ‘fFalse’ This error may be timing-related. If the error persists after rerunning the statement, use DBCC CHECKDB to check the database for structural integrity, or restart the server to ensure in-memory data structures are not corrupted
SQL Server Assertion: File: < .cpp>, line=2040 Failed Assertion =
}
To analyze the dump download and Install Windows Debugger from This Link
Step 1 (Load the memory dump file to debugger):
Open Windbg . Choose File menu –> select Open crash dump –>Select the Dump file (SQLDump000#.mdmp)
Note : You will find SQLDump000#.mdmp in your SQL Server error log when you get the Exception or assertion.
Step 2 (Set the symbol path to Microsoft symbols server):
on command window type
.sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
Step 3 (Load the symbols from Microsoft symbols server):
Type .reload /f and hit enter. This will force debugger to immediately load all the symbols.
Step 4 (check if symbols are loaded):
Verify if symbols are loaded for SQL Server by using the debugger command lmvm
0:002> lmvm sqlservr
start end module name
00000000`01000000 00000000`03679000 sqlservr T (pdb symbols) c:\websymbols\sqlservr.pdb\21E4AC6E96294A529C9D99826B5A7C032\sqlservr.pdb
Loaded symbol image file: sqlservr.exe
Image path: C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Wed Oct 07 21:15:52 2009 (4ACD6778)
CheckSum: 025FEB5E
ImageSize: 02679000
File version: 2005.90.4266.0
Product version: 9.0.4266.0
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Step 5 (Switch to exception context):
Type .ecxr
Step 6(Get the stack of thread which caused exception or assertion):
Type kC 1000 //You will get the stack of thread which raised exception or assertion .
I have pasted one of the sample stack below, from the exception dump which I worked recently. First thing to identify from stack is who is raising the exception. In the below stack look at the portion which is highlighted in red (In each frame before the ! symbol), that is the module which raised the exception (Exe or DLL name ).
If Exe/DLL name is Non Microsoft module (Exe or DLL name ) then the exception is being caused by a third party component, you will need to work with the company that provided that component to get a solution. lmvm Exe/DLL name will give you the company name. For example: lmvm wininet
If Exe/DLL name is SQLServr (or) any other SQL Server modules then the exception is raised by SQL Server, In that case type kC 1000 and paste the stack in comments session of this blog (or) When you start thread in MSDN forums (or) In This face book group. If you don’t get any prompt reply from the community, you may need to open a support ticket with Microsoft.
Note: When you get Assertion make sure you post message line which contains SQL Server Assertion: File: <Filename.cpp>, line = 2576 Failed Assertion = ”
0:000> kC 1000
Call Site
wininet!InternetFreeThreadInfo+0×26
wininet!InternetDestroyThreadInfo+0×40
wininet!DllMain_wininet+0xb5
wininet!__DllMainCRTStartup+0xdb
ntdll!LdrShutdownThread+0×155
ntdll!RtlExitUserThread+0×38
msvcr80!_endthreadex+0×27
msvcr80!_callthreadstartex+0x1e
msvcr80!_threadstartex+0×84
kernel32!BaseThreadInitThunk+0xd
ntdll!RtlUserThreadStart+0x1d
If you liked this post, do like us on Facebook at https://www.facebook.com/mssqlwiki and join our Facebook group https://www.facebook.com/mssqlwiki#!/groups/454762937884205/
Related posts:
- How to Analyze “Deadlocked Schedulers” Dumps?
- How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps ……
- Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes
- SQL Server generated Access Violation dumps while accessing oracle linked servers.
- SQL Server Latch & Debugging latch time out
Thank you,
Karthick P.K |My Facebook Page |My Site| Blog space| Twitter
Disclaimer
The views expressed on this website/blog are mine alone and do not reflect the views of my company. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.
Scott McCormick said
Hello Karthik –
This is a great explanation of the windbg tool. I have a couple of follow-up questions:
1 — You say that the red highlighted text is the application causing the problem. Do you mean the bold text? None of the text in the sample is red.
2 — Assuming you mean the bold text, does that mean the output is read from the top down?
3 — I honestly have no idea if the Stack Dump I’m seeing is being caused by a 3rd party, but I don’t believe it is. Would you please take a look?
0:000> kC 1000
Call Site
KERNELBASE!RaiseException
sqlservr!DmpRemoteDumpRequest
sqlservr!SQLExit
sqlservr!SQLExit
sqlservr!SQLExit
sqlservr!SQLExit
sqlservr!SQLExit
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
winhttp!RemoveFromSerializedList
winhttp!InternetDestroyThreadInfo
winhttp!DllMain
winhttp!CRT_INIT
ntdll!LdrShutdownThread
ntdll!RtlExitUserThread
msvcr80!endthreadex
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Karthick P.K said
Hello Scott,
WINHTTP.DLL is loaded in SQL Server’s internal address apace and crash is occuring WINHTTP.DLL.
I recommended to move this module outside SQL Server address space. The stored procedure ‘sp_oacreate’ should have been used to bring in the module to SQL Server address space, so to move out in the syntax you need to mention 4 with sp_oacreate
{
Here is the KB article that gives a workaround on How to move extended stored procedures out of process-
http://support.microsoft.com/kb/243428
}
You can use belowquery to identify the objects in which you have used SP_OA create
{
EXEC sp_MSforeachdb ‘
IF EXISTS (SELECT * FROM ?.dbo.syscomments WHERE CHARINDEX (”sp_OA”, [text])>0)
SELECT
LEFT (”?”, 30) AS db, LEFT (o.name, 40) AS sproc,
LEFT (REPLACE (REPLACE (SUBSTRING ([text], CHARINDEX (”SP_OA”, UPPER ([text])) – 10, 60), CHAR(10), ””), CHAR(13), ””), 60)
FROM ?.dbo.syscomments c
INNER JOIN ?.dbo.sysobjects o ON c.id = o.id
WHERE CHARINDEX (”sp_OA”, [text]) > 0′
}
Scott McCormick said
Thanks for the quick response. Unfortunately, it looks like sp_oa is not used in any SP or function. Or at least, it isn’t showing up in sys.comments. Can you explain how you found that information?
I was able to pull more information out after posting this question, and I agree that it’s winhttp. I was thinking it might be one of the CLRs? There are a few of them.
Here is the rest of what I could find.
FAULTING_IP:
winhttp!RemoveFromSerializedList+26
000007fe`f7de1376 4c8918 mov qword ptr [rax],r11
EXCEPTION_RECORD: 0000000021edf880 — (.exr 0x21edf880)
ExceptionAddress: 000007fef7de1376 (winhttp!RemoveFromSerializedList+0×0000000000000026)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000
DEFAULT_BUCKET_ID: APPLICATION_FAULT
PROCESS_NAME: sqlservr.exe
ERROR_CODE: (NTSTATUS) 0x42ac –
EXCEPTION_CODE: (Win32) 0x42ac (17068) –
NTGLOBALFLAG: 0
MANAGED_STACK: !dumpstack -EE
No export dumpstack found
MANAGED_BITNESS_MISMATCH:
Managed code needs matching platform of sos.dll for proper analysis. Use ‘x64′ debugger.
CONTEXT: 0000000021edf390 — (.cxr 0x21edf390)
rax=0000000000000000 rbx=000000001e5e1d00 rcx=000000000000412c
rdx=000000001e5e1d00 rsi=000007fef7e44730 rdi=0000000000000003
rip=000007fef7de1376 rsp=0000000021edf950 rbp=0000000000000000
r8=0000000000000000 r9=00000000053f1a30 r10=0000000000000000
r11=0000000000000000 r12=000007fffffda000 r13=000000001e1df960
r14=000007fffff40000 r15=0000000077d72670
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
winhttp!RemoveFromSerializedList+0×26:
000007fe`f7de1376 4c8918 mov qword ptr [rax],r11 ds:00000000`00000000=????????????????
Resetting default scope
WRITE_ADDRESS: 0000000000000000
FOLLOWUP_IP:
winhttp!RemoveFromSerializedList+26
000007fe`f7de1376 4c8918 mov qword ptr [rax],r11
BUGCHECK_STR: ACCESS_VIOLATION
FAULTING_THREAD: ffffffffffffffff
LAST_CONTROL_TRANSFER: from 000007fef7de13a9 to 000007fef7de1376
STACK_TEXT:
000007fe`f7de1376 winhttp!RemoveFromSerializedList+0×26
000007fe`f7de13a9 winhttp!InternetDestroyThreadInfo+0x2e
000007fe`f7de1190 winhttp!DllMain+0×20
000007fe`f7de10d2 winhttp!CRT_INIT+0x2af
00000000`77c86a28 ntdll!LdrShutdownThread+0×155
00000000`77c86968 ntdll!RtlExitUserThread+0×38
00000000`758537b7 msvcr80!endthreadex+0×27
00000000`758537de msvcr80!endthreadex+0x4e
00000000`75853894 msvcr80!endthreadex+0×104
00000000`77b3652d kernel32!BaseThreadInitThunk+0xd
00000000`77c6c521 ntdll!RtlUserThreadStart+0x1d
STACK_COMMAND: .cxr 21EDF390 ; kb ; dds 21edf950 ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: winhttp!RemoveFromSerializedList+26
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: winhttp
IMAGE_NAME: winhttp.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ca23
FAILURE_BUCKET_ID: APPLICATION_FAULT_42ac_winhttp.dll!RemoveFromSerializedList
BUCKET_ID: X64_ACCESS_VIOLATION_winhttp!RemoveFromSerializedList+26
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_4000_0/4fecc5ba/sqlservr_exe/2009_100_4000_0/4fecc5ba/42ac/020eae68.htm?Retriage=1
Followup: MachineOwner
———
0:000> .exr 0x21edf880
ExceptionAddress: 000007fef7de1376 (winhttp!RemoveFromSerializedList+0×0000000000000026)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000
0:000> .exr 0x21edf880
ExceptionAddress: 000007fef7de1376 (winhttp!RemoveFromSerializedList+0×0000000000000026)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 0000000000000000
Attempt to write to address 0000000000000000
MODULE_NAME: winhttp
IMAGE_NAME: winhttp.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ca23
FAILURE_BUCKET_ID: APPLICATION_FAULT_42ac_winhttp.dll!RemoveFromSerializedList
BUCKET_ID: X64_ACCESS_VIOLATION_winhttp!RemoveFromSerializedList+26
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_4000_0/4fecc5ba/sqlservr_exe/2009_100_4000_0/4fecc5ba/42ac/020eae68.htm?Retriage=1
Followup: MachineOwner
———
Karthick P.K said
I think you have run “!analyze -v”.
I found it by looking at module name(I missed to mark in red, will edit it).
In this case winhttp.dll is referencing a NULL pointer so AV.
If you suspect CLR then Disable CLR – >restart SQL – > check if winhttp.dll is still loaded. You can use below query. If you dont find winhttp.dll after disabling the CLR, Then its CLR.
{
select name as name1, * from sys.dm_os_loaded_modules where company ‘Microsoft Corporation’ or name like ‘%winhttp%’
}
I have seen in past application firewall’s loading winhttp.dll. Do you have any type of “application firewall” installed on the SQL server?
Scott McCormick said
There is no firewall.
Also, at the moment, winhttp is not loaded. My understanding of CLRs is that they are loaded up when the instance starts and are only unloaded due to memory pressure, but maybe that’s wrong? Is it possible the winhttp module isn’t loaded because the CLR has not been called since the last reboot (yesterday afternoon)? Or maybe it’s just not a CLR, but I don’t know what else it could be.
There aren’t that many things that go outside of SQL Server.
Karthick P.K said
Hello Scott,
By any chance do you see MSO.dll loaded in SQL Server .
Scott McCormick said
Hey Karthik –
It’s a giant coincidence that you would mention mso.dll today. Yesterday, there was a crash that didn’t generate a Stack Dump, but did apparently crash due to a mso.dll call. The event viewer message is:
Faulting application name: sqlservr.exe, version: 2009.100.4000.0, time stamp: 0x4fecc5ba
Faulting module name: mso.dll, version: 14.0.4760.1000, time stamp: 0x4ba906d8
Exception code: 0xc0000005
Fault offset: 0x0000000000050d4a
Faulting process id: 0x3b78
Faulting application start time: 0x01cdcc96f46da952
Faulting application path: C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL2008\MSSQL\Binn\sqlservr.exe
Faulting module path: C:\Program Files\Common Files\Microsoft Shared\office14\mso.dll
Report Id: fcfb0fc2-5e76-11e2-8623-a4badb0bce84
Karthick P.K said
Hello Scott,
Few days back one of the SQL DBA came to me with same issue which you faced SQLServer carsh by by wihttp.dll (winhttp!RemoveFromSerializedList). He shared the memory dump with me. We looked at the dump and found TLS (thread local storage, a mechanism by which threads can store data that is unique to each thread) is corrupted for winhttp.dll thread.When winhttp tried to do cleanup in the TLS, it encountered AV as the TLS is corrupted (Same issue as yours) but who corrupted the thread is still not clear. I am suspecting it to be mso.dll because “App verifier” pointed that this dll is corrupting TLS when I was debugging an other TLS corruption issue. There is also a FIX http://support.microsoft.com/kb/2516475 which will update MSO.dll
Can you please confirm if the crash by MSO.dll happened in the same server in which we got dump because of winhttp.dll? If yes please confirm me and apply the above fix for MSO.dll
Also can you let me know if below two options are checked in (Internet explorer–>Tools–>Internet options –>Advanced)
1. “Check for publisher’s certificate revocation”
2. “Check for server certificate revocation”
Thanks
Scott McCormick said
Hey Karthick –
This is the same server, and I’ll start pushing to apply this patch.
Also, those options are selected in the IE on the server.
Scott
Karthick P.K said
If those options are selected please uncheck them
“Check for publisher’s certificate revocation”
“Check for server certificate revocation”
Karthick P.K said
Any update Scott?
Scott McCormick said
Yeah, I flipped the switches on IE and installed the patch and it looks like that might have resolved it. It comes and goes, so it’s hard to say for sure, but I don’t think it’s an issue any more. Thanks a lot for your help, I really appreciate it.
In Depth SQL said
Hi Karthick – I follow mentioned procedure and got below dump, Please let me know what should I do next.
0:000> kC 1000
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqlservr!CDmpDump::DumpInternal
sqlservr!CDmpDump::DumpFilter
sqlservr!CDmpDump::Dump
msvcr80!__C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!RtlRaiseException
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_terminator
sqlservr!SOS_SEHTranslator
msvcr80!_CallSETranslator
msvcr80!FindHandlerForForeignException
msvcr80!FindHandler
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
sqlservr!CSlotPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!CMemThread::Alloc
sqlservr!CCriticalSectionSOS::Initialize
sqlservr!Tcp::FInit
sqlservr!Tcp::AcceptConnection
sqlservr!Tcp::AcceptDone
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Karthick P.K said
Hey Indepth,
This issue is fixed in
Cumulative Update package 6 for SQL Server 2008 R2
Cumulative update package 13 for SQL Server 2008 Service Pack 1
Cumulative update package 3 for SQL Server 2008 Service Pack 2
Thanks
KarthickP.K
In Depth SQL said
Hey Karthick – Thanks for the reply but what is the problem and how it can fix in CU. i.e. where/how did you fid that information, please share.
Karthick P.K said
Answered to Indepth offline
In Depth SQL said
Thanks Karthick for your help !!!
Jason Chamiak said
Hi Karthick,
This was a great article. After following your processes, I retrieved the following thread stack responsible for my recent crash.Can you tell me what happened here and what recommendations you have?
0:000> kC 1000
Call Site
sqlservr!CMEDIndexStatsCollection::GetProxyIndexWithLock
sqlservr!CMEDProxyRelation::GetProxyIndexWithLock
sqlservr!CMEDProxyRelation::GetIndex
sqlservr!CMetadataUtil::GetIndexFromIMED
sqlservr!CAlgTableMetadata::PimidxBaseIndex
sqlservr!CAlgTableMetadata::LoadSpecialColumns
sqlservr!CAlgTableMetadata::LoadColumns
sqlservr!CAlgTableMetadata::Bind
sqlservr!CRelOp_Get::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_FromList::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Select::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_QuerySpec::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_DerivedTable::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_BaseJoin::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_FromList::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_QuerySpec::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Insert::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_DMLQuery::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Query::FAlgebrizeQuery
sqlservr!CProchdr::FNormQuery
sqlservr!CProchdr::FNormalizeStep
sqlservr!CSQLSource::FCompile
sqlservr!CSQLSource::FCompWrapper
sqlservr!CSQLSource::Transform
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Thanks,
Jason Chamiak
Karthick P.K said
Hello Jason, This looks like a known issue, can let me know your SQL Server build number (select @@version). Do you get this dump when you fire query with “select TOP n on a table with identity column? ThanksKarthick > Date: Fri, 16 Nov 2012 19:53:19 +0000 > To: karthick_pk83@hotmail.com >
Jason Chamiak said
Hi Karthick,
This is the output of @@version:
Microsoft SQL Server 2008 R2 (SP1) – 10.50.2500.0 (X64) Jun 17 2011 00:54:03 Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows NT 6.1 (Build 7600: )
I do not see “SELECT TOP” in the stored procedures referenced in the dumps in the SQL Server log.
Thanks,
Jason Chamiak
Karthick P.K said
Hello Jason,
Apply Service pack 2 for SQL server 2008 R2 . If issue happens again after applying SP2 please let me know
Nuno Gomes said
0:000> kC 1000
Call Site
sqlservr!CTableVariableManager::GetRowsetidListAndReturnTableInfo
sqlservr!CRangeTableVariable::XretSchemaChanged
sqlservr!CEnvCollection::XretSchemaChanged
sqlservr!CXStmtCondWithQuery::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
And the version is: 10.50.1600.1
Thanks,
Nuno Gomes
Karthick P.K said
Quick question… Have you set the MAX DOP to 1?
nmgomes said
Yes … This error happens when Max DOP is set to 1.
Karthick P.K said
This is known issue in SQL Server 2008
For SQL Server 2008 R2, this issue is fixed in Service Pack 1
For SQL Server 2008, this issue is fixed in Service Pack 3
How to Analyze "Deadlocked Schedulers" Dumps? « MSSQLWIKI said
[...] SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion [...]
Meera Sinhasane said
Hi Karthick ,
SQL Server in our environment crashed generating an access violation. From the dump file it appears to be crashed due to msxml component but i am not pretty sure on it. Could you please take a look at the below stack and guide me on this.
Microsoft SQL Server 2008 R2 (RTM) – 10.50.1600.1 (X64)
Apr 2 2010 15:48:46
Copyright (c) Microsoft Corporation
Enterprise Edition (64-bit) on Windows NT 6.1 (Build 7600: ) (Hypervisor)
2013-01-14 21:33:51.74 Server Error: 17311, Severity: 16, State: 1.
2013-01-14 21:33:51.74 Server SQL Server is terminating because of fatal exception c0000005. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).
2013-01-14 21:33:51.79 Server Using ‘dbghelp.dll’ version ’4.0.5′
2013-01-14 21:33:51.83 Server **Dump thread – spid = 0, EC = 0×0000000000000000
2013-01-14 21:33:51.83 Server ***Stack Dump being sent to D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\LOG\SQLDump0001.txt
2013-01-14 21:33:51.83 Server * *******************************************************************************
2013-01-14 21:33:51.83 Server *
2013-01-14 21:33:51.83 Server * BEGIN STACK DUMP:
2013-01-14 21:33:51.83 Server * 01/14/13 21:33:51 spid 2636
2013-01-14 21:33:51.83 Server *
2013-01-14 21:33:51.83 Server * ex_handle_except encountered exception C0000005 – Server terminating
0:000> kc 100
Call Site
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_handle_except
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
msxml6!Base::freeRentalObjects
msxml6!Base::StackExitNormal
msxml6!ModelInit::~ModelInit
msxml6!SAXReader::parse
MSO
0×0
0×0
0×0
Karthick P.K said
Do you have MSO.dll loaded in SQL Server? what is the version?
Is this the first and only occurrence or do we have more dumps?
Karthick P.K said
any update?
Laurens said
Hi Karthick,
Could you please have a look at the debug below. I see that the wininet is probably causing the problem? There are no jobs running during this proces apart from a ms windows server backup. We do use CDO during the day, could the VC be cleaning the memory during the night at 1:30? Or could the windows update process use wininet?
Microsoft (R) Windows Debugger Version 6.2.9200.16384 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [D:\Data\MSSQL10_50.MSSQLSERVER\MSSQL\Log\SQLDump0003.mdmp]
Comment: ‘Stack Trace’
Comment: ‘ex_handle_except encountered exception C0000005 – Server terminating’
User Mini Dump File: Only registers, stack and portions of memory are available
Symbol search path is: *** Invalid ***
****************************************************************************
* Symbol loading may be unreliable without a symbol search path. *
* Use .symfix to have the debugger choose a symbol path. *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) UP Free x64
Product: Server, suite: TerminalServer SingleUserTS Blade
Machine Name:
Debug session time: Tue Jan 29 01:31:18.000 2013 (UTC + 1:00)
System Uptime: 3 days 16:22:27.234
Process Uptime: 3 days 16:21:44.000
……………………………………………………….
……………………………………………………….
…………
Loading unloaded module list
……………………………………………………………
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(598.e94): Unknown exception – code 000042ac (first/second chance not available)
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll –
*** ERROR: Symbol file could not be found. Defaulted to export symbols for KERNELBASE.dll –
ntdll!NtWaitForSingleObject+0xa:
00000000`7707135a c3 ret
0:000> .sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
Symbol search path is: srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
0:000> .reload /f
.*** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlservr.exe –
…….
Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.
…*** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlos.dll –
……………………..*** ERROR: Symbol file could not be found. Defaulted to export symbols for instapi10.dll –
..*** ERROR: Module load completed but symbols could not be loaded for sqlevn70.rll
…………………*** ERROR: Symbol file could not be found. Defaulted to export symbols for msfte.dll –
….
………..*** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlncli10.dll –
…*** ERROR: Module load completed but symbols could not be loaded for SQLNCLIR10.RLL
…….*** ERROR: Module load completed but symbols could not be loaded for odbcint.dll
…*** ERROR: Module load completed but symbols could not be loaded for XPStar.RLL
..*** ERROR: Module load completed but symbols could not be loaded for xplog70.rll
……..*** ERROR: Module load completed but symbols could not be loaded for oledb32r.dll
..*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEOLEDB.DLL –
..*** ERROR: Symbol file could not be found. Defaulted to export symbols for MSO.DLL –
…*** ERROR: Module load completed but symbols could not be loaded for ACECORE.DLL
..*** ERROR: Module load completed but symbols could not be loaded for ACEWSTR.DLL
.*** ERROR: Module load completed but symbols could not be loaded for MSORES.DLL
.*** ERROR: Module load completed but symbols could not be loaded for MSOINTL.DLL
.*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEINTL.DLL –
………..*** ERROR: Module load completed but symbols could not be loaded for INETRES.dll
…….
*** ERROR: Module load completed but symbols could not be loaded for ACCTRES.dll
…..*** ERROR: Module load completed but symbols could not be loaded for odsole70.rll
..*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEERR.DLL –
.*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEES.DLL –
….
Loading unloaded module list
……………………………………………………………
0:000> lmvm sqlservr
start end module name
00000000`00b10000 00000000`04722000 sqlservr (export symbols) sqlservr.exe
Loaded symbol image file: sqlservr.exe
Mapped memory image file: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Image path: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Thu Jun 28 22:59:38 2012 (4FECC5BA)
CheckSum: 03B58282
ImageSize: 03C12000
File version: 2009.100.4000.0
Product version: 10.50.4000.0
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft SQL Server
InternalName: SQLSERVR
OriginalFilename: SQLSERVR.EXE
ProductVersion: 10.50.4000.0
FileVersion: 2009.0100.4000.00 ((KJ_PCU_Main).120628-0827 )
FileDescription: SQL Server Windows NT – 64 Bit
LegalCopyright: Microsoft Corp. All rights reserved.
LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
Comments: SQL
0:000> Type .ecxr
^ No runnable debuggees error in ‘Type .ecxr ‘
0:000> .ecxr
rax=000000006e6313fe rbx=00000000040f5650 rcx=000000001976ca90
rdx=0000000000000000 rsi=0000000003cffcfc rdi=0000000000000440
rip=000007fefd7a9e5d rsp=000000001976d0a0 rbp=000000001976fe30
r8=0000000000000000 r9=0000000000000000 r10=00000000c000007c
r11=00000000013fdf60 r12=0000000000000e94 r13=0000000000000000
r14=0000000000000000 r15=0000000000000e94
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
KERNELBASE!RaiseException+0×39:
000007fe`fd7a9e5d 4881c4c8000000 add rsp,0C8h
0:000> kC 1000
Call Site
KERNELBASE!RaiseException
sqlservr!DmpRemoteDumpRequest
sqlservr!SQLExit
sqlservr!SQLExit
sqlservr!SQLExit
sqlservr!SQLExit
sqlservr!SQLExit
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatch
wininet!InternetFreeThreadInfo
wininet!InternetDestroyThreadInfo
wininet!DllMain_wininet
wininet!CRT_INIT
ntdll!LdrShutdownThread
ntdll!RtlExitUserThread
msvcr80!endthreadex
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
0:000> kL
*** Stack trace for last set context – .thread/.cxr resets it
Child-SP RetAddr Call Site
00000000`1976d0a0 00000000`036c440c KERNELBASE!RaiseException+0×39
00000000`1976d170 00000000`02bfae68 sqlservr!DmpRemoteDumpRequest+0x2fec
00000000`1976d1c0 00000000`02bfb9bc sqlservr!SQLExit+0x303a8
00000000`1976d1f0 00000000`02bee097 sqlservr!SQLExit+0x30efc
00000000`1976d3f0 00000000`02bed7e9 sqlservr!SQLExit+0x235d7
00000000`1976e940 00000000`03032206 sqlservr!SQLExit+0x22d29
00000000`1976e980 00000000`769d9460 sqlservr!SQLExit+0×467746
00000000`1976ebd0 00000000`770b43b8 kernel32!UnhandledExceptionFilter+0×160
00000000`1976ecb0 00000000`770385a8 ntdll! ?? ::FNODOBFM::`string’+0×2365
00000000`1976ece0 00000000`77049d0d ntdll!_C_specific_handler+0x8c
00000000`1976ed50 00000000`770391af ntdll!RtlpExecuteHandlerForException+0xd
00000000`1976ed80 00000000`77071278 ntdll!RtlDispatchException+0x45a
00000000`1976f460 00000000`76ee88cb ntdll!KiUserExceptionDispatch+0x2e
00000000`1976fa20 00000000`76ee8892 wininet!InternetFreeThreadInfo+0x1f
00000000`1976fa50 00000000`76ec128e wininet!InternetDestroyThreadInfo+0×40
00000000`1976fa80 00000000`76ec10d3 wininet!DllMain_wininet+0xb5
00000000`1976fad0 00000000`77066a28 wininet!CRT_INIT+0x2b3
00000000`1976fc30 00000000`77066968 ntdll!LdrShutdownThread+0×155
00000000`1976fd30 00000000`747b37b7 ntdll!RtlExitUserThread+0×38
00000000`1976fd70 00000000`747b37de msvcr80!endthreadex+0×27
00000000`1976fda0 00000000`747b3894 msvcr80!endthreadex+0x4e
00000000`1976fdd0 00000000`7695652d msvcr80!endthreadex+0×104
00000000`1976fe00 00000000`7704c521 kernel32!BaseThreadInitThunk+0xd
00000000`1976fe30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
Karthick P.K said
Can you share the output of
1. lmvm mso
2. lmvm cdosys
Laurens said
0:000> lmvm cdosys
start end module name
000007fe`e7c50000 000007fe`e7d69000 cdosys (pdb symbols) c:\websymbols\cdosys.pdb\3C0CCADEEF824576A6729525EBB42A8C1\cdosys.pdb
Loaded symbol image file: cdosys.dll
Mapped memory image file: c:\websymbols\cdosys.dll\4FCEEFFF119000\cdosys.dll
Image path: C:\Windows\System32\cdosys.dll
Image name: cdosys.dll
Timestamp: Wed Jun 06 07:51:59 2012 (4FCEEFFF)
CheckSum: 00117D93
ImageSize: 00119000
File version: 6.6.7601.17857
Product version: 6.6.7601.17857
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: CDOSYS.DLL
OriginalFilename: CDOSYS.DLL
ProductVersion: 6.6.7601.17857
FileVersion: 6.6.7601.17857 (win7sp1_gdr.120605-1503)
FileDescription: Microsoft CDO for Windows Library
LegalCopyright: © Microsoft Corporation. All rights reserved.
0:000> lmvm mso
start end module name
000007fe`e8090000 000007fe`e98c2000 MSO (export symbols) MSO.DLL
Loaded symbol image file: MSO.DLL
Mapped memory image file: C:\Program Files\Common Files\Microsoft Shared\OFFICE14\MSO.DLL
Image path: C:\Program Files\Common Files\Microsoft Shared\OFFICE14\MSO.DLL
Image name: MSO.DLL
Timestamp: Tue Mar 23 19:22:16 2010 (4BA906D8)
CheckSum: 01832834
ImageSize: 01832000
File version: 14.0.4760.1000
Product version: 14.0.4760.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04e4
CompanyName: Microsoft Corporation
ProductName: Microsoft Office 2010
InternalName: MSO
OriginalFilename: MSO.DLL
ProductVersion: 14.0.4760.1000
FileVersion: 14.0.4760.1000
FileDescription: Microsoft Office 2010 component
LegalCopyright: © 2010 Microsoft Corporation. All rights reserved.
Karthick P.K said
Apply http://support.microsoft.com/kb/2516475 it should resolve your issue.
Laurens said
Brilliant, thanks for the analysis, how did you figure it’s the mso.dll? Did the debugger indicate this or is it on the basis that there’s a newer version of the mso.dll available?
Karthick P.K said
If you look at your stack “wininet!InternetFreeThreadInfo” exception is raised when the threads local storage (TLS) is cleared,So basically some one else has already corrupted TLS.
Who corrupted it? Most probably MSO.DLL because “App verifier” pointed that this dll is corrupting TLS when I was debugging an other TLS corruption issue. There is also a FIX http://support.microsoft.com/kb/2516475 which will update MSO.dll.
Laurens said
Tanks for the reply, I’ve tried to install the update, states that it is already installed, I believe I read somewhere you have to place files manualy?
Karthick P.K said
Hello Lauren,
I do know how I missed your comment.Iam sorry.
Try running repair on Microsoft access database engine,It should update your MSO.DLL version
Laurens said
Hi Karthick,
So far so good, the server has stopped crashing. In order to install the update, you have to extract it, until you can extract it no longer, run it, then use the repair function from the installed applications in the control panel.
Thanks,
Laurens
manu0417Manohar said
Hi Karthick,
Can you please help me with below access violation exception.
0:000> kC 1000
Call Site
sqlservr!CTableVariableManager::InsertTableRef
sqlservr!alloca_probe
sqlservr!CFormalParamTbl::InitParams
sqlservr!alloca_probe
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!alloca_probe
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
0×0
0×0
Karthick P.K said
What is ur sqlversion?
Sent from my Windows Phone ________________________________
manu0417 said
SQL Server 2008 SP2 CU3
Version: 10.0.4279
Karthick P.K said
Have you set Max degree of parallelism to 1 ?
manu0417 said
Yes, We need that across out environment.
Karthick P.K said
I think we have two options here .If you can’t use any of below workaround I would suggest contacting Microsoft support to investigate further. 1. Use TEMP Tables instead of Table Variables or UDF’s 2. Change the MAX DOP to > 1 [2 for example]
manu0417 said
Thanks Karthick for the quick turn around. I will try if anything works out or will contact support.
manu0417 said
Karthick,
Can you please shed some light on how the max dop is resulting the access violation when we use table variables? Greatly appreciate any inputs on this.
Regards,
Manu
Karthick P.K said
This is known issue
manu0417 said
Hi Karthick,
Today we had an assertion on one of our Test servers and below is the kC 1000 output. The server went unresponsive after that.
0:000> kC 1000
Call Site
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!utassert_fail
sqlservr!purecall
sqlservr!GetRemoteTaskProxyWithRef
sqlservr!TasksTable::MoveToNextRow
sqlservr!TasksTable::InternalGetRow
sqlservr!CQScanTVFStreamNew::GetRow
sqlservr!CQScanSortNew::BuildSortTable
sqlservr!CQScanSortNew::OpenHelper
sqlservr!CQScanMergeJoinNew::Open
sqlservr!CQScanMergeJoinNew::Open
sqlservr!CQScanMergeJoinNew::Open
sqlservr!CQScanNLJoinNew::Open
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanTopNew::Open
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanUpdateNew::Open
sqlservr!CQueryScan::Startup
sqlservr!CXStmtQuery::SetupQueryScanAndExpression
sqlservr!CXStmtQuery::ErsqExecuteQuery
sqlservr!CXStmtDML::XretDMLExecute
sqlservr!CXStmtSelectInto::XretSelectIntoExecute
sqlservr!CXStmtSelectInto::XretExecute
sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!alloca_probe
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!alloca_probe
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
0×0
0×0
Karthick P.K said
Share the Build of SQLServer and let me know if it used as backend for Biztalk.
manu0417 said
We don’t use a BizTalk. Build is 10.0.4279, 2008 SP2 CU3.
The procedure which raised this Assertion does the auditing on the session activity and inserts the sampled data in to an audit table, bunch of simple select and insert statements.
Karthick P.K said
Hello Manu,
Can you please share the error message you got in errorlog for this assert?
It should be some thig like
SQL Server Assertion: File: , line
in your errorlog when dump was generated.
manu0417 said
2013-03-01 04:00:03.01 spid231 ***Stack Dump being sent to D:\MSSQL10.MSSQLSERVER\MSSQL\LOG\SQLDump1160.txt
2013-03-01 04:00:03.01 spid231 * *******************************************************************************
2013-03-01 04:00:03.01 spid231 *
2013-03-01 04:00:03.01 spid231 * BEGIN STACK DUMP:
2013-03-01 04:00:03.01 spid231 * 03/01/13 04:00:03 spid 231
2013-03-01 04:00:03.01 spid231 *
2013-03-01 04:00:03.01 spid231 * Location: purecall.cpp:51
2013-03-01 04:00:03.01 spid231 * Expression: !”purecall”
2013-03-01 04:00:03.01 spid231 * SPID: 231
2013-03-01 04:00:03.01 spid231 * Process ID: 21520
2013-03-01 04:00:03.01 spid231 * Description: Pure virtual function call
Karthick P.K said
Do you have any antivirus which detours in sqlserver address space like sophose etc?
Post the output of lm from your dump. Let me check
manu0417 said
I did not find any Antivirus. Below is the result of lm
0:000> lm
start end module name
00000000`002a0000 00000000`03b0f000 sqlservr (deferred)
00000000`1f320000 00000000`1f369000 System_Transactions (deferred)
00000000`2de70000 00000000`2e16f000 System_Data (deferred)
00000000`48060000 00000000`4806d000 instapi (deferred)
00000000`49980000 00000000`49d31000 msfte (deferred)
00000000`60000000 00000000`60025000 ftimport (deferred)
00000000`6de10000 00000000`6e00f000 sqlevn70 (deferred)
00000000`6f3d0000 00000000`6f52e000 DBGHELP (deferred)
00000000`6fbf0000 00000000`6fefe000 System (deferred)
00000000`6ff00000 00000000`7003e000 comres (deferred)
00000000`70520000 00000000`70562000 System_Security (deferred)
00000000`70570000 00000000`705dc000 System_Configuration (deferred)
00000000`705e0000 00000000`707d8000 System_Xml (deferred)
00000000`714b0000 00000000`714c4000 oledb32r (deferred)
00000000`71640000 00000000`71791000 msxmlsql (deferred)
00000000`71840000 00000000`71857000 xprepl (deferred)
00000000`71920000 00000000`719a8000 XPStar (deferred)
00000000`71e70000 00000000`71e72000 xplog70_71e70000 (deferred)
00000000`71e80000 00000000`71e90000 xplog70 (deferred)
00000000`71f20000 00000000`71f45000 XPStar_71f20000 (deferred)
00000000`71f50000 00000000`71f58000 xpsqlbot (deferred)
00000000`71fe0000 00000000`71fec000 sqlscm (deferred)
00000000`720e0000 00000000`7210c000 BatchParser (deferred)
00000000`72110000 00000000`72147000 SQLNCLIR10 (deferred)
00000000`72150000 00000000`7245a000 sqlncli10 (deferred)
00000000`724b0000 00000000`7260e000 dbghelp_724b0000 (deferred)
00000000`727f0000 00000000`727f7000 sqlos (deferred)
00000000`72800000 00000000`72808000 opends60 (deferred)
00000000`72fa0000 00000000`72fb3000 ssdebugps (deferred)
00000000`73640000 00000000`73678000 odbcint (deferred)
00000000`737d0000 00000000`73837000 SqlAccess (deferred)
00000000`73840000 00000000`73843000 security (deferred)
00000000`73cf0000 00000000`73cfd000 instapi10 (deferred)
00000000`74850000 00000000`74870000 ATL80 (deferred)
00000000`74ad0000 00000000`74bd9000 msvcp80 (deferred)
00000000`74be0000 00000000`74ca9000 msvcr80 (deferred)
00000000`76d70000 00000000`76e6a000 user32 (deferred)
00000000`76e70000 00000000`76f8f000 kernel32 (deferred)
00000000`76f90000 00000000`77139000 ntdll (pdb symbols) your local symbol folder\ntdll.pdb\15EB43E23B12409C84E3CC7635BAF5A32\ntdll.pdb
00000000`77160000 00000000`77167000 psapi (deferred)
00000642`ffaf0000 00000642`ffb09000 mscorsec (deferred)
00000642`ffff0000 00000642`ffffd000 wminet_utils (deferred)
000007fe`ebd00000 000007fe`ebde6000 System_Security_ni (deferred)
000007fe`ebdf0000 000007fe`ebefe000 oledb32 (deferred)
000007fe`ecae0000 000007fe`ecc64000 mscorjit (deferred)
000007fe`ecc70000 000007fe`eccbe000 pdh (deferred)
000007fe`eccc0000 000007fe`ecce8000 msdart (deferred)
000007fe`eccf0000 000007fe`ecd90000 comctl32 (deferred)
000007fe`ed8d0000 000007fe`ed90b000 winmm (deferred)
000007fe`edc20000 000007fe`edc8a000 msdaps (deferred)
000007fe`ef450000 000007fe`ef501000 odbc32 (deferred)
000007fe`ef530000 000007fe`ef5ec000 msdtcprx (deferred)
000007fe`ef5f0000 000007fe`ef650000 mtxclu (deferred)
000007fe`efb60000 000007fe`efb6f000 cscapi (deferred)
000007fe`efb70000 000007fe`efb8b000 cabinet (deferred)
000007fe`efb90000 000007fe`efbb6000 cryptnet (deferred)
000007fe`efd10000 000007fe`efd19000 SensApi (deferred)
000007fe`efe30000 000007fe`efe42000 browcli (deferred)
000007fe`efe50000 000007fe`efe5a000 netbios (deferred)
000007fe`efe80000 000007fe`efe94000 xolehlp (deferred)
000007fe`f0c00000 000007fe`f0c19000 resutils (deferred)
000007fe`f0c20000 000007fe`f0c70000 clusapi (deferred)
000007fe`f0e00000 000007fe`f0e26000 wmiutils (deferred)
000007fe`f0e30000 000007fe`f0e44000 wbemsvc (deferred)
000007fe`f0f00000 000007fe`f0f27000 ntdsapi (deferred)
000007fe`f0f30000 000007fe`f1012000 fastprox (deferred)
000007fe`f6f60000 000007fe`f7985000 System_ni (deferred)
000007fe`f7990000 000007fe`f886c000 mscorlib_ni (deferred)
000007fe`f8870000 000007fe`f920d000 mscorwks (deferred)
000007fe`f9210000 000007fe`f92a0000 mscoreei (deferred)
000007fe`f92a0000 000007fe`f930f000 mscoree (deferred)
000007fe`f9310000 000007fe`f9396000 wbemcomn (deferred)
000007fe`f93a0000 000007fe`f93af000 wbemprox (deferred)
000007fe`f9600000 000007fe`f961d000 samlib (deferred)
000007fe`f9620000 000007fe`f9634000 samcli (deferred)
000007fe`f9a70000 000007fe`f9a78000 rasadhlp (deferred)
000007fe`fa0d0000 000007fe`fa134000 webio (deferred)
000007fe`fa140000 000007fe`fa1b1000 winhttp (deferred)
000007fe`fa1c0000 000007fe`fa1d8000 dhcpcsvc (deferred)
000007fe`fa1e0000 000007fe`fa1f1000 dhcpcsvc6 (deferred)
000007fe`fa200000 000007fe`fa253000 FWPUCLNT (deferred)
000007fe`fa690000 000007fe`fa6a5000 wkscli (deferred)
000007fe`fa6b0000 000007fe`fa6c6000 netapi32 (deferred)
000007fe`fa710000 000007fe`fa71b000 winnsi (deferred)
000007fe`fa720000 000007fe`fa747000 IPHLPAPI (deferred)
000007fe`fae60000 000007fe`fae69000 msxactps (deferred)
000007fe`fb150000 000007fe`fb15a000 ktmw32 (deferred)
000007fe`fb820000 000007fe`fb82c000 dsrole (deferred)
000007fe`fb870000 000007fe`fba64000 comctl32_7fefb870000 (deferred)
000007fe`fbff0000 000007fe`fc01d000 ntmarta (deferred)
000007fe`fc020000 000007fe`fc02c000 version (deferred)
000007fe`fc0f0000 000007fe`fc0f7000 WSHTCPIP (deferred)
000007fe`fc1f0000 000007fe`fc20b000 gpapi (deferred)
000007fe`fc210000 000007fe`fc22e000 userenv (deferred)
000007fe`fc230000 000007fe`fc242000 devrtl (deferred)
000007fe`fc270000 000007fe`fc2bc000 bcryptprimitives (deferred)
000007fe`fc370000 000007fe`fc37a000 credssp (deferred)
000007fe`fc3a0000 000007fe`fc3fb000 dnsapi (deferred)
000007fe`fc560000 000007fe`fc5a7000 rsaenh (deferred)
000007fe`fc5f0000 000007fe`fc648000 schannel (deferred)
000007fe`fc650000 000007fe`fc6a1000 msv1_0 (deferred)
000007fe`fc6b0000 000007fe`fc6bc000 netutils (deferred)
000007fe`fc6d0000 000007fe`fc700000 logoncli (deferred)
000007fe`fc700000 000007fe`fc755000 mswsock (deferred)
000007fe`fc760000 000007fe`fc814000 kerberos (deferred)
000007fe`fc890000 000007fe`fc8a7000 cryptsp (deferred)
000007fe`fc8b0000 000007fe`fc8fe000 ncrypt (deferred)
000007fe`fc900000 000007fe`fc92f000 authz (deferred)
000007fe`fc930000 000007fe`fc937000 wship6 (deferred)
000007fe`fc940000 000007fe`fc962000 bcrypt (deferred)
000007fe`fc9f0000 000007fe`fca04000 cryptdll (deferred)
000007fe`fca10000 000007fe`fca33000 srvcli (deferred)
000007fe`fccf0000 000007fe`fccfb000 secur32 (deferred)
000007fe`fcd40000 000007fe`fcd65000 sspicli (deferred)
000007fe`fcd70000 000007fe`fcdc7000 apphelp (deferred)
000007fe`fcdd0000 000007fe`fcddf000 CRYPTBASE (deferred)
000007fe`fcec0000 000007fe`fced4000 RpcRtRemote (deferred)
000007fe`fcee0000 000007fe`fceef000 profapi (deferred)
000007fe`fcf80000 000007fe`fcf8f000 msasn1 (deferred)
000007fe`fcfd0000 000007fe`fd03c000 KERNELBASE (pdb symbols) your local symbol folder\kernelbase.pdb\61044362232B410AA600843CEBFD11612\kernelbase.pdb
000007fe`fd040000 000007fe`fd07a000 wintrust (deferred)
000007fe`fd140000 000007fe`fd2a7000 crypt32 (deferred)
000007fe`fd2b0000 000007fe`fd2be000 lpk (deferred)
000007fe`fd2c0000 000007fe`fd389000 usp10 (deferred)
000007fe`fd390000 000007fe`fd398000 nsi (deferred)
000007fe`fd550000 000007fe`fd627000 oleaut32 (deferred)
000007fe`fd630000 000007fe`fd70b000 advapi32 (deferred)
000007fe`fd710000 000007fe`fd7a7000 comdlg32 (deferred)
000007fe`fd930000 000007fe`fe6b8000 shell32 (deferred)
000007fe`fe6c0000 000007fe`fe7c9000 msctf (deferred)
000007fe`fe7d0000 000007fe`fe86f000 msvcrt (deferred)
000007fe`fe870000 000007fe`fe8d7000 gdi32 (deferred)
000007fe`fe8e0000 000007fe`fe979000 clbcatq (deferred)
000007fe`fe980000 000007fe`feb83000 ole32 (deferred)
000007fe`feb90000 000007fe`febdd000 ws2_32 (deferred)
000007fe`febe0000 000007fe`fec32000 Wldap32 (deferred)
000007fe`fec40000 000007fe`fec5f000 sechost (deferred)
000007fe`fec60000 000007fe`fec8e000 imm32 (deferred)
000007fe`feef0000 000007fe`fef07000 imagehlp (deferred)
000007fe`fef10000 000007fe`fef81000 shlwapi (deferred)
000007fe`ff170000 000007fe`ff29d000 rpcrt4 (deferred)
Unloaded modules:
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`73880000 00000000`73898000 sqlboot.dll
00000000`73860000 00000000`73878000 sqlboot.dll
00000000`73880000 00000000`73898000 sqlboot.dll
000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
00000000`72500000 00000000`72518000 sqlboot.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
00000000`72500000 00000000`72518000 sqlboot.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
00000000`72500000 00000000`72518000 sqlboot.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
00000000`72500000 00000000`72518000 sqlboot.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
00000000`72560000 00000000`72578000 sqlboot.dll
00000000`725b0000 00000000`725c8000 sqlboot.dll
000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
000007fe`fd120000 000007fe`fd13a000 DEVOBJ.dll
000007fe`fef90000 000007fe`ff167000 setupapi.dll
00000000`71870000 00000000`71888000 sqlboot.dll
00000000`71800000 00000000`71818000 sqlboot.dll
00000000`71870000 00000000`71888000 sqlboot.dll
00000000`71800000 00000000`71818000 sqlboot.dll
00000000`71870000 00000000`71888000 sqlboot.dll
00000000`71800000 00000000`71818000 sqlboot.dll
00000000`71870000 00000000`71888000 sqlboot.dll
00000000`71800000 00000000`71818000 sqlboot.dll
00000000`71870000 00000000`71888000 sqlboot.dll
00000000`71800000 00000000`71818000 sqlboot.dll
00000000`71870000 00000000`71888000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`6e050000 00000000`6e1ae000 dbghelp.dll
00000000`72540000 00000000`7269e000 dbghelp.dll
00000000`6e050000 00000000`6e1ae000 dbghelp.dll
00000000`6b4d0000 00000000`6b62e000 dbghelp.dll
000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
000007fe`fd120000 000007fe`fd13a000 DEVOBJ.dll
000007fe`fef90000 000007fe`ff167000 setupapi.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`715b0000 00000000`715c8000 sqlboot.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`715b0000 00000000`715c8000 sqlboot.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`715b0000 00000000`715c8000 sqlboot.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`715b0000 00000000`715c8000 sqlboot.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`715b0000 00000000`715c8000 sqlboot.dll
00000000`71490000 00000000`714a8000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`72c80000 00000000`72c98000 sqlboot.dll
00000000`73770000 00000000`73788000 sqlboot.dll
00000000`73880000 00000000`73898000 sqlboot.dll
00000000`73860000 00000000`73878000 sqlboot.dll
00000000`73880000 00000000`73898000 sqlboot.dll
000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
Karthick P.K said
Share the dump with me I look at it. If there is no security restriction.
manu0417 said
I have mailed it yesterday to karthick_pk83@hotmail.com
Karthick P.K said
Maanu I analyzed the dump and this dump looks like a victim of a dump which occurred earlier. Did this assert occurred after some Access violation? If so you have to fix the access violation and this assert will resolve by itself.
Dee said
What about this…I have 53 stack dumps all with different output…this is the results of the first one for the day..
running SQL server 2008 sp1 32 bit…
0:000> lmvm sqlservr
start end module name
01000000 03970000 sqlservr T (no symbols)
Loaded symbol image file: sqlservr.exe
Image path: E:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Sun Mar 29 17:29:20 2009 (49CFF640)
CheckSum: 029079F8
ImageSize: 02970000
File version: 2007.100.2531.0
Product version: 10.0.2531.0
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
0:000> .ecxr
eax=693ad01c ebx=00000000 ecx=03550ab4 edx=03550ab4 esi=00000000 edi=0357b640
eip=77e4bee7 esp=693ad018 ebp=693ad06c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
kernel32!RaiseException+0×53:
77e4bee7 5e pop esi
0:000> kC 1000
kernel32!RaiseException
WARNING: Stack unwind information not available. Following frames may be wrong.
sqlservr
sqlservr
sqlservr
sqlservr
sqlservr
sqlservr
msvcr80!_CallSETranslator
msvcr80!FindHandlerForForeignException
msvcr80!FindHandler
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler3
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
sqlservr
another one….
eax=4cb7a000 ebx=07dde040 ecx=005c0033 edx=00000000 esi=4cb7b020 edi=07dde0a4
eip=015ab31c esp=69dcf534 ebp=69dcf55c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
sqlservr!XVB::`scalar deleting destructor’+0×21:
015ab31c 8b4204 mov eax,dword ptr [edx+4] ds:0023:00000004=????????
0:000> kC 1000
sqlservr!XVB::`scalar deleting destructor’
sqlservr!ReadOnlyXactImp::~ReadOnlyXactImp
sqlservr!ReadOnlyXactImp::`vector deleting destructor’
sqlservr!AutoReadOnlyXact::Cleanup
Dee said
1st one from in January…right after upgrated to 2008 sp1 32bit from 2005. on same server
someone thought this was caused by 3rd party wonderware.. cant tell if new dumps are related….
0:000> .ecxr
eax=00b1dffc ebx=00000000 ecx=03550ab4 edx=03550ab4 esi=00000000 edi=0357b640
eip=77e4bee7 esp=00b1dff8 ebp=00b1e04c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
kernel32!RaiseException+0×53:
77e4bee7 5e pop esi
0:000> kC 1000
kernel32!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_terminator
msvcr80!_CallSETranslator
msvcr80!FindHandlerForForeignException
msvcr80!FindHandler
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler3
ntdll!ExecuteHandler2
ntdll!ExecuteHandler
ntdll!KiUserExceptionDispatcher
sqlservr!CVariableInfo::PbGetBlock
Karthick P.K said
Paste the full stack
Nitin said
Hi karthick
I need your help to analyze the sql server dump thrown indicating ‘Exception_Access_Violation’. I have tried to use Windebug to analyze the dump.
But it is giving error unable to load symbol.
Could you please help me to analyze the root caause for this dump.
Karthick P.K said
What is your sqlserver version?
SQL Server assert in Location: purecall.cpp:51 | SQLServerScribbles said
[...] 3. If there is no antivirus or memory errors follow the steps in http://mssqlwiki.com/2012/10/16/sql-server-exception_access_violation-and-sql-server-assertion/ [...]
Farhan said
Hi Karthick,
Very nice article – thanks for sharing.
Could you please have a look on following dump file contents and advise what actually caused our SQL server to crash.
In log folder I can see two dump files, one is related to todays crash and second one is 2 months old.
Running kC 1000 on dumps both generates identical result.
Version Information (select @@version):
Microsoft SQL Server 2008 R2 (SP1) – 10.50.2500.0 (X64) Jun 17 2011 00:54:03
Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1)
0:000> lmvm sqlservr
start end module name
00000000`00ed0000 00000000`04aad000 sqlservr T (pdb symbols) d:\debug_symbols\sqlservr.pdb\13165A9929E847D997E11887F2F963AE2\sqlservr.pdb
Loaded symbol image file: sqlservr.exe
Image path: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Sat Jun 18 00:18:09 2011 (4DFB6221)
CheckSum: 03B3E8A1
ImageSize: 03BDD000
File version: 2009.100.2500.0
Product version: 10.50.2500.0
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
0:000> .ecxr
rax=00000000ee9521f4 rbx=000000000449ec90 rcx=0000000027dcc910
rdx=0000000000000000 rsi=000000000407ccbc rdi=0000000000000440
rip=000007fefdf1cacd rsp=0000000027dccf20 rbp=0000000027dcfc80
r8=0000000000000000 r9=0000000000000000 r10=00000000c000007c
r11=00000000017b92d0 r12=0000000000004a40 r13=0000000000000000
r14=0000000000000000 r15=0000000000004a40
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
KERNELBASE!RaiseException+0×39:
000007fe`fdf1cacd 4881c4c8000000 add rsp,0C8h
0:000> kC 1000
Call Site
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_handle_except
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
msxml6!Base::freeRentalObjects
msxml6!Base::StackExitNormal
msxml6!ModelInit::~ModelInit
msxml6!SAXReader::parse
MSO
0×0
0×0
0×0
0:000> lmvm msxml6
start end module name
000007fe`f6af0000 000007fe`f6ce2000 msxml6 (private pdb symbols) d:\debug_symbols\msxml6.pdb\8EF4A994F018436F8083CD5C89BF8EB72\msxml6.pdb
Loaded symbol image file: msxml6.dll
Mapped memory image file: D:\Debug_Symbols\msxml6.dll\4CE7C8411f2000\msxml6.dll
Image path: C:\Windows\System32\msxml6.dll
Image name: msxml6.dll
Timestamp: Sun Nov 21 00:08:17 2010 (4CE7C841)
CheckSum: 001F86CD
ImageSize: 001F2000
File version: 6.30.7601.17514
Product version: 6.30.7601.17514
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft(R) MSXML 6.0 SP3
InternalName: MSXML6.dll
OriginalFilename: MSXML6.dll
ProductVersion: 6.30.7601.17514
FileVersion: 6.30.7601.17514
FileDescription: MSXML 6.0 SP3
LegalCopyright: Copyright (C) Microsoft Corporation. 1981-2008
0:000> lmvm mso
start end module name
000007fe`df6d0000 000007fe`e0f02000 MSO T (no symbols)
Loaded symbol image file: MSO.DLL
Image path: C:\Program Files\Common Files\Microsoft Shared\OFFICE14\MSO.DLL
Image name: MSO.DLL
Timestamp: Wed Mar 24 05:22:16 2010 (4BA906D8)
CheckSum: 01832834
ImageSize: 01832000
File version: 14.0.4760.1000
Product version: 14.0.4760.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
Thanks
Farhan
Karthick P.K said
Hello Farhan,
Apply http://support.microsoft.com/kb/2516475 ..It should solve the issue.
Farhan said
Thank you very much Karthick, I will apply this hotfix and will let you know how it goes.
Thanks
Regards,
Farhan
Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes « MSSQLWIKI said
[...] SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion [...]
Roi said
Hi Karthick,
Great article, thanks!
I will appreciate help with the following dump:
=====================================================================
BugCheck Dump
=====================================================================
This file is generated by Microsoft SQL Server
version 9.00.3077.00
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.
Computer type is AT/AT COMPATIBLE.
Current time is 11:22:15 04/09/13.
16 Unknown CPU 9., 2400 Mhz processor (s).
Windows NT 5.2 Build 3790 CSD Service Pack 2.
Memory
MemoryLoad = 98%
Total Physical = 32767 MB
Available Physical = 582 MB
Total Page File = 80823 MB
Available Page File = 48552 MB
Total Virtual = 8388607 MB
Available Virtual = 8354540 MB
***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0018.txt
SqlDumpExceptionHandler: Process 476 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is
terminating this process.
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 04/09/13 11:22:15 spid 476
*
*
* Exception Address = 000000007814E3A4 Module(MSVCR80+000000000001E3A4)
* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
* Access Violation occurred reading address 0000000000000014
* Input Buffer 510 bytes –
*
Here comes a query which runs every day for several years now – executed by a job in a loop through working hours.
after this error occured SQL Server rejected all new connections (except for local login).
Call Site
msvcr80!memcpy
sqlservr!CSession::CopyDeepMember
sqlservr!CSession::GetUserSid
sqlservr!CSTVFSysSess::GetRow
sqlservr!CQScanTVFStreamNew::GetRow
sqlservr!CQScanSortNew::BuildSortTable
sqlservr!CQScanSortNew::OpenHelper
sqlservr!CQScanMergeJoinNew::Open
sqlservr!CQScanNLJoinNew::Open
sqlservr!CQScanNLJoinNew::Open
sqlservr!CQScanMergeJoinNew::Open
sqlservr!CQScanNLJoinNew::Open
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanTopNew::Open
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanSpoolNew::LoadSpool
sqlservr!_chkstk
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanUpdateNew::Open
sqlservr!CQueryScan::Startup
sqlservr!CXStmtQuery::SetupQueryScanAndExpression
sqlservr!CXStmtQuery::ErsqExecuteQuery
sqlservr!CXStmtDML::XretDMLExecute
sqlservr!CXStmtDML::XretExecute
sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!_chkstk
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExec::XretExecute
sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!_chkstk
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadStart
I would like to understand what really happend there,
and to prevent it from happening again.
Thanks very much,
Roi
Karthick P.K said
Hello Roi,
I think you are hitting a know issue mentioned in http://support.microsoft.com/kb/940281
Update your SQLServer to Cumulative update package 3 for SQL Server 2005 Service Pack 2 or higher to resolve this.
Roi said
Hi Karthick,
Thanks for the very quick response!
I believe that you are correct and that this is my issue,
as i do have a second dump and error messages resembling
the “…appears to be non-yielding on Scheduler…” as mentioned in the link you posted.
My next step is to temporarily disable the Agent Job which is monitoring sys.dm_exec_sessions,
and inform my manager of the issue.
He will have to decide if he wants to apply the update,
or wait for our scheduled upgrade to SQL Server 2012
(which is due next month and will probably be delayed forever…)
I would love to know how you found that this is the issue,
and would appreciate it if you can direct me to some documentation
which will help me solve such problems on my own in the future.
Again, Thank you very much!!!
Roi
Roi said
Hello Karthick,
You must have missed my previous reply,
could you please direct me to some documentation?
I would really love to learn in-depth internals.
Thanks,
Roi
Drake Teran said
Some assistance; i also received an EXCEPTION_ACCESS_VIOLATION with errors (Error: 17310, Severity: 20, State: 1) and (Error: 17311, Severity: 16, State: 1). I am curious if this is related to the known SQL bug identified http://support.microsoft.com/kb/972936 (many thanks in advance)
SQLDump0110:
0:000> kC 1000
Call Site
fs4odbc64
0×0
kernel32!HeapFree
fs4odbc64
SQLDump0111:
0:000> kC 1000
Call Site
fs4odbc64
fs4odbc64
0×0
ntdll!_C_specific_handler
0×0
0×0
0×0
0×0
sspicli!_dyn_tls_init_callback (sspicli+0x217d4)
0×0
ntdll!RtlDispatchException
0×0
0×0
0×0
SQLDump0112:
0:000> kc 1000
Call Site
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_handle_except
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
ntdll!RtlEnterCriticalSection
fs4odbc64
SQLDump0113:
0:000> kC 1000
Call Site
fs4odbc64
fs4odbc64
0×0
ntdll!_C_specific_handler
0×0
0×0
0×0
0×0
sspicli!_dyn_tls_init_callback (sspicli+0x217d4)
0×0
ntdll!RtlDispatchException
ntdll!RtlpAllocateUserBlock
0×0
0×0
0×0
SQLDump0114:
0:000> kC 1000
Call Site
fs4odbc64
fs4odbc64
0×0
ntdll!_C_specific_handler
0×0
0×0
0×0
0×0
sspicli!_dyn_tls_init_callback (sspicli+0x217d4)
0×0
ntdll!RtlDispatchException
0×0
0×0
0×0
SQLDump0115:
0:000> kC 1000
Call Site
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_handle_except
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
ntdll!RtlEnterCriticalSection
fs4odbc64
Drake Teran said
Current time is 01:34:21 04/15/13.
=====================================================================
BugCheck Dump
=====================================================================
This file is generated by Microsoft SQL Server
version 10.50.2500.0
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.
Computer type is Intel(R) Xeon(R) CPU X5660 @ 2.80GHz.
Bios Version is INTEL – 6040000
PhoenixBIOS 4.0 Release 6.0
4 X64 level 8664, 14 Mhz processor (s).
Windows NT 6.1 Build 7601 CSD Service Pack 1.
Memory
MemoryLoad = 80%
Total Physical = 12287 MB
Available Physical = 2429 MB
Total Page File = 24573 MB
Available Page File = 14413 MB
Total Virtual = 8388607 MB
Available Virtual = 8375683 MB
***Stack Dump being sent to F:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\LOG\SQLDump0110.tx
t
SqlDumpExceptionHandler: Process 62 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is
terminating this process.
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 04/15/13 01:34:21 spid 62
*
*
* Exception Address = 000007FEF1A06BA3 Module(fs4odbc64+0000000000006BA3)
* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
* Access Violation occurred writing address 0000000016243304
* Input Buffer 110 bytes –
* exec DataDispatch.dbo.dp_MasterDispatcher 3;
Drake Teran said
*
* MODULE BASE END SIZE
* sqlservr 0000000000D00000 00000000048DCFFF 03bdd000
* ntdll 0000000076F20000 00000000770C8FFF 001a9000
* kernel32 0000000076E00000 0000000076F1EFFF 0011f000
* KERNELBASE 000007FEFD0D0000 000007FEFD13BFFF 0006c000
* ADVAPI32 000007FEFDFB0000 000007FEFE08AFFF 000db000
* msvcrt 000007FEFEF90000 000007FEFF02EFFF 0009f000
* sechost 000007FEFDF90000 000007FEFDFAEFFF 0001f000
* RPCRT4 000007FEFE0D0000 000007FEFE1FCFFF 0012d000
* MSVCR80 0000000074B70000 0000000074C38FFF 000c9000
* MSVCP80 00000000747E0000 00000000748E8FFF 00109000
* sqlos 0000000074740000 0000000074746FFF 00007000
* Secur32 000007FEFCA20000 000007FEFCA2AFFF 0000b000
* SSPICLI 000007FEFCCD0000 000007FEFCCF4FFF 00025000
* pdh 000007FEF7660000 000007FEF76ADFFF 0004e000
* SHLWAPI 000007FEFDA90000 000007FEFDB00FFF 00071000
* GDI32 000007FEFDEA0000 000007FEFDF06FFF 00067000
* USER32 0000000076D00000 0000000076DF9FFF 000fa000
* LPK 000007FEFDC20000 000007FEFDC2DFFF 0000e000
* USP10 000007FEFD9C0000 000007FEFDA88FFF 000c9000
* USERENV 000007FEFC210000 000007FEFC22DFFF 0001e000
* profapi 000007FEFCE30000 000007FEFCE3EFFF 0000f000
* WINMM 000007FEF7620000 000007FEF765AFFF 0003b000
* IPHLPAPI 000007FEFB2F0000 000007FEFB316FFF 00027000
* NSI 000007FEFE0C0000 000007FEFE0C7FFF 00008000
* WINNSI 000007FEFB260000 000007FEFB26AFFF 0000b000
* opends60 0000000074720000 0000000074727FFF 00008000
* NETAPI32 000007FEFAD70000 000007FEFAD85FFF 00016000
* netutils 000007FEFC830000 000007FEFC83BFFF 0000c000
* srvcli 000007FEFC910000 000007FEFC932FFF 00023000
* wkscli 000007FEFAD10000 000007FEFAD24FFF 00015000
* LOGONCLI 000007FEFC4C0000 000007FEFC4EFFFF 00030000
* SAMCLI 000007FEFA390000 000007FEFA3A3FFF 00014000
* BatchParser 00000000746D0000 00000000746FCFFF 0002d000
* IMM32 000007FEFE090000 000007FEFE0BDFFF 0002e000
* MSCTF 000007FEFDB10000 000007FEFDC18FFF 00109000
* psapi 00000000770E0000 00000000770E6FFF 00007000
* instapi10 0000000074730000 000000007473CFFF 0000d000
* cscapi 000007FEFA740000 000007FEFA74EFFF 0000f000
* sqlevn70 00000000744C0000 00000000746C2FFF 00203000
* CRYPTSP 000007FEFC8E0000 000007FEFC8F6FFF 00017000
* rsaenh 000007FEFC3D0000 000007FEFC416FFF 00047000
* CRYPTBASE 000007FEFCD60000 000007FEFCD6EFFF 0000f000
* BROWCLI 000007FEF6FF0000 000007FEF7001FFF 00012000
* AUTHZ 000007FEFC8B0000 000007FEFC8DEFFF 0002f000
* MSCOREE 000007FEFA230000 000007FEFA29EFFF 0006f000
* ole32 000007FEFD470000 000007FEFD672FFF 00203000
* credssp 000007FEFC2F0000 000007FEFC2F9FFF 0000a000
* msv1_0 000007FEFC600000 000007FEFC650FFF 00051000
* cryptdll 000007FEFCA30000 000007FEFCA43FFF 00014000
* kerberos 000007FEFC6C0000 000007FEFC773FFF 000b4000
* MSASN1 000007FEFCF10000 000007FEFCF1EFFF 0000f000
* schannel 000007FEFC460000 000007FEFC4B7FFF 00058000
* CRYPT32 000007FEFCF60000 000007FEFD0C9FFF 0016a000
* security 00000000744B0000 00000000744B2FFF 00003000
* WS2_32 000007FEFDDB0000 000007FEFDDFCFFF 0004d000
* SHELL32 000007FEFE200000 000007FEFEF87FFF 00d88000
* OLEAUT32 000007FEFD8E0000 000007FEFD9B6FFF 000d7000
* ftimport 0000000060000000 0000000060024FFF 00025000
* MSFTE 0000000049980000 0000000049D2DFFF 003ae000
* VERSION 000007FEFBFB0000 000007FEFBFBBFFF 0000c000
* dbghelp 0000000073110000 000000007326DFFF 0015e000
* WINTRUST 000007FEFCF20000 000007FEFCF59FFF 0003a000
* ncrypt 000007FEFC7B0000 000007FEFC7FDFFF 0004e000
* bcrypt 000007FEFC880000 000007FEFC8A1FFF 00022000
* mswsock 000007FEFC660000 000007FEFC6B4FFF 00055000
* wship6 000007FEFC820000 000007FEFC826FFF 00007000
* wshtcpip 000007FEFC080000 000007FEFC086FFF 00007000
* ntmarta 000007FEFBF80000 000007FEFBFACFFF 0002d000
* WLDAP32 000007FEFD370000 000007FEFD3C1FFF 00052000
* ntdsapi 000007FEFA7A0000 000007FEFA7C6FFF 00027000
* DNSAPI 000007FEFC4F0000 000007FEFC54AFFF 0005b000
* rasadhlp 000007FEFA7D0000 000007FEFA7D7FFF 00008000
* fwpuclnt 000007FEFB1B0000 000007FEFB202FFF 00053000
* bcryptprimitives 000007FEFC310000 000007FEFC35BFFF 0004c000
* SAMLIB 000007FEFA370000 000007FEFA38CFFF 0001d000
* CLBCatQ 000007FEFDE00000 000007FEFDE98FFF 00099000
* sqlncli10 00000000725A0000 000000007285BFFF 002bc000
* COMCTL32 000007FEFA2A0000 000007FEFA33FFFF 000a0000
* COMDLG32 000007FEFD3D0000 000007FEFD466FFF 00097000
* SQLNCLIR10 0000000072190000 00000000721C6FFF 00037000
* netbios 000007FEF4AB0000 000007FEF4AB9FFF 0000a000
* xpsqlbot 0000000072080000 0000000072087FFF 00008000
* xpstar 0000000071FF0000 0000000072077FFF 00088000
* SQLSCM 0000000072430000 000000007243DFFF 0000e000
* ODBC32 000007FEF4D10000 000007FEF4DC0FFF 000b1000
* ATL80 0000000074B20000 0000000074B3FFFF 00020000
* odbcint 00000000722E0000 0000000072317FFF 00038000
* clusapi 000007FEF6170000 000007FEF61BFFFF 00050000
* resutils 000007FEF6150000 000007FEF6168FFF 00019000
* xpstar 0000000071FC0000 0000000071FE4FFF 00025000
* xplog70 0000000071FB0000 0000000071FBFFFF 00010000
* xplog70 0000000071FA0000 0000000071FA1FFF 00002000
* dsrole 000007FEFBBD0000 000007FEFBBDBFFF 0000c000
* oledb32 000007FEF1A50000 000007FEF1B5DFFF 0010e000
* MSDART 000007FEF1EA0000 000007FEF1EC7FFF 00028000
* Comctl32 000007FEFB7F0000 000007FEFB9E3FFF 001f4000
* OLEDB32R 0000000066C80000 0000000066C93FFF 00014000
* comsvcs 000007FEF45F0000 000007FEF479FFFF 001b0000
* msdasql 000007FEF0E10000 000007FEF0EC5FFF 000b6000
* MSDATL3 000007FEF1E80000 000007FEF1E9FFFF 00020000
* MSDASQLR 0000000066C70000 0000000066C7EFFF 0000f000
* RpcRtRemote 000007FEFCE10000 000007FEFCE23FFF 00014000
* fs4odbc64 000007FEF1A00000 000007FEF1A4CFFF 0004d000
* LIBEAY32 000000000FBC0000 000000000FD36FFF 00177000
* WSOCK32 000007FEF1E70000 000007FEF1E78FFF 00009000
* SSLEAY32 0000000005020000 0000000005075FFF 00056000
* odbccp32 000007FEF17A0000 000007FEF17C7FFF 00028000
* instapi 0000000048060000 000000004806CFFF 0000d000
* dbghelp 00000000715A0000 00000000716FDFFF 0015e000
*
* P1Home: 00000000297B7AD0: 00000000C0000005 0000000000000000 000007FEF1A06BA3 0000000000000002 00000000
00000001 0000000016243304
* P2Home: 00000000297B75E0: 00000000297B7AD0 00000000297B75E0 0000000000000000 0000000000000000 00000000
297B79A8 007400780045005C
* P3Home: 0000000000000000:
* P4Home: 0000000000000000:
* P5Home: 00000000297B79A8: 0000000000010000 000000000000002C 00000000297B79D0 00000000297B79E0 00000078
C12C9653 00000000297B7BE0
* P6Home: 007400780045005C:
* ContextFlags: 000000000010001F:
* MxCsr: 0000000000001F80:
* SegCs: 0000000000000033:
* SegDs: 000000000000002B:
* SegEs: 000000000000002B:
* SegFs: 0000000000000053:
* SegGs: 000000000000002B:
* SegSs: 000000000000002B:
* EFlags: 0000000000010246: 0000000000000000 0000000101580000 0000000102680000 0000000102780000 00000000
00000000 0000001000000000
* Rax: 0000000000000000:
* Rcx: 0000000016243070:
* Rdx: 0000000000000000:
* Rbx: 0000000000376B20: 0000000000000002 0000000000776CC0 0000000000000000 0000000000000000 00000000
00000000 0000000000000000
* Rsp: 00000000297B7BA0: 0000000016232E30 0000000076E2301A 0000000000376B20 0000000000376B20 00000000
00000000 000007FEF1A043D5
* Rbp: 0000000000000001:
* Rsi: 000000000038D650: 000000000F4FC570 0000000000376B20 0000000000000000 00000000162E12F0 00000000
00376B20 00000000162E12F0
* Rdi: 000000000037AC20: 0000000000000003 0000000000776CC0 0000000000000000 0000000000000000 00000000
00000000 0000000000000000
* R8: 00000000297B7A98: 0000000000000000 0000000000000000 0000000000000000 000005C0FFFFFB30 000004D0
FFFFFB30 00000000000000F0
* R9: 0000000000008000:
* R10: 0000000000000000:
* R11: 0000000000000286:
* R12: 0000000000000010:
* R13: 0000000000000000:
* R14: 0000000000000000:
* R15: 0000000000000000:
* Rip: 000007FEF1A06BA3: 3300000294818966 8B480000014DE9C0 029480B70F702444 077E782444390000 33000001
34E9C033 B70F5024448966C0
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump
000007FEF1A06BA3 Module(fs4odbc64+0000000000006BA3)
000007FEF1A15867 Module(fs4odbc64+0000000000015867)
000007FEF4D13BA9 Module(ODBC32+0000000000003BA9)
000007FEF4D7BA8E Module(ODBC32+000000000006BA8E)
000007FEF0E49D53 Module(msdasql+0000000000039D53)
000007FEF0E34679 Module(msdasql+0000000000024679)
000007FEF0E34584 Module(msdasql+0000000000024584)
000007FEF0E354A4 Module(msdasql+00000000000254A4)
000007FEF1A5647E Module(oledb32+000000000000647E)
000007FEF1A563D2 Module(oledb32+00000000000063D2)
000007FEF1A55E56 Module(oledb32+0000000000005E56)
00000000017804E0 Module(sqlservr+0000000000A804E0)
0000000001873015 Module(sqlservr+0000000000B73015)
0000000001871999 Module(sqlservr+0000000000B71999)
0000000001871410 Module(sqlservr+0000000000B71410)
000000000186DCE8 Module(sqlservr+0000000000B6DCE8)
000000000134160E Module(sqlservr+000000000064160E)
0000000000F70E36 Module(sqlservr+0000000000270E36)
0000000000F755B7 Module(sqlservr+00000000002755B7)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
0000000000FB7183 Module(sqlservr+00000000002B7183)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
0000000000FC6720 Module(sqlservr+00000000002C6720)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
000000000104C380 Module(sqlservr+000000000034C380)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
0000000000FB7183 Module(sqlservr+00000000002B7183)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
00000000012DA59A Module(sqlservr+00000000005DA59A)
000000000123A181 Module(sqlservr+000000000053A181)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
00000000012386F7 Module(sqlservr+00000000005386F7)
0000000000F6BA93 Module(sqlservr+000000000026BA93)
0000000000F6B958 Module(sqlservr+000000000026B958)
0000000000F6BB0E Module(sqlservr+000000000026BB0E)
0000000000ED37A5 Module(sqlservr+00000000001D37A5)
0000000000E96644 Module(sqlservr+0000000000196644)
0000000000E97259 Module(sqlservr+0000000000197259)
0000000000E970FC Module(sqlservr+00000000001970FC)
00000000021DC97F Module(sqlservr+00000000014DC97F)
00000000021FF7E8 Module(sqlservr+00000000014FF7E8)
00000000021FB741 Module(sqlservr+00000000014FB741)
0000000000D69A86 Module(sqlservr+0000000000069A86)
0000000000D6B4AF Module(sqlservr+000000000006B4AF)
0000000000D6AD6C Module(sqlservr+000000000006AD6C)
00000000021FDA1C Module(sqlservr+00000000014FDA1C)
00000000021FD37A Module(sqlservr+00000000014FD37A)
00000000021FB36D Module(sqlservr+00000000014FB36D)
0000000000D69A86 Module(sqlservr+0000000000069A86)
0000000000D6B4AF Module(sqlservr+000000000006B4AF)
0000000000D6AD6C Module(sqlservr+000000000006AD6C)
00000000021FDA1C Module(sqlservr+00000000014FDA1C)
00000000021FD37A Module(sqlservr+00000000014FD37A)
00000000021FB36D Module(sqlservr+00000000014FB36D)
0000000000D69A86 Module(sqlservr+0000000000069A86)
0000000000D6B4AF Module(sqlservr+000000000006B4AF)
0000000000D6AD6C Module(sqlservr+000000000006AD6C)
00000000021FDA1C Module(sqlservr+00000000014FDA1C)
00000000021FD37A Module(sqlservr+00000000014FD37A)
00000000021FB36D Module(sqlservr+00000000014FB36D)
0000000000D69A86 Module(sqlservr+0000000000069A86)
0000000000D6B4AF Module(sqlservr+000000000006B4AF)
0000000000D6AD6C Module(sqlservr+000000000006AD6C)
0000000000D8CFA6 Module(sqlservr+000000000008CFA6)
0000000000DB65E2 Module(sqlservr+00000000000B65E2)
0000000000D0B450 Module(sqlservr+000000000000B450)
0000000000D0B116 Module(sqlservr+000000000000B116)
0000000000D0AF5B Module(sqlservr+000000000000AF5B)
0000000000E444FA Module(sqlservr+00000000001444FA)
0000000000E447DD Module(sqlservr+00000000001447DD)
000000000128C0CD Module(sqlservr+000000000058C0CD)
0000000000E453D2 Module(sqlservr+00000000001453D2)
0000000074B737D7 Module(MSVCR80+00000000000037D7)
0000000074B73894 Module(MSVCR80+0000000000003894)
0000000076E1652D Module(kernel32+000000000001652D)
0000000076F4C521 Module(ntdll+000000000002C521)
CSession @0x00000001EF71A3B0
—————————-
m_spid = 62 m_cRef = 14 m_rgcRefType[0] = 1
m_rgcRefType[1] = 1 m_rgcRefType[2] = 11 m_rgcRefType[3] = 1
m_rgcRefType[4] = 0 m_rgcRefType[5] = 0 m_pmo = 0x00000001EF71A080
m_pstackBhfPool = 0x00000001EEBA7120 m_dwLoginFlags = 0x83e0 m_fBackground = 0
m_eConnResetOption = 0 m_fUserProc = -1 m_fConnReset = 0
m_fIsConnReset = 0 m_fInLogin = 0 m_fAuditLoginSent = 1
m_fAuditLoginFailedSent = 0 m_fReplRelease = 0 m_fKill = 0
m_ulLoginStamp = 5045 m_eclClient = 6 m_protType = 6
m_hHttpToken = FFFFFFFFFFFFFFFF
m_pV7LoginRec
———————
0000000000000000: 9a010000 03000b73 00100000 00000007 e0070000 †……s……..à…
0000000000000014: 00000000 e0830000 e0010000 09040000 5e000900 †….à..à… …^. .
0000000000000028: 00000000 00000000 70004900 02010000 00000000 †……..p.I………
000000000000003C: 02010400 0a010000 0a010c00 0050568d 6eec2201 †………….PV.nì”.
0000000000000050: 78002201 00000000 00000000 0000†††††††††††††††x.”………..
CPhysicalConnection @0x00000001EF71A270
—————————————
m_pPhyConn->m_pmo = 0x00000001EF71A080
m_pPhyConn->m_pNetConn = 0x00000001EF71AAD0
m_pPhyConn->m_pConnList = 0x00000001EF71A380
m_pPhyConn->m_pSess = 0x00000001EF71A3B0 m_pPhyConn->m_fTracked = -1
m_pPhyConn->m_cbPacketsize = 4096 m_pPhyConn->m_fMars = 0 m_pPhyConn->m_fKill = 0
CBatch @0x00000001EF71AFA0
————————–
m_pSess = 0x00000001EF71A3B0 m_pConn = 0x00000001EF71AEA0 m_cRef = 3
m_rgcRefType[0] = 1 m_rgcRefType[1] = 1 m_rgcRefType[2] = 1
m_rgcRefType[3] = 0 m_rgcRefType[4] = 0 m_pTask = 0x000000000043F708
EXCEPT (null) @0x00000000297B4938
———————————
exc_number = 0 exc_severity = 0 exc_func = 0x000000000320F2A4
Task @0x000000000043F708
————————
CPU Ticks used (ms) = 28929974 Task State = 2
WAITINFO_INTERNAL: WaitResource = 0×0000000000000000 WAITINFO_INTERNAL: WaitType = 0×0
WAITINFO_INTERNAL: WaitSpinlock = 0×0000000000000000
WAITINFO_EXTERNAL: WaitResource = 0×0000000000000000
WAITINFO_EXTERNAL: WaitType = 0x200001D9
WAITINFO_EXTERNAL: WaitSpinlock = 0×0000000000000000 SchedulerId = 0×0
ThreadId = 0xfe0 m_state = 0 m_eAbortSev = 0
EC @0x00000001EF71BEC0
———————-
spid = 0 ecid = 0 ec_stat = 0×0
ec_stat2 = 0×0 ec_atomic = 0×0 __fSubProc = 1
__pSETLS = 0x00000001EF71AF10 __pSEParams = 0x00000001EF71B350
SEInternalTLS @0x00000001EF71AF10
———————————
m_flags = 0 m_TLSstatus = 3 m_owningTask = 0x000000000043F708
m_activeHeapDatasetList = 0x00000001EF71AF10
m_activeIndexDatasetList = 0x00000001EF71AF20 m_pDbccContext = 0×0000000000000000
SEParams @0x00000001EF71B350
—————————-
m_lockTimeout = -1 m_isoLevel = 4096 m_logDontReplicate = 0
m_neverReplicate = 0 m_XactWorkspace = 0x00000001EB7BF090
m_pSessionLocks = 0x00000001EEBA6A70 m_pDbLocks = 0x00000001EF71B870
m_execStats = 0x0000000084E82240 m_pAllocFileLimit = 0×0000000000000000
green accounting and financing said
Hiya! I simply wish to give an enormous thumbs up for the nice
information you may have right here on this post.
I can be coming again to your blog for extra soon.
Neret said
Hi Karthick !
Coul you help me with this, please???
This is the stack of thread which caused exception:
0:000> kC 1000
Call Site
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_handle_except
kernel32!UnhandledExceptionFilter
ntdll! ?? ::FNODOBFM::`string’
ntdll!_C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
msxml6!Base::freeRentalObjects
msxml6!Base::StackExitNormal
msxml6!ModelInit::~ModelInit
msxml6!SAXReader::parse
MSO!Ordinal398
MSO!Ordinal9364
MSO!Ordinal9364
MSO!Ordinal9364
MSO!Ordinal9694
MSO!Ordinal4349
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Thanks!
Karthick P.K said
Hello Neret, Apply http://support.microsoft.com/kb/2516475 .It should solve the issue. Note: After applying the fix check if MSO.dll is update to version mentioned in KB. If not try running repair on Microsoft access database engine, It should update your MSO.DLL version Thanks
Marek_Warszawa said
Dear Karthick,
I have done the steps described in your article and got the below stack trace.
Could you please help solve the problem?
0:000> kC 1000
Call Site
0×0
0×0
sqlservr!CConstraintProp::FAndNonItvlConst
sqlservr!CConstraintProp::FAndNonItvlConstraint
sqlservr!CConstraintProp::AndNewConstraint
sqlservr!CConstraintProp::PcnstrDeriveSelect
sqlservr!CLogOp_Select::PcnstrDerive
sqlservr!CLogOpArg::PcnstrDeriveHandler
sqlservr!CLogOpArg::DeriveGroupProperties
sqlservr!COpArg::DeriveNormalizedGroupProperties
sqlservr!COptExpr::DeriveGroupProperties
sqlservr!COptExpr::DeriveGroupProperties
sqlservr!COptExpr::DeriveGroupProperties
sqlservr!COptExpr::DeriveGroupProperties
sqlservr!COptExpr::DeriveGroupProperties
sqlservr!CQuery::PqoBuild
sqlservr!CStmtQuery::InitQuery
sqlservr!alloca_probe
sqlservr!CStmtCursorSelect::Init
sqlservr!CCompPlan::FCompileStep
sqlservr!CSQLSource::FCompile
sqlservr!CSQLSource::FCompWrapper
sqlservr!CSQLSource::Transform
sqlservr!CSQLSource::Execute
sqlservr!ExecuteSql
sqlservr!SPCursor::CursorExecPrepExecOpen
sqlservr!SPCursor::Open
sqlservr!CSpecProc::ExecuteSpecial
sqlservr!CSpecProc::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
0×0
0×0
Thanks in advance,
Marek
Karthick P.K said
Marek,
What is the version of SQL of your SQLServer (select @@version). I think you are in RTM version of SQLServer2005. I remember this is fixed in 9.0.1500. Please upgrade to latest service pack for SQLServer 2005
Marek_Warszawa said
Hi,
This is information from SQL Server log files:
Microsoft SQL Server 2008 (SP3) – 10.0.5512.0 (X64)
Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1) (VM)
Computer type is AMD Opteron(TM) Processor 6274 .
Bios Version is INTEL – 6040000
PhoenixBIOS 4.0 Release 6.0
2 X64 level 8664, 14 Mhz processor (s).
Windows NT 6.1 Build 7601 CSD Service Pack 1.
Memory:
MemoryLoad = 96%
Total Physical = 4095 MB
Available Physical = 129 MB
Total Page File = 8189 MB
Available Page File = 2214 MB
Total Virtual = 8388607 MB
Available Virtual = 8380146 MB
What could be the reason in this case?
BR,
Marek
Karthick P.K said
Two frames are corrupted so it’s difficult to crack
Call Site
0×0 ====>corrupted
0×0 0×0 ====>corrupted
sqlservr!CConstraintProp::FAndNonItvlConst
sqlservr!CConstraintProp::FAndNonItvlConstraint
Do you pass so many values in the IN clause of the query which causing this access violation? (You will find the input buffer in errorlog along with access violation message)
Can you reproduce this issue consistently?
Michiel said
Hello Karthick,
Great blog!
Can you please help me with this dump?
Our sql-server did not respond to request from other servers anymore.
After restarting MSSQLSERVER service it all worked again.
***************
Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [D:\Temp\SQLDump0005.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available
Comment: ‘Program fault handler’
Comment: ’05/13/13 17:17:42 spid 0 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION writing address 0000000000C4DA54 at 0×0000000000649889
‘
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Version 7600 MP (2 procs) Free x64
Product: Server, suite: Enterprise TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon May 13 17:17:56.000 2013 (GMT+2)
System Uptime: 66 days 18:15:01.669
Process Uptime: 66 days 18:11:58.000
……………………………………………………….
………………………….
Loading unloaded module list
……………………………………………………………
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(4f4.1fe8): Access violation – code c0000005 (first/second chance not available)
ntdll!ZwWaitForSingleObject+0xa:
00000000`7780f6fa c3 ret
***************
0:000> .ecxr
rax=00000000002afde0 rbx=00000004a1980528 rcx=00000000008122c0
rdx=0000000080018001 rsi=00000003609686a0 rdi=00000004a19804e0
rip=0000000000649889 rsp=000000001a1bde20 rbp=00000004a19803a0
r8=0000000000649960 r9=0000000000000254 r10=000007fffff70000
r11=000000001a1bde06 r12=0000000000000006 r13=000000001a1bdfa0
r14=0000000000000000 r15=000000028f156fa0
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
Unable to load image D:\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe, Win32 error 0n2
*** WARNING: Unable to verify timestamp for sqlservr.exe
sqlservr!CCriticalSectionSOS::Initialize+0×71:
00000000`00649889 488908 mov qword ptr [rax],rcx ds:00000000`002afde0=????????????????
***************
0:000> lmvm sqlservr
start end module name
00000000`00260000 00000000`03e10000 sqlservr T (pdb symbols) c:\symbols\sqlservr.pdb\E0E45C4EB0C64908A90279B30F42FACA2\sqlservr.pdb
Loaded symbol image file: sqlservr.exe
Image path: D:\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Sat Apr 23 14:44:29 2011 (4DB2C9AD)
CheckSum: 03B10238
ImageSize: 03BB0000
File version: 2009.100.1617.0
Product version: 10.50.1617.0
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
***************
0:000> .ecxr
rax=00000000002afde0 rbx=00000004a1980528 rcx=00000000008122c0
rdx=0000000080018001 rsi=00000003609686a0 rdi=00000004a19804e0
rip=0000000000649889 rsp=000000001a1bde20 rbp=00000004a19803a0
r8=0000000000649960 r9=0000000000000254 r10=000007fffff70000
r11=000000001a1bde06 r12=0000000000000006 r13=000000001a1bdfa0
r14=0000000000000000 r15=000000028f156fa0
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
sqlservr!CCriticalSectionSOS::Initialize+0×71:
00000000`00649889 488908 mov qword ptr [rax],rcx ds:00000000`002afde0=????????????????
***************
0:000> kc 1000
Call Site
sqlservr!CCriticalSectionSOS::Initialize
sqlservr!CryptoBase::FInit
sqlservr!SNIAddProvider
sqlservr!TDSSNIClient::FAddSSLProvider
sqlservr!process_prelogin
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Thanks for your help!
Michiel.
Karthick P.K said
Hello Michiel,
This issue has been fixed in CU6 for SQL Server 2008 R2. Please apply the patch
Michiel said
Hi Karthick,
Thanks for the fast reply!
A colleague of mine just came up with the same solution
If the problem returns I will let you know but I think that won’t be necessary
Thanks again.
Michiel.
Jose P said
does anyone can help find what wrong here?
0:000> kc 1000
Call Site
sqldk!CMemObj::Free
sqlmin!commondelete
sqlmin!IndexAnalysis::ExecuteCleanupAction
sqlmin!IndexAnalysis::GetNextRow
sqlmin!CIndexAnalysisSTVF::InternalGetRow
sqlmin!CQScanTVFStreamNew::GetRow
sqlmin!CQScanFilterNew::GetRowHelper
sqlmin!CQScanNLJoinNew::GetRowHelper
sqlmin!CQScanRangePrefetchDelayNew::GetRow
sqlmin!CQScanNLJoinTrivialNew::GetRow
sqlmin!CQScanNLJoinNew::GetRowHelper
sqlmin!CQueryScan::GetRow
sqllang!CXStmtQuery::ErsqExecuteQuery
sqllang!CXStmtCondWithQuery::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!process_request
sqllang!process_commands
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Karthick P.K said
I think you are hitting known issue documented in http://support.microsoft.com/kb/2837604 .Apply CU4 for 2102 SP1 it should solve your issue
Chris Page said
Hi thanks for the blog – gives me some hope of finding an answer
Stack below appeared today – this is the second time I have run a set of code on this server (is running on others of lower / same SQL versions with no problems) so was quite surprised to encounter an issue, many thanks :
0:000> KC 1000
Call Site
sqlservr!CVariableInfo::CVarBlock::PvbJoin
sqlservr!CVariableInfo::PviRelease
sqlservr!CVarPageMgr::Release
sqlservr!CMemObj::Free
sqlservr!IndexAnalysis::Cleanup
sqlservr!IndexAnalysis::ObjectIterator
sqlservr!IndexAnalysis::GetNextRow
sqlservr!CRowsetIndexAnalysis::FGetNextRow
sqlservr!CUtRowset::GetNextRows
sqlservr!CQScanRmtScanNew::GetRowHelper
sqlservr!CQScanStreamAggregateNew::GetRowHelper
sqlservr!CQScanStreamAggregateNew::GetCalculatedRow
sqlservr!CQScanStreamAggregateNew::Open
sqlservr!CQScanNLJoinTrivialNew::GetRow
sqlservr!CQScanStreamAggregateNew::GetRowHelper
sqlservr!CQScanStreamAggregateNew::GetCalculatedRow
sqlservr!CQScanStreamAggregateNew::Open
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanUpdateNew::Open
sqlservr!CQueryScan::Startup
sqlservr!CXStmtQuery::SetupQueryScanAndExpression
sqlservr!CXStmtQuery::InitForExecute
sqlservr!CXStmtQuery::ErsqExecuteQuery
sqlservr!CXStmtDML::XretDMLExecute
sqlservr!CXStmtDML::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecStr::XretExecStrExecute
sqlservr!CXStmtExecStr::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Karthick P.K said
Hello Chris,
Do you get this dump when you use linked server queries?
Note: You can find the query in SQL Server error log after the dump
Chris Page said
Hi karthick,
The code is running against the same server as the agent is on so no linked server activity of any sort. The query in the log didn’t tell me much because it contains the whole query from the agent task which calls 2 stored procs that themselves lead to individuals tables and indexes so doesn’t align with what appears to be the failure point from the default trace.
Karthick P.K said
Chris,
This issue is very unique and very difficult to crack without analyzing the dump. I am not sure how feasible it is for you to share the dump with me (Though I am interested I will not recommend). I would recommend you to create a support case with Microsoft SQL Server support.
This looks like a BUG for me but cant confirm unless I look at the dump.
Chris Page said
Thanks kathrick, the problem hasn’t recurred here, however, I have seen the message that the agent was giving me in a previous role but not picked up core dumps on those occassions so I’ve gone back to an ex-colleague to see if the problem is actually manifesting the same there so we’ll be able to have more material to co-ordinate any issues raised.
Chris