MSSQLWIKI

Karthick P.K on SQL Server

SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion

Posted by Karthick P.K on October 16, 2012

 

I have got few request’s from  SQL Server DBA’s in past to blog about analyzing SQL Server exceptions and assertions . After seeing lot of DBA’s getting stuck when they get EXCEPTION_ACCESS_VIOLATION (or) Assertion in SQL ServersI decided to write this blog.

This blog is published with intention to make DBA’s analyze and resolve EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion before contacting Microsoft support.  Exception and assertion are two different things. SQL handles both assertions and exceptions by writing the current thread’s stack to the Error log and generating a dump.  In simple An exception is an event that occurs during the execution of a program, and requires the execution of code outside the normal flow of control and assertion is the check that the programmer inserted into the code to make sure that some condition is true, If it returns false an assert is raised. SQL handles both assertions and exceptions by writing the current thread’s stack to the Error log and generating a dump, so trouble shooting steps are similar. 

 

You will find messages similar to one below in SQL Serve error logs when you get Exception or EXCEPTION_ACCESS_VIOLATION .

{

Error

External dump process returned no errors.
Using ‘dbghelp.dll’ version ’4.0.5′
SqlDumpExceptionHandler: Process 510 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process.
* *******************************************************************************
* BEGIN STACK DUMP:
*  Exception Address = 000000007752485C Module(ntdll+000000000002285C)

*   Exception Code    = c0000005 EXCEPTION_ACCESS_VIOLATION

*   Access Violation occurred reading address 0000041EA9AE2EF0

* Input Buffer 510 bytes –

ex_terminator – Last chance exception handling

}

You will find messages similar to one below in SQL Server error logs when you get an Assertion.

{

Error

spid323     Error: 17065, Severity: 16, State: 1.

spid323     SQL Server Assertion: File: < .cpp>, line = 2576 Failed Assertion = ‘fFalse’  This error may be timing-related. If the error persists after rerunning the statement, use DBCC CHECKDB to check the database for structural integrity, or restart the server to ensure in-memory data structures are not corrupted

SQL Server Assertion: File: <   .cpp>, line=2040 Failed Assertion =

}

To analyze the dump download and Install Windows Debugger from This Link 

 

Step 1 (Load the memory dump file to debugger):

Open Windbg .  Choose File menu –> select Open crash dump –>Select the Dump file (SQLDump000#.mdmp)

Note : You will find SQLDump000#.mdmp in your SQL Server error log when you get the Exception or assertion.

Step 2 (Set the symbol path to Microsoft symbols server):

on command window type

.sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;

Step 3 (Load the symbols from Microsoft symbols server):

Type .reload /f and hit enter. This will force debugger to immediately load all the symbols.

 

Step 4 (check if symbols are loaded):

Verify if symbols are loaded for  SQL Server by using the debugger command lmvm

0:002> lmvm sqlservr

start             end                 module name

00000000`01000000 00000000`03679000   sqlservr T (pdb symbols)          c:\websymbols\sqlservr.pdb\21E4AC6E96294A529C9D99826B5A7C032\sqlservr.pdb

    Loaded symbol image file: sqlservr.exe

    Image path: C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe

    Image name: sqlservr.exe

    Timestamp:        Wed Oct 07 21:15:52 2009 (4ACD6778)

    CheckSum:         025FEB5E

    ImageSize:        02679000

    File version:     2005.90.4266.0

    Product version:  9.0.4266.0

    File flags:       0 (Mask 3F)

    File OS:          40000 NT Base

    File type:        1.0 App

    File date:        00000000.00000000

    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

 

Step 5 (Switch to exception context):

Type .ecxr

Step 6(Get the stack of thread which caused exception or assertion):

Type  kC  1000    //You will get the stack of thread which raised exception or assertion .

I have pasted one of the sample stack below, from the exception dump which I worked recently.  First thing to identify from stack is who is raising the exception. In the below stack look at the portion which is highlighted in red (In each frame before the ! symbol), that is the module which raised the exception (Exe or DLL name ).

If Exe/DLL name is Non Microsoft  module (Exe or DLL name ) then the exception is being caused by a third party component, you will need to work with the company that provided that component to get a solution. lmvm Exe/DLL name will give you the company name. For example: lmvm wininet

If Exe/DLL name is  SQLServr  (or) any other SQL Server modules then the exception is raised by SQL Server, In that case type kC 1000 and paste the stack in comments session of this blog (or) When you start thread in MSDN forums (or) In This face book group. If you don’t get any prompt reply from the community, you may need to open a support ticket with Microsoft.

Note: When you get Assertion make sure you post message line which contains   SQL Server Assertion: File: <Filename.cpp>, line = 2576 Failed Assertion =  ”  

 

0:000> kC 1000

Call Site

wininet!InternetFreeThreadInfo+0×26

wininet!InternetDestroyThreadInfo+0×40

wininet!DllMain_wininet+0xb5

wininet!__DllMainCRTStartup+0xdb

ntdll!LdrShutdownThread+0×155

ntdll!RtlExitUserThread+0×38

msvcr80!_endthreadex+0×27

msvcr80!_callthreadstartex+0x1e

msvcr80!_threadstartex+0×84

kernel32!BaseThreadInitThunk+0xd

ntdll!RtlUserThreadStart+0x1d

If you liked this post, do like us on Facebook at https://www.facebook.com/mssqlwiki and join our Facebook group https://www.facebook.com/mssqlwiki#!/groups/454762937884205/

Related posts:

Thank you,

Karthick P.K |My Facebook Page |My Site| Blog space| Twitter

Disclaimer

The views expressed on this website/blog are mine alone and do not reflect the views of my company. All postings on this blog are provided “AS IS” with no warranties, and confers no rights.

About these ads

91 Responses to “SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion”

  1. Scott McCormick said

    Hello Karthik –

    This is a great explanation of the windbg tool. I have a couple of follow-up questions:

    1 — You say that the red highlighted text is the application causing the problem. Do you mean the bold text? None of the text in the sample is red.

    2 — Assuming you mean the bold text, does that mean the output is read from the top down?

    3 — I honestly have no idea if the Stack Dump I’m seeing is being caused by a 3rd party, but I don’t believe it is. Would you please take a look?

    0:000> kC 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!DmpRemoteDumpRequest
    sqlservr!SQLExit
    sqlservr!SQLExit
    sqlservr!SQLExit
    sqlservr!SQLExit
    sqlservr!SQLExit
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    winhttp!RemoveFromSerializedList
    winhttp!InternetDestroyThreadInfo
    winhttp!DllMain
    winhttp!CRT_INIT
    ntdll!LdrShutdownThread
    ntdll!RtlExitUserThread
    msvcr80!endthreadex
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    • Hello Scott,

      WINHTTP.DLL is loaded in SQL Server’s internal address apace and crash is occuring WINHTTP.DLL.

      I recommended to move this module outside SQL Server address space. The stored procedure ‘sp_oacreate’ should have been used to bring in the module to SQL Server address space, so to move out in the syntax you need to mention 4 with sp_oacreate

      {
      Here is the KB article that gives a workaround on How to move extended stored procedures out of process-
      http://support.microsoft.com/kb/243428
      }

      You can use belowquery to identify the objects in which you have used SP_OA create

      {
      EXEC sp_MSforeachdb ‘
      IF EXISTS (SELECT * FROM ?.dbo.syscomments WHERE CHARINDEX (”sp_OA”, [text])>0)
      SELECT
      LEFT (”?”, 30) AS db, LEFT (o.name, 40) AS sproc,
      LEFT (REPLACE (REPLACE (SUBSTRING ([text], CHARINDEX (”SP_OA”, UPPER ([text])) – 10, 60), CHAR(10), ””), CHAR(13), ””), 60)
      FROM ?.dbo.syscomments c
      INNER JOIN ?.dbo.sysobjects o ON c.id = o.id
      WHERE CHARINDEX (”sp_OA”, [text]) > 0′
      }

      • Scott McCormick said

        Thanks for the quick response. Unfortunately, it looks like sp_oa is not used in any SP or function. Or at least, it isn’t showing up in sys.comments. Can you explain how you found that information?

        I was able to pull more information out after posting this question, and I agree that it’s winhttp. I was thinking it might be one of the CLRs? There are a few of them.

        Here is the rest of what I could find.

        FAULTING_IP:
        winhttp!RemoveFromSerializedList+26
        000007fe`f7de1376 4c8918 mov qword ptr [rax],r11

        EXCEPTION_RECORD: 0000000021edf880 — (.exr 0x21edf880)
        ExceptionAddress: 000007fef7de1376 (winhttp!RemoveFromSerializedList+0×0000000000000026)
        ExceptionCode: c0000005 (Access violation)
        ExceptionFlags: 00000000
        NumberParameters: 2
        Parameter[0]: 0000000000000001
        Parameter[1]: 0000000000000000
        Attempt to write to address 0000000000000000

        DEFAULT_BUCKET_ID: APPLICATION_FAULT

        PROCESS_NAME: sqlservr.exe

        ERROR_CODE: (NTSTATUS) 0x42ac –

        EXCEPTION_CODE: (Win32) 0x42ac (17068) –

        NTGLOBALFLAG: 0

        MANAGED_STACK: !dumpstack -EE
        No export dumpstack found

        MANAGED_BITNESS_MISMATCH:
        Managed code needs matching platform of sos.dll for proper analysis. Use ‘x64′ debugger.

        CONTEXT: 0000000021edf390 — (.cxr 0x21edf390)
        rax=0000000000000000 rbx=000000001e5e1d00 rcx=000000000000412c
        rdx=000000001e5e1d00 rsi=000007fef7e44730 rdi=0000000000000003
        rip=000007fef7de1376 rsp=0000000021edf950 rbp=0000000000000000
        r8=0000000000000000 r9=00000000053f1a30 r10=0000000000000000
        r11=0000000000000000 r12=000007fffffda000 r13=000000001e1df960
        r14=000007fffff40000 r15=0000000077d72670
        iopl=0 nv up ei pl nz na pe nc
        cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
        winhttp!RemoveFromSerializedList+0×26:
        000007fe`f7de1376 4c8918 mov qword ptr [rax],r11 ds:00000000`00000000=????????????????
        Resetting default scope

        WRITE_ADDRESS: 0000000000000000

        FOLLOWUP_IP:
        winhttp!RemoveFromSerializedList+26
        000007fe`f7de1376 4c8918 mov qword ptr [rax],r11

        BUGCHECK_STR: ACCESS_VIOLATION

        FAULTING_THREAD: ffffffffffffffff

        LAST_CONTROL_TRANSFER: from 000007fef7de13a9 to 000007fef7de1376

        STACK_TEXT:
        000007fe`f7de1376 winhttp!RemoveFromSerializedList+0×26
        000007fe`f7de13a9 winhttp!InternetDestroyThreadInfo+0x2e
        000007fe`f7de1190 winhttp!DllMain+0×20
        000007fe`f7de10d2 winhttp!CRT_INIT+0x2af
        00000000`77c86a28 ntdll!LdrShutdownThread+0×155
        00000000`77c86968 ntdll!RtlExitUserThread+0×38
        00000000`758537b7 msvcr80!endthreadex+0×27
        00000000`758537de msvcr80!endthreadex+0x4e
        00000000`75853894 msvcr80!endthreadex+0×104
        00000000`77b3652d kernel32!BaseThreadInitThunk+0xd
        00000000`77c6c521 ntdll!RtlUserThreadStart+0x1d

        STACK_COMMAND: .cxr 21EDF390 ; kb ; dds 21edf950 ; kb

        SYMBOL_STACK_INDEX: 0

        SYMBOL_NAME: winhttp!RemoveFromSerializedList+26

        FOLLOWUP_NAME: MachineOwner

        MODULE_NAME: winhttp

        IMAGE_NAME: winhttp.dll

        DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ca23

        FAILURE_BUCKET_ID: APPLICATION_FAULT_42ac_winhttp.dll!RemoveFromSerializedList

        BUCKET_ID: X64_ACCESS_VIOLATION_winhttp!RemoveFromSerializedList+26

        WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_4000_0/4fecc5ba/sqlservr_exe/2009_100_4000_0/4fecc5ba/42ac/020eae68.htm?Retriage=1

        Followup: MachineOwner
        ———

        0:000> .exr 0x21edf880
        ExceptionAddress: 000007fef7de1376 (winhttp!RemoveFromSerializedList+0×0000000000000026)
        ExceptionCode: c0000005 (Access violation)
        ExceptionFlags: 00000000
        NumberParameters: 2
        Parameter[0]: 0000000000000001
        Parameter[1]: 0000000000000000
        Attempt to write to address 0000000000000000
        0:000> .exr 0x21edf880
        ExceptionAddress: 000007fef7de1376 (winhttp!RemoveFromSerializedList+0×0000000000000026)
        ExceptionCode: c0000005 (Access violation)
        ExceptionFlags: 00000000
        NumberParameters: 2
        Parameter[0]: 0000000000000001
        Parameter[1]: 0000000000000000
        Attempt to write to address 0000000000000000

        MODULE_NAME: winhttp

        IMAGE_NAME: winhttp.dll

        DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7ca23

        FAILURE_BUCKET_ID: APPLICATION_FAULT_42ac_winhttp.dll!RemoveFromSerializedList

        BUCKET_ID: X64_ACCESS_VIOLATION_winhttp!RemoveFromSerializedList+26

        WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_4000_0/4fecc5ba/sqlservr_exe/2009_100_4000_0/4fecc5ba/42ac/020eae68.htm?Retriage=1

        Followup: MachineOwner
        ———

        • I think you have run “!analyze -v”.

          I found it by looking at module name(I missed to mark in red, will edit it).
          In this case winhttp.dll is referencing a NULL pointer so AV.

          If you suspect CLR then Disable CLR – >restart SQL – > check if winhttp.dll is still loaded. You can use below query. If you dont find winhttp.dll after disabling the CLR, Then its CLR.
          {
          select name as name1, * from sys.dm_os_loaded_modules where company ‘Microsoft Corporation’ or name like ‘%winhttp%’
          }

          I have seen in past application firewall’s loading winhttp.dll. Do you have any type of “application firewall” installed on the SQL server?

          • Scott McCormick said

            There is no firewall.

            Also, at the moment, winhttp is not loaded. My understanding of CLRs is that they are loaded up when the instance starts and are only unloaded due to memory pressure, but maybe that’s wrong? Is it possible the winhttp module isn’t loaded because the CLR has not been called since the last reboot (yesterday afternoon)? Or maybe it’s just not a CLR, but I don’t know what else it could be.

            There aren’t that many things that go outside of SQL Server.

            • Hello Scott,
              By any chance do you see MSO.dll loaded in SQL Server .

              • Scott McCormick said

                Hey Karthik –

                It’s a giant coincidence that you would mention mso.dll today. Yesterday, there was a crash that didn’t generate a Stack Dump, but did apparently crash due to a mso.dll call. The event viewer message is:

                Faulting application name: sqlservr.exe, version: 2009.100.4000.0, time stamp: 0x4fecc5ba
                Faulting module name: mso.dll, version: 14.0.4760.1000, time stamp: 0x4ba906d8
                Exception code: 0xc0000005
                Fault offset: 0x0000000000050d4a
                Faulting process id: 0x3b78
                Faulting application start time: 0x01cdcc96f46da952
                Faulting application path: C:\Program Files\Microsoft SQL Server\MSSQL10_50.SQL2008\MSSQL\Binn\sqlservr.exe
                Faulting module path: C:\Program Files\Common Files\Microsoft Shared\office14\mso.dll
                Report Id: fcfb0fc2-5e76-11e2-8623-a4badb0bce84

                • Hello Scott,

                  Few days back one of the SQL DBA came to me with same issue which you faced SQLServer carsh by by wihttp.dll (winhttp!RemoveFromSerializedList). He shared the memory dump with me. We looked at the dump and found TLS (thread local storage, a mechanism by which threads can store data that is unique to each thread) is corrupted for winhttp.dll thread.When winhttp tried to do cleanup in the TLS, it encountered AV as the TLS is corrupted (Same issue as yours) but who corrupted the thread is still not clear. I am suspecting it to be mso.dll because “App verifier” pointed that this dll is corrupting TLS when I was debugging an other TLS corruption issue. There is also a FIX http://support.microsoft.com/kb/2516475 which will update MSO.dll

                  Can you please confirm if the crash by MSO.dll happened in the same server in which we got dump because of winhttp.dll? If yes please confirm me and apply the above fix for MSO.dll

                  Also can you let me know if below two options are checked in (Internet explorer–>Tools–>Internet options –>Advanced)

                  1. “Check for publisher’s certificate revocation”
                  2. “Check for server certificate revocation”

                  Thanks

                  • Scott McCormick said

                    Hey Karthick –

                    This is the same server, and I’ll start pushing to apply this patch.

                    Also, those options are selected in the IE on the server.

                    Scott

                    • If those options are selected please uncheck them
                      “Check for publisher’s certificate revocation”
                      “Check for server certificate revocation”

                    • Any update Scott?

                    • Scott McCormick said

                      Yeah, I flipped the switches on IE and installed the patch and it looks like that might have resolved it. It comes and goes, so it’s hard to say for sure, but I don’t think it’s an issue any more. Thanks a lot for your help, I really appreciate it.

  2. Hi Karthick – I follow mentioned procedure and got below dump, Please let me know what should I do next.

    0:000> kC 1000
    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqlservr!CDmpDump::DumpInternal
    sqlservr!CDmpDump::DumpFilter
    sqlservr!CDmpDump::Dump
    msvcr80!__C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!RtlRaiseException
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_terminator
    sqlservr!SOS_SEHTranslator
    msvcr80!_CallSETranslator
    msvcr80!FindHandlerForForeignException
    msvcr80!FindHandler
    msvcr80!__InternalCxxFrameHandler
    msvcr80!__CxxFrameHandler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    sqlservr!CSlotPageMgr::PbAllocate
    sqlservr!CMemObj::Alloc
    sqlservr!CMemThread::Alloc
    sqlservr!CCriticalSectionSOS::Initialize
    sqlservr!Tcp::FInit
    sqlservr!Tcp::AcceptConnection
    sqlservr!Tcp::AcceptDone
    sqlservr!SNIAcceptDoneWithReturnCode
    sqlservr!SNIAcceptDoneWrapper
    sqlservr!SNIAcceptDoneRouter
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  3. Jason Chamiak said

    Hi Karthick,

    This was a great article. After following your processes, I retrieved the following thread stack responsible for my recent crash.Can you tell me what happened here and what recommendations you have?

    0:000> kC 1000
    Call Site
    sqlservr!CMEDIndexStatsCollection::GetProxyIndexWithLock
    sqlservr!CMEDProxyRelation::GetProxyIndexWithLock
    sqlservr!CMEDProxyRelation::GetIndex
    sqlservr!CMetadataUtil::GetIndexFromIMED
    sqlservr!CAlgTableMetadata::PimidxBaseIndex
    sqlservr!CAlgTableMetadata::LoadSpecialColumns
    sqlservr!CAlgTableMetadata::LoadColumns
    sqlservr!CAlgTableMetadata::Bind
    sqlservr!CRelOp_Get::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_FromList::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_Select::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_QuerySpec::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_DerivedTable::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_BaseJoin::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_FromList::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_QuerySpec::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_Insert::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_DMLQuery::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_Query::FAlgebrizeQuery
    sqlservr!CProchdr::FNormQuery
    sqlservr!CProchdr::FNormalizeStep
    sqlservr!CSQLSource::FCompile
    sqlservr!CSQLSource::FCompWrapper
    sqlservr!CSQLSource::Transform
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    Thanks,

    Jason Chamiak

    • Hello Jason, This looks like a known issue, can let me know your SQL Server build number (select @@version). Do you get this dump when you fire query with “select TOP n on a table with identity column? ThanksKarthick > Date: Fri, 16 Nov 2012 19:53:19 +0000 > To: karthick_pk83@hotmail.com >

      • Jason Chamiak said

        Hi Karthick,

        This is the output of @@version:

        Microsoft SQL Server 2008 R2 (SP1) – 10.50.2500.0 (X64) Jun 17 2011 00:54:03 Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows NT 6.1 (Build 7600: )

        I do not see “SELECT TOP” in the stored procedures referenced in the dumps in the SQL Server log.

        Thanks,

        Jason Chamiak

  4. Nuno Gomes said

    0:000> kC 1000
    Call Site
    sqlservr!CTableVariableManager::GetRowsetidListAndReturnTableInfo
    sqlservr!CRangeTableVariable::XretSchemaChanged
    sqlservr!CEnvCollection::XretSchemaChanged
    sqlservr!CXStmtCondWithQuery::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    And the version is: 10.50.1600.1

    Thanks,
    Nuno Gomes

  5. [...] SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion [...]

  6. Meera Sinhasane said

    Hi Karthick ,

    SQL Server in our environment crashed generating an access violation. From the dump file it appears to be crashed due to msxml component but i am not pretty sure on it. Could you please take a look at the below stack and guide me on this.

    Microsoft SQL Server 2008 R2 (RTM) – 10.50.1600.1 (X64)
    Apr 2 2010 15:48:46
    Copyright (c) Microsoft Corporation
    Enterprise Edition (64-bit) on Windows NT 6.1 (Build 7600: ) (Hypervisor)

    2013-01-14 21:33:51.74 Server Error: 17311, Severity: 16, State: 1.
    2013-01-14 21:33:51.74 Server SQL Server is terminating because of fatal exception c0000005. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).
    2013-01-14 21:33:51.79 Server Using ‘dbghelp.dll’ version ’4.0.5′
    2013-01-14 21:33:51.83 Server **Dump thread – spid = 0, EC = 0×0000000000000000
    2013-01-14 21:33:51.83 Server ***Stack Dump being sent to D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\LOG\SQLDump0001.txt
    2013-01-14 21:33:51.83 Server * *******************************************************************************
    2013-01-14 21:33:51.83 Server *
    2013-01-14 21:33:51.83 Server * BEGIN STACK DUMP:
    2013-01-14 21:33:51.83 Server * 01/14/13 21:33:51 spid 2636
    2013-01-14 21:33:51.83 Server *
    2013-01-14 21:33:51.83 Server * ex_handle_except encountered exception C0000005 – Server terminating

    0:000> kc 100
    Call Site
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_handle_except
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    msxml6!Base::freeRentalObjects
    msxml6!Base::StackExitNormal
    msxml6!ModelInit::~ModelInit
    msxml6!SAXReader::parse
    MSO
    0×0
    0×0
    0×0

  7. Laurens said

    Hi Karthick,

    Could you please have a look at the debug below. I see that the wininet is probably causing the problem? There are no jobs running during this proces apart from a ms windows server backup. We do use CDO during the day, could the VC be cleaning the memory during the night at 1:30? Or could the windows update process use wininet?

    Microsoft (R) Windows Debugger Version 6.2.9200.16384 AMD64
    Copyright (c) Microsoft Corporation. All rights reserved.

    Loading Dump File [D:\Data\MSSQL10_50.MSSQLSERVER\MSSQL\Log\SQLDump0003.mdmp]
    Comment: ‘Stack Trace’
    Comment: ‘ex_handle_except encountered exception C0000005 – Server terminating’
    User Mini Dump File: Only registers, stack and portions of memory are available

    Symbol search path is: *** Invalid ***
    ****************************************************************************
    * Symbol loading may be unreliable without a symbol search path. *
    * Use .symfix to have the debugger choose a symbol path. *
    * After setting your symbol path, use .reload to refresh symbol locations. *
    ****************************************************************************
    Executable search path is:
    Windows 7 Version 7601 (Service Pack 1) UP Free x64
    Product: Server, suite: TerminalServer SingleUserTS Blade
    Machine Name:
    Debug session time: Tue Jan 29 01:31:18.000 2013 (UTC + 1:00)
    System Uptime: 3 days 16:22:27.234
    Process Uptime: 3 days 16:21:44.000
    ……………………………………………………….
    ……………………………………………………….
    …………
    Loading unloaded module list
    ……………………………………………………………
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (598.e94): Unknown exception – code 000042ac (first/second chance not available)
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll –
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for KERNELBASE.dll –
    ntdll!NtWaitForSingleObject+0xa:
    00000000`7707135a c3 ret
    0:000> .sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
    Symbol search path is: srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
    Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
    0:000> .reload /f
    .*** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlservr.exe –
    …….

    Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
    Run !sym noisy before .reload to track down problems loading symbols.

    …*** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlos.dll –
    ……………………..*** ERROR: Symbol file could not be found. Defaulted to export symbols for instapi10.dll –
    ..*** ERROR: Module load completed but symbols could not be loaded for sqlevn70.rll
    …………………*** ERROR: Symbol file could not be found. Defaulted to export symbols for msfte.dll –
    ….
    ………..*** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlncli10.dll –
    …*** ERROR: Module load completed but symbols could not be loaded for SQLNCLIR10.RLL
    …….*** ERROR: Module load completed but symbols could not be loaded for odbcint.dll
    …*** ERROR: Module load completed but symbols could not be loaded for XPStar.RLL
    ..*** ERROR: Module load completed but symbols could not be loaded for xplog70.rll
    ……..*** ERROR: Module load completed but symbols could not be loaded for oledb32r.dll
    ..*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEOLEDB.DLL –
    ..*** ERROR: Symbol file could not be found. Defaulted to export symbols for MSO.DLL –
    …*** ERROR: Module load completed but symbols could not be loaded for ACECORE.DLL
    ..*** ERROR: Module load completed but symbols could not be loaded for ACEWSTR.DLL
    .*** ERROR: Module load completed but symbols could not be loaded for MSORES.DLL
    .*** ERROR: Module load completed but symbols could not be loaded for MSOINTL.DLL
    .*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEINTL.DLL –
    ………..*** ERROR: Module load completed but symbols could not be loaded for INETRES.dll
    …….
    *** ERROR: Module load completed but symbols could not be loaded for ACCTRES.dll
    …..*** ERROR: Module load completed but symbols could not be loaded for odsole70.rll
    ..*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEERR.DLL –
    .*** ERROR: Symbol file could not be found. Defaulted to export symbols for ACEES.DLL –
    ….
    Loading unloaded module list
    ……………………………………………………………
    0:000> lmvm sqlservr
    start end module name
    00000000`00b10000 00000000`04722000 sqlservr (export symbols) sqlservr.exe
    Loaded symbol image file: sqlservr.exe
    Mapped memory image file: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
    Image path: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
    Image name: sqlservr.exe
    Timestamp: Thu Jun 28 22:59:38 2012 (4FECC5BA)
    CheckSum: 03B58282
    ImageSize: 03C12000
    File version: 2009.100.4000.0
    Product version: 10.50.4000.0
    File flags: 0 (Mask 3F)
    File OS: 40000 NT Base
    File type: 1.0 App
    File date: 00000000.00000000
    Translations: 0409.04b0
    CompanyName: Microsoft Corporation
    ProductName: Microsoft SQL Server
    InternalName: SQLSERVR
    OriginalFilename: SQLSERVR.EXE
    ProductVersion: 10.50.4000.0
    FileVersion: 2009.0100.4000.00 ((KJ_PCU_Main).120628-0827 )
    FileDescription: SQL Server Windows NT – 64 Bit
    LegalCopyright: Microsoft Corp. All rights reserved.
    LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
    Comments: SQL
    0:000> Type .ecxr
    ^ No runnable debuggees error in ‘Type .ecxr ‘
    0:000> .ecxr
    rax=000000006e6313fe rbx=00000000040f5650 rcx=000000001976ca90
    rdx=0000000000000000 rsi=0000000003cffcfc rdi=0000000000000440
    rip=000007fefd7a9e5d rsp=000000001976d0a0 rbp=000000001976fe30
    r8=0000000000000000 r9=0000000000000000 r10=00000000c000007c
    r11=00000000013fdf60 r12=0000000000000e94 r13=0000000000000000
    r14=0000000000000000 r15=0000000000000e94
    iopl=0 nv up ei pl nz na pe nc
    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
    KERNELBASE!RaiseException+0×39:
    000007fe`fd7a9e5d 4881c4c8000000 add rsp,0C8h
    0:000> kC 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!DmpRemoteDumpRequest
    sqlservr!SQLExit
    sqlservr!SQLExit
    sqlservr!SQLExit
    sqlservr!SQLExit
    sqlservr!SQLExit
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatch
    wininet!InternetFreeThreadInfo
    wininet!InternetDestroyThreadInfo
    wininet!DllMain_wininet
    wininet!CRT_INIT
    ntdll!LdrShutdownThread
    ntdll!RtlExitUserThread
    msvcr80!endthreadex
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart
    0:000> kL
    *** Stack trace for last set context – .thread/.cxr resets it
    Child-SP RetAddr Call Site
    00000000`1976d0a0 00000000`036c440c KERNELBASE!RaiseException+0×39
    00000000`1976d170 00000000`02bfae68 sqlservr!DmpRemoteDumpRequest+0x2fec
    00000000`1976d1c0 00000000`02bfb9bc sqlservr!SQLExit+0x303a8
    00000000`1976d1f0 00000000`02bee097 sqlservr!SQLExit+0x30efc
    00000000`1976d3f0 00000000`02bed7e9 sqlservr!SQLExit+0x235d7
    00000000`1976e940 00000000`03032206 sqlservr!SQLExit+0x22d29
    00000000`1976e980 00000000`769d9460 sqlservr!SQLExit+0×467746
    00000000`1976ebd0 00000000`770b43b8 kernel32!UnhandledExceptionFilter+0×160
    00000000`1976ecb0 00000000`770385a8 ntdll! ?? ::FNODOBFM::`string’+0×2365
    00000000`1976ece0 00000000`77049d0d ntdll!_C_specific_handler+0x8c
    00000000`1976ed50 00000000`770391af ntdll!RtlpExecuteHandlerForException+0xd
    00000000`1976ed80 00000000`77071278 ntdll!RtlDispatchException+0x45a
    00000000`1976f460 00000000`76ee88cb ntdll!KiUserExceptionDispatch+0x2e
    00000000`1976fa20 00000000`76ee8892 wininet!InternetFreeThreadInfo+0x1f
    00000000`1976fa50 00000000`76ec128e wininet!InternetDestroyThreadInfo+0×40
    00000000`1976fa80 00000000`76ec10d3 wininet!DllMain_wininet+0xb5
    00000000`1976fad0 00000000`77066a28 wininet!CRT_INIT+0x2b3
    00000000`1976fc30 00000000`77066968 ntdll!LdrShutdownThread+0×155
    00000000`1976fd30 00000000`747b37b7 ntdll!RtlExitUserThread+0×38
    00000000`1976fd70 00000000`747b37de msvcr80!endthreadex+0×27
    00000000`1976fda0 00000000`747b3894 msvcr80!endthreadex+0x4e
    00000000`1976fdd0 00000000`7695652d msvcr80!endthreadex+0×104
    00000000`1976fe00 00000000`7704c521 kernel32!BaseThreadInitThunk+0xd
    00000000`1976fe30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

    • Can you share the output of

      1. lmvm mso
      2. lmvm cdosys

      • Laurens said

        0:000> lmvm cdosys
        start end module name
        000007fe`e7c50000 000007fe`e7d69000 cdosys (pdb symbols) c:\websymbols\cdosys.pdb\3C0CCADEEF824576A6729525EBB42A8C1\cdosys.pdb
        Loaded symbol image file: cdosys.dll
        Mapped memory image file: c:\websymbols\cdosys.dll\4FCEEFFF119000\cdosys.dll
        Image path: C:\Windows\System32\cdosys.dll
        Image name: cdosys.dll
        Timestamp: Wed Jun 06 07:51:59 2012 (4FCEEFFF)
        CheckSum: 00117D93
        ImageSize: 00119000
        File version: 6.6.7601.17857
        Product version: 6.6.7601.17857
        File flags: 0 (Mask 3F)
        File OS: 40004 NT Win32
        File type: 2.0 Dll
        File date: 00000000.00000000
        Translations: 0409.04b0
        CompanyName: Microsoft Corporation
        ProductName: Microsoft® Windows® Operating System
        InternalName: CDOSYS.DLL
        OriginalFilename: CDOSYS.DLL
        ProductVersion: 6.6.7601.17857
        FileVersion: 6.6.7601.17857 (win7sp1_gdr.120605-1503)
        FileDescription: Microsoft CDO for Windows Library
        LegalCopyright: © Microsoft Corporation. All rights reserved.
        0:000> lmvm mso
        start end module name
        000007fe`e8090000 000007fe`e98c2000 MSO (export symbols) MSO.DLL
        Loaded symbol image file: MSO.DLL
        Mapped memory image file: C:\Program Files\Common Files\Microsoft Shared\OFFICE14\MSO.DLL
        Image path: C:\Program Files\Common Files\Microsoft Shared\OFFICE14\MSO.DLL
        Image name: MSO.DLL
        Timestamp: Tue Mar 23 19:22:16 2010 (4BA906D8)
        CheckSum: 01832834
        ImageSize: 01832000
        File version: 14.0.4760.1000
        Product version: 14.0.4760.0
        File flags: 0 (Mask 3F)
        File OS: 40004 NT Win32
        File type: 2.0 Dll
        File date: 00000000.00000000
        Translations: 0000.04e4
        CompanyName: Microsoft Corporation
        ProductName: Microsoft Office 2010
        InternalName: MSO
        OriginalFilename: MSO.DLL
        ProductVersion: 14.0.4760.1000
        FileVersion: 14.0.4760.1000
        FileDescription: Microsoft Office 2010 component
        LegalCopyright: © 2010 Microsoft Corporation. All rights reserved.

  8. Laurens said

    Brilliant, thanks for the analysis, how did you figure it’s the mso.dll? Did the debugger indicate this or is it on the basis that there’s a newer version of the mso.dll available?

    • If you look at your stack “wininet!InternetFreeThreadInfo” exception is raised when the threads local storage (TLS) is cleared,So basically some one else has already corrupted TLS.
      Who corrupted it? Most probably MSO.DLL because “App verifier” pointed that this dll is corrupting TLS when I was debugging an other TLS corruption issue. There is also a FIX http://support.microsoft.com/kb/2516475 which will update MSO.dll.

      • Laurens said

        Tanks for the reply, I’ve tried to install the update, states that it is already installed, I believe I read somewhere you have to place files manualy?

        • Hello Lauren,
          I do know how I missed your comment.Iam sorry.
          Try running repair on Microsoft access database engine,It should update your MSO.DLL version

          • Laurens said

            Hi Karthick,

            So far so good, the server has stopped crashing. In order to install the update, you have to extract it, until you can extract it no longer, run it, then use the repair function from the installed applications in the control panel.

            Thanks,

            Laurens

  9. Hi Karthick,

    Can you please help me with below access violation exception.

    0:000> kC 1000
    Call Site
    sqlservr!CTableVariableManager::InsertTableRef
    sqlservr!alloca_probe
    sqlservr!CFormalParamTbl::InitParams
    sqlservr!alloca_probe
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!alloca_probe
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    0×0
    0×0

  10. manu0417 said

    Hi Karthick,

    Today we had an assertion on one of our Test servers and below is the kC 1000 output. The server went unresponsive after that.

    0:000> kC 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!utassert_fail
    sqlservr!purecall
    sqlservr!GetRemoteTaskProxyWithRef
    sqlservr!TasksTable::MoveToNextRow
    sqlservr!TasksTable::InternalGetRow
    sqlservr!CQScanTVFStreamNew::GetRow
    sqlservr!CQScanSortNew::BuildSortTable
    sqlservr!CQScanSortNew::OpenHelper
    sqlservr!CQScanMergeJoinNew::Open
    sqlservr!CQScanMergeJoinNew::Open
    sqlservr!CQScanMergeJoinNew::Open
    sqlservr!CQScanNLJoinNew::Open
    sqlservr!CQScanNew::OpenHelper
    sqlservr!CQScanTopNew::Open
    sqlservr!CQScanNew::OpenHelper
    sqlservr!CQScanUpdateNew::Open
    sqlservr!CQueryScan::Startup
    sqlservr!CXStmtQuery::SetupQueryScanAndExpression
    sqlservr!CXStmtQuery::ErsqExecuteQuery
    sqlservr!CXStmtDML::XretDMLExecute
    sqlservr!CXStmtSelectInto::XretSelectIntoExecute
    sqlservr!CXStmtSelectInto::XretExecute
    sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!alloca_probe
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!alloca_probe
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    0×0
    0×0

    • Share the Build of SQLServer and let me know if it used as backend for Biztalk.

      • manu0417 said

        We don’t use a BizTalk. Build is 10.0.4279, 2008 SP2 CU3.
        The procedure which raised this Assertion does the auditing on the session activity and inserts the sampled data in to an audit table, bunch of simple select and insert statements.

        • Hello Manu,
          Can you please share the error message you got in errorlog for this assert?
          It should be some thig like
          SQL Server Assertion: File: , line
          in your errorlog when dump was generated.

          • manu0417 said

            2013-03-01 04:00:03.01 spid231 ***Stack Dump being sent to D:\MSSQL10.MSSQLSERVER\MSSQL\LOG\SQLDump1160.txt
            2013-03-01 04:00:03.01 spid231 * *******************************************************************************
            2013-03-01 04:00:03.01 spid231 *
            2013-03-01 04:00:03.01 spid231 * BEGIN STACK DUMP:
            2013-03-01 04:00:03.01 spid231 * 03/01/13 04:00:03 spid 231
            2013-03-01 04:00:03.01 spid231 *
            2013-03-01 04:00:03.01 spid231 * Location: purecall.cpp:51
            2013-03-01 04:00:03.01 spid231 * Expression: !”purecall”
            2013-03-01 04:00:03.01 spid231 * SPID: 231
            2013-03-01 04:00:03.01 spid231 * Process ID: 21520
            2013-03-01 04:00:03.01 spid231 * Description: Pure virtual function call

            • Do you have any antivirus which detours in sqlserver address space like sophose etc?
              Post the output of lm from your dump. Let me check

              • manu0417 said

                I did not find any Antivirus. Below is the result of lm

                0:000> lm
                start end module name
                00000000`002a0000 00000000`03b0f000 sqlservr (deferred)
                00000000`1f320000 00000000`1f369000 System_Transactions (deferred)
                00000000`2de70000 00000000`2e16f000 System_Data (deferred)
                00000000`48060000 00000000`4806d000 instapi (deferred)
                00000000`49980000 00000000`49d31000 msfte (deferred)
                00000000`60000000 00000000`60025000 ftimport (deferred)
                00000000`6de10000 00000000`6e00f000 sqlevn70 (deferred)
                00000000`6f3d0000 00000000`6f52e000 DBGHELP (deferred)
                00000000`6fbf0000 00000000`6fefe000 System (deferred)
                00000000`6ff00000 00000000`7003e000 comres (deferred)
                00000000`70520000 00000000`70562000 System_Security (deferred)
                00000000`70570000 00000000`705dc000 System_Configuration (deferred)
                00000000`705e0000 00000000`707d8000 System_Xml (deferred)
                00000000`714b0000 00000000`714c4000 oledb32r (deferred)
                00000000`71640000 00000000`71791000 msxmlsql (deferred)
                00000000`71840000 00000000`71857000 xprepl (deferred)
                00000000`71920000 00000000`719a8000 XPStar (deferred)
                00000000`71e70000 00000000`71e72000 xplog70_71e70000 (deferred)
                00000000`71e80000 00000000`71e90000 xplog70 (deferred)
                00000000`71f20000 00000000`71f45000 XPStar_71f20000 (deferred)
                00000000`71f50000 00000000`71f58000 xpsqlbot (deferred)
                00000000`71fe0000 00000000`71fec000 sqlscm (deferred)
                00000000`720e0000 00000000`7210c000 BatchParser (deferred)
                00000000`72110000 00000000`72147000 SQLNCLIR10 (deferred)
                00000000`72150000 00000000`7245a000 sqlncli10 (deferred)
                00000000`724b0000 00000000`7260e000 dbghelp_724b0000 (deferred)
                00000000`727f0000 00000000`727f7000 sqlos (deferred)
                00000000`72800000 00000000`72808000 opends60 (deferred)
                00000000`72fa0000 00000000`72fb3000 ssdebugps (deferred)
                00000000`73640000 00000000`73678000 odbcint (deferred)
                00000000`737d0000 00000000`73837000 SqlAccess (deferred)
                00000000`73840000 00000000`73843000 security (deferred)
                00000000`73cf0000 00000000`73cfd000 instapi10 (deferred)
                00000000`74850000 00000000`74870000 ATL80 (deferred)
                00000000`74ad0000 00000000`74bd9000 msvcp80 (deferred)
                00000000`74be0000 00000000`74ca9000 msvcr80 (deferred)
                00000000`76d70000 00000000`76e6a000 user32 (deferred)
                00000000`76e70000 00000000`76f8f000 kernel32 (deferred)
                00000000`76f90000 00000000`77139000 ntdll (pdb symbols) your local symbol folder\ntdll.pdb\15EB43E23B12409C84E3CC7635BAF5A32\ntdll.pdb
                00000000`77160000 00000000`77167000 psapi (deferred)
                00000642`ffaf0000 00000642`ffb09000 mscorsec (deferred)
                00000642`ffff0000 00000642`ffffd000 wminet_utils (deferred)
                000007fe`ebd00000 000007fe`ebde6000 System_Security_ni (deferred)
                000007fe`ebdf0000 000007fe`ebefe000 oledb32 (deferred)
                000007fe`ecae0000 000007fe`ecc64000 mscorjit (deferred)
                000007fe`ecc70000 000007fe`eccbe000 pdh (deferred)
                000007fe`eccc0000 000007fe`ecce8000 msdart (deferred)
                000007fe`eccf0000 000007fe`ecd90000 comctl32 (deferred)
                000007fe`ed8d0000 000007fe`ed90b000 winmm (deferred)
                000007fe`edc20000 000007fe`edc8a000 msdaps (deferred)
                000007fe`ef450000 000007fe`ef501000 odbc32 (deferred)
                000007fe`ef530000 000007fe`ef5ec000 msdtcprx (deferred)
                000007fe`ef5f0000 000007fe`ef650000 mtxclu (deferred)
                000007fe`efb60000 000007fe`efb6f000 cscapi (deferred)
                000007fe`efb70000 000007fe`efb8b000 cabinet (deferred)
                000007fe`efb90000 000007fe`efbb6000 cryptnet (deferred)
                000007fe`efd10000 000007fe`efd19000 SensApi (deferred)
                000007fe`efe30000 000007fe`efe42000 browcli (deferred)
                000007fe`efe50000 000007fe`efe5a000 netbios (deferred)
                000007fe`efe80000 000007fe`efe94000 xolehlp (deferred)
                000007fe`f0c00000 000007fe`f0c19000 resutils (deferred)
                000007fe`f0c20000 000007fe`f0c70000 clusapi (deferred)
                000007fe`f0e00000 000007fe`f0e26000 wmiutils (deferred)
                000007fe`f0e30000 000007fe`f0e44000 wbemsvc (deferred)
                000007fe`f0f00000 000007fe`f0f27000 ntdsapi (deferred)
                000007fe`f0f30000 000007fe`f1012000 fastprox (deferred)
                000007fe`f6f60000 000007fe`f7985000 System_ni (deferred)
                000007fe`f7990000 000007fe`f886c000 mscorlib_ni (deferred)
                000007fe`f8870000 000007fe`f920d000 mscorwks (deferred)
                000007fe`f9210000 000007fe`f92a0000 mscoreei (deferred)
                000007fe`f92a0000 000007fe`f930f000 mscoree (deferred)
                000007fe`f9310000 000007fe`f9396000 wbemcomn (deferred)
                000007fe`f93a0000 000007fe`f93af000 wbemprox (deferred)
                000007fe`f9600000 000007fe`f961d000 samlib (deferred)
                000007fe`f9620000 000007fe`f9634000 samcli (deferred)
                000007fe`f9a70000 000007fe`f9a78000 rasadhlp (deferred)
                000007fe`fa0d0000 000007fe`fa134000 webio (deferred)
                000007fe`fa140000 000007fe`fa1b1000 winhttp (deferred)
                000007fe`fa1c0000 000007fe`fa1d8000 dhcpcsvc (deferred)
                000007fe`fa1e0000 000007fe`fa1f1000 dhcpcsvc6 (deferred)
                000007fe`fa200000 000007fe`fa253000 FWPUCLNT (deferred)
                000007fe`fa690000 000007fe`fa6a5000 wkscli (deferred)
                000007fe`fa6b0000 000007fe`fa6c6000 netapi32 (deferred)
                000007fe`fa710000 000007fe`fa71b000 winnsi (deferred)
                000007fe`fa720000 000007fe`fa747000 IPHLPAPI (deferred)
                000007fe`fae60000 000007fe`fae69000 msxactps (deferred)
                000007fe`fb150000 000007fe`fb15a000 ktmw32 (deferred)
                000007fe`fb820000 000007fe`fb82c000 dsrole (deferred)
                000007fe`fb870000 000007fe`fba64000 comctl32_7fefb870000 (deferred)
                000007fe`fbff0000 000007fe`fc01d000 ntmarta (deferred)
                000007fe`fc020000 000007fe`fc02c000 version (deferred)
                000007fe`fc0f0000 000007fe`fc0f7000 WSHTCPIP (deferred)
                000007fe`fc1f0000 000007fe`fc20b000 gpapi (deferred)
                000007fe`fc210000 000007fe`fc22e000 userenv (deferred)
                000007fe`fc230000 000007fe`fc242000 devrtl (deferred)
                000007fe`fc270000 000007fe`fc2bc000 bcryptprimitives (deferred)
                000007fe`fc370000 000007fe`fc37a000 credssp (deferred)
                000007fe`fc3a0000 000007fe`fc3fb000 dnsapi (deferred)
                000007fe`fc560000 000007fe`fc5a7000 rsaenh (deferred)
                000007fe`fc5f0000 000007fe`fc648000 schannel (deferred)
                000007fe`fc650000 000007fe`fc6a1000 msv1_0 (deferred)
                000007fe`fc6b0000 000007fe`fc6bc000 netutils (deferred)
                000007fe`fc6d0000 000007fe`fc700000 logoncli (deferred)
                000007fe`fc700000 000007fe`fc755000 mswsock (deferred)
                000007fe`fc760000 000007fe`fc814000 kerberos (deferred)
                000007fe`fc890000 000007fe`fc8a7000 cryptsp (deferred)
                000007fe`fc8b0000 000007fe`fc8fe000 ncrypt (deferred)
                000007fe`fc900000 000007fe`fc92f000 authz (deferred)
                000007fe`fc930000 000007fe`fc937000 wship6 (deferred)
                000007fe`fc940000 000007fe`fc962000 bcrypt (deferred)
                000007fe`fc9f0000 000007fe`fca04000 cryptdll (deferred)
                000007fe`fca10000 000007fe`fca33000 srvcli (deferred)
                000007fe`fccf0000 000007fe`fccfb000 secur32 (deferred)
                000007fe`fcd40000 000007fe`fcd65000 sspicli (deferred)
                000007fe`fcd70000 000007fe`fcdc7000 apphelp (deferred)
                000007fe`fcdd0000 000007fe`fcddf000 CRYPTBASE (deferred)
                000007fe`fcec0000 000007fe`fced4000 RpcRtRemote (deferred)
                000007fe`fcee0000 000007fe`fceef000 profapi (deferred)
                000007fe`fcf80000 000007fe`fcf8f000 msasn1 (deferred)
                000007fe`fcfd0000 000007fe`fd03c000 KERNELBASE (pdb symbols) your local symbol folder\kernelbase.pdb\61044362232B410AA600843CEBFD11612\kernelbase.pdb
                000007fe`fd040000 000007fe`fd07a000 wintrust (deferred)
                000007fe`fd140000 000007fe`fd2a7000 crypt32 (deferred)
                000007fe`fd2b0000 000007fe`fd2be000 lpk (deferred)
                000007fe`fd2c0000 000007fe`fd389000 usp10 (deferred)
                000007fe`fd390000 000007fe`fd398000 nsi (deferred)
                000007fe`fd550000 000007fe`fd627000 oleaut32 (deferred)
                000007fe`fd630000 000007fe`fd70b000 advapi32 (deferred)
                000007fe`fd710000 000007fe`fd7a7000 comdlg32 (deferred)
                000007fe`fd930000 000007fe`fe6b8000 shell32 (deferred)
                000007fe`fe6c0000 000007fe`fe7c9000 msctf (deferred)
                000007fe`fe7d0000 000007fe`fe86f000 msvcrt (deferred)
                000007fe`fe870000 000007fe`fe8d7000 gdi32 (deferred)
                000007fe`fe8e0000 000007fe`fe979000 clbcatq (deferred)
                000007fe`fe980000 000007fe`feb83000 ole32 (deferred)
                000007fe`feb90000 000007fe`febdd000 ws2_32 (deferred)
                000007fe`febe0000 000007fe`fec32000 Wldap32 (deferred)
                000007fe`fec40000 000007fe`fec5f000 sechost (deferred)
                000007fe`fec60000 000007fe`fec8e000 imm32 (deferred)
                000007fe`feef0000 000007fe`fef07000 imagehlp (deferred)
                000007fe`fef10000 000007fe`fef81000 shlwapi (deferred)
                000007fe`ff170000 000007fe`ff29d000 rpcrt4 (deferred)

                Unloaded modules:
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`73880000 00000000`73898000 sqlboot.dll
                00000000`73860000 00000000`73878000 sqlboot.dll
                00000000`73880000 00000000`73898000 sqlboot.dll
                000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                00000000`72500000 00000000`72518000 sqlboot.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                00000000`72500000 00000000`72518000 sqlboot.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                00000000`72500000 00000000`72518000 sqlboot.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                00000000`72500000 00000000`72518000 sqlboot.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                00000000`72560000 00000000`72578000 sqlboot.dll
                00000000`725b0000 00000000`725c8000 sqlboot.dll
                000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
                000007fe`fd120000 000007fe`fd13a000 DEVOBJ.dll
                000007fe`fef90000 000007fe`ff167000 setupapi.dll
                00000000`71870000 00000000`71888000 sqlboot.dll
                00000000`71800000 00000000`71818000 sqlboot.dll
                00000000`71870000 00000000`71888000 sqlboot.dll
                00000000`71800000 00000000`71818000 sqlboot.dll
                00000000`71870000 00000000`71888000 sqlboot.dll
                00000000`71800000 00000000`71818000 sqlboot.dll
                00000000`71870000 00000000`71888000 sqlboot.dll
                00000000`71800000 00000000`71818000 sqlboot.dll
                00000000`71870000 00000000`71888000 sqlboot.dll
                00000000`71800000 00000000`71818000 sqlboot.dll
                00000000`71870000 00000000`71888000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`6e050000 00000000`6e1ae000 dbghelp.dll
                00000000`72540000 00000000`7269e000 dbghelp.dll
                00000000`6e050000 00000000`6e1ae000 dbghelp.dll
                00000000`6b4d0000 00000000`6b62e000 dbghelp.dll
                000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll
                000007fe`fd120000 000007fe`fd13a000 DEVOBJ.dll
                000007fe`fef90000 000007fe`ff167000 setupapi.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`715b0000 00000000`715c8000 sqlboot.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`715b0000 00000000`715c8000 sqlboot.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`715b0000 00000000`715c8000 sqlboot.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`715b0000 00000000`715c8000 sqlboot.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`715b0000 00000000`715c8000 sqlboot.dll
                00000000`71490000 00000000`714a8000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`72c80000 00000000`72c98000 sqlboot.dll
                00000000`73770000 00000000`73788000 sqlboot.dll
                00000000`73880000 00000000`73898000 sqlboot.dll
                00000000`73860000 00000000`73878000 sqlboot.dll
                00000000`73880000 00000000`73898000 sqlboot.dll
                000007fe`fcf90000 000007fe`fcfc6000 CFGMGR32.dll

  11. Dee said

    What about this…I have 53 stack dumps all with different output…this is the results of the first one for the day..
    running SQL server 2008 sp1 32 bit…

    0:000> lmvm sqlservr
    start end module name
    01000000 03970000 sqlservr T (no symbols)
    Loaded symbol image file: sqlservr.exe
    Image path: E:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
    Image name: sqlservr.exe
    Timestamp: Sun Mar 29 17:29:20 2009 (49CFF640)
    CheckSum: 029079F8
    ImageSize: 02970000
    File version: 2007.100.2531.0
    Product version: 10.0.2531.0
    File flags: 0 (Mask 3F)
    File OS: 40000 NT Base
    File type: 1.0 App
    File date: 00000000.00000000
    Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4
    0:000> .ecxr
    eax=693ad01c ebx=00000000 ecx=03550ab4 edx=03550ab4 esi=00000000 edi=0357b640
    eip=77e4bee7 esp=693ad018 ebp=693ad06c iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
    kernel32!RaiseException+0×53:
    77e4bee7 5e pop esi
    0:000> kC 1000

    kernel32!RaiseException
    WARNING: Stack unwind information not available. Following frames may be wrong.
    sqlservr
    sqlservr
    sqlservr
    sqlservr
    sqlservr
    sqlservr
    msvcr80!_CallSETranslator
    msvcr80!FindHandlerForForeignException
    msvcr80!FindHandler
    msvcr80!__InternalCxxFrameHandler
    msvcr80!__CxxFrameHandler3
    ntdll!ExecuteHandler2
    ntdll!ExecuteHandler
    ntdll!KiUserExceptionDispatcher
    sqlservr

    another one….
    eax=4cb7a000 ebx=07dde040 ecx=005c0033 edx=00000000 esi=4cb7b020 edi=07dde0a4
    eip=015ab31c esp=69dcf534 ebp=69dcf55c iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
    sqlservr!XVB::`scalar deleting destructor’+0×21:
    015ab31c 8b4204 mov eax,dword ptr [edx+4] ds:0023:00000004=????????
    0:000> kC 1000

    sqlservr!XVB::`scalar deleting destructor’
    sqlservr!ReadOnlyXactImp::~ReadOnlyXactImp
    sqlservr!ReadOnlyXactImp::`vector deleting destructor’
    sqlservr!AutoReadOnlyXact::Cleanup

  12. Dee said

    1st one from in January…right after upgrated to 2008 sp1 32bit from 2005. on same server
    someone thought this was caused by 3rd party wonderware.. cant tell if new dumps are related….

    0:000> .ecxr
    eax=00b1dffc ebx=00000000 ecx=03550ab4 edx=03550ab4 esi=00000000 edi=0357b640
    eip=77e4bee7 esp=00b1dff8 ebp=00b1e04c iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
    kernel32!RaiseException+0×53:
    77e4bee7 5e pop esi
    0:000> kC 1000

    kernel32!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_terminator
    msvcr80!_CallSETranslator
    msvcr80!FindHandlerForForeignException
    msvcr80!FindHandler
    msvcr80!__InternalCxxFrameHandler
    msvcr80!__CxxFrameHandler3
    ntdll!ExecuteHandler2
    ntdll!ExecuteHandler
    ntdll!KiUserExceptionDispatcher
    sqlservr!CVariableInfo::PbGetBlock

  13. Nitin said

    Hi karthick

    I need your help to analyze the sql server dump thrown indicating ‘Exception_Access_Violation’. I have tried to use Windebug to analyze the dump.
    But it is giving error unable to load symbol.
    Could you please help me to analyze the root caause for this dump.

  14. [...] 3. If there is no antivirus or memory errors follow the steps in http://mssqlwiki.com/2012/10/16/sql-server-exception_access_violation-and-sql-server-assertion/ [...]

  15. Farhan said

    Hi Karthick,

    Very nice article – thanks for sharing.

    Could you please have a look on following dump file contents and advise what actually caused our SQL server to crash.
    In log folder I can see two dump files, one is related to todays crash and second one is 2 months old.

    Running kC 1000 on dumps both generates identical result.

    Version Information (select @@version):
    Microsoft SQL Server 2008 R2 (SP1) – 10.50.2500.0 (X64) Jun 17 2011 00:54:03
    Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1)

    0:000> lmvm sqlservr
    start end module name
    00000000`00ed0000 00000000`04aad000 sqlservr T (pdb symbols) d:\debug_symbols\sqlservr.pdb\13165A9929E847D997E11887F2F963AE2\sqlservr.pdb
    Loaded symbol image file: sqlservr.exe
    Image path: C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
    Image name: sqlservr.exe
    Timestamp: Sat Jun 18 00:18:09 2011 (4DFB6221)
    CheckSum: 03B3E8A1
    ImageSize: 03BDD000
    File version: 2009.100.2500.0
    Product version: 10.50.2500.0
    File flags: 0 (Mask 3F)
    File OS: 40000 NT Base
    File type: 1.0 App
    File date: 00000000.00000000
    Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

    0:000> .ecxr
    rax=00000000ee9521f4 rbx=000000000449ec90 rcx=0000000027dcc910
    rdx=0000000000000000 rsi=000000000407ccbc rdi=0000000000000440
    rip=000007fefdf1cacd rsp=0000000027dccf20 rbp=0000000027dcfc80
    r8=0000000000000000 r9=0000000000000000 r10=00000000c000007c
    r11=00000000017b92d0 r12=0000000000004a40 r13=0000000000000000
    r14=0000000000000000 r15=0000000000004a40
    iopl=0 nv up ei pl nz na pe nc
    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202
    KERNELBASE!RaiseException+0×39:
    000007fe`fdf1cacd 4881c4c8000000 add rsp,0C8h

    0:000> kC 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_handle_except
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    msxml6!Base::freeRentalObjects
    msxml6!Base::StackExitNormal
    msxml6!ModelInit::~ModelInit
    msxml6!SAXReader::parse
    MSO
    0×0
    0×0
    0×0

    0:000> lmvm msxml6
    start end module name
    000007fe`f6af0000 000007fe`f6ce2000 msxml6 (private pdb symbols) d:\debug_symbols\msxml6.pdb\8EF4A994F018436F8083CD5C89BF8EB72\msxml6.pdb
    Loaded symbol image file: msxml6.dll
    Mapped memory image file: D:\Debug_Symbols\msxml6.dll\4CE7C8411f2000\msxml6.dll
    Image path: C:\Windows\System32\msxml6.dll
    Image name: msxml6.dll
    Timestamp: Sun Nov 21 00:08:17 2010 (4CE7C841)
    CheckSum: 001F86CD
    ImageSize: 001F2000
    File version: 6.30.7601.17514
    Product version: 6.30.7601.17514
    File flags: 0 (Mask 3F)
    File OS: 40004 NT Win32
    File type: 2.0 Dll
    File date: 00000000.00000000
    Translations: 0000.04b0
    CompanyName: Microsoft Corporation
    ProductName: Microsoft(R) MSXML 6.0 SP3
    InternalName: MSXML6.dll
    OriginalFilename: MSXML6.dll
    ProductVersion: 6.30.7601.17514
    FileVersion: 6.30.7601.17514
    FileDescription: MSXML 6.0 SP3
    LegalCopyright: Copyright (C) Microsoft Corporation. 1981-2008

    0:000> lmvm mso
    start end module name
    000007fe`df6d0000 000007fe`e0f02000 MSO T (no symbols)
    Loaded symbol image file: MSO.DLL
    Image path: C:\Program Files\Common Files\Microsoft Shared\OFFICE14\MSO.DLL
    Image name: MSO.DLL
    Timestamp: Wed Mar 24 05:22:16 2010 (4BA906D8)
    CheckSum: 01832834
    ImageSize: 01832000
    File version: 14.0.4760.1000
    Product version: 14.0.4760.0
    File flags: 0 (Mask 3F)
    File OS: 40004 NT Win32
    File type: 2.0 Dll
    File date: 00000000.00000000
    Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

    Thanks
    Farhan

  16. [...] SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion [...]

  17. Roi said

    Hi Karthick,

    Great article, thanks!

    I will appreciate help with the following dump:

    =====================================================================
    BugCheck Dump
    =====================================================================

    This file is generated by Microsoft SQL Server
    version 9.00.3077.00
    upon detection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is AT/AT COMPATIBLE.
    Current time is 11:22:15 04/09/13.
    16 Unknown CPU 9., 2400 Mhz processor (s).
    Windows NT 5.2 Build 3790 CSD Service Pack 2.

    Memory
    MemoryLoad = 98%
    Total Physical = 32767 MB
    Available Physical = 582 MB
    Total Page File = 80823 MB
    Available Page File = 48552 MB
    Total Virtual = 8388607 MB
    Available Virtual = 8354540 MB
    ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0018.txt
    SqlDumpExceptionHandler: Process 476 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is
    terminating this process.
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 04/09/13 11:22:15 spid 476
    *
    *
    * Exception Address = 000000007814E3A4 Module(MSVCR80+000000000001E3A4)
    * Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
    * Access Violation occurred reading address 0000000000000014
    * Input Buffer 510 bytes –
    *

    Here comes a query which runs every day for several years now – executed by a job in a loop through working hours.

    after this error occured SQL Server rejected all new connections (except for local login).

    Call Site
    msvcr80!memcpy
    sqlservr!CSession::CopyDeepMember
    sqlservr!CSession::GetUserSid
    sqlservr!CSTVFSysSess::GetRow
    sqlservr!CQScanTVFStreamNew::GetRow
    sqlservr!CQScanSortNew::BuildSortTable
    sqlservr!CQScanSortNew::OpenHelper
    sqlservr!CQScanMergeJoinNew::Open
    sqlservr!CQScanNLJoinNew::Open
    sqlservr!CQScanNLJoinNew::Open
    sqlservr!CQScanMergeJoinNew::Open
    sqlservr!CQScanNLJoinNew::Open
    sqlservr!CQScanNew::OpenHelper
    sqlservr!CQScanTopNew::Open
    sqlservr!CQScanNew::OpenHelper
    sqlservr!CQScanSpoolNew::LoadSpool
    sqlservr!_chkstk
    sqlservr!CQScanNew::OpenHelper
    sqlservr!CQScanUpdateNew::Open
    sqlservr!CQueryScan::Startup
    sqlservr!CXStmtQuery::SetupQueryScanAndExpression
    sqlservr!CXStmtQuery::ErsqExecuteQuery
    sqlservr!CXStmtDML::XretDMLExecute
    sqlservr!CXStmtDML::XretExecute
    sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!_chkstk
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExec::XretExecute
    sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!_chkstk
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadStart

    I would like to understand what really happend there,
    and to prevent it from happening again.

    Thanks very much,
    Roi

    • Hello Roi,
      I think you are hitting a know issue mentioned in http://support.microsoft.com/kb/940281
      Update your SQLServer to Cumulative update package 3 for SQL Server 2005 Service Pack 2 or higher to resolve this.

      • Roi said

        Hi Karthick,

        Thanks for the very quick response!

        I believe that you are correct and that this is my issue,
        as i do have a second dump and error messages resembling
        the “…appears to be non-yielding on Scheduler…” as mentioned in the link you posted.

        My next step is to temporarily disable the Agent Job which is monitoring sys.dm_exec_sessions,
        and inform my manager of the issue.
        He will have to decide if he wants to apply the update,
        or wait for our scheduled upgrade to SQL Server 2012
        (which is due next month and will probably be delayed forever…)

        I would love to know how you found that this is the issue,
        and would appreciate it if you can direct me to some documentation
        which will help me solve such problems on my own in the future.

        Again, Thank you very much!!!
        Roi

      • Roi said

        Hello Karthick,

        You must have missed my previous reply,
        could you please direct me to some documentation?
        I would really love to learn in-depth internals.

        Thanks,
        Roi

  18. Drake Teran said

    Some assistance; i also received an EXCEPTION_ACCESS_VIOLATION with errors (Error: 17310, Severity: 20, State: 1) and (Error: 17311, Severity: 16, State: 1). I am curious if this is related to the known SQL bug identified http://support.microsoft.com/kb/972936 (many thanks in advance)

    SQLDump0110:
    0:000> kC 1000
    Call Site
    fs4odbc64
    0×0
    kernel32!HeapFree
    fs4odbc64

    SQLDump0111:

    0:000> kC 1000
    Call Site
    fs4odbc64
    fs4odbc64
    0×0
    ntdll!_C_specific_handler
    0×0
    0×0
    0×0
    0×0
    sspicli!_dyn_tls_init_callback (sspicli+0x217d4)
    0×0
    ntdll!RtlDispatchException
    0×0
    0×0
    0×0

    SQLDump0112:
    0:000> kc 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_handle_except
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    ntdll!RtlEnterCriticalSection
    fs4odbc64

    SQLDump0113:

    0:000> kC 1000
    Call Site
    fs4odbc64
    fs4odbc64
    0×0
    ntdll!_C_specific_handler
    0×0
    0×0
    0×0
    0×0
    sspicli!_dyn_tls_init_callback (sspicli+0x217d4)
    0×0
    ntdll!RtlDispatchException
    ntdll!RtlpAllocateUserBlock
    0×0
    0×0
    0×0

    SQLDump0114:

    0:000> kC 1000
    Call Site
    fs4odbc64
    fs4odbc64
    0×0
    ntdll!_C_specific_handler
    0×0
    0×0
    0×0
    0×0
    sspicli!_dyn_tls_init_callback (sspicli+0x217d4)
    0×0
    ntdll!RtlDispatchException
    0×0
    0×0
    0×0

    SQLDump0115:

    0:000> kC 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_handle_except
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    ntdll!RtlEnterCriticalSection
    fs4odbc64

    • Drake Teran said

      Current time is 01:34:21 04/15/13.
      =====================================================================
      BugCheck Dump
      =====================================================================

      This file is generated by Microsoft SQL Server
      version 10.50.2500.0
      upon detection of fatal unexpected error. Please return this file,
      the query or program that produced the bugcheck, the database and
      the error log, and any other pertinent information with a Service Request.

      Computer type is Intel(R) Xeon(R) CPU X5660 @ 2.80GHz.
      Bios Version is INTEL – 6040000
      PhoenixBIOS 4.0 Release 6.0
      4 X64 level 8664, 14 Mhz processor (s).
      Windows NT 6.1 Build 7601 CSD Service Pack 1.

      Memory
      MemoryLoad = 80%
      Total Physical = 12287 MB
      Available Physical = 2429 MB
      Total Page File = 24573 MB
      Available Page File = 14413 MB
      Total Virtual = 8388607 MB
      Available Virtual = 8375683 MB
      ***Stack Dump being sent to F:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\LOG\SQLDump0110.tx
      t
      SqlDumpExceptionHandler: Process 62 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is
      terminating this process.
      * *******************************************************************************
      *
      * BEGIN STACK DUMP:
      * 04/15/13 01:34:21 spid 62
      *
      *
      * Exception Address = 000007FEF1A06BA3 Module(fs4odbc64+0000000000006BA3)
      * Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION
      * Access Violation occurred writing address 0000000016243304
      * Input Buffer 110 bytes –
      * exec DataDispatch.dbo.dp_MasterDispatcher 3;

      • Drake Teran said

        *
        * MODULE BASE END SIZE
        * sqlservr 0000000000D00000 00000000048DCFFF 03bdd000
        * ntdll 0000000076F20000 00000000770C8FFF 001a9000
        * kernel32 0000000076E00000 0000000076F1EFFF 0011f000
        * KERNELBASE 000007FEFD0D0000 000007FEFD13BFFF 0006c000
        * ADVAPI32 000007FEFDFB0000 000007FEFE08AFFF 000db000
        * msvcrt 000007FEFEF90000 000007FEFF02EFFF 0009f000
        * sechost 000007FEFDF90000 000007FEFDFAEFFF 0001f000
        * RPCRT4 000007FEFE0D0000 000007FEFE1FCFFF 0012d000
        * MSVCR80 0000000074B70000 0000000074C38FFF 000c9000
        * MSVCP80 00000000747E0000 00000000748E8FFF 00109000
        * sqlos 0000000074740000 0000000074746FFF 00007000
        * Secur32 000007FEFCA20000 000007FEFCA2AFFF 0000b000
        * SSPICLI 000007FEFCCD0000 000007FEFCCF4FFF 00025000
        * pdh 000007FEF7660000 000007FEF76ADFFF 0004e000
        * SHLWAPI 000007FEFDA90000 000007FEFDB00FFF 00071000
        * GDI32 000007FEFDEA0000 000007FEFDF06FFF 00067000
        * USER32 0000000076D00000 0000000076DF9FFF 000fa000
        * LPK 000007FEFDC20000 000007FEFDC2DFFF 0000e000
        * USP10 000007FEFD9C0000 000007FEFDA88FFF 000c9000
        * USERENV 000007FEFC210000 000007FEFC22DFFF 0001e000
        * profapi 000007FEFCE30000 000007FEFCE3EFFF 0000f000
        * WINMM 000007FEF7620000 000007FEF765AFFF 0003b000
        * IPHLPAPI 000007FEFB2F0000 000007FEFB316FFF 00027000
        * NSI 000007FEFE0C0000 000007FEFE0C7FFF 00008000
        * WINNSI 000007FEFB260000 000007FEFB26AFFF 0000b000
        * opends60 0000000074720000 0000000074727FFF 00008000
        * NETAPI32 000007FEFAD70000 000007FEFAD85FFF 00016000
        * netutils 000007FEFC830000 000007FEFC83BFFF 0000c000
        * srvcli 000007FEFC910000 000007FEFC932FFF 00023000
        * wkscli 000007FEFAD10000 000007FEFAD24FFF 00015000
        * LOGONCLI 000007FEFC4C0000 000007FEFC4EFFFF 00030000
        * SAMCLI 000007FEFA390000 000007FEFA3A3FFF 00014000
        * BatchParser 00000000746D0000 00000000746FCFFF 0002d000
        * IMM32 000007FEFE090000 000007FEFE0BDFFF 0002e000
        * MSCTF 000007FEFDB10000 000007FEFDC18FFF 00109000
        * psapi 00000000770E0000 00000000770E6FFF 00007000
        * instapi10 0000000074730000 000000007473CFFF 0000d000
        * cscapi 000007FEFA740000 000007FEFA74EFFF 0000f000
        * sqlevn70 00000000744C0000 00000000746C2FFF 00203000
        * CRYPTSP 000007FEFC8E0000 000007FEFC8F6FFF 00017000
        * rsaenh 000007FEFC3D0000 000007FEFC416FFF 00047000
        * CRYPTBASE 000007FEFCD60000 000007FEFCD6EFFF 0000f000
        * BROWCLI 000007FEF6FF0000 000007FEF7001FFF 00012000
        * AUTHZ 000007FEFC8B0000 000007FEFC8DEFFF 0002f000
        * MSCOREE 000007FEFA230000 000007FEFA29EFFF 0006f000
        * ole32 000007FEFD470000 000007FEFD672FFF 00203000
        * credssp 000007FEFC2F0000 000007FEFC2F9FFF 0000a000
        * msv1_0 000007FEFC600000 000007FEFC650FFF 00051000
        * cryptdll 000007FEFCA30000 000007FEFCA43FFF 00014000
        * kerberos 000007FEFC6C0000 000007FEFC773FFF 000b4000
        * MSASN1 000007FEFCF10000 000007FEFCF1EFFF 0000f000
        * schannel 000007FEFC460000 000007FEFC4B7FFF 00058000
        * CRYPT32 000007FEFCF60000 000007FEFD0C9FFF 0016a000
        * security 00000000744B0000 00000000744B2FFF 00003000
        * WS2_32 000007FEFDDB0000 000007FEFDDFCFFF 0004d000
        * SHELL32 000007FEFE200000 000007FEFEF87FFF 00d88000
        * OLEAUT32 000007FEFD8E0000 000007FEFD9B6FFF 000d7000
        * ftimport 0000000060000000 0000000060024FFF 00025000
        * MSFTE 0000000049980000 0000000049D2DFFF 003ae000
        * VERSION 000007FEFBFB0000 000007FEFBFBBFFF 0000c000
        * dbghelp 0000000073110000 000000007326DFFF 0015e000
        * WINTRUST 000007FEFCF20000 000007FEFCF59FFF 0003a000
        * ncrypt 000007FEFC7B0000 000007FEFC7FDFFF 0004e000
        * bcrypt 000007FEFC880000 000007FEFC8A1FFF 00022000
        * mswsock 000007FEFC660000 000007FEFC6B4FFF 00055000
        * wship6 000007FEFC820000 000007FEFC826FFF 00007000
        * wshtcpip 000007FEFC080000 000007FEFC086FFF 00007000
        * ntmarta 000007FEFBF80000 000007FEFBFACFFF 0002d000
        * WLDAP32 000007FEFD370000 000007FEFD3C1FFF 00052000
        * ntdsapi 000007FEFA7A0000 000007FEFA7C6FFF 00027000
        * DNSAPI 000007FEFC4F0000 000007FEFC54AFFF 0005b000
        * rasadhlp 000007FEFA7D0000 000007FEFA7D7FFF 00008000
        * fwpuclnt 000007FEFB1B0000 000007FEFB202FFF 00053000
        * bcryptprimitives 000007FEFC310000 000007FEFC35BFFF 0004c000
        * SAMLIB 000007FEFA370000 000007FEFA38CFFF 0001d000
        * CLBCatQ 000007FEFDE00000 000007FEFDE98FFF 00099000
        * sqlncli10 00000000725A0000 000000007285BFFF 002bc000
        * COMCTL32 000007FEFA2A0000 000007FEFA33FFFF 000a0000
        * COMDLG32 000007FEFD3D0000 000007FEFD466FFF 00097000
        * SQLNCLIR10 0000000072190000 00000000721C6FFF 00037000
        * netbios 000007FEF4AB0000 000007FEF4AB9FFF 0000a000
        * xpsqlbot 0000000072080000 0000000072087FFF 00008000
        * xpstar 0000000071FF0000 0000000072077FFF 00088000
        * SQLSCM 0000000072430000 000000007243DFFF 0000e000
        * ODBC32 000007FEF4D10000 000007FEF4DC0FFF 000b1000
        * ATL80 0000000074B20000 0000000074B3FFFF 00020000
        * odbcint 00000000722E0000 0000000072317FFF 00038000
        * clusapi 000007FEF6170000 000007FEF61BFFFF 00050000
        * resutils 000007FEF6150000 000007FEF6168FFF 00019000
        * xpstar 0000000071FC0000 0000000071FE4FFF 00025000
        * xplog70 0000000071FB0000 0000000071FBFFFF 00010000
        * xplog70 0000000071FA0000 0000000071FA1FFF 00002000
        * dsrole 000007FEFBBD0000 000007FEFBBDBFFF 0000c000
        * oledb32 000007FEF1A50000 000007FEF1B5DFFF 0010e000
        * MSDART 000007FEF1EA0000 000007FEF1EC7FFF 00028000
        * Comctl32 000007FEFB7F0000 000007FEFB9E3FFF 001f4000
        * OLEDB32R 0000000066C80000 0000000066C93FFF 00014000
        * comsvcs 000007FEF45F0000 000007FEF479FFFF 001b0000
        * msdasql 000007FEF0E10000 000007FEF0EC5FFF 000b6000
        * MSDATL3 000007FEF1E80000 000007FEF1E9FFFF 00020000
        * MSDASQLR 0000000066C70000 0000000066C7EFFF 0000f000
        * RpcRtRemote 000007FEFCE10000 000007FEFCE23FFF 00014000
        * fs4odbc64 000007FEF1A00000 000007FEF1A4CFFF 0004d000
        * LIBEAY32 000000000FBC0000 000000000FD36FFF 00177000
        * WSOCK32 000007FEF1E70000 000007FEF1E78FFF 00009000
        * SSLEAY32 0000000005020000 0000000005075FFF 00056000
        * odbccp32 000007FEF17A0000 000007FEF17C7FFF 00028000
        * instapi 0000000048060000 000000004806CFFF 0000d000
        * dbghelp 00000000715A0000 00000000716FDFFF 0015e000
        *
        * P1Home: 00000000297B7AD0: 00000000C0000005 0000000000000000 000007FEF1A06BA3 0000000000000002 00000000
        00000001 0000000016243304
        * P2Home: 00000000297B75E0: 00000000297B7AD0 00000000297B75E0 0000000000000000 0000000000000000 00000000
        297B79A8 007400780045005C
        * P3Home: 0000000000000000:
        * P4Home: 0000000000000000:
        * P5Home: 00000000297B79A8: 0000000000010000 000000000000002C 00000000297B79D0 00000000297B79E0 00000078
        C12C9653 00000000297B7BE0
        * P6Home: 007400780045005C:
        * ContextFlags: 000000000010001F:
        * MxCsr: 0000000000001F80:
        * SegCs: 0000000000000033:
        * SegDs: 000000000000002B:
        * SegEs: 000000000000002B:
        * SegFs: 0000000000000053:
        * SegGs: 000000000000002B:
        * SegSs: 000000000000002B:
        * EFlags: 0000000000010246: 0000000000000000 0000000101580000 0000000102680000 0000000102780000 00000000
        00000000 0000001000000000
        * Rax: 0000000000000000:
        * Rcx: 0000000016243070:
        * Rdx: 0000000000000000:
        * Rbx: 0000000000376B20: 0000000000000002 0000000000776CC0 0000000000000000 0000000000000000 00000000
        00000000 0000000000000000
        * Rsp: 00000000297B7BA0: 0000000016232E30 0000000076E2301A 0000000000376B20 0000000000376B20 00000000
        00000000 000007FEF1A043D5
        * Rbp: 0000000000000001:
        * Rsi: 000000000038D650: 000000000F4FC570 0000000000376B20 0000000000000000 00000000162E12F0 00000000
        00376B20 00000000162E12F0
        * Rdi: 000000000037AC20: 0000000000000003 0000000000776CC0 0000000000000000 0000000000000000 00000000
        00000000 0000000000000000
        * R8: 00000000297B7A98: 0000000000000000 0000000000000000 0000000000000000 000005C0FFFFFB30 000004D0
        FFFFFB30 00000000000000F0
        * R9: 0000000000008000:
        * R10: 0000000000000000:
        * R11: 0000000000000286:
        * R12: 0000000000000010:
        * R13: 0000000000000000:
        * R14: 0000000000000000:
        * R15: 0000000000000000:
        * Rip: 000007FEF1A06BA3: 3300000294818966 8B480000014DE9C0 029480B70F702444 077E782444390000 33000001
        34E9C033 B70F5024448966C0
        * *******************************************************************************
        * ——————————————————————————-
        * Short Stack Dump
        000007FEF1A06BA3 Module(fs4odbc64+0000000000006BA3)
        000007FEF1A15867 Module(fs4odbc64+0000000000015867)
        000007FEF4D13BA9 Module(ODBC32+0000000000003BA9)
        000007FEF4D7BA8E Module(ODBC32+000000000006BA8E)
        000007FEF0E49D53 Module(msdasql+0000000000039D53)
        000007FEF0E34679 Module(msdasql+0000000000024679)
        000007FEF0E34584 Module(msdasql+0000000000024584)
        000007FEF0E354A4 Module(msdasql+00000000000254A4)
        000007FEF1A5647E Module(oledb32+000000000000647E)
        000007FEF1A563D2 Module(oledb32+00000000000063D2)
        000007FEF1A55E56 Module(oledb32+0000000000005E56)
        00000000017804E0 Module(sqlservr+0000000000A804E0)
        0000000001873015 Module(sqlservr+0000000000B73015)
        0000000001871999 Module(sqlservr+0000000000B71999)
        0000000001871410 Module(sqlservr+0000000000B71410)
        000000000186DCE8 Module(sqlservr+0000000000B6DCE8)
        000000000134160E Module(sqlservr+000000000064160E)
        0000000000F70E36 Module(sqlservr+0000000000270E36)
        0000000000F755B7 Module(sqlservr+00000000002755B7)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        0000000000FB7183 Module(sqlservr+00000000002B7183)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        0000000000FC6720 Module(sqlservr+00000000002C6720)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        000000000104C380 Module(sqlservr+000000000034C380)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        0000000000FB7183 Module(sqlservr+00000000002B7183)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        00000000012DA59A Module(sqlservr+00000000005DA59A)
        000000000123A181 Module(sqlservr+000000000053A181)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        00000000012386F7 Module(sqlservr+00000000005386F7)
        0000000000F6BA93 Module(sqlservr+000000000026BA93)
        0000000000F6B958 Module(sqlservr+000000000026B958)
        0000000000F6BB0E Module(sqlservr+000000000026BB0E)
        0000000000ED37A5 Module(sqlservr+00000000001D37A5)
        0000000000E96644 Module(sqlservr+0000000000196644)
        0000000000E97259 Module(sqlservr+0000000000197259)
        0000000000E970FC Module(sqlservr+00000000001970FC)
        00000000021DC97F Module(sqlservr+00000000014DC97F)
        00000000021FF7E8 Module(sqlservr+00000000014FF7E8)
        00000000021FB741 Module(sqlservr+00000000014FB741)
        0000000000D69A86 Module(sqlservr+0000000000069A86)
        0000000000D6B4AF Module(sqlservr+000000000006B4AF)
        0000000000D6AD6C Module(sqlservr+000000000006AD6C)
        00000000021FDA1C Module(sqlservr+00000000014FDA1C)
        00000000021FD37A Module(sqlservr+00000000014FD37A)
        00000000021FB36D Module(sqlservr+00000000014FB36D)
        0000000000D69A86 Module(sqlservr+0000000000069A86)
        0000000000D6B4AF Module(sqlservr+000000000006B4AF)
        0000000000D6AD6C Module(sqlservr+000000000006AD6C)
        00000000021FDA1C Module(sqlservr+00000000014FDA1C)
        00000000021FD37A Module(sqlservr+00000000014FD37A)
        00000000021FB36D Module(sqlservr+00000000014FB36D)
        0000000000D69A86 Module(sqlservr+0000000000069A86)
        0000000000D6B4AF Module(sqlservr+000000000006B4AF)
        0000000000D6AD6C Module(sqlservr+000000000006AD6C)
        00000000021FDA1C Module(sqlservr+00000000014FDA1C)
        00000000021FD37A Module(sqlservr+00000000014FD37A)
        00000000021FB36D Module(sqlservr+00000000014FB36D)
        0000000000D69A86 Module(sqlservr+0000000000069A86)
        0000000000D6B4AF Module(sqlservr+000000000006B4AF)
        0000000000D6AD6C Module(sqlservr+000000000006AD6C)
        0000000000D8CFA6 Module(sqlservr+000000000008CFA6)
        0000000000DB65E2 Module(sqlservr+00000000000B65E2)
        0000000000D0B450 Module(sqlservr+000000000000B450)
        0000000000D0B116 Module(sqlservr+000000000000B116)
        0000000000D0AF5B Module(sqlservr+000000000000AF5B)
        0000000000E444FA Module(sqlservr+00000000001444FA)
        0000000000E447DD Module(sqlservr+00000000001447DD)
        000000000128C0CD Module(sqlservr+000000000058C0CD)
        0000000000E453D2 Module(sqlservr+00000000001453D2)
        0000000074B737D7 Module(MSVCR80+00000000000037D7)
        0000000074B73894 Module(MSVCR80+0000000000003894)
        0000000076E1652D Module(kernel32+000000000001652D)
        0000000076F4C521 Module(ntdll+000000000002C521)

        CSession @0x00000001EF71A3B0
        —————————-
        m_spid = 62 m_cRef = 14 m_rgcRefType[0] = 1
        m_rgcRefType[1] = 1 m_rgcRefType[2] = 11 m_rgcRefType[3] = 1
        m_rgcRefType[4] = 0 m_rgcRefType[5] = 0 m_pmo = 0x00000001EF71A080
        m_pstackBhfPool = 0x00000001EEBA7120 m_dwLoginFlags = 0x83e0 m_fBackground = 0
        m_eConnResetOption = 0 m_fUserProc = -1 m_fConnReset = 0
        m_fIsConnReset = 0 m_fInLogin = 0 m_fAuditLoginSent = 1
        m_fAuditLoginFailedSent = 0 m_fReplRelease = 0 m_fKill = 0
        m_ulLoginStamp = 5045 m_eclClient = 6 m_protType = 6
        m_hHttpToken = FFFFFFFFFFFFFFFF

        m_pV7LoginRec
        ———————
        0000000000000000: 9a010000 03000b73 00100000 00000007 e0070000 †š……s……..à…

        0000000000000014: 00000000 e0830000 e0010000 09040000 5e000900 †….àƒ..à… …^. .

        0000000000000028: 00000000 00000000 70004900 02010000 00000000 †……..p.I………

        000000000000003C: 02010400 0a010000 0a010c00 0050568d 6eec2201 †………….PV.nì”.

        0000000000000050: 78002201 00000000 00000000 0000†††††††††††††††x.”………..

        CPhysicalConnection @0x00000001EF71A270
        —————————————
        m_pPhyConn->m_pmo = 0x00000001EF71A080
        m_pPhyConn->m_pNetConn = 0x00000001EF71AAD0
        m_pPhyConn->m_pConnList = 0x00000001EF71A380
        m_pPhyConn->m_pSess = 0x00000001EF71A3B0 m_pPhyConn->m_fTracked = -1
        m_pPhyConn->m_cbPacketsize = 4096 m_pPhyConn->m_fMars = 0 m_pPhyConn->m_fKill = 0

        CBatch @0x00000001EF71AFA0
        ————————–
        m_pSess = 0x00000001EF71A3B0 m_pConn = 0x00000001EF71AEA0 m_cRef = 3
        m_rgcRefType[0] = 1 m_rgcRefType[1] = 1 m_rgcRefType[2] = 1
        m_rgcRefType[3] = 0 m_rgcRefType[4] = 0 m_pTask = 0x000000000043F708

        EXCEPT (null) @0x00000000297B4938
        ———————————
        exc_number = 0 exc_severity = 0 exc_func = 0x000000000320F2A4

        Task @0x000000000043F708
        ————————
        CPU Ticks used (ms) = 28929974 Task State = 2
        WAITINFO_INTERNAL: WaitResource = 0×0000000000000000 WAITINFO_INTERNAL: WaitType = 0×0
        WAITINFO_INTERNAL: WaitSpinlock = 0×0000000000000000
        WAITINFO_EXTERNAL: WaitResource = 0×0000000000000000
        WAITINFO_EXTERNAL: WaitType = 0x200001D9
        WAITINFO_EXTERNAL: WaitSpinlock = 0×0000000000000000 SchedulerId = 0×0
        ThreadId = 0xfe0 m_state = 0 m_eAbortSev = 0

        EC @0x00000001EF71BEC0
        ———————-
        spid = 0 ecid = 0 ec_stat = 0×0
        ec_stat2 = 0×0 ec_atomic = 0×0 __fSubProc = 1
        __pSETLS = 0x00000001EF71AF10 __pSEParams = 0x00000001EF71B350

        SEInternalTLS @0x00000001EF71AF10
        ———————————
        m_flags = 0 m_TLSstatus = 3 m_owningTask = 0x000000000043F708
        m_activeHeapDatasetList = 0x00000001EF71AF10
        m_activeIndexDatasetList = 0x00000001EF71AF20 m_pDbccContext = 0×0000000000000000

        SEParams @0x00000001EF71B350
        —————————-
        m_lockTimeout = -1 m_isoLevel = 4096 m_logDontReplicate = 0
        m_neverReplicate = 0 m_XactWorkspace = 0x00000001EB7BF090
        m_pSessionLocks = 0x00000001EEBA6A70 m_pDbLocks = 0x00000001EF71B870
        m_execStats = 0x0000000084E82240 m_pAllocFileLimit = 0×0000000000000000

  19. Hiya! I simply wish to give an enormous thumbs up for the nice
    information you may have right here on this post.
    I can be coming again to your blog for extra soon.

  20. Neret said

    Hi Karthick !

    Coul you help me with this, please???

    This is the stack of thread which caused exception:

    0:000> kC 1000
    Call Site
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_handle_except
    kernel32!UnhandledExceptionFilter
    ntdll! ?? ::FNODOBFM::`string’
    ntdll!_C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    msxml6!Base::freeRentalObjects
    msxml6!Base::StackExitNormal
    msxml6!ModelInit::~ModelInit
    msxml6!SAXReader::parse
    MSO!Ordinal398
    MSO!Ordinal9364
    MSO!Ordinal9364
    MSO!Ordinal9364
    MSO!Ordinal9694
    MSO!Ordinal4349
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    Thanks!

  21. Marek_Warszawa said

    Dear Karthick,
    I have done the steps described in your article and got the below stack trace.
    Could you please help solve the problem?

    0:000> kC 1000
    Call Site
    0×0
    0×0
    sqlservr!CConstraintProp::FAndNonItvlConst
    sqlservr!CConstraintProp::FAndNonItvlConstraint
    sqlservr!CConstraintProp::AndNewConstraint
    sqlservr!CConstraintProp::PcnstrDeriveSelect
    sqlservr!CLogOp_Select::PcnstrDerive
    sqlservr!CLogOpArg::PcnstrDeriveHandler
    sqlservr!CLogOpArg::DeriveGroupProperties
    sqlservr!COpArg::DeriveNormalizedGroupProperties
    sqlservr!COptExpr::DeriveGroupProperties
    sqlservr!COptExpr::DeriveGroupProperties
    sqlservr!COptExpr::DeriveGroupProperties
    sqlservr!COptExpr::DeriveGroupProperties
    sqlservr!COptExpr::DeriveGroupProperties
    sqlservr!CQuery::PqoBuild
    sqlservr!CStmtQuery::InitQuery
    sqlservr!alloca_probe
    sqlservr!CStmtCursorSelect::Init
    sqlservr!CCompPlan::FCompileStep
    sqlservr!CSQLSource::FCompile
    sqlservr!CSQLSource::FCompWrapper
    sqlservr!CSQLSource::Transform
    sqlservr!CSQLSource::Execute
    sqlservr!ExecuteSql
    sqlservr!SPCursor::CursorExecPrepExecOpen
    sqlservr!SPCursor::Open
    sqlservr!CSpecProc::ExecuteSpecial
    sqlservr!CSpecProc::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    0×0
    0×0

    Thanks in advance,
    Marek

    • Marek,
      What is the version of SQL of your SQLServer (select @@version). I think you are in RTM version of SQLServer2005. I remember this is fixed in 9.0.1500. Please upgrade to latest service pack for SQLServer 2005

      • Marek_Warszawa said

        Hi,
        This is information from SQL Server log files:

        Microsoft SQL Server 2008 (SP3) – 10.0.5512.0 (X64)
        Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1) (VM)

        Computer type is AMD Opteron(TM) Processor 6274 .
        Bios Version is INTEL – 6040000
        PhoenixBIOS 4.0 Release 6.0
        2 X64 level 8664, 14 Mhz processor (s).
        Windows NT 6.1 Build 7601 CSD Service Pack 1.

        Memory:
        MemoryLoad = 96%
        Total Physical = 4095 MB
        Available Physical = 129 MB
        Total Page File = 8189 MB
        Available Page File = 2214 MB
        Total Virtual = 8388607 MB
        Available Virtual = 8380146 MB

        What could be the reason in this case?

        BR,
        Marek

        • Two frames are corrupted so it’s difficult to crack
          Call Site
          0×0 ====>corrupted
          0×0 0×0 ====>corrupted
          sqlservr!CConstraintProp::FAndNonItvlConst
          sqlservr!CConstraintProp::FAndNonItvlConstraint

          Do you pass so many values in the IN clause of the query which causing this access violation? (You will find the input buffer in errorlog along with access violation message)
          Can you reproduce this issue consistently?

  22. Michiel said

    Hello Karthick,

    Great blog!
    Can you please help me with this dump?

    Our sql-server did not respond to request from other servers anymore.
    After restarting MSSQLSERVER service it all worked again.

    ***************
    Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
    Copyright (c) Microsoft Corporation. All rights reserved.

    Loading Dump File [D:\Temp\SQLDump0005.mdmp]
    User Mini Dump File: Only registers, stack and portions of memory are available

    Comment: ‘Program fault handler’
    Comment: ’05/13/13 17:17:42 spid 0 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION writing address 0000000000C4DA54 at 0×0000000000649889

    Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
    Executable search path is:
    Windows 7 Version 7600 MP (2 procs) Free x64
    Product: Server, suite: Enterprise TerminalServer SingleUserTS
    Machine Name:
    Debug session time: Mon May 13 17:17:56.000 2013 (GMT+2)
    System Uptime: 66 days 18:15:01.669
    Process Uptime: 66 days 18:11:58.000
    ……………………………………………………….
    ………………………….
    Loading unloaded module list
    ……………………………………………………………
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (4f4.1fe8): Access violation – code c0000005 (first/second chance not available)
    ntdll!ZwWaitForSingleObject+0xa:
    00000000`7780f6fa c3 ret

    ***************

    0:000> .ecxr
    rax=00000000002afde0 rbx=00000004a1980528 rcx=00000000008122c0
    rdx=0000000080018001 rsi=00000003609686a0 rdi=00000004a19804e0
    rip=0000000000649889 rsp=000000001a1bde20 rbp=00000004a19803a0
    r8=0000000000649960 r9=0000000000000254 r10=000007fffff70000
    r11=000000001a1bde06 r12=0000000000000006 r13=000000001a1bdfa0
    r14=0000000000000000 r15=000000028f156fa0
    iopl=0 nv up ei pl nz na pe nc
    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
    Unable to load image D:\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe, Win32 error 0n2
    *** WARNING: Unable to verify timestamp for sqlservr.exe
    sqlservr!CCriticalSectionSOS::Initialize+0×71:
    00000000`00649889 488908 mov qword ptr [rax],rcx ds:00000000`002afde0=????????????????

    ***************

    0:000> lmvm sqlservr
    start end module name
    00000000`00260000 00000000`03e10000 sqlservr T (pdb symbols) c:\symbols\sqlservr.pdb\E0E45C4EB0C64908A90279B30F42FACA2\sqlservr.pdb
    Loaded symbol image file: sqlservr.exe
    Image path: D:\MSSQL10_50.MSSQLSERVER\MSSQL\Binn\sqlservr.exe
    Image name: sqlservr.exe
    Timestamp: Sat Apr 23 14:44:29 2011 (4DB2C9AD)
    CheckSum: 03B10238
    ImageSize: 03BB0000
    File version: 2009.100.1617.0
    Product version: 10.50.1617.0
    File flags: 0 (Mask 3F)
    File OS: 40000 NT Base
    File type: 1.0 App
    File date: 00000000.00000000
    Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

    ***************

    0:000> .ecxr
    rax=00000000002afde0 rbx=00000004a1980528 rcx=00000000008122c0
    rdx=0000000080018001 rsi=00000003609686a0 rdi=00000004a19804e0
    rip=0000000000649889 rsp=000000001a1bde20 rbp=00000004a19803a0
    r8=0000000000649960 r9=0000000000000254 r10=000007fffff70000
    r11=000000001a1bde06 r12=0000000000000006 r13=000000001a1bdfa0
    r14=0000000000000000 r15=000000028f156fa0
    iopl=0 nv up ei pl nz na pe nc
    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202
    sqlservr!CCriticalSectionSOS::Initialize+0×71:
    00000000`00649889 488908 mov qword ptr [rax],rcx ds:00000000`002afde0=????????????????

    ***************

    0:000> kc 1000
    Call Site
    sqlservr!CCriticalSectionSOS::Initialize
    sqlservr!CryptoBase::FInit
    sqlservr!SNIAddProvider
    sqlservr!TDSSNIClient::FAddSSLProvider
    sqlservr!process_prelogin
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    Thanks for your help!

    Michiel.

    • Hello Michiel,
      This issue has been fixed in CU6 for SQL Server 2008 R2. Please apply the patch

      • Michiel said

        Hi Karthick,

        Thanks for the fast reply!
        A colleague of mine just came up with the same solution :)

        If the problem returns I will let you know but I think that won’t be necessary

        Thanks again.

        Michiel.

  23. Jose P said

    does anyone can help find what wrong here?
    0:000> kc 1000
    Call Site
    sqldk!CMemObj::Free
    sqlmin!commondelete
    sqlmin!IndexAnalysis::ExecuteCleanupAction
    sqlmin!IndexAnalysis::GetNextRow
    sqlmin!CIndexAnalysisSTVF::InternalGetRow
    sqlmin!CQScanTVFStreamNew::GetRow
    sqlmin!CQScanFilterNew::GetRowHelper
    sqlmin!CQScanNLJoinNew::GetRowHelper
    sqlmin!CQScanRangePrefetchDelayNew::GetRow
    sqlmin!CQScanNLJoinTrivialNew::GetRow
    sqlmin!CQScanNLJoinNew::GetRowHelper
    sqlmin!CQueryScan::GetRow
    sqllang!CXStmtQuery::ErsqExecuteQuery
    sqllang!CXStmtCondWithQuery::XretExecute
    sqllang!CMsqlExecContext::ExecuteStmts
    sqllang!CMsqlExecContext::FExecute
    sqllang!CSQLSource::Execute
    sqllang!process_request
    sqllang!process_commands
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  24. Chris Page said

    Hi thanks for the blog – gives me some hope of finding an answer :-) Stack below appeared today – this is the second time I have run a set of code on this server (is running on others of lower / same SQL versions with no problems) so was quite surprised to encounter an issue, many thanks :
    0:000> KC 1000
    Call Site
    sqlservr!CVariableInfo::CVarBlock::PvbJoin
    sqlservr!CVariableInfo::PviRelease
    sqlservr!CVarPageMgr::Release
    sqlservr!CMemObj::Free
    sqlservr!IndexAnalysis::Cleanup
    sqlservr!IndexAnalysis::ObjectIterator
    sqlservr!IndexAnalysis::GetNextRow
    sqlservr!CRowsetIndexAnalysis::FGetNextRow
    sqlservr!CUtRowset::GetNextRows
    sqlservr!CQScanRmtScanNew::GetRowHelper
    sqlservr!CQScanStreamAggregateNew::GetRowHelper
    sqlservr!CQScanStreamAggregateNew::GetCalculatedRow
    sqlservr!CQScanStreamAggregateNew::Open
    sqlservr!CQScanNLJoinTrivialNew::GetRow
    sqlservr!CQScanStreamAggregateNew::GetRowHelper
    sqlservr!CQScanStreamAggregateNew::GetCalculatedRow
    sqlservr!CQScanStreamAggregateNew::Open
    sqlservr!CQScanNew::OpenHelper
    sqlservr!CQScanUpdateNew::Open
    sqlservr!CQueryScan::Startup
    sqlservr!CXStmtQuery::SetupQueryScanAndExpression
    sqlservr!CXStmtQuery::InitForExecute
    sqlservr!CXStmtQuery::ErsqExecuteQuery
    sqlservr!CXStmtDML::XretDMLExecute
    sqlservr!CXStmtDML::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecStr::XretExecStrExecute
    sqlservr!CXStmtExecStr::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    • Hello Chris,
      Do you get this dump when you use linked server queries?
      Note: You can find the query in SQL Server error log after the dump

      • Chris Page said

        Hi karthick,

        The code is running against the same server as the agent is on so no linked server activity of any sort. The query in the log didn’t tell me much because it contains the whole query from the agent task which calls 2 stored procs that themselves lead to individuals tables and indexes so doesn’t align with what appears to be the failure point from the default trace.

        • Chris,
          This issue is very unique and very difficult to crack without analyzing the dump. I am not sure how feasible it is for you to share the dump with me (Though I am interested I will not recommend). I would recommend you to create a support case with Microsoft SQL Server support.
          This looks like a BUG for me but cant confirm unless I look at the dump.

          • Chris Page said

            Thanks kathrick, the problem hasn’t recurred here, however, I have seen the message that the agent was giving me in a previous role but not picked up core dumps on those occassions so I’ve gone back to an ex-colleague to see if the problem is actually manifesting the same there so we’ll be able to have more material to co-ordinate any issues raised.

            Chris

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 1,700 other followers

%d bloggers like this: