Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes

Posted by Karthick P.K on August 21, 2012

Do you see below errors in SQL error along with dumps and stuck?

Non-yielding IOCP Listener

* BEGIN STACK DUMP:
* 05/06/12 03:54:59 spid 0
* Non-yielding IOCP Listener

Non-yielding Scheduler
* BEGIN STACK DUMP:
* 04/16/12 10:09:58 spid 6256
* Non-yielding Scheduler

Non-yielding Resource Monitor

* BEGIN STACK DUMP

* 01/22/09 19:11:16 spid 0

* Non-yielding Resource Monitor

External dump process returned no errors.
Date Time Server Process 0:0:0 (0x31e8) Worker 0x000000016F41d140 appears to be non-yielding on Scheduler 4. Thread creation time: 12010668087858. Approx Thread CPU Used: kernel 2 ms, user 60516 ms. Process Utilization 11%. System Idle 83%. Interval: 71227 ms.

Refer “How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps” for analyzing the Non-yielding Scheduler, Non-yielding IOCP Listener and Non-yielding Resource Monitor Dumps.

If you are interested in just finding a quick resolution follow the below steps to get the Non-Yield stack from the dump and check if it is matching with any existing known issues in SQL Server.

To analyze the dump download and Install Windows Debugger from This link

Step 1:

Open Windbg

step 2:

Choose File menu –> select Open crash dump –>Select the Dump file (SQLDump000#.mdmp)

Step 3:

on command window type
.sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;

Step 4:

Type .reload /f and hit enter. This will force debugger to immediately load all the symbols.

Step 5:

Type .cxr sqlservr!g_copiedStackInfo+0X20 for SQL Server2005 and SQL Server2008/2008R2 (or) .cxr sqlmin!g_copiedStackInfo+0X20 for SQL Server2012.

Type kc 100 and look at the stack to see if it matches with the stack of any of known issues in SQL Server listed below.

If kc 100 doesn’t display any stack and throws “WARNING: Frame IP not in any known module. Following frames may be wrong” type .cxr to reset to default scope and try .cxr sqlservr!g_copiedStackInfo+0X00c (In 32-Bit (X86) SQL server valid offset for context is 0X00c Look at This blog to see how we identified the offset)

Note:If your stack doesn’t match with any of the stack listed below then paste the stack in comments session of this blog (or) In This face book group.We will try to find the cause for you. If you don’t get any prompt reply from the community, you may need to open a support ticket with Microsoft.

Stack 1

sqlservr!COptExpr::DetachPointersIntoMemo

sqlservr!COptContext::PcxteOptimizeQuery

sqlservr!CQuery::Optimize

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtDML::InitNormal

sqlservr!CStmtDML::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

sqlservr!CSQLSource::Execute

sqlservr!ExecuteSql

sqlservr!CSpecProc::ExecuteSpecial

sqlservr!CXProc::Execute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExecProc::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB :2344600:FIX: "Non-yielding Scheduler" error may occur when you use the CONTAINSTABLE function together with many OR and AND predicates in SQL Server 2008 or in SQL Server 2008 R2

Stack 2

sqlservr!TMatchPattern

sqlservr!FMatchStrTxt

sqlservr!I8CharindexStrBhI8

sqlservr!CEs::GeneralEval4

sqlservr!CXStmtCond::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,0>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands 0x12a

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

2633357 FIX: "Non-yielding Scheduler" error might occur when you run a query that uses the CHARINDEX function in SQL Server 2008 R2

Stack 3

sqlservr!CItvlVal::Copy

sqlservr!CConstraintItvl::PcnstrItvlUnion

sqlservr!CConstraintProp::FBuildItvlFromOr

sqlservr!CConstraintProp::FBuildItvlFromPexpr

sqlservr!CConstraintProp::FAndItvlConstraint

sqlservr!CConstraintProp::AndNewConstraint

sqlservr!CConstraintProp::PcnstrDeriveSelect

sqlservr!CLogOp_Select::PcnstrDerive

sqlservr!CLogOpArg::PcnstrDeriveHandler

sqlservr!CLogOpArg::DeriveGroupProperties

sqlservr!COpArg::DeriveNormalizedGroupProperties

sqlservr!COptExpr::DeriveGroupProperties

sqlservr!COptExpr::DeriveGroupProperties 0xc6

sqlservr!COptExpr::DeriveGroupProperties

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtDML::InitNormal

sqlservr!CStmtDML::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

KB: 982376 FIX: A non-yielding scheduler error or an error 8623 occurs when you run a query that contains a large IN clause in SQL Server 2005,SQL Server 2008, or SQL Server 2008 R2

Stack 4

sqlservr!COptExpr::AdjustParallelPlan

sqlservr!COptContext::PcxteOptimizeQuery

sqlservr!CQuery::Optimize

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtSelect::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB: 943060 FIX: A query that has many outer joins takes a long time to compile in SQL Server 2005

Stack 5

sqlservr!CXid::GetBlockingTask

sqlservr!SNode::SearchForDeadlock

sqlservr!DeadlockMonitor::SearchForDeadlock

sqlservr!DeadlockMonitor::SearchAndResolve

sqlservr!DeadlockMonitor::SearchTaskAndResolve

sqlservr!DeadlockMonitor::WorkLoop

sqlservr!lockMonitor

sqlservr!lockMonitorThread

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

956854 Cumulative update package 10 for SQL Server 2005 Service Pack 2

Stack 6

ntdll!ZwQueryAttributesFile

ntdll!RtlDoesFileExists_UstrEx

ntdll!LdrpSearchPath

ntdll!LdrpCheckForLoadedDll

ntdll!LdrpLoadDll

ntdll!LdrLoadDll

kernel32!LoadLibraryExW

mswsock!SockLoadHelperDll

mswsock!SockGetTdiName

mswsock!SockSocket

mswsock!WSPSocket

ws2_32!WSASocketW

ws2_32!WSASocketA

sqlservr!CreateSocket

sqlservr!AcceptObject::AsyncAccept

sqlservr!Tcp::AcceptDone

sqlservr!SNIAcceptDoneWithReturnCode

sqlservr!SNIAcceptDoneWrapper

sqlservr!SNIAcceptDoneRouter

sqlservr!SOS_Node::ListenOnIOCompletionPort

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB 2711549 FIX: An error message is logged when you start SQL Server 2008 R2 or when a client sends a request to SQL Server 2008 R2

Stack 7

ntdll!ZwOpenKey

advapi32!LocalBaseRegOpenKey

advapi32!RegOpenKeyExW

sqlservr!COledbConnect::GetProviderOptions

sqlservr!COledbConnect::SetClsidFromProvider

sqlservr!COledbConnect::Init

sqlservr!CStmtExecProc::XretRemoteExec

sqlservr!CRemoteProcExecLevel::Execute

sqlservr!CStmtExecProc::XretWrapRemoteExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExec::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

KB2468047 FIX: Error code 17883 or "Non-yielding Scheduler" error may occur when you use the OPENQUERY function on SQL Server 2005

Stack 8

ntdll!ZwQueryVirtualMemory

psapi!QueryWorkingSetEx

sqlservr!BPool::Shrink

sqlservr!BPool::ReleaseAwayBufs

sqlservr!BPool::LazyWriter

sqlservr!lazywriter

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

967908 Cumulative update package 13 for SQL Server 2005 Service Pack 2 or 970279 Cumulative update package 4 for SQL Server 2005 Service Pack 3

Stack 9

sqlservr!LatchBase::ReleaseInternal

sqlservr!XVB::GetRecord

sqlservr!RowsetVersionScan::GetData

sqlservr!CQScanRowsetNew::GetRowWithPrefetch

sqlservr!CQScanRowsetNew::GetRow

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRow

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRow

sqlservr!CQueryScan::GetRow

sqlservr!CXStmtQuery::InitForExecute

sqlservr!CXStmtQuery::ErsqExecuteQuery

sqlservr!CXStmtCondWithQuery::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CXStmtDML::FExecTrigger

sqlservr!CXStmtDML::FExecAllTriggers

sqlservr!CXStmtDML::XretDMLExecute

sqlservr!CXStmtDML::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<0,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtPrepQuery::XretExecute

sqlservr!CExecuteStatement::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecStr::XretExecStrExecute

sqlservr!CXStmtExecStr::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB : 949595 FIX: Error message when you run a query that uses a join condition in SQL Server 2005: "Non-yielding Scheduler"

Stack 10

sqlservr!SQLServerLogIter::LookupScanCache

sqlservr!SQLServerLogIterForward::GetNextBlock

sqlservr!SQLServerLogIterForward::GetNext

sqlservr!LsMgr::GetEndOfLog

sqlservr!LsMgr::ProcessInternalRollForward

sqlservr!LsWorkRequest::Execute

sqlservr!LsWorker::ThreadRoutine

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB 970044 FIX: Error message when you try to break database mirroring between two servers that are running SQL Server 2008: "Non-yielding Scheduler"

Stack 11

sqlservr!CLinkedMap

sqlservr!CCheckReadersAndWriters::Release

sqlservr!CMainIlb::~CMainIlb

sqlservr!CBlobHandleFactoryMain::ReleaseILockBytes

sqlservr!CMainIlb::Release

sqlservr!CTraceRpcBinaryStream::~CTraceRpcBinaryStream

sqlservr!CTraceTvpData::~CTraceTvpData

sqlservr!CRpcTraceHelper::CleanUpTraceTvpData

sqlservr!CRpcTraceHelper::TracePostExec

sqlservr!CRPCExecEnv::OnExecFinish

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!endthreadex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2520808 FIX: Non-yielding scheduler error when you run a query that uses a TVP in SQL Server 2008 or in SQL Server 2008 R2 if SQL Profiler or SQL Server Extended Events is used

Stack 12

ntdll!ZwFreeVirtualMemory

KERNELBASE!VirtualFree

sqlservr!MemoryNode::VirtualFree

sqlservr!ReservedMemBlock::FreeMemory

sqlservr!MultiPageAllocator::FreePagesInternal

sqlservr!MultiPageAllocator::FreePages

sqlservr!MemoryNode::FreePagesInternal

sqlservr!MemoryClerkInternal::FreePagesInline

sqlservr!CVarPageMgr::Release

sqlservr!CMemObj::Free

sqlservr!CMemThread<CMemObj>::Free

sqlservr!LockBytesSS::~LockBytesSS

sqlservr!LockBytesHolder::`scalar deleting destructor’

sqlservr!LockBytesHolder::DestroyCallback

sqlservr!CacheLbss

sqlservr!LockBytesSS::Release

sqlservr!CQueryIlb::~CQueryIlb

sqlservr!CBlobHandleFactoryMain::ReleaseILockBytes

sqlservr!CMainIlb::Release

sqlservr!CTraceRpcBinaryStream::~CTraceRpcBinaryStream

sqlservr!CTraceTvpData::~CTraceTvpData

sqlservr!CRpcTraceHelper::CleanUpTraceTvpData

sqlservr!CRpcTraceHelper::TracePostExec

sqlservr!CRPCExecEnv::OnExecFinish

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!endthreadex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2520808 FIX: Non-yielding scheduler error when you run a query that uses a TVP in SQL Server 2008 or in SQL Server 2008 R2 if SQL Profiler or SQL Server Extended Events is used

Stack 13

sqlservr!CompareStringWEnglishNoCase

sqlservr!CTypeInfo::ICompW

sqlservr!CDefaultCollation::ICompW

sqlservr!CDependElem::ICompare

sqlservr!CDependList::Find

sqlservr!CDependList::Insert

sqlservr!CDependList::Concat

sqlservr!CDependList::CollectDependencies

sqlservr!FillSysdepends

sqlservr!CProchdr::CreateProc

sqlservr!CSQLSource::PerformPphFakeExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2306162 FIX: Poor performance and some occasional non-yielding scheduler errors occur when you create a complex view that references a large amount of nested views or tables in SQL Server 2008 or in SQL Server 2008 R2

Stack 14

sqlservr!BaseSharedHoBt::GetHoBtId

sqlservr!HoBtFactory::GetDeferredDropCacheHobt

sqlservr!DropDeferredWorkTables

sqlservr!GhostRecordCleanupTask

sqlservr!CGhostCleanupTask::ProcessTskPkt

sqlservr!TaskReqPktTimer::ExecuteTask

sqlservr!OnDemandTaskContext::ProcessTskPkt

sqlservr!SystemTaskContext::ExecuteFunc

sqlservr!SystemTaskEntryPoint

sqlservr!OnDemandTaskContext::FuncEntryPoint

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SchedulerManager::FiberEntryPoint

kernel32!BaseFiberStart

kernel32!RtlCompareMemoryStub

KB 2505256 FIX: Poor performance when worktables that are marked for deferred drop are cleaned up in SQL Server 2008 R2

Stack 15

ntdll!ZwReadFile

kernel32!ReadFile

sqlservr!DiskReadAsync

sqlservr!FCB::AsyncRead

sqlservr!BackupIoRequest::StartDatabaseRead

sqlservr!BackupCopyMachine::CopyFileToBackupSet0

sqlservr!BackupCopyMachine::CopyFileToBackupSet

KB 960543 FIX: SQL Server 2005 or SQL Server 2008 may stop responding when you are performing a backup

Stack 16

sqlservr!Worker::ProfilingCPUTicks::ProfilingCpuTicksCallback

sqlservr!SOS_Scheduler::TaskTransition

sqlservr!SOS_Scheduler::Switch

sqlservr!SOS_Scheduler::SuspendNonPreemptive

sqlservr!SOS_Scheduler::Suspend

sqlservr!SOS_Task::Sleep

sqlservr!BTreeMgr::Seek

sqlservr!BTreeMgr::GetHPageIdWithKey

sqlservr!IndexPageManager::GetPageWithKey

sqlservr!GetRowForKeyValue

sqlservr!IndexRowScanner::EstablishInitialKeyOrderPosition

sqlservr!IndexDataSetSession::GetNextRowValuesInternal

sqlservr!RowsetNewSS::GetNextRows

sqlservr!CMEDScan::FGetRow

sqlservr!CMEDCatalogOwner::GetOwnerAliasIdFromSid

sqlservr!CMEDCatalogOwner::LookupPrimaryIdInCatalog

sqlservr!CMEDCacheEntryFactory::GetProxiedCacheEntryByAltKey

sqlservr!CMEDCatalogOwner::GetProxyOwnerBySID

sqlservr!CMEDProxyDatabase::GetOwnerBySID

sqlservr!GetDefaultSchemaIdCrossDb

sqlservr!GetCtxtSchemaId

sqlservr!CMEDAccess::GetMultiNameObject

sqlservr!CRangeObject::CImplName::FSameObject

sqlservr!CRangeObject::FCheckImplNames

sqlservr!CRangeObject::XretPostSchemaChecks

sqlservr!CRangeObject::XretSchemaChanged

sqlservr!CRangeTable::XretSchemaChanged

sqlservr!CEnvCollection::XretSchemaChanged

sqlservr!CXStmtQuery::XretSchemaChanged

sqlservr!CXStmtSelect::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExecProc::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

KB 2699013 FIX: SQL Server 2008 R2 or SQL Server 2008 stops responding and a "Non-yielding Scheduler" error is logged

Stack 17

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQueryScan::GetRow

sqlservr!CXStmtQuery::ErsqExecuteQuery

sqlservr!CXStmtCondWithQuery::XretExecute

sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn

sqlservr!CMsqlExecContext::ExecuteStmts<1,0>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CXStmtDML::FExecTrigger

sqlservr!CXStmtDML::FExecAllTriggers

sqlservr!CXStmtDML::XretDMLExecute

sqlservr!CXStmtDML::XretExecute

sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn

sqlservr!CMsqlExecContext::ExecuteStmts<0,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtPrepQuery::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

KB 967169 FIX: When you run an UPDATE statement against a table that has a FOR UPDATE trigger that joins the DELETED and INSERTED tables, the query takes a long time to finish

Stack 18

msvcr80!memcpy

BackupString::vswcatf

BackupString::swcatf

BackupHistory::GenerateBackupDetails

sqlservr!BackupHistory::GenerateBackupSet

KB 917971 FIX: You may receive more than 100,000 page faults when you try to back up a SQL Server 2005 database that contains hundreds of files and file groups.

Stack 20

mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!Smux::InternalClose
sqlservr!Smux::ReadDone

"Non-yielding Scheduler" error and SQL Server 2008 or SQL Server 2008 R2 stops responding intermittently in Windows Server 2008 or in Windows Server 2008 R2

Stack 21

mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!Smux::InternalClose
sqlservr!Smux::ReadDone

"Non-yielding Scheduler" error and SQL Server 2008 or SQL Server 2008 R2 stops responding intermittently in Windows Server 2008 or in Windows Server 2008 R2

How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps ……

SQL Server Latch & Debugging latch time out

How to Analyze "Deadlocked Schedulers" Dumps

If you liked this post, do like us on Facebook at https://www.facebook.com/mssqlwiki and join our Facebook group to get answers for all your SQL Server related questions.

Thank you,

Karthick P.K |My Facebook Page |My Site| Blog space| Twitter

Disclaimer:

The views expressed on this website/blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights

This entry was posted on August 21, 2012 at 9:08 AM and is filed under Debugging, Performance, SQL General, SQL Server Tools. Tagged: 17883, 17884, 17887, 17888, Configuration, Debugging, error 17883 non yielding, External dump process returned no errors, External dump process returned no errors.DoMiniDump () encountered error, Interval:, IO Completion Listener () Worker appears to be non-yielding on Node . Approx CPU Used: kernel ms, memory dump, non-yielding, Non-yielding IOCP Listener, non-yielding resource monitor, Non-yielding Scheduler, Process 0:0:0 ( ) Worker appears to be non-yielding on Scheduler, Process :0:0 () Worker appears to be non-yielding on Scheduler n. Thread creation time: . Approx Thread CPU Used: kernel 0 ms, SQL General, sql scheduler, SQL Server 2005, SQL Server 2008, SQL Server Engine. Non-yielding Scheduler, SQLServer dump, SQLServer mdmp, Stack Dump, user 0 ms. Process Utilization 0%. System Idle %. Interval: ms., user ms. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

161 Responses to “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”

Pavel Chromy said

October 27, 2012 at 6:53 AM
Hi, our stack doesn’t match with any of the stack :
Call Site
sqlservr!SpinlockBase::SpinToAcquire
sqlservr!SOS_Event::Signal
sqlservr!BPool::Steal
sqlservr!SQLSinglePageAllocator::AllocatePages
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CMemThread::PbGetNewPages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemThread::Alloc
sqlservr!operator new
sqlservr!SNI_Conn::InitObject
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Can you help me ?
Thanks
Pavel Chromy

Reply
- Karthick P.K said
  
  October 27, 2012 at 7:29 AM
  Will look at this in few minutes. I am driving now
  
  Sent from my Windows Phone ________________________________
  
  Reply
- Karthick P.K said
  
  October 27, 2012 at 8:50 AM
  Can you paste the following info
  1. SQL Server version and windows version?
  2. Amont of physical server memory on this server?
  3.What is sQL Server max server memory and Max dop?
  4. Are we suing SQLClr in this server?
  5. How frequently are we getting this dump and errorlog message similar to Thread CPU Used: kernel n ms, user n ms
  
  Reply
  - Pavel Chromy said
    
    October 27, 2012 at 9:17 AM
    Sending additional info :
    
    Microsoft SQL Server 2005 version 9.00.5296.05
    Computer type is AT/AT COMPATIBLE.
    Bios Version is DELL – 1
    Phoenix ROM BIOS PLUS Version 1.10 2.2.3
    40 Unknown CPU 9., 2 Mhz processor (s).
    Windows NT 6.1 Build 7601 CSD Service Pack 1.
    Memory Total Physical = 262130 MB
    
    sp_configure :
    name minimum maximum config_value run_value
    ———————————– ———– ———– ———— ———–
    Ad Hoc Distributed Queries 0 1 0 0
    affinity I/O mask -2147483648 2147483647 0 0
    affinity mask -2147483648 2147483647 0 0
    affinity64 I/O mask -2147483648 2147483647 0 0
    affinity64 mask -2147483648 2147483647 0 0
    Agent XPs 0 1 1 1
    allow updates 0 1 0 0
    awe enabled 0 1 0 0
    blocked process threshold 0 86400 5 5
    c2 audit mode 0 1 0 0
    clr enabled 0 1 0 0
    cost threshold for parallelism 0 32767 5 5
    cross db ownership chaining 0 1 0 0
    cursor threshold -1 2147483647 -1 -1
    Database Mail XPs 0 1 1 1
    default full-text language 0 2147483647 1033 1033
    default language 0 9999 0 0
    default trace enabled 0 1 1 1
    disallow results from triggers 0 1 0 0
    fill factor (%) 0 100 0 0
    ft crawl bandwidth (max) 0 32767 100 100
    ft crawl bandwidth (min) 0 32767 0 0
    ft notify bandwidth (max) 0 32767 100 100
    ft notify bandwidth (min) 0 32767 0 0
    index create memory (KB) 704 2147483647 0 0
    in-doubt xact resolution 0 2 0 0
    lightweight pooling 0 1 0 0
    locks 5000 2147483647 0 0
    max degree of parallelism 0 64 4 4
    max full-text crawl range 0 256 4 4
    max server memory (MB) 16 2147483647 2147483647 2147483647
    max text repl size (B) 0 2147483647 65536 65536
    max worker threads 128 32767 0 0
    media retention 0 365 0 0
    min memory per query (KB) 512 2147483647 1024 1024
    min server memory (MB) 0 2147483647 16 16
    nested triggers 0 1 1 1
    network packet size (B) 512 32767 4096 4096
    Ole Automation Procedures 0 1 0 0
    open objects 0 2147483647 0 0
    PH timeout (s) 1 3600 60 60
    precompute rank 0 1 0 0
    priority boost 0 1 1 1
    query governor cost limit 0 2147483647 0 0
    query wait (s) -1 2147483647 -1 -1
    recovery interval (min) 0 32767 0 0
    remote access 0 1 1 1
    remote admin connections 0 1 0 0
    remote login timeout (s) 0 2147483647 20 20
    remote proc trans 0 1 0 0
    remote query timeout (s) 0 2147483647 10800 10800
    Replication XPs 0 1 0 0
    scan for startup procs 0 1 0 0
    server trigger recursion 0 1 1 1
    set working set size 0 1 0 0
    show advanced options 0 1 1 1
    SMO and DMO XPs 0 1 1 1
    SQL Mail XPs 0 1 0 0
    transform noise words 0 1 0 0
    two digit year cutoff 1753 9999 2049 2049
    user connections 0 32767 0 0
    user options 0 32767 0 0
    Web Assistant Procedures 0 1 0 0
    xp_cmdshell 0 1 0 0
    
    SQLClr do not use.
    
    Over the past 14 days are few stiff SQL Server (Non-yielding IOCP Listener) – in the log then appears “IO Completion Listener (0x830) Worker 0x00000000112DC1C0 Appears to be non-yielding on Node 0th Approx CPU Used: kernel 187 ms, user 924898 ms, Interval: 925116th “processors working at 100% and the server is not responding.
    
    Thank you for any idea
    Pavel
    
    Reply
    - Karthick P.K said
      
      October 27, 2012 at 12:45 PM
      Limit your SQL Server “Max server Memory” (Start with 200,000 MB . We can increase or reduce based on OS and other applications need).
      Most likely since all OS physical memory has been used and OS is struggling with intensive paging SQL Server is affected.
      So if we will limit SQL Server for amount of memory it will resolve this issue.
      Let me know the output of below command as well !findstack sqlservr!BPool::ReplenishFreeList.
      We will get this dump with this stack even if the lazywritter is not able to keep up to the speed of updates/Inserts because of I/O bottlenecks.
      
      Reply
      - Pavel Chromy said
        
        October 28, 2012 at 9:18 AM
        Thank you for the advice, but unfortunately SQL server froze again. Max server memory I set to 150,000 megabytes. In the dump file is only “Non-yielding IOCP Listener”.
        
        Reply
        
        Karthick P.K said
        
        October 28, 2012 at 11:05 AM
        Did you get “Non-yielding IOCP Listener” dump after you set max server memory to 150000? how much was available physical memory in server when you got the dump?
        Check if stack is same in new dump ,Also run !findstack sqlservr!BPool::ReplenishFreeList and paste the output
        
        Reply
        
        Pavel Chromy said
        
        October 28, 2012 at 11:41 AM
        Yes, max memory has already been set to 150.000 MB. Physical memory was available 95.500 MB.
        Sending listing :
        Call Site
        ntdll!NtWaitForSingleObject
        KERNELBASE!WaitForSingleObjectEx
        sqlservr!CDmpDump::DumpInternal
        sqlservr!CDmpDump::Dump
        sqlservr!CImageHelper::DoMiniDump
        sqlservr!stackTrace
        sqlservr!SQL_SOSNonYieldIOCPCallback
        sqlservr!SOS_OS::ExecuteNonYieldIOCPCallbacks
        sqlservr!alloca_probe
        sqlservr!SchedulerMonitor::Run
        sqlservr!SchedulerMonitor::EntryPoint
        sqlservr!SOS_Task::Param::Execute
        sqlservr!SOS_Scheduler::RunTask
        sqlservr!SOS_Scheduler::ProcessTasks
        sqlservr!SchedulerManager::WorkerEntryPoint
        sqlservr!SystemThread::RunWorker
        sqlservr!SystemThreadDispatcher::ProcessWorker
        sqlservr!SchedulerManager::ThreadEntryPoint
        msvcr80!endthreadex
        msvcr80!endthreadex
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart
        
        Reply
        
        Karthick P.K said
        
        October 28, 2012 at 12:25 PM
        This look like scheduler monitor stack. You need to use .cxr sqlservr!g_copiedStackInfo+0X20 and then use kC to get the Non-Yield stack
        
        Reply
        
        Pavel Chromy said
        
        October 28, 2012 at 12:56 PM
        Sorry, here is the correct listing :
        Call Site
        sqlservr!SpinlockBase::SpinToAcquire
        sqlservr!SOS_Event::Signal
        sqlservr!alloca_probe
        sqlservr!SQLSinglePageAllocator::AllocatePages
        sqlservr!MemoryNode::AllocatePagesInternal
        sqlservr!MemoryClerkInternal::AllocatePages
        sqlservr!CMemThread::PbGetNewPages
        sqlservr!CMemFixed::Alloc
        sqlservr!CMemThread::Alloc
        sqlservr!operator new
        sqlservr!LockManager::AllocateLockOwnerBlocks
        sqlservr!LockManager::GetLockOwnerBlocks
        sqlservr!alloca_probe
        sqlservr!XactWorkspaceImp::GetSharedDBLockFromLockManager
        sqlservr!XactWorkspaceImp::GetDBLock
        sqlservr!lockdb
        sqlservr!CMEDUserStoreCallbacks::StartClockRun
        sqlservr!ClockAlgorithm::MoveHand
        sqlservr!ClockAlgorithm::ProcessTick
        sqlservr!SOS_UserStore::Notify
        sqlservr!ResourceMonitor::NotifyMemoryConsumers
        sqlservr!ResourceMonitor::ResourceMonitorTask
        sqlservr!SetupResourceMonitorTaskContext
        sqlservr!SOS_Task::Param::Execute
        sqlservr!SOS_Scheduler::RunTask
        sqlservr!SOS_Scheduler::ProcessTasks
        sqlservr!SchedulerManager::WorkerEntryPoint
        sqlservr!SystemThread::RunWorker
        sqlservr!SystemThreadDispatcher::ProcessWorker
        sqlservr!SchedulerManager::ThreadEntryPoint
        msvcr80!endthreadex
        msvcr80!endthreadex
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart
        
        We’re pretty desperate, do you think it would help to re-install Windows or SQL Server ?
        
        Reply
        
        Karthick P.K said
        
        October 28, 2012 at 2:06 PM
        There is still a memory pressure on this server.Not sure what is causing memory pressure? By any chance do you see OOM errors or dbcc memory status output in SQL Server error log?
        
        1. Enable Lock pages in memory for startup account of SQL Server and restart.
        2. Are you very sure available physical memory was 95GB when the dump occurred?
        3. I would say apply the latest SP and hotfix to windows instead of reinstalling windows. (https://mssqlwiki.com/sqlwiki/sql-performance/windows-2008-and-windows-2008-r2-known-issues-related-to-working-set-memory/)
        4. If you have not restarted SQL Server after the dump please run two SQL Scripts mentioned in https://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/ and share the output.
        5. Also share the output of sys.dm_os_memory_clerks
Pavel Chromy said

October 29, 2012 at 6:05 AM
The error log contains only::
…
2012-10-29 04:25:41.63 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2012-10-29 04:25:41.65 Server **Dump thread – spid = 0, PSS = 0x0000000000000000, EC = 0x0000000000000000
2012-10-29 04:25:41.65 Server ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.txt
2012-10-29 04:25:41.65 Server * *******************************************************************************
2012-10-29 04:25:41.65 Server *
2012-10-29 04:25:41.65 Server * BEGIN STACK DUMP:
2012-10-29 04:25:41.65 Server * 10/29/12 04:25:41 spid 0
2012-10-29 04:25:41.65 Server *
2012-10-29 04:25:41.65 Server * Non-yielding IOCP Listener
2012-10-29 04:25:41.65 Server *
2012-10-29 04:25:41.65 Server * *******************************************************************************
2012-10-29 04:25:41.65 Server * ——————————————————————————-
2012-10-29 04:25:41.65 Server * Short Stack Dump
2012-10-29 04:25:41.70 Server Stack Signature for the dump is 0x0000000000000207
2012-10-29 04:25:44.80 Server External dump process return code 0x20000001.
…
The dump file contains only:
=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 9.00.5296.05
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is AT/AT COMPATIBLE.
Bios Version is DELL – 1
Phoenix ROM BIOS PLUS Version 1.10 2.7.7
Current time is 04:25:41 10/29/12.
40 Unknown CPU 9., 2 Mhz processor (s).
Windows NT 6.1 Build 7600 CSD .

Memory
MemoryLoad = 90%
Total Physical = 262130 MB
Available Physical = 24467 MB
Total Page File = 264176 MB
Available Page File = 25893 MB
Total Virtual = 8388607 MB
Available Virtual = 8124449 MB
**Dump thread – spid = 0, PSS = 0x0000000000000000, EC = 0x0000000000000000
***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 10/29/12 04:25:41 spid 0
*
* Non-yielding IOCP Listener
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Lock pages is enabled. Yesterday we reinstalled the entire server (windows and SQL server – we apply the latest SP and hotfix) and after 3 hours the server again froze – unfortunately, in this state he can not run anything.

Reply
- Karthick P.K said
  
  October 29, 2012 at 2:58 PM
  Just noticing you have standard edition did you enable trace flag 845 ? while you enabled lock pages in memory
  
  Reply
- Karthick P.K said
  
  October 29, 2012 at 3:45 PM
  Also disable priority boost for SQL Server.
  
  Reply
  - Pavel Chromy said
    
    October 30, 2012 at 10:43 AM
    I did everything you wrote (Add traceflag, Enable lockpages in memory,Restart SQL Server and remove priority boost) and from that time’s running server great !!!
    Thank you.
    
    Reply
In Depth SQL said

October 31, 2012 at 1:38 PM
Hi Karthick – I have stack dump but that is not matching anuy one of above listed dumps. Below is the dump that I got it, can you please help on this.

Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqlservr!CDmpDump::DumpInternal
sqlservr!CDmpDump::DumpFilter
sqlservr!CDmpDump::Dump
msvcr80!__C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!RtlRaiseException
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_terminator
sqlservr!SOS_SEHTranslator
msvcr80!_CallSETranslator
msvcr80!FindHandlerForForeignException
msvcr80!FindHandler
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
sqlservr!CSlotPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!CMemThread::Alloc
sqlservr!CCriticalSectionSOS::Initialize
sqlservr!Tcp::FInit
sqlservr!Tcp::AcceptConnection
sqlservr!Tcp::AcceptDone
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  October 31, 2012 at 2:05 PM
  Hello InDepth,
  
  This is Exception stack
  {
  ntdll!KiUserExceptionDispatcher
  sqlservr!CSlotPageMgr::PbAllocate
  }
  
  Please follow https://mssqlwiki.com/2012/10/16/sql-server-exception_access_violation-and-sql-server-assertion/
  
  Any ways looking at your stack, This issue is fixed in
  Cumulative Update package 6 for SQL Server 2008 R2
  Cumulative update package 13 for SQL Server 2008 Service Pack 1
  Cumulative update package 3 for SQL Server 2008 Service Pack 2
  
  Thanks
  
  Reply
Scott McCormick said

November 16, 2012 at 8:21 PM
This is the error log a few hours before the dump you asked for in a Facebook post (https://www.facebook.com/mssqlwiki/posts/335361023227918)

I’m assuming you don’t actually need usernames or IP addresses.

2012-11-14 12:45:39.54 Logon Login succeeded for user ‘***\****’. Connection: trusted. [CLIENT: ***.***.***.***]
2012-11-14 12:54:53.92 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2012-11-14 12:54:56.81 Server **Dump thread – spid = 0, PSS = 0x00000000, EC = 0x00000000
2012-11-14 12:54:56.81 Server ***Stack Dump being sent to ***\MSSQL.1\MSSQL\LOG\SQLDump0002.txt
2012-11-14 12:54:56.81 Server * *******************************************************************************
2012-11-14 12:54:56.81 Server *
2012-11-14 12:54:56.81 Server * BEGIN STACK DUMP:
2012-11-14 12:54:56.81 Server * 11/14/12 12:54:56 spid 0
2012-11-14 12:54:56.81 Server *
2012-11-14 12:54:56.81 Server * Non-yielding IOCP Listener
2012-11-14 12:54:56.81 Server *
2012-11-14 12:54:56.81 Server * *******************************************************************************
2012-11-14 12:54:56.81 Server * ——————————————————————————-
2012-11-14 12:54:56.81 Server * Short Stack Dump
2012-11-14 12:55:00.36 Server Stack Signature for the dump is 0x0000036D
2012-11-14 12:55:13.73 Logon Error: 18456, Severity: 14, State: 8.
2012-11-14 12:55:20.67 Logon Login succeeded for user ‘***\****’. Connection: trusted. [CLIENT: ***.***.***.***]
2012-11-14 13:03:04.04 Server Timeout waiting for external dump process 68908.

2012-11-14 13:03:04.04 Server IO Completion Listener (0x5b0) Worker 0x008980E8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
ntdll!KiFastSystemCallRet
ntdll!NtRequestWaitReplyPort
secur32!CallSPM
secur32!SecpAcceptSecurityContext
secur32!LsaAcceptSecurityContext
secur32!AcceptSecurityContext
sqlservr!Ssl::Handshake
sqlservr!Ssl::HandshakeReadDone
sqlservr!CryptoBase::ReadDone
sqlservr!SNIReadDone
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadStart

Reply
- Karthick P.K said
  
  November 19, 2012 at 5:21 AM
  Please update your SQLServer to latest service pack. Looks like you are hitting a known issue mentioned in http://support.microsoft.com/kb/929376
  
  Reply
Steve said

November 26, 2012 at 8:46 PM
Hi Karthick,

I have a stack which is not listed. Can you help me with that?

ntdll!KiFastSystemCallRet
ntdll!ZwClose
kernel32!CloseHandle
sqlservr!CTraceFileIoProvider::OpenNextFile
sqlservr!CTraceWriteRequest::InitForFileTrace
sqlservr!CTraceFileIoProvider::InitializeWriteRequest
sqlservr!CTrace::WriteRecord
sqlservr!CTraceController::ProduceRecord
sqlservr!CTraceController::ProduceRecordWithTextBase
sqlservr!AuditEventBase
sqlservr!AuditEventNoCheck
sqlservr!AuditEvent
sqlservr!DbccOperation::CleanupDbccOperation
sqlservr!CStmtDbcc::XretExecute
sqlservr!CExecuteStatement::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_endthreadex
msvcr80!_endthreadex
kernel32!BaseThreadStart

Happend once.

SQL Version: Microsoft SQL Server 2005 – 9.00.3080.00 (Intel X86)
Sep 6 2009 01:43:32 Copyright (c) 1988-2005 Microsoft Corporation
Enterprise Edition on Windows NT 5.2 (Build 3790: Service Pack 2)

Windows Server 2003 Enterprise
Build 379.srv03_sp2_gdr.120503-0334 SP2

Physical Memory: 15.727.764 KB
SQL Server 10GB

Thanks

Reply
- Karthick P.K said
  
  November 27, 2012 at 2:12 AM
  When profiler tace is collected to network location or to a disk which is very slow, we might end with this.
  1. Make sure you allways collect server side trace instead of client side traces when required.
  2. Dont collect multiple trace files to gether.
  
  Reply
How to Analyze "Deadlocked Schedulers" Dumps? « MSSQLWIKI said

January 17, 2013 at 8:12 AM
[…] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… […]

Reply
Ewan Gibb said

February 14, 2013 at 5:09 PM
Hi,

I’ve been through the steps in this blog and have the following stack trace:

sqldk!CVariableInfo::PbGetBlock
sqldk!TVarPageMgr::PbAllocate
sqldk!CMemObj::Alloc
sqlmin!BTreeRowFactory::AllocateNewPageContext
sqlmin!SplitPage
sqlmin!IndexDataSetSession::InsertSmallRecord
sqlmin!IndexDataSetSession::InsertRowInternal
sqlmin!DatasetSession::InsertRow
sqlmin!RowsetNewSS::InsertRow
sqlmin!CValRowMult::SetDataX
sqltses!CEsExec::GeneralEval
sqlmin!CQScanUpdateNew::GetRow
sqlmin!CQScanNLJoinNew::GetRowHelper
sqlmin!CQScanAssertNew::GetRow
sqlmin!CQueryScan::GetRow
sqllang!CXStmtQuery::ErsqExecuteQuery
sqllang!CXStmtDML::XretDMLExecute
sqllang!CXStmtDML::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!ExecuteSql
sqllang!CSpecProc::ExecuteSpecial
sqllang!CSpecProc::Execute
sqllang!process_request
sqllang!process_commands
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

This is on a 64-bit Windows Server 2008 R2 Standard Edition server, 32gb ram, running SQL Server 2012 SP1.

Microsoft SQL Server 2012 (SP1) – 11.0.3321.0 (X64)
Nov 14 2012 19:41:38
Copyright (c) Microsoft Corporation
Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1)

Any help would be much appreciated.

Thanks!

Reply
- Karthick P.K said
  
  February 15, 2013 at 4:07 AM
  Ewan,
  1. In SQLServer errolog immediately after this dump you will find message for “Approx CPU Used” can you share that line please. I am intrested in kernel and user time,
  2. Share your max server memory and min server memory config.
  3. Have you enabled LPIM for SQLServer startup account?
  
  Reply
  - Ewan Gibb said
    
    March 5, 2013 at 1:33 PM
    Karthik,
    
    Thank you so much for the prompt response. I haven’t noticed it until today as I lost the link to this page. We are still having the problem only now it is happening with a higher regularity.
    
    To answer your questions:
    
    1 – I hope this is the log entry you are after:
    
    Process 0:0:0 (0x1fd0) Worker 0x00000007839F6160 appears to be non-yielding on Scheduler 2. Thread creation time: 13006905344342. Approx Thread CPU Used: kernel 0 ms, user 70465 ms. Process Utilization 6%. System Idle 92%. Interval: 70465 ms.
    
    2 – Minimum server memory (in MB): 0
    Maximum server memory (in MB): 2147483647
    
    3 – LPIM is not enabled.
    
    Kind regards.
    
    Reply
    - Karthick P.K said
      
      March 5, 2013 at 2:20 PM
      How much RAM do we have on this system?
      
      Reply
      - Ewan Gibb said
        
        March 5, 2013 at 4:18 PM
        The server has 32gb RAM.
        
        Reply
        
        Karthick P.K said
        
        March 6, 2013 at 5:57 AM
        Cap the MaxServermemory of SQL Server after considering the memory requirements by OS, other applications drivers, Non-bpool etc. If it is dedicated SQL Server start with 26GB. Let us increase or decrease on need later.
        
        Reply
        
        Ewan Gibb said
        
        March 6, 2013 at 8:11 AM
        Thanks! It is a dedicated SQL Server, so I have set it to 26gb. I will see how it goes today and monitor it closely.
        
        Reply
        
        Ewan Gibb said
        
        March 6, 2013 at 4:04 PM
        Things looked fine for almost 8 hours, then the non yielding-scheduler message struck again. The stack trace is slightly different this time:
        
        sqldk!CVariableInfo::PbGetBlock
        sqldk!TVarPageMgr::PbAllocate
        sqldk!CMemObj::Alloc
        sqlmin!BTreeRowFactory::AllocateNewBTreeRow
        sqlmin!IndexDataSetSession::InsertSmallRecord
        sqlmin!IndexDataSetSession::InsertRowInternal
        sqlmin!DatasetSession::InsertRow
        sqlmin!RowsetNewSS::InsertRow
        sqlmin!CValRowMult::SetDataX
        sqltses!CEsExec::GeneralEval
        sqlmin!CQScanUpdateNew::GetRow
        sqlmin!CQScanNLJoinNew::GetRowHelper
        sqlmin!CQScanAssertNew::GetRow
        sqlmin!CQueryScan::GetRow
        sqllang!CXStmtQuery::ErsqExecuteQuery
        sqllang!CXStmtDML::XretDMLExecute
        sqllang!CXStmtDML::XretExecute
        sqllang!CMsqlExecContext::ExecuteStmts
        sqllang!CMsqlExecContext::FExecute
        sqllang!CSQLSource::Execute
        sqllang!ExecuteSql
        sqllang!CSpecProc::ExecuteSpecial
        sqllang!CSpecProc::Execute
        sqllang!process_request
        sqllang!process_commands
        sqldk!SOS_Task::Param::Execute
        sqldk!SOS_Scheduler::RunTask
        sqldk!SOS_Scheduler::ProcessTasks
        sqldk!SchedulerManager::WorkerEntryPoint
        sqldk!SystemThread::RunWorker
        sqldk!SystemThreadDispatcher::ProcessWorker
        sqldk!SchedulerManager::ThreadEntryPoint
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart
        
        The entry in the log this time reads:
        
        “Process 0:0:0 (0x1d80) Worker 0x00000004BBBEA160 appears to be non-yielding on Scheduler 19. Thread creation time: 13007047217492. Approx Thread CPU Used: kernel 0 ms, user 71339 ms. Process Utilization 7%. System Idle 92%. Interval: 71338 ms.”
        
        Entries such as the above are logged at regular intervals until the SQL Server service is restarted.
        
        In addition there are multiple examples of this:
        
        “A time-out occurred while waiting for buffer latch — type 2, bp 000000002D049000, page 1:3704, stat 0xb, database id: 34, allocation unit Id: 20700475686912/302175452397568, task 0x00000008EE12B498 : 0, waittime 300 seconds, flags 0x3a, owning task 0x0000000482C42558. Not continuing to wait.”
        
        If I run DBCC Opentran against all databases, the same database always has an open transaction. Using DBCC INPUTBUFFER(xxx) I can see that this is also the same each time.
        
        Are there any other diagnostics I can perform?
        
        The server was updated to SQL Server 2012 SP1 CU2 earlier in the week to see if that would help.
        
        Reply
        
        Karthick P.K said
        
        March 7, 2013 at 5:59 AM
        1. Again there is delay in memory allocation. ADD LPIM privilege to startup account of SQL Server. SQL Server uses different mechanism to allocate memory when we have LPIM, most probably it should resolve your issue.
        2. Did Available memory on the system dropped very low when you got this dump? If so you may have to further reduce max server memory.
        If issue doesn’t get resolve after enabling LPIM for startup account of SQL Server and restarting , share the dump and error log using your FTP share.I will review it for you. If there are security restrictions you may have to open a case with Microsoft support and can share the case number with me.
        
        For the latch time out dump ““A time-out occurred while waiting for buffer latch” check if you have I/O request taking longer than %d seconds error message in SQL Error log? Follow the steps in https://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ once if you get the dump.
        
        Reply
        
        Ewan Gibb said
        
        March 7, 2013 at 9:05 AM
        I’ve granted LPIM for the SQL Server service account. Memory levels were fine all day yesterday. Didn’t breach 27gb in total. SQL Server sat at around 25gb.
        
        I will continue to monitor.
        
        Many thanks!
        
        Karthick P.K said
        
        March 7, 2013 at 9:51 AM
        Yep , LPIM should solve it. If by any chance you get dump again share the dump.
        
        Ewan Gibb said
        
        March 12, 2013 at 1:34 PM
        The configuration was set at the end of last week; I’ve been monitoring the server for 2 working days now, and the error has not arisen again. I will continue to monitor.
        
        Thanks for your help Karthick!
        
        Karthick P.K said
        
        March 12, 2013 at 4:17 PM
        Thanks Ewan😊 . Continue monitoring and and let me know how it goes. It can help other SQL users who read your comment. Sent from my Windows Phone ________________________________
        
        Karthick P.K said
        
        June 1, 2013 at 2:15 AM
        Hello Ewan,
        Fix for our issue is released today. http://support.microsoft.com/kb/2846378 has details of our issue and you can apply CU4 from http://support.microsoft.com/kb/2833645
        to fix it.
Bian said

February 20, 2013 at 6:39 PM
Hi Karthik,

Can you also help me on this stack?

ntdll!ZwClose
KERNELBASE!CloseHandle
kernel32!CloseHandleImplementation
sqlservr!CTraceFileIoProvider::OpenNextFile
sqlservr!CTraceWriteRequest::InitForFileTrace
sqlservr!CTraceFileIoProvider::InitializeWriteRequest
sqlservr!CTrace::WriteRecord
sqlservr!CTraceController::ProduceRecord
sqlservr!CTraceDDL::TraceDDL
sqlservr!CStatsDef::CreateStats
sqlservr!CStmtCreateStats::XretExecute
sqlservr!CXStmtDDL::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CSQLSource::SeExecute
sqlservr!CExecAutoCreateStats::FExecute
sqlservr!COptContext::FInstantiateGuessedStats
sqlservr!COptContext::PexprSimplify
sqlservr!COptContext::PcxteOptimizeQuery
sqlservr!CQuery::Optimize
sqlservr!CQuery::PqoBuild
sqlservr!CStmtQuery::InitQuery
sqlservr!CStmtDML::InitNormal
sqlservr!CStmtDML::Init
sqlservr!CCompPlan::FCompileStep
sqlservr!CSQLSource::FCompile
sqlservr!CSQLSource::FCompWrapper
sqlservr!CSQLSource::Transform
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex

Thanks much!

Reply
- Bian said
  
  February 20, 2013 at 7:34 PM
  Sorry Karthik. The actual stack is this. I followed your instruction and got this;
  
  ntdll!ZwClose
  KERNELBASE!CloseHandle
  kernel32!CloseHandleImplementation
  sqlservr!CTraceFileIoProvider::OpenNextFile
  sqlservr!CTraceWriteRequest::InitForFileTrace
  sqlservr!CTraceFileIoProvider::InitializeWriteRequest
  sqlservr!CTrace::WriteRecord
  sqlservr!CTraceController::ProduceRecord
  sqlservr!CTraceDDL::CommitOrUndo
  sqlservr!XdesRMReadWrite::CommitAllVirtualLogRecs
  sqlservr!XdesRMReadWrite::PreDestruct
  sqlservr!XdesRMFull::~XdesRMFull
  sqlservr!XdesRMFull::`scalar deleting destructor’
  sqlservr!XactRM::SinglePhaseCommit
  sqlservr!XactRM::CommitInternal
  sqlservr!FullXactImp::Commit
  sqlservr!CMsqlXactInternalBase::CommitInternal
  sqlservr!CMsqlXactInternalReadWrite::Commit
  sqlservr!CMsqlXactImp::Commit
  sqlservr!CAutoMsqlXact::CommitNestedXact
  sqlservr!COptContext::FInstantiateGuessedStats
  sqlservr!COptContext::PexprSimplify
  sqlservr!COptContext::PcxteOptimizeQuery
  sqlservr!CQuery::Optimize
  sqlservr!CQuery::PqoBuild
  sqlservr!CStmtQuery::InitQuery
  sqlservr!CStmtDML::InitNormal
  sqlservr!CStmtDML::Init
  sqlservr!CCompPlan::FCompileStep
  sqlservr!CSQLSource::FCompile
  sqlservr!CSQLSource::FCompWrapper
  sqlservr!CSQLSource::Transform
  sqlservr!CSQLSource::Execute
  sqlservr!process_request
  sqlservr!process_commands
  sqlservr!SOS_Task::Param::Execute
  sqlservr!SOS_Scheduler::RunTask
  sqlservr!SOS_Scheduler::ProcessTasks
  sqlservr!SchedulerManager::WorkerEntryPoint
  sqlservr!SystemThread::RunWorker
  sqlservr!SystemThreadDispatcher::ProcessWorker
  *** WARNING: Unable to verify timestamp for msvcr80.dll
  sqlservr!SchedulerManager::ThreadEntryPoint
  msvcr80!_callthreadstartex
  
  thanks!!
  
  Reply
  - Karthick P.K said
    
    February 21, 2013 at 3:00 PM
    sqlservr!CTraceFileIoProvider::OpenNextFile you are running a profiler trace and it looks like you have set the maximum file size , So SQL engine is trying to create a new file.
    Have you configured to capture profiler trace on local drive or network share? If local system check the speed of destination drive. If remote system (Network share) change it to local drive.
    
    Reply
    - bian said
      
      February 21, 2013 at 3:13 PM
      ah, that makes sense. we’ll change the destination to a local drive and see if that helps.
      
      I really appreicate for your help but here’s one more. can you also help me on this as well?
      
      Child-SP RetAddr Call Site
      00000081`683fd0e8 000007fe`fda84bf6 ntdll!NtCreateFile+0xa
      00000081`683fd0f0 00000000`76c718ed KERNELBASE!CreateFileW+0x2cd
      *** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlservr.exe –
      00000081`683fd250 00000000`01f72617 kernel32!CreateFileWImplementation+0x7d
      00000081`683fd2b0 00000000`01f7513f sqlservr!GetIUMSForMsxml+0x8055f3
      00000081`683fd340 00000000`00c53aa0 sqlservr!GetIUMSForMsxml+0x80811b
      00000081`683fd390 00000000`00c54a5a sqlservr+0xbd3aa0
      00000081`683fd3e0 00000000`00c5547b sqlservr+0xbd4a5a
      00000081`683fd420 00000000`00c550e1 sqlservr+0xbd547b
      00000081`683fd4f0 00000000`01989cc5 sqlservr+0xbd50e1
      00000081`683fd580 00000000`000c67a4 sqlservr!GetIUMSForMsxml+0x21cca1
      00000081`683fd5d0 00000000`001027b4 sqlservr+0x467a4
      00000081`683fd600 00000000`00146525 sqlservr+0x827b4
      00000081`683fd660 00000000`001460db sqlservr+0xc6525
      00000081`683fd690 00000000`00101e05 sqlservr+0xc60db
      00000081`683fd730 00000000`001021d8 sqlservr+0x81e05
      00000081`683fd760 00000000`0014cef5 sqlservr+0x821d8
      00000081`683fd8f0 00000000`015b7c97 sqlservr+0xccef5
      00000081`683fd990 00000000`015b7f0b sqlservr!_____SQL______Process______Available+0x713d47
      00000081`683fda80 00000000`01527c45 sqlservr!_____SQL______Process______Available+0x713fbb
      00000081`683fdab0 00000000`006eacba sqlservr!_____SQL______Process______Available+0x683cf5
      00000081`683fea60 00000000`000ec3bc sqlservr+0x66acba
      00000081`683febe0 00000000`000ed3ab sqlservr+0x6c3bc
      00000081`683fed10 00000000`0012a4c2 sqlservr+0x6d3ab
      00000081`683ff370 00000000`0008f280 sqlservr+0xaa4c2
      00000081`683ff580 00000000`0008ee3e sqlservr+0xf280
      00000081`683ff690 00000000`0008eb3b sqlservr+0xee3e
      00000081`683ff6f0 00000000`001b5caa sqlservr+0xeb3b
      00000081`683ff760 00000000`001b5e25 sqlservr+0x135caa
      00000081`683ff840 00000000`001b567d sqlservr+0x135e25
      00000081`683ff880 00000000`001b6586 sqlservr+0x13567d
      00000081`683ff930 00000000`74b137d7 sqlservr+0x136586
      00000081`683ff9d0 00000000`74b13894 msvcr80!_callthreadstartex+0x17
      00000081`683ffa00 00000000`76c7652d msvcr80!_threadstartex+0x84
      00000081`683ffa30 00000000`7736c521 kernel32!BaseThreadInitThunk+0xd
      00000081`683ffa60 00000000`00000000 ntdll!RtlUserThreadStart+0x1
      
      Again, thank you for your help.
      
      Reply
      - bian said
        
        February 22, 2013 at 11:42 AM
        Also, is there a way to get sqlservr.pdb for SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64)?
        
        Reply
      - Karthick P.K said
        
        February 22, 2013 at 1:33 PM
        Hello Bian,
        Second stack which you posted us Access violation dump or Non-Yielding scheduler dump?
        
        Reply
        
        Brian Kang said
        
        February 22, 2013 at 2:52 PM
        that is Non-Yielding scheduler dump as well.
        
        thanks!
        
        Reply
        
        Karthick P.K said
        
        February 24, 2013 at 6:28 AM
        Hello Brian, Do you have any antivirus which traces CreateFile and cause delay in file creation. Both the stacks are pointing that the thread are waiting on CreateFileWImplementation. I couldnt find the cause for second dump from stack because symbols are not aligned. Please share the memory dump and errorlog file. I will look in to it. Thanks > Date: Fri, 22 Feb 2013 14:52:45 +0000 > To: karthick_pk83@hotmail.com >
        
        Reply
        
        Karthick P.K said
        
        February 25, 2013 at 4:02 AM
        Hello Brian,
        
        I doubt if I can locate the public symbols for your build any time soon. So I would suggest to share the dump (As long as there is no security constraint).
        
        Thanks
        Karthick
        
        ——————————————————————————–
        
        Subject: Re: [MSSQLWIKI] Comment: “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”
        Date: Sun, 24 Feb 2013 11:40:37 -0800
        
        Hi Karthick,
        
        Thanks a lot for your help on this, first of all.
        
        Well, for the second dump, you’re right. We don’t have a correct symbols for it, which our server is Microsoft SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64) running on 2-node clustering. The size of dump file is about 3 GB (original size is 33 GB so I zipped it and split them into 4 files). This is because we had to enable filtered dump since the mini dump with default setting didn’t contain anything.
        
        If you have a shared folder that I can drop the dump files with you. However, if you can share the correct symbols for our server version with me, I would analyzed it and then share the output with you. I’m sure this will save lots of your time and effort. Is this something possible?
        
        thanks again,
        
        Brian
        
        Reply
        
        Brian Kang said
        
        February 25, 2013 at 6:09 PM
        ah, unfortunately, we can’t share the dump/log to the public. what do you think, is there any other way? thanks,Brian Date: Mon, 25 Feb 2013 04:02:43 +0000 To: bsk64@hotmail.com
        
        Reply
        
        Karthick P.K said
        
        February 26, 2013 at 1:07 AM
        I think the only option u have is contact “Microsoft SQL Server support”
        
        Brian Kang said
        
        February 26, 2013 at 11:56 AM
        yeah, I guess that’s the only option for us.
        
        thanks for your help and suggestion, we’ll open up a case for this.
        
        Brian Kang said
        
        February 24, 2013 at 7:42 PM
        Hi Karthick,
        
        Thanks a lot for your help on this, first of all.
        
        Well, for the second dump, you’re right. We don’t have a correct symbols for it, which our server is Microsoft SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64) running on 2-node clustering. The size of dump file is about 3 GB (original size is 33 GB so I zipped it and split them into 4 files). This is because we had to enable filtered dump since the mini dump with default setting didn’t contain anything.
        
        If you have a shared folder that I can drop the dump files with you. However, if you can share the correct symbols for our server version with me, I would analyzed it and then share the output with you. I’m sure this will save lots of your time and effort. Is this something possible?
        
        thanks again, Brian
        
        From: karthick pk Sent: Saturday, February 23, 2013 10:28 PM To: comment+rw812t0753pis_xm_l4uxebl0@comment.wordpress.com ; bsk64@hotmail.com Subject: RE: [MSSQLWIKI] Comment: “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”
        
        Hello Brian, Do you have any antivirus which traces CreateFile and cause delay in file creation. Both the stacks are pointing that the thread are waiting on CreateFileWImplementation. I couldnt find the cause for second dump from stack because symbols are not aligned. Please share the memory dump and errorlog file. I will look in to it. Thanks
        
        > Date: Fri, 22 Feb 2013 14:52:45 +0000 > To: karthick_pk83@hotmail.com >
        
        Reply
    - Brian Kang said
      
      February 22, 2013 at 11:44 AM
      Thanks Karthick.
      
      I posted another question but wonder if you can help me to get sqlservr.pdb for SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64)?
      
      Thanks much!! -Brian
      
      Reply
MAdler said

March 1, 2013 at 6:19 PM
Hi- i hope you can help me – this is the stack i am getting (seems much shorter then the rest)
Sqlserver 2005 6bit latest SP (0.5)

0:021> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqlservr!SystemThread::Suspend
sqlservr!SOS_Scheduler::SuspendPreemptive
sqlservr!SOS_Event::Wait
sqlservr!alloca_probe
sqlservr!CMemThread::Alloc
sqlservr!operator new
sqlservr!SNI_Conn::InitObject
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Brian Kang said
  
  March 4, 2013 at 5:01 AM
  Hi Karthick,
  
  No, we don’t see any “Non-yielding IOCP Listener” messages at all. just flood of these errors,
  
  2013-02-20 18:12:57.63 Server Process 0:0:0 (0x28ac) Worker 0x000000129E0781A0 appears to be non-yielding on Scheduler 35. Thread creation time: 13005864565060. Approx Thread CPU Used: kernel 0 ms, user 0 ms. Process Utilization 17%. System Idle 98%. Interval: 70358 ms. 2013-02-20 18:12:57.65 spid39s Database mirroring is inactive for database ‘XXXX’. This is an informational message only. No user action is required.
  
  we’re experiencing lots of cxpacket waits so this could be due to a parallelism issue?
  
  Thanks, Brian
  
  Reply
  - Karthick P.K said
    
    March 4, 2013 at 11:51 AM
    Hello Brian,
    
    Excessive parallelism can cause deadlocked scheduler but not non-Yielding scheduler.
    Remember SQL Server will generate dump only once when it detects non-yielding condition. Once the dump is generated warnings will be printed on error log for successive occurrence’s but dumps will not be generated. In our case it has detected the condition and printed the warning in SQL Server error log but has not generated the dump so ideally it would have generated dump in past.
    
    You can enabled tracefalg 1262 to generate dump on every non-Yield condition. But it may generate dumps continously and cause performance problem. So if you see the above error continoulsy
    
    1. Enable the traceflag
    2. Wait for first dump
    3. Disable the trace
    4. Paste the Non-yield stack.
    
    Did you change the min-server memory to default?
    Also let me know the verision of mscorwks which is loaded in SQLServer.
    
    Reply
- Karthick P.K said
  
  March 4, 2013 at 9:20 AM
  Hello Madler,
  
  Stack which you pasted is current or copied stack? From stack it looks like there is a memory pressure on the system. Can you share the SQL-Error log and output of two queries in https://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/
  
  Reply
  - madler said
    
    March 4, 2013 at 4:01 PM
    It was the stack trace off of the memdump that sql created at the time of the error.
    We hade sql set to default max memory – we limited sql to maintain 4 GB for the operating system and the errors seem to have gone away – so i guess the issue you dexcribed in that link was indeed what was happending.
    
    here are the results of the queries:
    RunTime Notification_Time Notification_type MemoryUtilization % Node Id Process_Indicator System_Indicator type state reserved Effect type state reserved Effect type state reserved Effect SQL_ReservedMemory_KB SQL_CommittedMemory_KB SQL_AWEMemory SinglePagesMemory MultiplePagesMemory TotalPhysicalMemory_KB AvailablePhysicalMemory_KB TotalPageFile_KB AvailablePageFile_KB TotalVirtualAddressSpace_KB AvailableVirtualAddressSpace_KB Record Id Type Record Time Current Time
    2013-03-04 10:50:35.310 2013-03-01 10:55:45.683 RESOURCE_MEMPHYSICAL_HIGH 100 1 NULL NULL APPLY_LOWPM EFFECT_OFF 0 0 APPLY_HIGHPM EFFECT_ON 0 0 REVERT_HIGHPM EFFECT_OFF 0 0 33683712 118416 0 5056 14048 33541144 28767956 67080432 32551328 8589934464 8556024928 0 RING_BUFFER_RESOURCE_MONITOR 6984298 265873923
    and:
    
    runtime Notification_time ProcessUtilization SystemIdle % UserModeTime KernelModeTime PageFaults WorkingSetDelta MemoryUtilization (%workingset) Record Time Current Time
    2013-03-04 10:57:19.680 2013-03-04 06:41:27.380 0 99 0 5148033 68 -16 98 250922562 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:42:27.383 5 91 28392182 526971378 2339 36 98 250982565 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:43:27.387 0 92 10920070 37908243 68 0 98 251042568 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:44:27.390 6 88 27144174 622755992 2325 -8 98 251102572 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:45:27.393 0 98 7176046 47580305 153 20 98 251162575 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:46:27.397 4 94 25116161 442106834 2293 36 98 251222579 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:47:27.403 20 78 13104084 1945332470 584 -404 98 251282583 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:48:27.417 12 86 28080180 1192783646 2221 -72 98 251342599 266274861
    
    Reply
Kile said

March 13, 2013 at 6:02 PM
Hi – I couldn’t find anything that matched the stack trace that I received. Any help is greatly appreciated.

Microsoft SQL Server 2012 – 11.0.2100.60 (X64)

0:525> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqldk!SOS_Scheduler::SwitchContext
sqldk!SOS_Scheduler::SuspendNonPreemptive
sqldk!EventInternal::Wait
sqldk!EventInternal::WaitAllowPrematureWakeup
sqlmin!CXPacketList::RemoveHead
sqlmin!CXPipe::ReceivePacket
sqlmin!CXTransLocal::ReceiveBuffers
sqlmin!CQScanExchangeNew::GetRowFromProducer
sqlmin!CQScanExchangeNew::GetRowHelper
sqlmin!CQScanRangePrefetchDelayNew::GetRow
sqlmin!CQScanNLJoinTrivialNew::GetRow
sqlmin!CQScanXProducerNew::GetRowHelper
sqlmin!CQScanXProducerNew::GetRow
sqlmin!FnProducerOpen
sqlmin!FnProducerThread
sqlmin!SubprocEntrypoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  March 14, 2013 at 4:03 AM
  Hello Kile,
  Is this copied stack . I mean did u do .cxr sqlmin!g_copiedStackInfo+0X20 and kc 100?
  
  Reply
Donny said

March 15, 2013 at 3:58 PM
Hi, I could not match my stack trace also. Thanks for any help/insight you may be able to provide!

This was in error log:
2013-03-14 12:45:51.480 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2013-03-14 12:45:51.630 Server **Dump thread – spid = 0, EC = 0x0000000000000000
2013-03-14 12:45:51.630 Server ***Stack Dump being sent to J:\MSSQL11.NAPSQL24\MSSQL\LOG\SQLDump0002.txt
2013-03-14 12:45:51.630 Server *
2013-03-14 12:45:51.630 Server * BEGIN STACK DUMP:
2013-03-14 12:45:51.630 Server * 03/14/13 12:45:51 spid 50876
2013-03-14 12:45:51.630 Server *
2013-03-14 12:45:51.630 Server * Non-yielding IOCP Listener
2013-03-14 12:45:51.630 Server *
2013-03-14 12:45:51.630 Server * Short Stack Dump
2013-03-14 12:45:51.690 Server Stack Signature for the dump is 0x00000000000003E1
2013-03-14 12:46:16.270 Server External dump process return code 0x20000001. External dump process returned no errors.
2013-03-14 12:46:16.270 Server IO Completion Listener (0x15ec0) Worker 0x00000080F9280160 appears to be non-yielding on Node 1. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15085.
2013-03-14 12:46:41.800 Server IO Completion Listener (0x15ec0) Worker 0x00000080F9280160 appears to be non-yielding on Node 1. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 50280.
2013-03-14 12:46:44.690 Server IO Completion Listener (0xdd8c) Worker 0x00000080FBF00160 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 40982.

This is the debug session:

0:042> .cxr sqlmin!g_copiedStackInfo+0X20
rax=0000000000000000 rbx=0000000000000000 rcx=00000079a773f498
rdx=00000000ce11f238 rsi=00000000ffffffff rdi=0000000000000b1c
rip=0000000077af135a rsp=00000000ce11ee40 rbp=00000000ce11f018
r8=00000080faf20040 r9=0000000000000000 r10=0000000000000000
r11=00000000ce11f1b8 r12=0000000000000000 r13=0000000000000000
r14=00000080fb024130 r15=00000000ffffffff
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
ntdll!ZwWaitForSingleObject+0xa:
00000000`77af135a c3 ret
0:042> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqldk!SystemThread::Suspend
sqldk!SOS_Scheduler::SuspendPreemptive
sqldk!EventInternal::Wait
sqldk!SOS_UnfairMutexPair::LongWait
sqldk!SOS_UnfairMutexPair::AcquirePair
sqldk!CMemThread::Alloc
sqldk!SOS_Node::EnqueueTaskDirectInternal
sqllang!SOS_Node::EnqueueTaskDirect
sqllang!SNIAcceptDoneRouter
sqldk!SOS_Node::ListenOnIOCompletionPort
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
0:042> lmvm sqlservr
start end module name
00000000`ff080000 00000000`ff0be000 sqlservr (pdb symbols) c:\websymbols\sqlservr.pdb\5F808D3337994C2AB03A56D03A08B9192\sqlservr.pdb
Loaded symbol image file: sqlservr.exe
Mapped memory image file: c:\Websymbols\sqlservr.exe\4F36496E3e000\sqlservr.exe
Image path: C:\Program Files\Microsoft SQL Server\MSSQL11.NAPSQL24\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Sat Feb 11 05:56:46 2012 (4F36496E)
CheckSum: 00034130
ImageSize: 0003E000
File version: 2011.110.2100.60
Product version: 11.0.2100.60
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft SQL Server
InternalName: SQLSERVR
OriginalFilename: SQLSERVR.EXE
ProductVersion: 11.0.2100.60
FileVersion: 2011.0110.2100.060 ((SQL11_RTM).120210-1917 )
FileDescription: SQL Server Windows NT – 64 Bit
LegalCopyright: Microsoft Corp. All rights reserved.
LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
Comments: SQL
0:042> !findstack sqlservr!SQL_SOSNonYieldSchedulerCallback
0:042> !findstack sqlservr!SQL_SOSNonYield
Thread 006, 1 frame(s) match
* 13 00000000861eea10 000007fedf76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0x184

Thread 042, 1 frame(s) match
* 07 00000000ce13ee70 000007fedf76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0xeb

0:042> ~[006]s
ntdll!ZwAllocateVirtualMemory+0xa:
00000000`77af149a c3 ret
0:006> kP
Child-SP RetAddr Call Site
00000000`861edb48 000007fe`fe261935 ntdll!ZwAllocateVirtualMemory+0xa
00000000`861edb50 000007fe`df6d7275 KERNELBASE!VirtualAlloc+0x45
00000000`861edb90 000007fe`df6ca47b sqldk!MemoryNode::VirtualAllocNativeNoCheck+0xa0
00000000`861edbf0 000007fe`df6ca1b4 sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality+0x256
00000000`861edd10 000007fe`df6c9e39 sqldk!SOS_MemoryBlockAllocator::AllocateBlock+0x5bd
00000000`861ede00 000007fe`df6ca816 sqldk!SOS_MemoryWorkSpace::AllocatePage+0x273
00000000`861edfb0 000007fe`df6cac61 sqldk!MemoryNode::AllocatePagesInternal+0x187
00000000`861ee040 000007fe`df6cae86 sqldk!MemoryClerkInternal::AllocatePages+0xb1
00000000`861ee370 000007fe`df6cae2d sqldk!CVarPageMgr::PviNewVarPage+0x36
00000000`861ee3e0 000007fe`df6c0287 sqldk!CVarPageMgr::PbAllocate+0x16f
00000000`861ee420 000007fe`d4c878ee sqldk!CMemObj::Alloc+0x47
00000000`861ee450 00000000`ff098349 sqllang!CErrorReportingManager::CwchFormatAndPrint+0x472
00000000`861ee910 00000000`ff093504 sqlservr!scierrlog+0x79
00000000`861ee9b0 000007fe`df76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0x184
00000000`861eea20 000007fe`df704c8c sqldk!SOS_OS::ExecuteNonYieldIOCPCallbacks+0xde
00000000`861eecc0 000007fe`df6c8389 sqldk!SchedulerMonitor::CheckIoCompletionHangs+0x1d0
00000000`861eee00 000007fe`df79bf89 sqldk!SchedulerMonitor::Run+0x175
00000000`861eef30 000007fe`df6c3170 sqldk!SchedulerMonitor::EntryPoint+0x9
00000000`861eef60 000007fe`df6c2f64 sqldk!SOS_Task::Param::Execute+0x21e
00000000`861ef560 000007fe`df6c2d66 sqldk!SOS_Scheduler::RunTask+0xa8
0:006> kC
Call Site
ntdll!ZwAllocateVirtualMemory
KERNELBASE!VirtualAlloc
sqldk!MemoryNode::VirtualAllocNativeNoCheck
sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality
sqldk!SOS_MemoryBlockAllocator::AllocateBlock
sqldk!SOS_MemoryWorkSpace::AllocatePage
sqldk!MemoryNode::AllocatePagesInternal
sqldk!MemoryClerkInternal::AllocatePages
sqldk!CVarPageMgr::PviNewVarPage
sqldk!CVarPageMgr::PbAllocate
sqldk!CMemObj::Alloc
sqllang!CErrorReportingManager::CwchFormatAndPrint
sqlservr!scierrlog
sqlservr!SQL_SOSNonYieldIOCPCallback
sqldk!SOS_OS::ExecuteNonYieldIOCPCallbacks
sqldk!SchedulerMonitor::CheckIoCompletionHangs
sqldk!SchedulerMonitor::Run
sqldk!SchedulerMonitor::EntryPoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
0:006> ~[042]s
ntdll!ZwWaitForSingleObject+0xa:
00000000`77af135a c3 ret
0:066> kP
Child-SP RetAddr Call Site
00000000`d501dc18 000007fe`fe2610dc ntdll!ZwWaitForSingleObject+0xa
00000000`d501dc20 000007fe`d5995f51 KERNELBASE!WaitForSingleObjectEx+0x79
00000000`d501dcc0 000007fe`d59964cf sqllang!FRunCommunicationsManager+0xf1
00000000`d501de90 000007fe`df6c3170 sqllang!CommunicationsManager+0x3bf
00000000`d501f0f0 000007fe`df6c2f64 sqldk!SOS_Task::Param::Execute+0x21e
00000000`d501f6f0 000007fe`df6c2d66 sqldk!SOS_Scheduler::RunTask+0xa8
00000000`d501f760 000007fe`df6de22f sqldk!SOS_Scheduler::ProcessTasks+0x299
00000000`d501f7e0 000007fe`df6de5c0 sqldk!SchedulerManager::WorkerEntryPoint+0x261
00000000`d501f880 000007fe`df6dd9cb sqldk!SystemThread::RunWorker+0x8f
00000000`d501f8b0 000007fe`df6de059 sqldk!SystemThreadDispatcher::ProcessWorker+0x3c8
00000000`d501f960 00000000`7799652d sqldk!SchedulerManager::ThreadEntryPoint+0x236
00000000`d501fa00 00000000`77acc521 kernel32!BaseThreadInitThunk+0xd
00000000`d501fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
0:066> kC
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqllang!FRunCommunicationsManager
sqllang!CommunicationsManager
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  March 16, 2013 at 7:12 AM
  Hello Donny,
  How much RAM do we have on this server and how frequently you see non-yielding error in errorlog ?
  
  Reply
  - Donny said
    
    March 18, 2013 at 9:49 PM
    Physical memory on server: 524211 MB
    
    SQL memory setting:
    name minimum maximum config_value run_value
    max server memory (MB) 128 2147483647 409600 409600
    
    SSIS also runs on this server.
    Thanks for having a look!
    
    Reply
  - Donny said
    
    March 18, 2013 at 9:50 PM
    That was the first time we saw non-yielding on that server.
    
    Reply
    - Karthick P.K said
      
      April 1, 2013 at 1:44 PM
      Hello Donny,
      I missed your comment. Please apply the fix in http://support.microsoft.com/kb/2819662
      (Upgrade to 11.0.3344.0 or higher) .Also let me know if you have Lock pages enabled for startup account of SQLServer.
      
      Reply
ccpjeff said

March 25, 2013 at 1:22 PM
Just a thank you for the great resource… currently troubleshooting some non-yielding errors…

It has hit us once every 2 weeks for the last few months and only caused crash dumps twice, and never causes SQL to die or anything like that. It typically results in multiple ‘client unable to reuse a session’ errors, and some errors in the front end application.

I used your info to dig through the dump, and found this:

mswsock!_GSHandlerCheck_SEH
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close

Which references several of your included dumps and KB articles. Checking those, it shows that this should of been fixed in SQL 2008 SP2 CU2. Our server is @ SP2 CU5 and most of our clients as well, but it’s possible a new client/service/application is not @ the correct level.

I’m verifying the native client versions on those new machines and hoping they are outdated! If so, this should be an easy fix!

but again, thanks for the great resource!

Cheers,

jeff

Reply
- Karthick P.K said
  
  March 25, 2013 at 1:48 PM
  Remember to apply the windows fix along with SQL Fix mentioned in http://support.microsoft.com/kb/2491214?wa=wsignin1.0 If there is difficulties applying the windows fix you can use the workaround mentioned in same Kb.
  
  Reply
KK said

April 29, 2013 at 11:44 AM
I got 2 SQL 2005 instances and they both gave non-yielding IOCP Listener error. The server has 16GB of physical memory but the both SQL instances are limited to Min of 2GB & Max of 4GB memory. Below the info from one of the instances dumpfile.

0:008> KC 100
Call Site
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!operator new
sqlservr!operator new[]
sqlservr!CErrorReportingManager::SendErrorToErrLog
sqlservr!CErrorReportingManager::CwchFormatAndPrint
sqlservr!scierrlog
sqlservr!alloca_probe
sqlservr!Np::PrepareForNextAccept
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
0x0
0x0
0x0
0x0
msvcr80!_threadstartex
msvcr80!_threadstartex
msvcr80!__initialmbcinfo
0x0

=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 9.00.5000.00
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is AT/AT COMPATIBLE.
Current time is 20:25:03 04/29/13.
4 Unknown CPU 9., 2660 Mhz processor (s).
Windows NT 5.2 Build 3790 CSD Service Pack 2.

Memory
MemoryLoad = 98%
Total Physical = 16383 MB
Available Physical = 211 MB
Total Page File = 19864 MB
Available Page File = 3580 MB
Total Virtual = 8388607 MB
Available Virtual = 8371825 MB
**Dump thread – spid = 0, PSS = 0x0000000000000000, EC = 0x0000000000000000
***Stack Dump being sent to E:\MSSQL.1\MSSQL\LOG\SQLDump0008.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 04/29/13 20:26:07 spid 0
*
* Non-yielding IOCP Listener
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Can you advise if this is related to disk i/o performance or memory ?

Reply
- Karthick P.K said
  
  April 30, 2013 at 5:14 AM
  Hello KK,
  This is side effect of memory pressure on the box. What other applications are running in this server ? You have to control the memory usage by other applications (or) increase physical memory in the server and max server memory. Review your error logs and let me know if you see any of the memory errors mentioned in https://mssqlwiki.com/sqlwiki/sql-performance/troubleshooting-sql-server-memory/
  
  Reply
Danni said

May 7, 2013 at 1:24 PM
Hi Karthick,

thank you for that nice Tutorial.
I couldn’t find any stack that matches. Maybe you can help me..

Errorlog:
2013-04-30 18:30:23.83 Server **Dump thread – spid = 0, EC = 0x00000000
2013-04-30 18:30:23.83 Server ***Stack Dump being sent to c:\Programme\Microsoft SQL Server\MSSQL10_50.DESIGO\MSSQL\LOG\SQLDump0007.txt
2013-04-30 18:30:23.83 Server * *******************************************************************************
2013-04-30 18:30:23.83 Server *
2013-04-30 18:30:23.83 Server * BEGIN STACK DUMP:
2013-04-30 18:30:23.83 Server * 04/30/13 18:30:23 spid 3688
2013-04-30 18:30:23.83 Server *
2013-04-30 18:30:23.83 Server * Non-yielding IOCP Listener
2013-04-30 18:30:23.83 Server *
2013-04-30 18:30:23.83 Server * *******************************************************************************
2013-04-30 18:30:23.83 Server * ——————————————————————————-
2013-04-30 18:30:23.83 Server * Short Stack Dump
2013-04-30 18:30:23.84 Server Stack Signature for the dump is 0x00000180
2013-04-30 18:30:25.66 Server External dump process return code 0x20000001.
External dump process returned no errors.

2013-04-30 18:30:25.66 Server 2013-04-30 20:03:35.68 Server 2013-04-30 20:03:45.71 Server 2013-04-30 21:02:40.75 Server 2013-04-30 23:29:15.79 Server 2013-05-01 00:29:50.81 Server 2013-05-01 00:30:00.84 Server 2013-05-01 02:02:00.87 Server 2013-05-01 02:02:10.92 Server 2013-05-01 03:30:00.95 Server 2013-05-01 03:30:10.98 Server 2013-05-01 13:29:13.14 Server 2013-05-01 14:29:23.18 Server 2013-05-01 16:01:28.20 Server 2013-05-01 22:01:53.23 Server 2013-05-01 22:02:03.27 Server 2013-05-01 23:01:13.29 Server 2013-05-01 23:01:23.32 Server 2013-05-02 08:30:18.34 Server 2013-05-02 09:30:38.39 Server 2013-05-02 10:29:53.42 Server 2013-05-02 11:31:13.46 Server 2013-05-02 13:03:48.49 Server 2013-05-02 14:03:23.51 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25046.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25046.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.

Debugging:
0:004> .cxr sqlservr!g_copiedStackInfo+0X20
eax=03b6f8e0 ebx=7c94845c ecx=00000246 edx=0000001b esi=03b6f90c edi=03d8e988
eip=00000000 esp=00000000 ebp=00000023 iopl=0 nv up di pl nz na po nc
cs=0000 ss=0010 ds=b34e es=dc08 fs=0000 gs=0000 efl=00000000
00000000 ?? ???
0:004> kc 100

WARNING: Frame IP not in any known module. Following frames may be wrong.
0x0
0:004> kc 100

WARNING: Frame IP not in any known module. Following frames may be wrong.
0x0
0:004> .cxr
Resetting default scope
0:004> .cxr sqlservr!g_copiedStackInfo+0X00c
eax=03d8e988 ebx=00000000 ecx=0000b34e edx=7c94dc08 esi=00000000 edi=00bc8048
eip=7c94845c esp=03b6f8e0 ebp=03b6f90c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c94845c c3 ret
0:004> kc 100

ntdll!KiFastSystemCallRet
ntdll!NtRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadStart

Thank you for your Help

Reply
- Karthick P.K said
  
  May 7, 2013 at 3:12 PM
  Hello Danni,
  This is 32bit SQLServer or 64-Bit SQLServer?
  
  Reply
  - Danni said
    
    May 7, 2013 at 5:40 PM
    It’s 32-Bit
    
    Reply
    - Karthick P.K said
      
      May 7, 2013 at 11:41 PM
      Ok. In debugger rype thebelow commands andshare the output
      ~~[0xe84]s
      kc100
      
      Reply
- Karthick P.K said
  
  May 7, 2013 at 3:14 PM
  Also can you do ~~[0xe84]s and then kc100 and share the output
  
  Reply
  - Danni said
    
    May 8, 2013 at 6:04 AM
    Here it is:
    
    0:004> ~~[0xe84]s
    eax=053d3ff0 ebx=00000000 ecx=00000001 edx=00001fb0 esi=000002e8 edi=00000000
    eip=7c94845c esp=3361f16c ebp=3361f1e0 iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
    ntdll!KiFastSystemCallRet:
    7c94845c c3 ret
    0:013> kc 100
    
    ntdll!KiFastSystemCallRet
    ntdll!NtSignalAndWaitForSingleObject
    kernel32!SignalObjectAndWait
    sqlservr!SOS_Scheduler::Switch
    sqlservr!SOS_Scheduler::SwitchContext
    sqlservr!SOS_Scheduler::SuspendNonPreemptive
    sqlservr!SOS_Scheduler::Suspend
    sqlservr!EventInternal<Spinlock >::Wait
    sqlservr!ResQueueBase::Dequeue
    sqlservr!ResQueue::Dequeue
    sqlservr!systemTaskThread
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadStart
    
    Reply
    - Karthick P.K said
      
      May 10, 2013 at 6:45 AM
      Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
      
      From the above warming non-yielding thread is not consuming CPU at all (kernel 0 ms, user 0 ms).
      
      I think garbage collector is suspending all the other threads.
      
      type ~*kc 100 and search for mscorwks!SVR::GCHeap
      
      If you find atleast one garbage collector thread apply Apply the .Net fix mentioned in http://support.microsoft.com/kb/2504603
      
      If you dont find any garbage collector thread let me know.
      
      Reply
Adam Angelini said

May 16, 2013 at 7:42 PM
Do you knwo what cause this one? SQL2012 sp1 enterprise x64 bit

Call Site
ntdll!ZwWaitForSingleObject
ntdll!RtlpWaitOnCriticalSection
ntdll!RtlEnterCriticalSection
ws2_32!closesocket
sqllang!Tcp::FCloseTransport
sqllang!SNI::detail::Transport::FCloseRefHandle
sqllang!Tcp::Close
sqllang!Smux::InternalClose
sqllang!Smux::ReadDone
sqllang!SNIReadDone
sqldk!SOS_Node::ListenOnIOCompletionPort
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  May 17, 2013 at 5:49 AM
  Hello Adam,
  
  which windows version are you using?
  
  You need to apply the windows fix mentioned in below KB. (windows fix in KB is applicabel for SQL2012 as well)
  
  http://support.microsoft.com/kb/2491214/en-US
  
  {
  Hotfix for Windows Server 2008 R2
  976932 Information about Service Pack 1 for Windows 7 and for Windows Server 2008 R2
  Hotfix for Windows Server 2008
  2398202 An application may stop responding in Windows Server 2008 or in Windows Vista when the application closes a socket connection or shuts down
  }
  
  Reply
tilak said

May 31, 2013 at 8:57 AM
Hi Karthick,

Can you please let me know the cause of this. SQL version is 10.0.2531

Call Site
ntdll!NtWriteFile
KERNELBASE!WriteFile
kernel32!WriteFileImplementation
sqlservr!DiskWriteAsync
sqlservr!CTraceIoBuf::IncrementalFlush
sqlservr!CTraceFileIoProvider::IncrementalFlush
sqlservr!CTraceController::IncrementalFlush
sqlservr!CTraceManagementTask::Invoke
sqlservr!CTraceBatchTask::Task
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
0x0
0x0

Thanks,
Tilak M

Reply
- Karthick P.K said
  
  May 31, 2013 at 9:14 AM
  It is because of the profiler trace you are running.
  
  Reply
Raka said

July 2, 2013 at 10:14 AM
Hi,
could you please help me to fix this error

Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’, id 000000008082EEF0, type 2, Task 0x000000000568B948 : 0, waittime 600, flags 0x4018, owning task 0x0000000000000000. Continuing to wait.
2013-06-30 13:30:08.55 spid54 Time out occurred while waiting for buffer latch — type 4, bp 0000000084FDD740, page 1:412488, stat 0xc0000b, database id: 2, allocation unit id: 6488064, task 0x00000002E970D048 : 0, waittime 6900, flags 0x1000000039, owning task 0x00000000056A4E08. Continuing to wait.

Reply
- Karthick P.K said
  
  July 2, 2013 at 11:35 AM
  Hello Raka,
  
  This is latch timeout dump and you should be following https://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ and get the stack of owning task. Steps documented in above blog are very simple.
  
  You have posted two errors
  
  Error 1 Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’
  
  1. Check if there is any “The operating system returned error” mentioned in error log. 2. Check if data file is auto growing or I/O request taking longer than 60 seconds warning
  
  Error 2 Time out occurred while waiting for buffer latch — type 4, bp 0000000084FDD740, page 1:412488, stat 0xc0000b, database id: 2, allocation unit id: 6488064, task 0x00000002E970D048 : 0, waittime 6900, flags 0x1000000039, owning task 0x00000000056A4E08. If you look at above warning carefully we are waiting to occur latch on PFS page 1:412488 of tempdb (every 8088th page is PFS page). Did the error message disappear after some time?
  
  We can follow below steps to reduce contention on PFS page but if you find continuous time out on same pages (1:412488) in error log you have to find the stack of thread which is owning the latch on page 1:412488 follow https://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ in that case and share the stack else follow below steps.
  
  1.Increase the number of Tempdb data files to 8, size them equally and place them in different physical disk and then if contention continues, increase the number of data files by multiples of 4 (up to the number of logical processors) until the contention is reduced to acceptable levels or make changes to the workload/code.
  
  2. Enable trace flag 1118 Related KB: http://support.microsoft.com/kb/2154845
  
  Thanks Karthick
  
  Reply
Raka said

July 2, 2013 at 1:21 PM
Hi Karthick,

Thanks for reply.
we are facing performance issue and for that time iam able to see these error in sql server logs.

Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’

Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’, id 000000008082EEF0, type 2, Task 0x000000000568B948 : 0, waittime 600, flags 0×4018, owning task 0×0000000000000000. Continuing to wait.
2013-06-30 13:30:08.55 spid54 Time out occurred while waiting for buffer latch — type 4, bp 0000000084FDD740, page 1:412488, stat 0xc0000b, database id: 2, allocation unit id: 6488064, task 0x00000002E970D048 : 0, waittime 6900, flags 0×1000000039, owning task 0x00000000056A4E08. Continuing to wait.

no there is not any operating system related error in logs.

i have followed the steps suggested by you and found below logs but unable to find reason behind these errors.
could you please help me to expalin these error and how to resolve these.

Thanks,
Raka

157 Id: a78.3378 Suspend: 0 Teb: 000007ff`fff2e000 Unfrozen
# Child-SP RetAddr Call Site
00 00000000`42f57a58 00000000`77a44327 ntdll!NtSignalAndWaitForSingleObject+0xa
01 00000000`42f57a60 00000000`00194b08 kernel32!SignalObjectAndWait+0x117
02 00000000`42f57b20 00000000`00192d76 sqlservr!SOS_Scheduler::Switch+0x181
03 00000000`42f58270 00000000`00192700 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xca
04 00000000`42f582b0 00000000`001929ac sqlservr!SOS_Scheduler::Suspend+0x2d
05 00000000`42f582e0 00000000`001f7e46 sqlservr!EventInternal<Spinlock >::Wait+0x1a8
06 00000000`42f58330 00000000`001f7a29 sqlservr!LatchBase::Suspend+0x599
07 00000000`42f58fa0 00000000`001b0689 sqlservr!LatchBase::AcquireInternal+0x1ff
08 00000000`42f59040 00000000`001b1792 sqlservr!BUF::AcquireLatch+0x8d
09 00000000`42f59360 00000000`001b166c sqlservr!BPool::Get+0xc7
0a 00000000`42f593c0 00000000`0074d0ea sqlservr!PageRef::Fix+0xbc
0b 00000000`42f59420 00000000`0074cb53 sqlservr!DeAllocMgr::DropAllocUnitInternal+0x642
0c 00000000`42f59790 00000000`0066dd5c sqlservr!DropAllocationUnit+0x18b
0d 00000000`42f59850 00000000`0027c167 sqlservr!DeAllocateIndexNew+0x539
0e 00000000`42f59aa0 00000000`001f6e20 sqlservr!SETruncateWorktableRowset+0x777
0f 00000000`42f59eb0 00000000`00614673 sqlservr!SETruncateRowset+0x144
10 00000000`42f5d6f0 00000000`0060db49 sqlservr!RowsetWorkTableSS::Truncate+0x43
11 00000000`42f5d770 00000000`0060e5bb sqlservr!SqlProfTruncateWorktableRowset+0x50
12 00000000`42f5d7b0 00000000`01a8e82b sqlservr!CQScanSpoolNew::ChangeSpoolState+0xbc
13 00000000`42f5d7e0 00000000`0024f7db sqlservr!CQScanSpoolNew::Dormant+0x5f
06 0000000

Reply
- Karthick P.K said
  
  July 5, 2013 at 5:34 PM
  Hello Raka,
  Which command did you run?
  your owning task is “owning task 0x00000000056A4E08” from error.
  so you should run
  ~*e .echo ThreadId:; ?? @$tid; r? @$t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? @$t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; s -d @$t1 @$t2 0x00000000056A4E08
  
  and do kL only on threads which as reference to owning task.
  
  Reply
halabj said

July 16, 2013 at 11:36 PM
we got performace issue fo about 40 minutes, and created this:

ntdll!KiFastSystemCallRet
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
kernel32!WaitForSingleObject
sqlservr!SOS_Scheduler::SwitchContext
sqlservr!SOS_Scheduler::SuspendNonPreemptive
sqlservr!SOS_Scheduler::Suspend
sqlservr!SOS_Event::Wait
sqlservr!LatchBase::Suspend
sqlservr!LatchBase::AcquireInternal
sqlservr!CSbReentrantAutoLatch::Acquire
sqlservr!CSbTransmissionStoreNotifications::SteppedOnEntryCommon
sqlservr!ClockHand::Move
sqlservr!ClockAlgorithm::MoveHand
sqlservr!ClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadStart

Reply
- Brian Kang said
  
  July 17, 2013 at 4:44 AM
  Hi Karthik, We just got below dump and followed your instruction but i can’t find a matcing case so wonder if you can help us why we got this. Thanks much as always!! -Brian 2013-07-15 18:55:24.64 Server **Dump thread – spid = 0, EC = 0x0000000000000000 2013-07-15 18:55:24.95 Server ***Stack Dump being sent to E:\Microsoft SQL Server\MSSQL11.INT\MSSQL\LOG\SQLDump0001.txt 2013-07-15 18:55:24.97 Server * ******************************************************************************* 2013-07-15 18:55:24.97 Server * 2013-07-15 18:55:24.97 Server * BEGIN STACK DUMP: 2013-07-15 18:55:24.97 Server * 07/15/13 18:55:24 spid 2080 2013-07-15 18:55:24.97 Server * 2013-07-15 18:55:24.97 Server * Non-yielding Resource Monitor 2013-07-15 18:55:24.97 Server * 2013-07-15 18:55:24.97 Server * ******************************************************************************* 2013-07-15 18:55:24.97 Server * ——————————————————————————- 2013-07-15 18:55:24.97 Server * Short Stack Dump 2013-07-15 18:55:29.84 Server Stack Signature for the dump is 0x0000000000000268 2013-07-15 18:59:11.79 Server Timeout waiting for external dump process 9612. 2013-07-15 18:59:11.93 Server Resource Monitor (0x858) Worker 0x000000047F688160 appears to be non-yielding on Node 0. Memory freed: 6640 KB. Last wait: MISCELLANEOUS. Approx CPU Used: kernel 15 ms, user 0 ms, Interval: 60423.
  
  0:005> .cxr sqlmin!g_copiedStackInfo+0X20 rax=000007ff83773480 rbx=00000003ac59c060 rcx=000000046b078f80 rdx=000000023ac91ec0 rsi=0000000000000000 rdi=0000000000000000 rip=000007ff837aaa36 rsp=000000000684e950 rbp=0000000000000000 r8=000000000684ea30 r9=0000000000000000 r10=000000043688afe0 r11=000000000684ea20 r12=0000000000000000 r13=7fffffffffffffff r14=000000000000004f r15=7fffffffffffffff iopl=0 nv up ei pl nz na pe nc cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00010202 sqlmin!RowsetNewSS::ReleaseAccessor+0x6: 000007ff`837aaa36 4883baa800000000 cmp qword ptr [rdx+0xa8],0x0 ds:00000002`3ac91f68=???????????????? 0:005> kc 100 Child-SP RetAddr Call Site 00000000`0684e950 000007ff`837ab187 sqlmin!RowsetNewSS::ReleaseAccessor+0x6 00000000`0684e980 000007ff`837aa9c8 sqlmin!CAccessorRelease::ReleaseAccessor+0x10a 00000000`0684ea30 000007ff`837ab0b7 sqlmin!CAccessorRelease::`scalar deleting destructor’+0x18 00000000`0684ea60 000007ff`837aad33 sqlmin!CAccessorRelease::Release+0x17 00000000`0684ea90 000007ff`837aabce sqlmin!CQueryExecContext::~CQueryExecContext+0x23a 00000000`0684ece0 000007ff`837ab627 sqlmin!CQueryExecContext::`scalar deleting destructor’+0xe 00000000`0684ed10 000007ff`837ab81b sqlmin!CQueryScan::ShutdownQueryExecContext+0x76 00000000`0684ed40 000007ff`816cbe61 sqlmin!CQueryScan::DestroyQuery+0x1e2 00000000`0684ed90 000007ff`81bb1369 sqllang!CExecuteStatement::Destroy+0x61 00000000`0684edc0 000007ff`8247e135 sqllang!CMetaStmtSet::DeleteXstmtsInPartition+0x74 00000000`0684ee10 000007ff`8247fdd0 sqllang!CMetaStmtSet::StepOnStatements+0x106 00000000`0684ee70 000007ff`82452ad8 sqllang!CCompPlan::StepOnStatements+0x30 ^ Extra character error in ‘kc 100’
  
  Date: Tue, 16 Jul 2013 18:06:20 +0000 To: bsk64@hotmail.com
  
  Reply
  - Karthick P.K said
    
    August 1, 2013 at 12:01 PM
    Hello Brian,
    
    I reviewed the error log and memory dump which you send me. Non-yielding Resource Monitor dump is just a side effect of memory pressure on the system.
    
    When there was non-yielding dump just 10% of overall memory used by SQL Server was in RAM and remaining was in page file. Look at the error message in SQL Server just before the dump.
    
    Total memory used by SQL Server is 12080092 KB (12GB) and only 10 % of this memory is from RAM (Working set (KB): 1246168). So SQL Server has started responding to the memory pressure and releasing the memory back to OS, at the same time windows working set manger have started trimming the working set (Page out)all the processes including the working of SQL Server process. When the critical portions of SQL Server memory is paged it takes long time for SQL Server to respond to memory pressure and scale down its memory usage. I also see garbage collector is kicked and running (you are using CLR). When Garbage collector is kicked off during memory pressure all other threads in the process are suspended. So if Garbage collector is taking a long time resource monitor thread appears stuck and hence the non-yielding errors and dumps are generated. Look at error immediately after the dump resource monitor thread has got only 15ms of CPU time in interval of 60.5 seconds.
    
    { 2013-07-15 18:54:10.86 spid13s A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 333 seconds. Working set (KB): 1246168, committed (KB): 12080092, memory utilization: 10%.
    
    Resource Monitor (0x858) Worker 0x000000047F688160 appears to be non-yielding on Node 0. Memory freed: 6640 KB. Last wait: MISCELLANEOUS. Approx CPU Used: kernel 15 ms, user 0 ms, Interval: 60423. }
    
    How to fix this ?
    
    Look at the SQLDump0001.txt. I have copied a small portion for your reference. Total memory on this server is 32 GB and just 700 Mb is free. SQL Server is consuming around 1GB in RAM .Identify who is consuming remaining memory and fix that process. { MemoryLoad = 97% Total Physical = 32767 MB Available Physical = 773 MB }
    
    1. You can follow the steps in https://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/ 2. Since this is a VMware box I suspect that this issue could be because of VMware ballooning. Make sure you disable ballooning. More details in https://mssqlwiki.com/2013/03/31/sql-server-and-vmware-ballooning/ 3. You can temporarily use Locked pages in memory till you identify the process/Driver which is consuming most of the memory in the server making SQL Server and windows to starve for memory.
    
    Reply
- Karthick P.K said
  
  July 17, 2013 at 12:30 PM
  Hello Halabj,
  I know your issue existed in SQLServer2005 SP2 and got fixed in Cumulative update package 13 for SQL Server 2005 Service Pack 2.
  Are you on SQL2005 with out latest service pack? Apply latest SP for 2005 and you will not face it.
  
  @Brian Kang You have replied to Halabj comment instead of new comment. so you are getting this mail. Ignore this I will look at your stack in a while.
  
  Reply
  - halabj said
    
    July 17, 2013 at 3:13 PM
    indeed, we are at SP4 and altest hotfix, 9.00.5324. Anathor note is this is 32 bits machine
    
    Reply
Jose Ramon Vela Martin said

August 28, 2013 at 9:24 PM
Hello I have one SQL stack, is similar like you but is not the same, Could anybody help me?
ç

0:005> kC
*** Stack trace for last set context – .thread/.cxr resets it
Call Site
ntdll!ZwWaitForSingleObject
kernel32!WaitForSingleObjectEx
sqlservr!SystemThread::Suspend
sqlservr!SOS_Scheduler::SuspendPreemptive
sqlservr!SOS_Event::Wait
sqlservr!LatchBase::Suspend
sqlservr!LatchBase::AcquireInternal
sqlservr!CAutoLatch::GetAccess
sqlservr!BPool::ReplenishFreeList
sqlservr!BPool::Steal
sqlservr!SQLSinglePageAllocator::AllocatePages
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CMemThread::PbGetNewPages
sqlservr!CMemFixed::Alloc
sqlservr!CMemThread::Alloc
sqlservr!operator new
sqlservr!SOS_Node::EnqueueTaskDirectInternal
sqlservr!SOS_Node::EnqueueTaskDirect
sqlservr!TDSSNIClient::AcceptCommon

ection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is AT/AT COMPATIBLE.
Current time is 09:10:11 08/25/13.
2 Unknown CPU 9., 2800 Mhz processor (s).
Windows NT 5.2 Build 3790 CSD Service Pack 2.

Memory
MemoryLoad = 95%
Total Physical = 4095 MB
Available Physical = 184 MB
Total Page File = 9974 MB
Available Page File = 4497 MB
Total Virtual = 8388607 MB
Available Virtual = 8384006 MB
**Dump thread – spid = 0, PSS = 0x0000000000000000, EC = 0x0000000000000000
***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0083.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 08/25/13 09:10:11 spid 0
*
* Non-yielding IOCP Listener
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

ate 25/08/2013 8:30:34
Log SQL Server (Archive #1 – 25/08/2013 10:05:00)

Source spid2s

Message
CACHESTORE_STACKFRAMES (Total)
VM Reserved = 0 KB
VM Committed = 0 KB
AWE Allocated = 0 KB
SM Reserved = 0 KB
SM Committed = 0 KB
SinglePage Allocator = 0 KB
MultiPage Allocator = 8 KB

Reply
- Jose Ramon Vela Martin said
  
  September 2, 2013 at 9:07 PM
  More info—Please does anyone know this issue???
  
  0:006> kP
  Child-SP RetAddr Call Site
  00000000`047cf018 00000000`77d6d03b ntdll!ZwWaitForMultipleObjects+0xa
  00000000`047cf020 00000000`77d6bb41 kernel32!WaitForMultipleObjectsEx+0x1cf
  00000000`047cf140 00000000`014f6b64 kernel32!WaitForMultipleObjects+0x11
  00000000`047cf180 00000000`02b449d7 sqlservr!ResourceMonitor::CheckIndicators+0x184
  00000000`047cf230 00000000`02b49a51 sqlservr!ResourceMonitor::ResourceMonitorTask+0x127
  00000000`047cf360 00000000`015c483e sqlservr!SetupResourceMonitorTaskContext+0x481
  00000000`047cfaf0 00000000`015c4ad9 sqlservr!SOS_Task::Param::Execute+0xee
  00000000`047cfc00 00000000`015bb254 sqlservr!SOS_Scheduler::RunTask+0xc9
  00000000`047cfc90 00000000`014d4687 sqlservr!SOS_Scheduler::ProcessTasks+0xb4
  00000000`047cfd00 00000000`017be7b9 sqlservr!SchedulerManager::WorkerEntryPoint+0xe7
  00000000`047cfda0 00000000`017be585 sqlservr!SystemThread::RunWorker+0x59
  00000000`047cfde0 00000000`017eade8 sqlservr!SystemThreadDispatcher::ProcessWorker+0xb5
  00000000`047cfe80 00000000`781337d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x128
  00000000`047cff20 000007ff`fffa9498 msvcr80!_callthreadstartex(void)+0x17 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 348]
  00000000`047cff28 00000000`005732f0 0x7ff`fffa9498
  00000000`047cff30 00000000`005732f0 0x5732f0
  00000000`047cff38 00000000`005adce0 0x5732f0
  00000000`047cff40 00000000`78133810 0x5adce0
  00000000`047cff48 00000000`78133894 msvcr80!_threadstartex(void) [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 260]
  00000000`047cff50 00000000`781e95c0 msvcr80!_threadstartex(
  void * ptd = 0x00000000`77d6b6e0)+0x84 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 326]
  
  Reply
  - Karthick P.K said
    
    September 3, 2013 at 11:21 AM
    Jose,
    What is your sqlserver build number (@@version)?
    
    Reply
    - José Ramón Vela said
      
      September 3, 2013 at 1:18 PM
      Hi,
      My cuurent versión is SQL Server 2005 Service Pack 4 9.00.5000.00.
      Thaks for reply.
      
      Reply
      - Karthick P.K said
        
        September 3, 2013 at 1:28 PM
        Send the dump and error log to me. I will look at it tomorrow
        
        Sent from my Windows Phone ________________________________
        
        Reply
        
        Jose Ramon Vela Martin said
        
        September 3, 2013 at 2:39 PM
        You have in your hotmail inbox the log files (dmp and log)
        
        Thank you very much for your help!
        
        Reply
        
        Karthick P.K said
        
        September 4, 2013 at 6:29 PM
        Hello Jose,
        
        I reviewed the dumps and log which you uploaded.
        
        Set the SQL server max server memory to 3GB and you should not see the problem again. Non-Yieding scheduler issue which you faced was just a side effect of memory pressure on the system.
        You can use the script below to set max server memory.
        
        {
        EXEC sys.sp_configure N’max server memory (MB)’, N’3096′
        GO
        RECONFIGURE WITH OVERRIDE
        GO
        }
        
        Reply
        
        Jose Ramon Vela Martin said
        
        September 4, 2013 at 7:13 PM
        Thank you so much Karthick, You are the best support SQL Server!!!!!
        
        Reply
Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes - Karthick PK 's Blog - Site Home - MSDN Blogs said

January 11, 2014 at 10:53 AM
[…] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… […]

Reply
Top SQL Server blogs from MSSQLWIKI « MSSQLWIKI said

January 11, 2014 at 12:56 PM
[…] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… […]

Reply
Mudit said

January 15, 2014 at 2:39 AM
Hi Karthick,
I am facing similar problem, occurrences of bugcheck dump in sql server logs. I asked same question in msdn forum where you mentioned using winbdg tool.
Thank you,

Here is the Stack file:

sqltses!CEsCompValSeg::DestroyExecValSeg
sqltses!CEsExec::`vector deleting destructor’
sqltses!CEsRuntime::DestroyPesx
sqlmin!RowsetNewSS::~RowsetNewSS
sqlmin!RowsetNewSS::`vector deleting destructor’
sqlmin!RowsetNewSS::Release
sqlmin!CQueryExecContext::~CQueryExecContext
sqlmin!CQueryScan::ShutdownQueryExecContext
sqlmin!CQueryScan::DestroyQuery
sqllang!CExecuteStatement::Destroy
sqllang!CMetaStmtSet::DeleteXstmtsInPartition
sqllang!CMetaStmtSet::StepOnStatements
sqllang!CCompPlan::StepOnStatements
sqllang!CCacheStoreCPCallback::SteppedOnEntry
sqldk!CacheClockHand::MoveInternal
sqldk!CacheClockHand::Move
sqldk!TClockAlgorithm<CacheClockHand >::StartExternalPressure
sqldk!ResourceMonitor::NotifyMemoryConsumers
sqldk!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

And here is the bugcheck dump file:

Current time is 18:16:11 01/12/14.
=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 11.0.3393.0
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is AMD Opteron(TM) Processor 6276 .
Bios Version is INTEL – 6040000
PhoenixBIOS 4.0 Release 6.0
1 X64 level 8664, 14 Mhz processor (s).
Windows NT 6.2 Build 9200 CSD .

Memory
MemoryLoad = 75%
Total Physical = 4095 MB
Available Physical = 1023 MB
Total Page File = 6319 MB
Available Page File = 1725 MB
Total Virtual = 8388607 MB
Available Virtual = 8378657 MB
**Dump thread – spid = 0, EC = 0x0000000000000000
***Stack Dump being sent to E:\MSSQL\Data\MSSQL11.MSSQLSERVER\MSSQL\LOG\SQLDump0013.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 01/12/14 18:16:11 spid 4088
*
* Non-yielding Resource Monitor
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Reply
- Karthick P.K said
  
  January 15, 2014 at 3:47 PM
  Hello Mudit,
  Mail the dump to me and sql server error log. I will let you know the next steps. Basically your resource monitor thread is stuck trying to clear the procedure cache.
  
  Reply
  - Mudit said
    
    January 15, 2014 at 8:28 PM
    Hi Karthick,
    I sent an email to karthick_pk83 at hotmaildotcom. Thanks for looking into this issue.
    I am running dbcc freeproccache on this server. Trying to see, after clear cache, how long it take for resource monitor messages to reappear.
    Thanks,
    
    Reply
m60freeman said

January 22, 2014 at 12:27 AM
I am working with a SQL Server 2005 instance. I installed the debugger and followed the instructions, but I get this:

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.mdmp]
User Mini Dump File: Only registers, stack and portions of memory are available

Comment: ‘Stack Trace’
Comment: ‘Deadlocked Schedulers’
Symbol search path is: *** Invalid ***
****************************************************************************
* Symbol loading may be unreliable without a symbol search path. *
* Use .symfix to have the debugger choose a symbol path. *
* After setting your symbol path, use .reload to refresh symbol locations. *
****************************************************************************
Executable search path is:
Windows 7 Version 7601 (Service Pack 1) UP Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Thu Jan 16 15:06:51.000 2014 (GMT-5)
System Uptime: 1 days 11:40:53.029
Process Uptime: 1 days 11:41:45.000
……………………………………………………….
…………………………………………..
Loading unloaded module list
….
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(610.708): Unknown exception – code 00000000 (first/second chance not available)
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=0301fc28 edi=00000000
eip=7737fd91 esp=0301fbe4 ebp=0301fc4c iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll –
ntdll!NtDelayExecution+0x15:
7737fd91 83c404 add esp,4
0:004> .sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
Symbol search path is: srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
0:004> .reload /f
.

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

…………………………*** ERROR: Symbol file could not be found. Defaulted to export symbols for instapi.dll –
..*** ERROR: Module load completed but symbols could not be loaded for sqlevn70.rll
………………*** ERROR: Module load completed but symbols could not be loaded for comres.dll
……*** ERROR: Symbol file could not be found. Defaulted to export symbols for msfte.dll –
…….
…………..*** ERROR: Module load completed but symbols could not be loaded for sqlnclir.rll
…*** ERROR: Symbol file could not be found. Defaulted to export symbols for msftepxy.dll –
…….*** ERROR: Module load completed but symbols could not be loaded for odbcint.dll
.*** ERROR: Module load completed but symbols could not be loaded for XPStar90.RLL
..*** ERROR: Module load completed but symbols could not be loaded for xplog70.rll
….*** ERROR: Module load completed but symbols could not be loaded for oledb32r.dll
…*** WARNING: Unable to verify checksum for _IBProvider_v2_i.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for _IBProvider_v2_i.dll –
.*** WARNING: Unable to verify checksum for gds32.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for gds32.dll –
..*** WARNING: Unable to verify checksum for cc3250mt.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for cc3250mt.dll –
………….
Loading unloaded module list
….
0:004> .cxr sqlservr!g_copiedStackInfo+0X20
Unable to read context, Win32 error 0n30
0:004> kc 100

ntdll!NtDelayExecution
KERNELBASE!SleepEx
KERNELBASE!Sleep
sqlservr!SchedulerMonitor::Run
sqlservr!SchedulerMonitor::EntryPoint
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_endthreadex
msvcr80!_endthreadex
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

Reply
- m60freeman said
  
  January 22, 2014 at 12:32 AM
  I tried lmvm sqlservr and got this result:
  
  start end module name
  01000000 02c27000 sqlservr (pdb symbols) C:\Program Files (x86)\Debugging Tools for Windows (x86)\sym\sqlservr.pdb\C6512984CD7C4067AB9E7B96653A666F2\sqlservr.pdb
  Loaded symbol image file: sqlservr.exe
  Mapped memory image file: C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe
  Image path: C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe
  Image name: sqlservr.exe
  Timestamp: Fri Dec 10 17:23:31 2010 (4D02A863)
  CheckSum: 01BFD679
  ImageSize: 01C27000
  File version: 2005.90.5000.0
  Product version: 9.0.5000.0
  File flags: 0 (Mask 3F)
  File OS: 40000 NT Base
  File type: 1.0 App
  File date: 00000000.00000000
  Translations: 0409.04e4
  CompanyName: Microsoft Corporation
  ProductName: Microsoft SQL Server
  InternalName: SQLSERVR
  OriginalFilename: SQLSERVR.EXE
  ProductVersion: 9.00.5000.00
  FileVersion: 2005.090.5000.00
  FileDescription: SQL Server Windows NT
  LegalCopyright: © Microsoft Corp. All rights reserved.
  LegalTrademarks: Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
  Comments: NT INTEL X86
  
  I then tried !findstack sqlservr!SQL_SOSNonYieldSchedulerCallback, but it didn’t return a result.
  
  Reply
  - Karthick P.K said
    
    January 22, 2014 at 2:43 PM
    Hello Mark,
    
    I looked at the dump. You can ignore this dump if it is one time occurrence. I believe you haven’t changed the default max worker threads. If you see this dump again or if it has happened many times before then please let me know.
    
    Thanks Karthick
    
    Reply
    - m60freeman said
      
      January 22, 2014 at 7:59 PM
      Thank you very much for taking the time to look at the files I sent. I have not changed the default max worker threads. Would doing so help to avoid a future such problem with this instance?
      
      Do you have any suggestions as to what I can do to be able to debug these dumps so I do not get the “Unable to read context, Win32 error 0n30” error? I’d like to be able to analyze future dumps without bothering you.
      
      Thanks, Mark
      
      Reply
      - Karthick P.K said
        
        January 23, 2014 at 2:15 PM
        Mark,
        Reason you did not find “findstack sqlservr!SQL_SOSNonYieldSchedulerCallback” is because this dump was “Deadlocked Schedulers” dump (SQLDump0001.txt/errorlog will have the type), so ideally you should have followed https://mssqlwiki.com/2010/06/15/how-to-analyze-deadlocked-schedulers-dumps/
        
        Reply
Lina said

January 23, 2014 at 1:32 PM
Hi,

Could you please take a look to my stack:

msxmlsql!assignRO
msxmlsql!ChildNodeSet::advance
msxmlsql!AdvanceNodeSet::nextElement
msxmlsql!FilterMatchNodeSet::getIndex
msxmlsql!FilterMatcher::matches
msxmlsql!FilterQuery::matches
msxmlsql!ChildQuery::matches
msxmlsql!DescendantSelector::selectNextDescendant
msxmlsql!DescendantNodeSet::advance
msxmlsql!AdvanceNodeSet::nextElement
msxmlsql!XSLPatternWrapper::nextNode
sqlservr!CXMLRowset::FGetNextRow
sqlservr!CXMLRowset::GetNextRows
sqlservr!CQScanRmtScanNew::GetRowHelper
sqlservr!CQScanCountStarNew::GetRowHelper
sqlservr!CQScanStreamAggregateNew::GetCalculatedRow
sqlservr!CQScanStreamAggregateNew::Open
sqlservr!CQueryScan::Uncache
sqlservr!CXStmtQuery::SetupQueryScanAndExpression
sqlservr!CXStmtQuery::InitForExecute
sqlservr!CXStmtQuery::ErsqExecuteQuery
sqlservr!CXStmtSelect::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

I have Microsoft SQL Server 2008R2 SP2.
Cumulative update package 1 has been installed.

Thanks!

Reply
- Karthick P.K said
  
  January 23, 2014 at 2:19 PM
  Hello Polina,
  
  I think SQL server is Iterating through the Nodes of the XML document and the document is very large. How frequent the issue is? Can you reduce the size of XML document? Share the dump with me I will check If I can pull out the XML document Thanks Karthick
  
  Reply
  - Lina said
    
    January 23, 2014 at 4:34 PM
    Thanks Karthick!
    
    Yes you are right.
    We have a stored procedure which parses xml files. Files come from third party and we can’t affect the size.
    In normal situation there is no problem.
    Problem comes only if procedure execution is terminated. In our case it happens from application side by time out.
    I got same result when terminate procedure execution in Management Studio.
    Problem is not so critical for us, because we can control it by increasing time out.
    But we want to find better solution.
    
    I’ll send dump files by email.
    
    BR. Polina
    
    Reply
    - Karthick P.K said
      
      January 24, 2014 at 6:39 PM
      Hello Polina,
      
      I looked at the dumps and for me it looks like some sort of flaw in SQL. Let me confirm my understanding, I see oc_Set_Account_Statement_To_DB is executing. Do we get this dump when you kill the SP execution?
      
      I would need to see the repro to confirm.
      
      Which region are you in?
      
      Thanks
      
      Karthick
      
      Reply
      - Lina said
        
        January 27, 2014 at 1:46 PM
        Thank you Karthick.
        Yes, error can be reproduced by killing oc_Set_Account_Statement_To_DB stored procedure execution.
        I am from Finland.
        
        BR, Polina
        
        Reply
m60freeman said

January 24, 2014 at 12:01 AM
Karthick,

I have a stack dump from a Non-yielding Resource Monitor (Resource Monitor (0x5e4) Worker 0x7FFFC0E8 appears to be non-yielding on Node 0) on a SQL Server 2005 (32-bit) SP4 system. Using the debugger, I was able to get this from the dump:

sqlservr!CMxcContainer::UlDelete
sqlservr!CCacheStoreCPCallback::SteppedOnEntry
sqlservr!ClockHand::Move
sqlservr!ClockAlgorithm::MoveHand
sqlservr!ClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_endthreadex
msvcr80!_endthreadex
kernel32!BaseThreadInitThunk
ntdll!__RtlUserThreadStart
ntdll!_RtlUserThreadStart

This doesn’t match any of the stacks in your original post. What do you think might be causing it?

Thanks,
Mark

Reply
m60freeman said

January 24, 2014 at 12:25 AM
Karthick,

I have yet another stack dump. Really, I’m not trying to set a record. 🙂

This one is from a Non-yielding Resource Monitor (Resource Monitor (0x564) Worker 0x000000008004E1A0 appears to be non-yielding on Node 0) on a SQL Server 2008R2 (64-bit) SP2 system. (My other ones were from 2005 32-bit instances.) Using the debugger, I was able to get this from the dump:

Call Site
sqlservr!CExecuteStatement::Destroy
sqlservr!CMetaStmtSet::DeleteXstmtsInPartition
sqlservr!CMetaStmtSet::StepOnStatements
sqlservr!CCompPlan::StepOnStatements
sqlservr!CCacheStoreCPCallback::SteppedOnEntry
sqlservr!CacheClockHand::MoveInternal
sqlservr!CacheClockHand::Move
sqlservr!TClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

This doesn’t match any of the stacks in your original post. What do you think might be causing it?

Thanks,
Mark

Reply
- Karthick P.K said
  
  January 24, 2014 at 12:04 PM
  Hello Mark,
  
  If you look at the stack carefully, you will notice resource monitor thread is trying to clear the cache (Scaling down the memory usage) and could not complete the operation in expected time (Crossed 60 seconds without yielding), so certainly there should have been memory pressure on the system when this has happened, but that doesn’t mean thread can execute for long time without yielding. We have to look at the error log when the issue occurred , you will notice Kernel time, user time, process utilization next to ” appears to be non-yielding ” that will help us to understand if thread really did something (or) got affected because of some performance problem on OS. We also had some known issues with thread not yielding when there is large cache. Send me all the dumps, complete SQL Server error log and SQLDump00*.txt files. I will look at them and let you know the next steps.
  
  Thanks Karthick
  
  Reply
  - Karthick P.K said
    
    January 27, 2014 at 1:36 PM
    Hello Mark,
    
    Great you provided the logs.
    
    Previous analysis stand same. From the stack you would have noticed that the resource monitor is scaling down the memory usage of SQL Server. This certainly happens when the available memory on the system drops.
    Look at the error log just above the dump . You will see the below error message before the dump.
    A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 0 seconds. Working set (KB): 107520, committed (KB): 240840, memory utilization: 44%.”
    
    SQL Server is using only 240 MB, Just 107 Mb in RAM and remaining is paged out. Now you have to identify which other process is taking all the memory in the server, causing SQL Server to scale down its memory usage, making windows to move SQL Server working set to page file and fix that process. SQL Server is just a Victim in this scenario.
    Check if vmware balloming is taking the memory from server Follow: https://mssqlwiki.com/2013/03/31/sql-server-and-vmware-ballooning/ If it is not then we have to identify the culprit process (other process which is eating the memory on the sever) we have to follow https://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/
    
    Reply
Hugo Tremblay said

February 14, 2014 at 6:37 AM
I`m getting this one:
0:018> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqldk!SystemThread::Suspend
sqldk!SOS_Scheduler::SuspendPreemptive
sqldk!UnfairRecursiveMutexInternal::LongWait
sqldk!TAutoMutex<UnfairRecursiveMutexInternal,4294967295>::GetAccess
sqldk!PhysicalPageCache::AllocatePhysicalPages
sqldk!MemoryNode::AllocateUserPhysicalPages
sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality
sqldk!SOS_MemoryBlockAllocator::AllocateBlock
sqldk!SOS_MemoryWorkSpace::AllocatePage
sqldk!MemoryNode::AllocatePagesInternal
sqldk!MemoryClerkInternal::AllocatePages
sqldk!CMemThread::PbGetNewPages
sqldk!CSlotPageMgr::PbAllocate
sqldk!CMemThread::Alloc
sqllang
sqllang
sqllang
sqllang
sqllang
sqllang
sqllang
sqllang
sqldk!SOS_Node::ListenOnIOCompletionPort
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

This is version 11.0.3393

My errors are alternating between latch timeouts and non-yielding (the one I wrote about is a non-yielding)

Thanks in advance if you find something

Reply
- Karthick P.K said
  
  February 14, 2014 at 3:55 PM
  You have to apply Cumulative Update 9 for Microsoft SQL Server 2012 Service Pack 1 (SP1) which is yet to be released. You are facing the issue mentioned in https://support.microsoft.com/kb/2926223
  
  Reply
holisticdeveloper said

February 15, 2014 at 6:28 AM
I have the following dump from a non-yielding scheduler. This is on 10.0.5785. Immediately prior to this, there were several log entries regarding delayed I/O requests to tempdb.

Call Site
ntdll!NtClose
KERNELBASE!CloseHandle
kernel32!CloseHandleImplementation
sqlservr!CDiskDev::Close
sqlservr!CDiskDev::~CDiskDev
sqlservr!CDiskDev::`scalar deleting destructor’
sqlservr!CStreamDev::Release
sqlservr!CStreamBuf::~CStreamBuf
sqlservr!CStreamBuf::`scalar deleting destructor’
sqlservr!CStreamBuf::Release
sqlservr!CImportStream::~CImportStream
sqlservr!CImportStream::`scalar deleting destructor’
sqlservr!CAutoImportStream::~CAutoImportStream
sqlservr!CImpImportProvider::GetRowsetPtr
msvcr80!_CallSettingFrame
msvcr80!__FrameUnwindToState
msvcr80!__FrameUnwindToEmptyState
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler
sqlservr!_GSHandlerCheck_EH
ntdll!RtlpExecuteHandlerForUnwind
ntdll!RtlUnwindEx
kernel32!RtlUnwindExStub
msvcr80!_UnwindNestedFrames
msvcr80!FindHandler
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler
sqlservr!_GSHandlerCheck_EH
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!RtlRaiseException
KERNELBASE!RaiseException
msvcr80!_CxxThrowException
sqlservr!Worker::SetExceptionInfo
sqlservr!ex_raise2
sqlservr!ex_raise
sqlservr!CStreamBuf::CheckIOError
sqlservr!CImportStream::SwitchBuffer
sqlservr!CImpImportProvider::GetRowsetPtr
sqlservr!CreateInprocProvider
sqlservr!CBcpImport::CreateInprocProvider
sqlservr!CBcpImport::InitForOpenRowset
sqlservr!CreateBulkRowset
sqlservr!COledbRangeStreamBulk::DoNormalize
sqlservr!COledbRange::Normalize
sqlservr!alloca_probe
sqlservr!CAlgTableMetadata::Bind
sqlservr!CRelOp_Get::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_QuerySpec::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Insert::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_DMLQuery::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Query::FAlgebrizeQuery
sqlservr!CStmtBulkInsert::FBind
sqlservr!CProchdr::FNormalizeStep
sqlservr!CSQLSource::FCompile
sqlservr!CSQLSource::FCompWrapper
sqlservr!CSQLSource::Transform
sqlservr!CSQLSource::Execute
sqlservr!ExecuteSql
sqlservr!CSpecProc::ExecuteSpecial
sqlservr!CXProc::Execute
sqlservr!alloca_probe
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CStmtExecProc::XretLocalExec
sqlservr!CStmtExecProc::XretExecExecute
sqlservr!CXStmtExecProc::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
nathan.sunderman@des.wa.gov said

March 19, 2014 at 10:18 PM
Hi Karthick-

I am getting a non-yeilding IOCP dump in my SQL 2012 server 11.0.3128

I didn’t find the dump listed, but here is the stack please let me know if you have any advice.

Kind regards,

-Nathan

Call Site
ntdll!ZwClose
mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqllang!Tcp::FCloseTransport
sqllang!SNI::detail::Transport::FCloseRefHandle
sqllang!Tcp::Close
sqllang!Smux::InternalClose
sqllang!Smux::ReadDone
sqllang!SNIReadDone
sqldk!SOS_Node::ListenOnIOCompletionPort
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
B Jones said

May 6, 2014 at 9:32 PM
The stack we received recently along with Non-yielding IOCP Listener is similar, but not quite the same as others I have seen here.

sqlservr!BPool::Steal
sqlservr!SQLSinglePageAllocator::AllocatePages
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CMemThread::PbGetNewPages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemThread::Alloc
sqlservr!operator new
sqlservr!SNI_Conn::InitObject
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadStart

SQL 2005 Standard (64-bit)
version 9.00.5000.00

Computer type is AT/AT COMPATIBLE.
Current time is 05:40:56 05/06/14.
4 Unknown CPU 9., 2328 Mhz processor (s).
Windows NT 5.2 Build 3790 CSD Service Pack 2.

Memory
MemoryLoad = 88%
Total Physical = 32762 MB
Available Physical = 3654 MB
Total Page File = 81450 MB
Available Page File = 47144 MB
Total Virtual = 8388607 MB
Available Virtual = 8355267 MB
**Dump thread – spid = 0, PSS = 0x0000000000000000, EC = 0x0000000000000000
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 05/06/14 05:40:56 spid 0
*
* Non-yielding IOCP Listener
*
* *******************************************************************************

This is not recurring often. It did happen approximately 5 months ago. The only difference between the stack from that time and the most recent is the previous stack included as the first line:
sqlservr!BPool::ReplenishFreeList

Note, we had not limited max memory, but based on many references, including much of what is here, I have today placed a max memory limit of 30715.

If you have a solid idea, that would be more than fantastic!

Reply
Kaitlyn said

June 6, 2014 at 5:26 AM
I don’t see a match for my call stack here. I can’t seem to figure it out, is it memory pressure?

from SQLDump0001.txt:
Current time is 01:07:38 06/05/14.
=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 11.0.3379.0
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz.
Bios Version is INTEL – 6040000
PhoenixBIOS 4.0 Release 6.0
4 X64 level 8664, 14 Mhz processor (s).
Windows NT 6.2 Build 9200 CSD .

Memory
MemoryLoad = 97%
Total Physical = 32767 MB
Available Physical = 791 MB
Total Page File = 37631 MB
Available Page File = 5777 MB
Total Virtual = 134217727 MB
Available Virtual = 134111055 MB
**Dump thread – spid = 0, EC = 0x0000000000000000
***Stack Dump being sent to J:\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\LOG\SQLDump0001.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 06/05/14 01:07:38 spid 1432
*
* Non-yielding Resource Monitor
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Call Site
sqlmin!CQueryExecContext::FinalCleanup
sqlmin!CQueryScan::DestroyQuery
sqllang!CExecuteStatement::Destroy
sqllang!CMetaStmtSet::DeleteXstmtsInPartition
sqllang!CMetaStmtSet::StepOnStatements
sqllang!CCompPlan::StepOnStatements
sqllang!CCacheStoreCPCallback::SteppedOnEntry
sqldk!CacheClockHand::MoveInternal
sqldk!CacheClockHand::Move
sqldk!TClockAlgorithm<CacheClockHand >::StartExternalPressure
sqldk!ResourceMonitor::NotifyMemoryConsumers
sqldk!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

We are on Windows 2012 R2 with SQL Server 2012 SP CU6 I believe. We have set max server memory at 28 GB. There’s 32gb available.
Thanks for any help or guidance on this!

Kaitlyn

Reply
Luke said

June 17, 2014 at 1:54 PM
Hi there,

I’m seeing this in the stack:

0:046> .cxr sqlservr!g_copiedStackInfo+0X20
rax=0000000000000002 rbx=0000000000000000 rcx=000000008018c240
rdx=0000000000000e2c rsi=000000002a8c5a80 rdi=0000000000000001
rip=00007ff87e41af5a rsp=00000000069ceb30 rbp=00000000069cebc1
r8=0000000000000000 r9=00000000069d0450 r10=000000008050f320
r11=00000000013c4af0 r12=00000000069cef90 r13=00007ff87bb62940
r14=00000000069cf0e9 r15=00000000069cefa8
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
ntdll!NtQueryAttributesFile+0xa:
00007ff8`7e41af5a c3 ret
0:046> KC 100
Call Site
ntdll!NtQueryAttributesFile
ntdll!RtlDoesFileExists_UstrEx
ntdll!RtlDosSearchPath_Ustr
KERNELBASE!BasepLoadLibraryAsDataFileInternal
KERNELBASE!LoadLibraryExW
KERNELBASE!GetFileVersionInfoSizeExW
sqlservr!CCLRHost::CheckSqlAccessVersion
sqlservr!CCLRHost::InitProductSpecific
sqlservr!CCLRHostBase::InitRunTime
sqlservr!SysClrTypes::GetAssemblyFullPathName
sqlservr!SystemAssemblyFileBuffer::GetSystemAssemblyFileBuffer
sqlservr!CMEDProxyAssemblyFile::GetSourceBytes
sqlservr!CAssemblyMetaInfo::LoadAssemblyFromDatabase
sqlservr!GetAssemblyProperty
sqlservr!SsvarAsmPropWstrStr
sqlservr!CEs::GeneralEval
sqlservr!CXStmtQuery::ErsqExecuteQuery
sqlservr!CXStmtSelect::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Other info:

=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 10.50.4305.0
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz.
Bios Version is INTEL – 6040000
PhoenixBIOS 4.0 Release 6.0
4 X64 level 8664, 2 Mhz processor (s).
Windows NT 6.2 Build 9200 CSD .

Memory
MemoryLoad = 92%
Total Physical = 16383 MB
Available Physical = 1270 MB
Total Page File = 33815 MB
Available Page File = 18096 MB
Total Virtual = 134217727 MB
Available Virtual = 134200178 MB
***Unable to get thread context for spid 0
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 06/17/14 02:12:31 spid 1372
*
* Non-yielding Scheduler
*
* *******************************************************************************

Any ideas as I’m having trouble finding a solution???

Many thanks,

Luke

Reply
- Karthick P.K said
  
  June 17, 2014 at 10:52 PM
  How many times this issue has occurred.
  
  Reply
  - Luke said
    
    June 17, 2014 at 11:00 PM
    I’ve had the server go non responsive a few times in the last few weeks (not good as it’s production!). I’d have to get more logs off the box tomorrow to find out if it’s a recurring issue.
    
    Reply
  - Luke said
    
    June 18, 2014 at 9:19 PM
    I’ve checked the other logs and I do have another non-yielding scheduler dump, but for reasons unknown the dump file is 0kb so I can’t give any more information sadly
    
    Reply
Oliver Jen said

June 21, 2014 at 4:08 AM
Hi Karthik,

Have you seen this one before?

Call Site
ntdll!ExpInterlockedPopEntrySListResume16
sqldk!TMemoryBlockAllocator::GetNewActiveBlockFromPartialListInt
sqldk!TMemoryBlockAllocator::GetNewActiveBlockFromPartialList
sqldk!SOS_MemoryBlockAllocator::AllocateBlockInt
sqldk!SOS_MemoryBlockAllocator::AllocateBlock
sqldk!SOS_MemoryWorkSpace::AllocatePage
sqldk!MemoryClerkInternal::AllocateReservedPages
sqlmin!BPool::Steal
sqlmin!PbufGetGrantedBufs
sqlmin!soGetBuf
sqlmin!soInitBobMgr
sqlmin!addrun
sqlmin!soAllocRecBuf
sqlmin!RowsetSorted::InsertRow
sqlmin!RowsetChangeSort::InsertRow
sqlmin!CValRowNoHrow::SetDataX
sqltses!CEsExec::GeneralEval
sqlmin!CQScanSortNew::PushRow
sqlmin!CQScanSortNew::BuildSortTable
sqlmin!CQScanSortNew::OpenHelper
sqlmin!CQScanMergeJoinNew::Open
sqlmin!CQueryScan::StartupQuery
sqllang!CXStmtQuery::SetupQueryScanAndExpression
sqllang!CXStmtQuery::InitForExecute
sqllang!CXStmtQuery::ErsqExecuteQuery
sqllang!CXStmtSelect::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!InternalPrepExec
sqllang!SpPrepExec
sqllang!CSpecProc::ExecuteSpecial
sqllang!CSpecProc::Execute
sqllang!process_request
sqllang!process_commands
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  June 21, 2014 at 4:42 AM
  Hello Oliver,
  
  Your are hitting the issue mentioned in http://support.microsoft.com/kb/2762557
  
  Apply the Fix mentioned in KB. It will solve your issue.
  
  Thanks
  
  Karthick
  
  Reply
  - Oliver Jen said
    
    June 26, 2014 at 11:16 PM
    Perfect; thank you, Karthick! 🙂
    
    Reply
Nishtha said

July 7, 2014 at 12:23 PM
Call Site
sqlservr!AccessorCache::Destroy
sqlservr!CEnvCollection::CleanupAccessorCache
sqlservr!CStmtQuery::~CStmtQuery
sqlservr!CStmtSelect::`vector deleting destructor’
sqlservr!CStatement::Destroy
sqlservr!CStatement::Release
sqlservr!CPlanSkeleton::~CPlanSkeleton
sqlservr!CPlanSkeleton::`scalar deleting destructor’
sqlservr!CPlanSkeleton::Release
sqlservr!CCompPlan::~CCompPlan
sqlservr!CCompPlan::`vector deleting destructor’
sqlservr!CCacheObject::DeleteObject
sqlservr!CCacheObject::Release
sqlservr!SOS_CacheStore::RemoveDescriptor
sqlservr!SOS_CacheStore::CacheEntryDescriptor::Destroy
sqlservr!ClockHand::Move
sqlservr!ClockAlgorithm::MoveHand
sqlservr!ClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
0x0
0x0
0x0
0x0
msvcr80!_threadstartex
msvcr80!_threadstartex
msvcr80!__initialmbcinfo
0x0

Reply
- Karthick P.K said
  
  July 8, 2014 at 2:40 AM
  Which version of SQL Server is this? Can you please share the build number.
  
  Reply
- Nishtha said
  
  July 9, 2014 at 6:22 AM
  .009.00.4035.00
  
  Reply
  - NIshtha said
    
    July 9, 2014 at 6:24 AM
    Sorry, this site doesn’t run very well on IE
    The version no; is 9.00.4035.00
    
    Reply
Michael Tang said

July 8, 2014 at 7:02 AM
Hi

Can you please look at my dump output?
SQL server 2005 SP4

ntdll!ZwWaitForSingleObject
ntdll!RtlpWaitOnCriticalSection
ntdll!RtlEnterCriticalSection
swi_ifslsp_64!GetLspGuid
swi_ifslsp_64!GetLspGuid
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!SNIClose
sqlservr!CNetConnection::CloseNetlibEP
sqlservr!CNetConnection::KillConnection
sqlservr!CPhysicalConnection::NotifySessionLogout
sqlservr!CSession::LogoutSession
sqlservr!kill_session
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
Shivam bhatia said

July 8, 2014 at 12:07 PM
hi Karthick,

got the below dump and sql server became unresponsive after that …..

can you please help me with this.

00000034`fdfe69c8 000007fe`fd0810dc ntdll!ZwWaitForSingleObject+0xa
00000034`fdfe69d0 00000000`0068ceea KERNELBASE!WaitForSingleObjectEx+0x79
00000034`fdfe6a70 00000000`00674485 sqlservr!SOS_Scheduler::SwitchContext+0x72f
00000034`fdfe6d30 00000000`00673db6 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xc5
00000034`fdfe6d70 00000000`00bba45d sqlservr!EventInternal<Spinlock >::Wait+0x436
00000034`fdfe7360 00000000`00bc32af sqlservr!LockOwner::Sleep+0xbd
00000034`fdfe73d0 00000000`006960c0 sqlservr!lck_lockInternal+0xe8d
00000034`fdfe8140 00000000`0154c562 sqlservr!BTreeRow::AcquireLock+0x478
00000034`fdfe82d0 00000000`00696227 sqlservr!alloca_probe+0x1366a6
00000034`fdfe8330 00000000`006a8f3d sqlservr!IndexDataSetSession::GetNextRowValuesInternal+0x77a
00000034`fdfe89f0 00000000`006c66a2 sqlservr!RowsetNewSS::FetchNextRow+0x15d
00000034`fdfe8c10 00000000`006b510c sqlservr!CQScanRangeNew::GetRow+0x6b6
00000034`fdfe92d0 00000000`006c7378 sqlservr!CQScanStreamAggregateNew::GetRowHelper+0x244
00000034`fdfe9630 00000000`007abb0a sqlservr!CQScanFilterNew::GetRow+0x28
00000034`fdfe9660 00000000`007596d5 sqlservr!CQScanNLJoinNew::GetRowHelper+0x218e
00000034`fdfe9be0 00000000`007589fc sqlservr!CQueryScan::GetRow+0x65
00000034`fdfe9c10 00000000`00bf0bd1 sqlservr!CXStmtQuery::ErsqExecuteQuery+0x357
00000034`fdfecc80 00000000`023c9243 sqlservr!CXStmtCondWithQuery::XretExecute+0xe4
00000034`fdfecfb0 00000000`023ccf7a sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn+0x13
00000034`fdfecfe0 00000000`017540f0 sqlservr!CMsqlExecContext::ExecuteStmts+0xbca
00000034`fdfedf90 00000000`00774629 sqlservr!alloca_probe+0x3d87fc
00000034`fdfee110 00000000`022b84fd sqlservr!CSQLSource::Execute+0x319
00000034`fdfee240 00000000`022c19d4 sqlservr!ExecuteSql+0x72d
00000034`fdfeece0 00000000`02d06461 sqlservr!CSpecProc::ExecuteSpecial+0x234
00000034`fdfeee00 00000000`00778d3f sqlservr!CSpecProc::Execute+0x1f1
00000034`fdfeef70 00000000`0072fb2a sqlservr!process_request+0x371
00000034`fdfef230 00000000`0067cc1b sqlservr!process_commands+0x1ba
00000034`fdfef430 00000000`0067dffa sqlservr!SOS_Task::Param::Execute+0x11b
00000034`fdfef550 00000000`0067cd69 sqlservr!SOS_Scheduler::RunTask+0xca
00000034`fdfef5e0 00000000`00b49940 sqlservr!SOS_Scheduler::ProcessTasks+0x99
00000034`fdfef650 00000000`00b48c30 sqlservr!SchedulerManager::WorkerEntryPoint+0x110
00000034`fdfef710 00000000`00b47426 sqlservr!SystemThread::RunWorker+0x60
00000034`fdfef740 00000000`00b49c4f sqlservr!SystemThreadDispatcher::ProcessWorker+0x130
00000034`fdfef7d0 00000000`74ac37d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x12f
00000034`fdfef860 00000000`74ac3894 msvcr80!endthreadex+0x47
00000034`fdfef890 00000000`76d6652d msvcr80!endthreadex+0x104
00000034`fdfef8c0 00000000`76e9c541 kernel32!BaseThreadInitThunk+0xd
00000034`fdfef8f0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Reply
Scott said

July 10, 2014 at 6:31 AM
Hello –

Just got a Non-Yielding IOCP Listener stack dump that appears to have caused a failover on a cluster.

Would you please look at it and give me any advice?

Windows Server 2003 R2 EE
SQL Server 2005 EE (9.00.2153)
Server Ram: 31.9 GB
SQL Server Max Server Memory: 28000
MaxDOP: 0
CLR objects : Many in several databases.

sqlservr!BPool::BulkUnmap
sqlservr!BPool::Map
sqlservr!BPool::Steal
sqlservr!SQLSinglePageAllocator::AllocatePages
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!IMemObj::PbGetNewPages
sqlservr!CMemThread::PbGetNewPages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!CMemThread::Alloc
sqlservr!Tcp::AcceptConnection
sqlservr!Tcp::AcceptDone
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDone
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadStart

Thank you,
Scott

Reply
Tilak said

July 10, 2014 at 5:15 PM
Hi Karthick,

My SQL version is 10.50.4000 and i see the below stack

Call Site
ntdll!NtClose
KERNELBASE!CloseHandle
kernel32!CloseHandleImplementation
sqlservr!CTraceFileIoProvider::OpenNextFile
sqlservr!CTraceWriteRequest::InitForFileTrace
sqlservr!CTraceFileIoProvider::InitializeWriteRequest
sqlservr!CTrace::WriteRecord
sqlservr!CTraceController::ProduceRecord
sqlservr!CTraceController::ProduceRecordWithTextBase
sqlservr!CSECAuditBase::PostTraceEvent
sqlservr!CSecRuleAudit::AuditEvent
sqlservr!_RuleAudit::AuditEvent
sqlservr!ISECManager::CheckPermRuleAuditOnly
sqlservr!DbccThreadContext::Cleanup
sqlservr!DbccThreadContext::`scalar deleting destructor’
sqlservr!ThreadContextFactory::UninstallContext
sqlservr!AutoDbccThreadContext::~AutoDbccThreadContext
sqlservr!CStmtDbcc::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
0x0
0x0
0x0

Can you please suggest what this means.

Thanks.
Tilak M

Reply
- Karthick P.K said
  
  July 10, 2014 at 7:46 PM
  It was because of profiler trace you are running.
  
  Reply
Saravnaa said

July 23, 2014 at 4:48 PM
Hi Karthick,

Below is the output of dump file which I got.
Please provide your suggestions on this,

ntdll!NtWaitForSingleObject
ntdll!RtlAcquireResourceExclusive
schannel!SpDeleteUserModeContext
sspicli!LsaDeleteSecurityContext
sspicli!DeleteSecurityContext
sqlservr!Ssl::~Ssl
sqlservr!Ssl::`vector deleting destructor’
sqlservr!CryptoBase::Release
sqlservr!SNI_Conn::Release
sqlservr!SNIPacketRelease
sqlservr!CNetConnection::Destroy
sqlservr!ProtocolHelpers::DestroyOrEnqueueIfIocp
sqlservr!CNetConnection::Release
sqlservr!CPhysicalConnection::NotifySessionLogout
sqlservr!CSession::LogoutSession
sqlservr!CSession::RemoveBatch
sqlservr!process_close_connection
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  July 24, 2014 at 12:42 AM
  What is your windows version?
  
  Reply
- Karthick P.K said
  
  July 24, 2014 at 12:45 AM
  This is not SQL Server issue. SQL Server is just a victim. Check if fix in http://support.microsoft.com/kb/2883492/en-us is applicable.
  
  Reply
  - Saravnaa said
    
    August 12, 2014 at 11:15 PM
    thanks for the suggestion Karthick
    
    Reply
  - DmitryE said
    
    November 16, 2016 at 1:54 PM
    Hi Karthick!
    Help please!
    
    I have almost the same problem, but `scalar deleting destructor’ instead of `vector deleting destructor’
    Windows Server 2003 R2 x64 ServicePack 2
    
    Microsoft SQL Server 2005 – 9.00.5000.00 (X64)
    Enterprise Edition (64-bit) on Windows NT 5.2 (Build 3790: Service Pack 2)
    
    fix in http://support.microsoft.com/kb/2883492/en-us for Windows 7 or Windows Server 2008 R2
    
    ntdll!NtRequestWaitReplyPort
    secur32!SecpDeleteSecurityContext
    secur32!LsaDeleteSecurityContext
    secur32!DeleteSecurityContext
    sqlservr!Ssl::~Ssl
    sqlservr!Ssl::`scalar deleting destructor’
    sqlservr!CryptoBase::Release
    sqlservr!SNIPacketRelease
    sqlservr!CNetConnection::Destroy
    sqlservr!ProtocolHelpers::DestroyOrEnqueueIfIocp
    sqlservr!CNetConnection::Release
    sqlservr!CPhysicalConnection::NotifySessionLogout
    sqlservr!CSession::LogoutSession
    sqlservr!process_close_connection
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    
    Reply
Jo said

August 28, 2014 at 3:29 PM
Hi Karthick,

Could you please take a look at this stack:

Call Site
sqlservr!CMetaStmtXStmtDPS::PstmtSafe
sqlservr!CMetaStmtSet::PstmtGetSafe
sqlservr!CCompPlan::PStmtCurSelectNext
sqlservr!CCompPlan::StepOnStatements
sqlservr!CCacheStoreCPCallback::SteppedOnEntry
sqlservr!CacheClockHand::MoveInternal
sqlservr!CacheClockHand::Move
sqlservr!TClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

SQL server 2008 R2, build 10.50.2861

Thanks,
Jo

Reply
- Rupert Hirst said
  
  July 8, 2015 at 7:14 PM
  Hi, did you ever get a reply to this one ~ I’ve been experiencing a similar dump ?
  
  Reply
Tayllor Myxter (Developer) said

September 5, 2014 at 5:46 AM
Call Site
sqlservr!CStmtQuery::~CStmtQuery
sqlservr!CStmtSelect::`vector deleting destructor’
sqlservr!CStatement::Destroy
sqlservr!CStatement::Release
sqlservr!CMetaStmtXStmtDPS::Cleanup
sqlservr!CMetaStmtSet::Cleanup
sqlservr!CCompPlan::~CCompPlan
sqlservr!CCompPlan::`vector deleting destructor’
sqlservr!CCacheObject::DeleteObject
sqlservr!CCacheObject::Release
sqlservr!SOS_CacheStore::RemoveDescriptor
sqlservr!SOS_CacheStore::CacheEntryDescriptor::Destroy
sqlservr!CacheClockHand::MoveInternal
sqlservr!CacheClockHand::Move
sqlservr!TClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Windows Server 2008r2 sp1
SQL Server: 10.0.4067

Reply
Marc said

September 25, 2014 at 9:47 PM
Can anyone help please?

Current time is 00:09:23 09/25/14.
=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 10.50.1600.1
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is Intel(R) Xeon(R) CPU E5504 @ 2.00GHz.
Bios Version is HP – 2
4 X64 level 8664, 1 Mhz processor (s).
Windows NT 6.1 Build 7601 CSD Service Pack 1.

Memory
MemoryLoad = 93%
Total Physical = 6133 MB
Available Physical = 395 MB
Total Page File = 12265 MB
Available Page File = 4572 MB
Total Virtual = 8388607 MB
Available Virtual = 8375783 MB
**Dump thread – spid = 0, EC = 0x0000000000000000
***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\LOG\SQLDump0019.tx
t
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 09/25/14 00:09:24 spid 1816
*
* Non-yielding IOCP Listener
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Call Site
sqlservr!Spinlock::SpinToAcquireOptimistic
sqlservr!EventInternal<Spinlock >::Signal
sqlservr!BPool::Steal
sqlservr!SQLSinglePageAllocator::AllocatePages
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CMemThread::PbGetNewPages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!CMemThread::Alloc
sqlservr!SNI_Conn::InitObject
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
Jason said

November 4, 2014 at 3:13 AM
Hi Karthick,

We could use some help with getting to the root of some recent crashes. This particular instance is suffering from disk performance issues and we are working with our client to remedy the situation. In the meantime, if we could pinpoint certain actions or determine that disk performance is causing these crashes, we could provide some feedback some much needed feedback to our client.

=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 11.0.5532.0
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz.
Bios Version is DELL – 1
Phoenix ROM BIOS PLUS Version 1.10 2.2.3
16 X64 level 8664, 3 Mhz processor (s).
Windows NT 6.2 Build 9200 CSD .

Memory
MemoryLoad = 91%
Total Physical = 393170 MB
Available Physical = 31705 MB
Total Page File = 434130 MB
Available Page File = 68512 MB
Total Virtual = 134217727 MB
Available Virtual = 133361833 MB
***Unable to get thread context for spid 0
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 11/03/14 10:26:41 spid 424
*
* Non-yielding Scheduler
*
* *******************************************************************************

Call Site
ntdll!ZwWriteFile
KERNELBASE!WriteFile
sqlmin!DiskWriteAsync
sqlmin!CTraceIoBuf::Release
sqlmin!CTraceWriteRequest::GetNextBlock
sqllang!CTrace::PutBytes
sqllang!CTrace::WriteColumn
sqllang!CTrace::WriteRecord
sqllang!CTraceController::ProduceRecord
sqllang!COptContext::PcxteOptimizeQuery
sqllang!COptContext::PqteOptimizeWrapper
sqllang!PqoBuild
sqllang!CStmtQuery::InitQuery
sqllang!CStmtDML::InitNormal
sqllang!CStmtDML::Init
sqllang!CCompPlan::FCompileStep
sqllang!CSQLSource::FCompile
sqllang!CSQLSource::FCompWrapper
sqllang!CSQLSource::Transform
sqllang!CSQLSource::Execute
sqllang!CSQLSource::SeExecute
sqllang!CIndexDDL::PopulateIndex
sqllang!CIndexDDL::ConstructIndicesOffline
sqllang!CIndexDDL::Create
sqllang!CIndexDef::CreateIndex
sqllang!CStmtCreateIndex::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!CStmtExecProc::XretLocalExec
sqllang!CStmtExecProc::XretExecExecute
sqllang!CXStmtExecProc::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!process_request
sqllang!process_commands
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
Kursat said

April 1, 2015 at 7:08 PM
Hi,

my stack does not match any of these above.

Could you have a look at please..

OS, Windows Server 2008 Enterprise x64
SQL, SQL Server 2008 SP3 (10.00.5500.00)

i passed the hotfix (CU4 – KB2673383)..

SQL increases the log files and windows alerts disk full..
here s the dump file (two of them, i deleted others)
http://1drv.ms/19Et9to

WinDbg result:

Call Site
sqlservr!CMEDIndexStatsCollection::GetProxyIndexWithLock
sqlservr!CMEDProxyRelation::GetIndex
sqlservr!CMetadataUtil::GetIndexFromIMED
sqlservr!CAlgTableMetadata::PimidxBaseIndex
sqlservr!CAlgTableMetadata::LoadSpecialColumns
sqlservr!CAlgTableMetadata::LoadColumns
sqlservr!CAlgTableMetadata::Bind
sqlservr!CRelOp_Get::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_BaseJoin::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_FromList::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Select::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_QuerySpec::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_SelectQuery::BindTree
sqlservr!COptExpr::BindTree
sqlservr!CRelOp_Query::FAlgebrizeQuery
sqlservr!CProchdr::FNormQuery
sqlservr!CProchdr::FNormalizeStep
sqlservr!CSQLSource::FCompile
sqlservr!CSQLSource::FCompWrapper
sqlservr!CSQLSource::Transform
sqlservr!CSQLSource::Execute
sqlservr!ExecuteSql
sqlservr!CSpecProc::ExecuteSpecial
sqlservr!CSpecProc::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
Karren said

July 16, 2015 at 5:22 PM
Hi Hi Karthick,

I see where you have been able to assist others and I am really seeking your assistance.
We have been getting non-yielding scheduler messages and queries that used to perform great have degraded severely. I didn’t see my stack here so here is the output:

rax=000000303f3b0370 rbx=000000303f3b0516 rcx=000000303f3b03d0
rdx=000000303f3b03f0 rsi=00000000000320c8 rdi=0000000000000000
rip=000007faf80a7d92 rsp=000000004057e1a0 rbp=000000004057e209
r8=000000303dfac840 r9=0000000000000001 r10=000000303dfac6b0
r11=000000303f3b0310 r12=0000000000000000 r13=0000000000000004
r14=000000303dfac840 r15=000000303f3b0160
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000202
sqlmin!RowsetColumnsetAdapter::OutputColumnData+0x79:
000007fa`f80a7d92 41ffc9 dec r9d

We are running SQL Server 2012 SP2 on Windows Server 2012.

Any insight will help greatly as one of our production system has bee unable to generate reports for the last 4 days.

Thanks.

Reply
Karren said

July 16, 2015 at 5:29 PM
Sorry I had overlooked the KC 100

My stack is
sqlmin!RowsetColumnsetAdapter::OutputColumnData
sqlmin!RowsetColumnsetAdapter::FetchNextRow
sqlmin!CXRowset::FetchNextRow
sqlmin!CQScanRowsetNew::GetRowWithPrefetch
sqlmin!CQScanRangeNew::GetRow
sqlmin!CQScanXProducerNew::GetRowHelper
sqlmin!CQScanXProducerNew::GetRow
sqlmin!FnProducerOpen
sqlmin!FnProducerThread
sqlmin!SubprocEntrypoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
Chen said

August 9, 2015 at 12:41 PM
Hi,
First thank you for your guide 🙂
If I go by your guide, I’m getting a lot exceptions on step 4:

……………………………………………………….

Loading unloaded module list
……………………………………………………………
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(784.8f0): Unknown exception – code 00000000 (first/second chance not available)
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll –
ntdll!NtWaitForSingleObject+0xa:
00000000`775212fa c3 ret
0:028> .sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
Symbol search path is: srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
0:028> .reload /f
………

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

……………………….*** ERROR: Symbol file could not be found. Defaulted to export symbols for instapi10.dll –
..*** ERROR: Module load completed but symbols could not be loaded for sqlevn70.rll
…………………*** ERROR: Symbol file could not be found. Defaulted to export symbols for msfte.dll –
….
……………..*** ERROR: Module load completed but symbols could not be loaded for SQLNCLIR10.RLL
…*** ERROR: Module load completed but symbols could not be loaded for xplog70.rll
…..*** ERROR: Module load completed but symbols could not be loaded for odbcint.dll
….*** ERROR: Module load completed but symbols could not be loaded for XPStar.RLL
….*** WARNING: Unable to verify checksum for mscorlib.ni.dll
………*** WARNING: Unable to verify checksum for System.ni.dll
..*** WARNING: Unable to verify checksum for System.Security.ni.dll
….*** ERROR: Symbol file could not be found. Defaulted to export symbols for instapi.dll –
…………….

Loading unloaded module list
……………………………………………………………

Does someone know why ???

thanks for your answers 🙂

Reply
- chenshaulian said
  
  August 9, 2015 at 12:51 PM
  Hi again ,,, ok, so I just added “kc 100” and now it’s working 🙂
  I’m getting:
  0:028> kc 100
  Call Site
  ntdll!ZwWaitForSingleObject
  KERNELBASE!WaitForSingleObjectEx
  sqlservr!SOS_Scheduler::SwitchContext
  sqlservr!SOS_Scheduler::SuspendNonPreemptive
  sqlservr!SOS_Scheduler::Suspend
  sqlservr!EventInternal<Spinlock >::Wait
  sqlservr!EventInternal<Spinlock >::WaitAllowPrematureWakeup
  sqlservr!CXPacketList::RemoveHead
  sqlservr!CXPipe::ReceivePacket
  sqlservr!CXTransLocal::ReceiveBuffers
  sqlservr!CQScanExchangeNew::GetRow
  sqlservr!CQScanBitmapNew::GetRow
  sqlservr!CQScanHash::ReadRow
  sqlservr!CQScanHash::ConsumeBuild
  sqlservr!CQScanHash::Iterate
  sqlservr!CQScanHash::Open
  sqlservr!CQScanSortNew::BuildSortTable
  sqlservr!CQScanSortNew::OpenHelper
  sqlservr!CQScanNew::OpenHelper
  sqlservr!CQScanXProducerNew::Open
  sqlservr!FnProducerOpen
  sqlservr!FnProducerThread
  sqlservr!SubprocEntrypoint
  sqlservr!SOS_Task::Param::Execute
  sqlservr!SOS_Scheduler::RunTask
  sqlservr!SOS_Scheduler::ProcessTasks
  sqlservr!SchedulerManager::WorkerEntryPoint
  sqlservr!SystemThread::RunWorker
  sqlservr!SystemThreadDispatcher::ProcessWorker
  sqlservr!SchedulerManager::ThreadEntryPoint
  msvcr80!endthreadex
  msvcr80!endthreadex
  kernel32!BaseThreadInitThunk
  ntdll!RtlUserThreadStart
  
  Do you know what it means ?
  
  Reply
Scott said

January 27, 2016 at 2:30 AM
Hi-

We found a couple of entries for “Non-yielding Resource Monitor” in our log over a span of only 4 months – but am still concerned as to what is happening.
Tried to figure out the the debug but am still unsure or have not found an answer from your site…please let me know if anyone can help.

See out details/info below:

From SQLDump0002.log…

2016-01-23 23:32:24.49 Server Using ‘dbghelp.dll’ version ‘4.0.5’
2016-01-23 23:32:30.01 Server **Dump thread – spid = 0, EC = 0x0000000000000000
2016-01-23 23:32:31.17 Server ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\MSSQL10_50.MSSQLSERVER\MSSQL\LOG\SQLDump0002.txt
2016-01-23 23:32:31.17 Server * *******************************************************************************
2016-01-23 23:32:31.17 Server *
2016-01-23 23:32:31.17 Server * BEGIN STACK DUMP:
2016-01-23 23:32:31.17 Server * 01/23/16 23:32:31 spid 1044
2016-01-23 23:32:31.17 Server *
2016-01-23 23:32:31.17 Server * Non-yielding Resource Monitor
2016-01-23 23:32:31.17 Server *
2016-01-23 23:32:31.17 Server * *******************************************************************************
2016-01-23 23:32:31.17 Server * ——————————————————————————-
2016-01-23 23:32:31.17 Server * Short Stack Dump
2016-01-23 23:32:50.88 Server Stack Signature for the dump is 0x000000000000024C
2016-01-23 23:33:55.70 Server Timeout waiting for external dump process 17188.

2016-01-23 23:33:56.24 Server
Resource Monitor (0x4c8) Worker 0x00000000800421A0 appears to be non-yielding on Node 0.
Memory freed: 96776 KB. Last wait: MISCELLANEOUS.
Last clerk: type CACHESTORE_SQLCP, name SQL Plans.
Approx CPU Used: kernel 62 ms, user 15 ms, Interval: 60106.

0:005> .cxr sqlservr!g_copiedStackInfo+0X20
rax=0000000000002000 rbx=00000003c1c90000 rcx=0000000b09809830
rdx=ffffffffffffffcc rsi=000000000000004b rdi=00000003c3dee090
rip=0000000000cd740c rsp=0000000008c6ed00 rbp=0000000b09809820
r8=0000000000000050 r9=0000000000000008 r10=00000000801e05f8
r11=000000000cb8e940 r12=00000003c3dbe220 r13=0000000000000070
r14=0000000008fa0098 r15=00000000000fffff
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00010206
sqlservr!CIncPageMgr::ReleaseAll+0x52:
00000000`00cd740c ?? ???

0:005> kc 100
Call Site
sqlservr!CIncPageMgr::ReleaseAll
sqlservr!CMemProc::DestroyObject
sqlservr!MemObjProxy::ReleaseExternal
sqlservr!IMemObj::Release
sqlservr!CPmoLock::Destroy
sqlservr!CStatement::Destroy
sqlservr!CStatement::Release
sqlservr!CMetaStmtSet::Cleanup
sqlservr!CCompPlan::~CCompPlan
sqlservr!CCompPlan::`vector deleting destructor’
sqlservr!CCacheObject::DeleteObject
sqlservr!CCacheObject::Release
sqlservr!SOS_CacheStore::RemoveDescriptor
sqlservr!SOS_CacheStore::CacheEntryDescriptor::Destroy
sqlservr!CacheClockHand::MoveInternal
sqlservr!CacheClockHand::Move
sqlservr!TClockAlgorithm::ProcessTick
sqlservr!SOS_CacheStore::Notify
sqlservr!ResourceMonitor::NotifyMemoryConsumers
sqlservr!ResourceMonitor::ResourceMonitorTask
sqlservr!SetupResourceMonitorTaskContext
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
0x0
0x0
0x0

Reply
venkat said

April 12, 2016 at 8:56 PM
Hi,
Can you please help me with this. I am unable to find any details related to below logs

00 ntdll!NtReadFile
01 KERNELBASE!ReadFile
02 kernel32!ReadFileImplementation
03 sqlservr!DiskReadAsync
04 sqlservr!FCB::AsyncRead
05 sqlservr!SQLServerLogMgr::LogBlockReadAheadAsync
06 sqlservr!SQLServerLogIterBackLink::ReadLog
07 sqlservr!SQLServerLogIterBackLink::TryLogReadAhead
08 sqlservr!SQLServerLogIterBackLink::GetNextBlock
09 sqlservr!SQLServerLogIterBackLink::GetNext
0a sqlservr!XdesRMReadWrite::RollbackToLsn
0b sqlservr!XdesRMFull::Rollback
0c sqlservr!XactRM::RollbackInternal
0d sqlservr!CMsqlXactInternalBase::RollbackInternal
0e sqlservr!CMsqlXactInternalReadWrite::Rollback
0f sqlservr!CMsqlXactInternalBase::DefectOrRollbackXact
10 sqlservr!CMsqlXactManager::DefectOrRollbackAll
11 sqlservr!CMsqlXact::DefectOrRollbackAllForLogout
12 sqlservr!CSession::LogoutSession
13 sqlservr!CSession::RemoveBatch
14 sqlservr!process_close_connection
15 sqlservr!process_commands
16 sqlservr!SOS_Task::Param::Execute
17 sqlservr!SOS_Scheduler::RunTask
18 sqlservr!SOS_Scheduler::ProcessTasks
19 sqlservr!SchedulerManager::WorkerEntryPoint
1a sqlservr!SystemThread::RunWorker
1b sqlservr!SystemThreadDispatcher::ProcessWorker
1c sqlservr!SchedulerManager::ThreadEntryPoint
1d msvcr80!_callthreadstartex
1e msvcr80!_threadstartex
1f kernel32!BaseThreadInitThunk
20 ntdll!RtlUserThreadStart

Reply
Brian Gale said

February 9, 2017 at 12:04 AM
I have an interesting one as well:
0:004> .cxr sqlservr!g_copiedStackInfo+0X20
rax=1a77f9c300020006 rbx=0000000000000000 rcx=0000000000000002
rdx=000000001a77f9c3 rsi=000000098e3b0140 rdi=0000000000000000
rip=000000000245baca rsp=000000000ac9ea70 rbp=0000000000000001
r8=0000000991d84140 r9=0000000000000000 r10=0000000250b20003
r11=000000000ac9eab8 r12=0000000000000000 r13=0000000000000006
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
sqlservr!CCache::GetPcoFromPlanHandle+0x14a:
00000000`0245baca ?? ???
0:004> kc
*** Stack trace for last set context – .thread/.cxr resets it
Call Site
sqlservr!CCache::GetPcoFromPlanHandle
sqlservr!CProcStatsSTVFBase::GetRow
sqlservr!CQScanTVFStreamNew::GetRow
sqlservr!CQScanSortNew::BuildSortTable
sqlservr!CQScanSortNew::OpenHelper
sqlservr!CQScanNLJoinNew::GetRowHelper
sqlservr!CQScanTopNew::GetRow
sqlservr!CQScanSpoolNew::LoadSpool
sqlservr!CQScanSpoolNew::Open
sqlservr!CQScanNew::OpenHelper
sqlservr!CQScanUpdateNew::Open
sqlservr!CQueryScan::Startup
sqlservr!CXStmtQuery::SetupQueryScanAndExpression
sqlservr!CXStmtQuery::InitForExecute
sqlservr!CXStmtQuery::ErsqExecuteQuery
sqlservr!CXStmtDML::XretDMLExecute
sqlservr!CXStmtDML::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
0x0
0x0
0x0

Reply
- Brian Gale said
  
  March 1, 2017 at 9:05 PM
  Figured this one out. The problem was with a 3rd party tool called Red-Gate SQL Monitor. Version 7.0.0 had a bug with SQL 2008 R2 and older. Upgraded to 7.0.2 and the problem was resolved. 7.0.4 is out now but we have not yet upgraded.
  
  Reply
  - Karthick P.K said
    
    March 2, 2017 at 1:53 AM
    There is IO bottleneck on your server. Fix the disk performance in server.
    
    Sent from my iPhone
    
    Reply
Marc Lafortune said

March 1, 2017 at 9:01 PM
Call Site
ntdll!NtWriteFile
KERNELBASE!WriteFile
sqlservr!CErrorReportingManager::WriteToErrLog
sqlservr!CErrorReportingManager::SendErrorToErrLog
sqlservr!CErrorReportingManager::CwchFormatAndPrint
sqlservr!scierrlog
sqlservr!FCB::ReportLongIO
sqlservr!IoRequestDispenser::CheckForLongIo
sqlservr!BPool::LazyWriter
sqlservr!lazywriter
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
Ken said

May 17, 2017 at 6:22 PM
Any help with this one is greatly appreciated – thanks in advance!

This is from a Windows Server 2008R2 Enterprise SP1 VM / SQL2012 (11.0.6537.0) with 16GB RAM and 4 cores.
Non-Yielding scheduler occurs nightly and coincides with execution of “DBCC CHECKDB (DbNameHere) WITH no_infomsgs”, setup to run against all databases (in a loop) on the primary replica of our AG.

# Call Site
00 ntdll!ZwSignalAndWaitForSingleObject
01 kernel32!SignalObjectAndWait
02 sqldk!SOS_Scheduler::Switch
03 sqldk!SOS_Scheduler::SuspendNonPreemptive
04 sqlmin!EventInternal::Wait
05 sqlmin!AsynchronousDiskPool::WaitUntilDoneOrTimeout
06 sqlmin!AsynchronousDiskPool::WaitUntilDone
07 sqlmin!DBMgr::SyncAndLinkReplicaSetupPhase
08 sqlmin!DBMgr::CreatePhasedTransientReplica
09 sqlmin!DBMgr::CreateTransientReplica
0a sqlmin!DBDDLAgent::CreateReplica
0b sqlmin!UtilDbccCreateReplica
0c sqlmin!UtilDbccCheckDatabase
0d sqlmin!DbccCheckDB
0e sqllang!DbccCommand::Execute
0f sqllang!CStmtDbcc::XretExecute
10 sqllang!CMsqlExecContext::ExecuteStmts
11 sqllang!CMsqlExecContext::FExecute
12 sqllang!CSQLSource::Execute
13 sqllang!CStmtExecStr::XretExecStrExecute
14 sqllang!CXStmtExecStr::XretExecute
15 sqllang!CMsqlExecContext::ExecuteStmts
16 sqllang!CMsqlExecContext::FExecute
17 sqllang!CSQLSource::Execute
18 sqllang!process_request
19 sqllang!process_commands
1a sqldk!SOS_Task::Param::Execute
1b sqldk!SOS_Scheduler::RunTask
1c sqldk!SOS_Scheduler::ProcessTasks
1d sqldk!SchedulerManager::WorkerEntryPoint
1e sqldk!SystemThread::RunWorker
1f sqldk!SystemThreadDispatcher::ProcessWorker
20 sqldk!SchedulerManager::ThreadEntryPoint
21 kernel32!BaseThreadInitThunk
22 ntdll!RtlUserThreadStart

Reply
laoki said

June 13, 2017 at 1:30 AM
Can you please help?

Thanks in advance.

00 ntdll!NtClose
01 KERNELBASE!CloseHandle
02 sqlmin!FCB::Close
03 sqlmin!FileMgr::CreateNewFile
04 sqlmin!AsynchronousDiskAction::ExecuteDeferredAction
05 sqlmin!AsynchronousDiskWorker::ThreadRoutine
06 sqlmin!SubprocEntrypoint
07 sqldk!SOS_Task::Param::Execute
08 sqldk!SOS_Scheduler::RunTask
09 sqldk!SOS_Scheduler::ProcessTasks
0a sqldk!SchedulerManager::WorkerEntryPoint
0b sqldk!SystemThread::RunWorker
0c sqldk!SystemThreadDispatcher::ProcessWorker
0d sqldk!SchedulerManager::ThreadEntryPoint
0e kernel32!BaseThreadInitThunk
0f ntdll!RtlUserThreadStart

Reply
MikeL said

June 19, 2018 at 10:03 PM
This is on a SQL 2012 server (11.0.3156.0 (X64))

0:081> .cxr sqlmin!g_copiedStackInfo+0X20
rax=0000000000000cc0 rbx=000000000111b028 rcx=00000006d6b01fe0
rdx=00000006d6b014f0 rsi=0000000000001430 rdi=0000000000000000
rip=00007ff9e90306fa rsp=000000000b5ce290 rbp=00007ff67c6c0000
r8=00000006d6b01310 r9=00000006d6b012f0 r10=00000006d6b42030
r11=000000001c47de50 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=000000007ffe0382
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
ntdll!NtWaitForSingleObject+0xa:
00007ff9`e90306fa c3 ret
0:081> kc 100
# Call Site
00 ntdll!NtWaitForSingleObject
01 ntdll!RtlpWaitOnCriticalSection
02 ntdll!RtlpEnterCriticalSectionContended
03 KERNELBASE!FindClose
04 sqlmin!CMemoryDumpInfoTable::InternalReleaseResources
05 sqlmin!CSTVFInternal::Release
06 sqlmin!CQueryExecContext::~CQueryExecContext
07 sqlmin!CQueryScan::ShutdownQueryExecContext
08 sqlmin!CQueryScan::DestroyQuery
09 sqllang!CExecuteStatement::Destroy
0a sqllang!CMetaStmtSet::DeleteXstmtsInPartition
0b sqllang!CMetaStmtSet::DeleteAllXstmts
0c sqllang!CMetaStmtSet::Cleanup
0d sqllang!CCompPlan::~CCompPlan
0e sqllang!CCompPlan::`scalar deleting destructor’
0f sqllang!CCacheObject::Release
10 sqllang!CMsqlExecContext::~CMsqlExecContext
11 sqllang!CMsqlExecContext::Destroy
12 sqllang!CMsqlExecContext::CacheOrRelease
13 sqllang!CSQLObject::CleanupMxcAndClrStuff
14 sqllang!CSQLObject::`vector deleting destructor’
15 sqllang!CXStmtExec::CleanupExecLevel
16 sqllang!CXStmtExec::FinishCommon
17 sqllang!CMsqlExecContext::ExecuteStmts
18 sqllang!CMsqlExecContext::FExecute
19 sqllang!CSQLSource::Execute
1a sqllang!process_request
1b sqllang!process_commands
1c sqldk!SOS_Task::Param::Execute
1d sqldk!SOS_Scheduler::RunTask
1e sqldk!SOS_Scheduler::ProcessTasks
1f sqldk!SchedulerManager::WorkerEntryPoint
20 sqldk!SystemThread::RunWorker
21 sqldk!SystemThreadDispatcher::ProcessWorker
22 sqldk!SchedulerManager::ThreadEntryPoint
23 kernel32!BaseThreadInitThunk
24 ntdll!RtlUserThreadStart

Thank you!
-Mike-

Reply
Russell Young said

February 28, 2019 at 9:25 PM
0:092> .cxr sqlmin!g_copiedStackInfo+0X20
rax=0000000000006338 rbx=00007ffb9c97a000 rcx=00000000000129a8
rdx=00007ffb9c9c46a0 rsi=0000000000000004 rdi=00007ffb9b0b2d01
rip=00007ffba7825d87 rsp=00000000091ee530 rbp=00000000091eec80
r8=0000000000692d01 r9=0000000000006338 r10=0000000000005d63
r11=000000000000690d r12=0000000000000000 r13=0000000000000000
r14=0000000000000014 r15=00000000091eec80
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00010202
ntdll!RtlpSearchFunctionTable+0x3f:
00007ffb`a7825d87 8b02 mov eax,dword ptr [rdx] ds:00007ffb`9c9c46a0=????????
0:092> Kc 100
# Call Site
00 ntdll!RtlpSearchFunctionTable
01 ntdll!RtlpLookupFunctionEntryForStackWalks
02 ntdll!RtlpWalkFrameChain
03 ntdll!RtlWalkFrameChain
04 ntdll!RtlCaptureStackBackTrace
05 sqldk!ExceptionBackout::GetCurrentException
06 sqldk!ex_raise2
07 sqldk!ex_raise
08 sqllang!RaiseExecutionAbortedError
09 sqllang!YieldAndCheckForAbort
0a sqllang!COptExpr::PqteConvert
0b sqllang!COptExpr::PqteConvertTree
0c sqllang!COptContext::PcxteOptimizeQuery
0d sqllang!COptContext::PqteOptimizeWrapper
0e sqllang!PqoBuild
0f sqllang!CStmtQuery::InitQuery
10 sqllang!CStmtSelect::Init
11 sqllang!CCompPlan::FCompileStep
12 sqllang!CSQLSource::FCompile
13 sqllang!CSQLSource::FCompWrapper
14 sqllang!CSQLSource::Transform
15 sqllang!CSQLSource::Execute
16 sqllang!CStatsUtil::PCStatManMaxdiffHistogram
17 sqllang!CStatsUtil::PersistQPStatsInCatalogs
18 sqllang!CStatsUtil::CreateQPStats
19 sqllang!CStatsDef::CreateStats
1a sqllang!CStmtCreateStats::XretExecute
1b sqllang!CMsqlExecContext::ExecuteStmts
1c sqllang!CMsqlExecContext::FExecute
1d sqllang!CSQLSource::Execute
1e sqllang!CSQLSource::SeExecute
1f sqllang!CExecAutoCreateStats::FExecute
20 sqllang!COptContext::FInstantiateGuessedStats
21 sqllang!COptContext::PexprSimplify
22 sqllang!COptContext::PcxteOptimizeQuery
23 sqllang!COptContext::PqteOptimizeWrapper
24 sqllang!PqoBuild
25 sqllang!CStmtQuery::InitQuery
26 sqllang!CStmtDML::InitNormal
27 sqllang!CStmtDML::Init
28 sqllang!CCompPlan::FCompileStep
29 sqllang!CSQLSource::FCompile
2a sqllang!CSQLSource::FCompWrapper
2b sqllang!CSQLSource::Transform
2c sqllang!CSQLSource::Execute
2d sqllang!process_request
2e sqllang!process_commands
2f sqldk!SOS_Task::Param::Execute
30 sqldk!SOS_Scheduler::RunTask
31 sqldk!SOS_Scheduler::ProcessTasks
32 sqldk!SchedulerManager::WorkerEntryPoint
33 sqldk!SystemThread::RunWorker
34 sqldk!SystemThreadDispatcher::ProcessWorker
35 sqldk!SchedulerManager::ThreadEntryPoint
36 kernel32!BaseThreadInitThunk
37 ntdll!RtlUserThreadStart

Reply
Charly Zv said

May 12, 2021 at 12:29 AM
Microsoft (R) Windows Debugger Version 10.0.19041.685 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Program Files\Microsoft SQL Server\MSSQL11.PANACIM\MSSQL\Log\SQLDump0001.mdmp]
Comment: ‘Stack Trace’
Comment: ‘Non-yielding IOCP Listener’
User Mini Dump File: Only registers, stack and portions of memory are available

************* Path validation summary **************
Response Time (ms) Location
Deferred srv*c:\symbols*http://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response Time (ms) Location
Deferred srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Windows 8 Version 9200 MP (4 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Machine Name:
Debug session time: Mon May 10 12:09:50.000 2021 (UTC – 5:00)
System Uptime: 7 days 5:42:56.159
Process Uptime: 7 days 5:42:33.000
……………………………………………………….
…………………………………….
Loading unloaded module list
……………………………………………………….
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(758.84c): Unknown exception – code 00000000 (first/second chance not available)
For analysis of this file, run !analyze -v
ntdll!NtWaitForSingleObject+0xa:
00007ffc`9b03062a c3 ret
0:005> .sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
Symbol search path is: srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols

************* Path validation summary **************
Response Time (ms) Location
Deferred srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
0:005> .reload /f
…………

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

…………………………………………….
…………………………………….
Loading unloaded module list
……………………………………………………….

************* Symbol Loading Error Summary **************
Module name Error
sqlevn70 The system cannot find the file specified
SQLNCLIR11 The system cannot find the file specified
XPStar The system cannot find the file specified
xplog70 The system cannot find the file specified

You can troubleshoot most symbol related issues by turning on symbol loading diagnostics (!sym noisy) and repeating the command that caused symbols to be loaded.
You should also verify that your symbol search path (.sympath) is correct.
0:005> lmvm sqlservr
Browse full module list
start end module name
00007ff6`72520000 00007ff6`7257a000 sqlservr (pdb symbols) c:\websymbols\sqlservr.pdb\43E27FD6623645F4986ADEF7EF0EE1852\sqlservr.pdb
Loaded symbol image file: sqlservr.exe
Mapped memory image file: C:\Program Files\Microsoft SQL Server\MSSQL11.PANACIM\MSSQL\Binn\sqlservr.exe
Image path: C:\Program Files\Microsoft SQL Server\MSSQL11.PANACIM\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Browse all global symbols functions data
Timestamp: Sat Oct 31 23:43:43 2020 (5F9E4B0F)
CheckSum: 00037AA4
ImageSize: 0005A000
File version: 2011.110.7507.2
Product version: 11.0.7507.2
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Microsoft Corporation
ProductName: Microsoft SQL Server
InternalName: SQLSERVR
OriginalFilename: SQLSERVR.EXE
ProductVersion: 11.0.7507.2
FileVersion: 2011.0110.7507.02 ((SQL11_SP4_QFE-OD).201101-0030 )
FileDescription: SQL Server Windows NT – 64 Bit
LegalCopyright: Microsoft Corp. All rights reserved.
LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
Comments: SQL
0:005> !findstack sqlservr!SQL_SOSNonYieldSchedulerCallback
0:005> .cxr sqlmin!g_copiedStackInfo+0X20
rax=0000000000000005 rbx=0000000000000000 rcx=0000000000000c08
rdx=0000000000000000 rsi=00000001c3748520 rdi=0000000000000c08
rip=00007ffc9b03064a rsp=0000000005c2eaf0 rbp=0000000005c2ec78
r8=0000000000000000 r9=0000000000000012 r10=0000000000000000
r11=000001c5c0a6af69 r12=0000000000000000 r13=00000002e414cec0
r14=0000000000000000 r15=0000000005c2ed38
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
ntdll!NtReadFile+0xa:
00007ffc`9b03064a c3 ret
0:005> kc 100
# Call Site
00 ntdll!NtReadFile
01 KERNELBASE!ReadFile
02 sqllang!Np::ReadAsync
03 sqllang!Smux::PostReadAsync
04 sqllang!Smux::ReadDone
05 sqllang!SNIReadDone
06 sqldk!SOS_Node::ListenOnIOCompletionPort
07 sqldk!SOS_Task::Param::Execute
08 sqldk!SOS_Scheduler::RunTask
09 sqldk!SOS_Scheduler::ProcessTasks
0a sqldk!SchedulerManager::WorkerEntryPoint
0b sqldk!SystemThread::RunWorker
0c sqldk!SystemThreadDispatcher::ProcessWorker
0d sqldk!SchedulerManager::ThreadEntryPoint
0e kernel32!BaseThreadInitThunk
0f ntdll!RtlUserThreadStart

Reply