MSSQLWIKI

Karthick P.K on SQL Server

Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes

Posted by Karthick P.K on August 21, 2012

Do you see below errors in SQL error along with dumps and stuck?

Non-yielding IOCP Listener

* BEGIN STACK DUMP:
*   05/06/12 03:54:59 spid 0
* Non-yielding IOCP Listener

Non-yielding Scheduler
* BEGIN STACK DUMP:
*   04/16/12 10:09:58 spid 6256
* Non-yielding Scheduler

Non-yielding Resource Monitor

* BEGIN STACK DUMP

*   01/22/09 19:11:16 spid 0

* Non-yielding Resource Monitor

External dump process returned no errors.
Date Time Server Process 0:0:0 (0x31e8) Worker 0x000000016F41d140 appears to be non-yielding on Scheduler 4. Thread creation time: 12010668087858. Approx Thread CPU Used: kernel 2 ms, user 60516 ms. Process Utilization 11%. System Idle 83%. Interval: 71227 ms.

Refer “How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps”  for analyzing the Non-yielding Scheduler, Non-yielding IOCP Listener and Non-yielding Resource Monitor Dumps.

If you are interested in just finding a quick resolution follow the below steps to get the Non-Yield stack from the dump and check if it is matching with any existing known issues in SQL Server. 

To analyze the dump download and Install Windows Debugger from This  link

Step 1:

Open Windbg

step 2:

Choose File menu –> select Open crash dump –>Select the Dump file (SQLDump000#.mdmp)

Step 3:

on command window type    
.sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;

Step 4:

Type .reload /f and hit enter. This will force debugger to immediately load all the symbols.

Step 5:

Type  .cxr sqlservr!g_copiedStackInfo+0X20   for SQL Server2005 and SQL Server2008/2008R2  (or)   .cxr sqlmin!g_copiedStackInfo+0X20  for SQL Server2012.

Type kc 100 and look at the stack to see if it matches with the stack of any of known issues in SQL Server listed below.

If kc 100 doesn’t display any stack and throws “WARNING: Frame IP not in any known module. Following frames may be wrong”  type .cxr to reset to default scope and try  .cxr sqlservr!g_copiedStackInfo+0X00c (In 32-Bit (X86)  SQL server valid offset for context is 0X00c Look at This blog to see how we identified the offset)

Note:If your stack doesn’t match with any of the stack  listed below then paste the stack in comments session of this blog (or) In This face book group.We will try to find the cause for you. If you don’t get any prompt reply from the community, you may need to open a support ticket with Microsoft.

 

Stack 1

sqlservr!COptExpr::DetachPointersIntoMemo

sqlservr!COptExpr::DetachPointersIntoMemo

sqlservr!COptExpr::DetachPointersIntoMemo

sqlservr!COptExpr::DetachPointersIntoMemo

sqlservr!COptContext::PcxteOptimizeQuery

sqlservr!CQuery::Optimize

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtDML::InitNormal

sqlservr!CStmtDML::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

sqlservr!CSQLSource::Execute

sqlservr!ExecuteSql

sqlservr!CSpecProc::ExecuteSpecial

sqlservr!CXProc::Execute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExecProc::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB :2344600:FIX: "Non-yielding Scheduler" error may occur when you use the CONTAINSTABLE function together with many OR and AND predicates in SQL Server 2008 or in SQL Server 2008 R2

Stack 2

sqlservr!TMatchPattern

sqlservr!FMatchStrTxt

sqlservr!I8CharindexStrBhI8

sqlservr!CEs::GeneralEval4

sqlservr!CXStmtCond::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,0>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands 0x12a

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

2633357 FIX: "Non-yielding Scheduler" error might occur when you run a query that uses the CHARINDEX function in SQL Server 2008 R2

Stack 3

sqlservr!CItvlVal::Copy

sqlservr!CConstraintItvl::PcnstrItvlUnion

sqlservr!CConstraintProp::FBuildItvlFromOr

sqlservr!CConstraintProp::FBuildItvlFromPexpr

sqlservr!CConstraintProp::FAndItvlConstraint

sqlservr!CConstraintProp::AndNewConstraint

sqlservr!CConstraintProp::PcnstrDeriveSelect

sqlservr!CLogOp_Select::PcnstrDerive

sqlservr!CLogOpArg::PcnstrDeriveHandler

sqlservr!CLogOpArg::DeriveGroupProperties

sqlservr!COpArg::DeriveNormalizedGroupProperties

sqlservr!COptExpr::DeriveGroupProperties

sqlservr!COptExpr::DeriveGroupProperties 0xc6

sqlservr!COptExpr::DeriveGroupProperties

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtDML::InitNormal

sqlservr!CStmtDML::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

KB: 982376 FIX: A non-yielding scheduler error or an error 8623 occurs when you run a query that contains a large IN clause in SQL Server 2005,SQL Server 2008, or SQL Server 2008 R2

Stack 4

sqlservr!COptExpr::AdjustParallelPlan

sqlservr!COptContext::PcxteOptimizeQuery

sqlservr!CQuery::Optimize

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtSelect::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB: 943060 FIX: A query that has many outer joins takes a long time to compile in SQL Server 2005

Stack 5

sqlservr!CXid::GetBlockingTask

sqlservr!SNode::SearchForDeadlock

sqlservr!DeadlockMonitor::SearchForDeadlock

sqlservr!DeadlockMonitor::SearchAndResolve

sqlservr!DeadlockMonitor::SearchTaskAndResolve

sqlservr!DeadlockMonitor::WorkLoop

sqlservr!lockMonitor

sqlservr!lockMonitorThread

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

956854 Cumulative update package 10 for SQL Server 2005 Service Pack 2

Stack 6

ntdll!ZwQueryAttributesFile

ntdll!RtlDoesFileExists_UstrEx

ntdll!LdrpSearchPath

ntdll!LdrpCheckForLoadedDll

ntdll!LdrpLoadDll

ntdll!LdrLoadDll

kernel32!LoadLibraryExW

mswsock!SockLoadHelperDll

mswsock!SockGetTdiName

mswsock!SockSocket

mswsock!WSPSocket

ws2_32!WSASocketW

ws2_32!WSASocketA

sqlservr!CreateSocket

sqlservr!AcceptObject::AsyncAccept

sqlservr!Tcp::AcceptDone

sqlservr!SNIAcceptDoneWithReturnCode

sqlservr!SNIAcceptDoneWrapper

sqlservr!SNIAcceptDoneRouter

sqlservr!SOS_Node::ListenOnIOCompletionPort

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB 2711549 FIX: An error message is logged when you start SQL Server 2008 R2 or when a client sends a request to SQL Server 2008 R2

Stack 7

ntdll!ZwOpenKey

advapi32!LocalBaseRegOpenKey

advapi32!RegOpenKeyExW

sqlservr!COledbConnect::GetProviderOptions

sqlservr!COledbConnect::SetClsidFromProvider

sqlservr!COledbConnect::Init

sqlservr!CStmtExecProc::XretRemoteExec

sqlservr!CRemoteProcExecLevel::Execute

sqlservr!CStmtExecProc::XretWrapRemoteExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExec::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

KB2468047 FIX: Error code 17883 or "Non-yielding Scheduler" error may occur when you use the OPENQUERY function on SQL Server 2005

Stack 8

ntdll!ZwQueryVirtualMemory

psapi!QueryWorkingSetEx

sqlservr!BPool::Shrink

sqlservr!BPool::ReleaseAwayBufs

sqlservr!BPool::LazyWriter

sqlservr!lazywriter

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

967908 Cumulative update package 13 for SQL Server 2005 Service Pack 2 or 970279 Cumulative update package 4 for SQL Server 2005 Service Pack 3

Stack 9

sqlservr!LatchBase::ReleaseInternal

sqlservr!XVB::GetRecord

sqlservr!RowsetVersionScan::GetData

sqlservr!CQScanRowsetNew::GetRowWithPrefetch

sqlservr!CQScanRowsetNew::GetRow

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRow

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRow

sqlservr!CQueryScan::GetRow

sqlservr!CXStmtQuery::InitForExecute

sqlservr!CXStmtQuery::ErsqExecuteQuery

sqlservr!CXStmtCondWithQuery::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CXStmtDML::FExecTrigger

sqlservr!CXStmtDML::FExecAllTriggers

sqlservr!CXStmtDML::XretDMLExecute

sqlservr!CXStmtDML::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<0,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtPrepQuery::XretExecute

sqlservr!CExecuteStatement::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecStr::XretExecStrExecute

sqlservr!CXStmtExecStr::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB : 949595 FIX: Error message when you run a query that uses a join condition in SQL Server 2005: "Non-yielding Scheduler"

Stack 10

sqlservr!SQLServerLogIter::LookupScanCache

sqlservr!SQLServerLogIterForward::GetNextBlock

sqlservr!SQLServerLogIterForward::GetNext

sqlservr!LsMgr::GetEndOfLog

sqlservr!LsMgr::ProcessInternalRollForward

sqlservr!LsWorkRequest::Execute

sqlservr!LsWorker::ThreadRoutine

sqlservr!LsWorker::ThreadRoutine

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB 970044 FIX: Error message when you try to break database mirroring between two servers that are running SQL Server 2008: "Non-yielding Scheduler"

Stack 11

sqlservr!CLinkedMap

sqlservr!CCheckReadersAndWriters::Release

sqlservr!CMainIlb::~CMainIlb

sqlservr!CBlobHandleFactoryMain::ReleaseILockBytes

sqlservr!CMainIlb::Release

sqlservr!CTraceRpcBinaryStream::~CTraceRpcBinaryStream

sqlservr!CTraceTvpData::~CTraceTvpData

sqlservr!CRpcTraceHelper::CleanUpTraceTvpData

sqlservr!CRpcTraceHelper::TracePostExec

sqlservr!CRPCExecEnv::OnExecFinish

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!endthreadex

msvcr80!endthreadex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2520808 FIX: Non-yielding scheduler error when you run a query that uses a TVP in SQL Server 2008 or in SQL Server 2008 R2 if SQL Profiler or SQL Server Extended Events is used

Stack 12

ntdll!ZwFreeVirtualMemory

KERNELBASE!VirtualFree

sqlservr!MemoryNode::VirtualFree

sqlservr!ReservedMemBlock::FreeMemory

sqlservr!MultiPageAllocator::FreePagesInternal

sqlservr!MultiPageAllocator::FreePages

sqlservr!MemoryNode::FreePagesInternal

sqlservr!MemoryClerkInternal::FreePagesInline

sqlservr!CVarPageMgr::Release

sqlservr!CMemObj::Free

sqlservr!CMemThread<CMemObj>::Free

sqlservr!LockBytesSS::~LockBytesSS

sqlservr!LockBytesHolder::`scalar deleting destructor’

sqlservr!LockBytesHolder::DestroyCallback

sqlservr!CacheLbss

sqlservr!LockBytesSS::Release

sqlservr!CQueryIlb::~CQueryIlb

sqlservr!CBlobHandleFactoryMain::ReleaseILockBytes

sqlservr!CMainIlb::Release

sqlservr!CTraceRpcBinaryStream::~CTraceRpcBinaryStream

sqlservr!CTraceTvpData::~CTraceTvpData

sqlservr!CRpcTraceHelper::CleanUpTraceTvpData

sqlservr!CRpcTraceHelper::TracePostExec

sqlservr!CRPCExecEnv::OnExecFinish

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!endthreadex

msvcr80!endthreadex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2520808 FIX: Non-yielding scheduler error when you run a query that uses a TVP in SQL Server 2008 or in SQL Server 2008 R2 if SQL Profiler or SQL Server Extended Events is used

Stack 13

sqlservr!CompareStringWEnglishNoCase

sqlservr!CTypeInfo::ICompW

sqlservr!CDefaultCollation::ICompW

sqlservr!CDependElem::ICompare

sqlservr!CDependList::Find

sqlservr!CDependList::Insert

sqlservr!CDependList::Concat

sqlservr!CDependList::CollectDependencies

sqlservr!FillSysdepends

sqlservr!CProchdr::CreateProc

sqlservr!CSQLSource::PerformPphFakeExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2306162 FIX: Poor performance and some occasional non-yielding scheduler errors occur when you create a complex view that references a large amount of nested views or tables in SQL Server 2008 or in SQL Server 2008 R2

Stack 14

sqlservr!BaseSharedHoBt::GetHoBtId

sqlservr!HoBtFactory::GetDeferredDropCacheHobt

sqlservr!DropDeferredWorkTables

sqlservr!GhostRecordCleanupTask

sqlservr!CGhostCleanupTask::ProcessTskPkt

sqlservr!TaskReqPktTimer::ExecuteTask

sqlservr!OnDemandTaskContext::ProcessTskPkt

sqlservr!SystemTaskContext::ExecuteFunc

sqlservr!SystemTaskEntryPoint

sqlservr!OnDemandTaskContext::FuncEntryPoint

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SchedulerManager::FiberEntryPoint

kernel32!BaseFiberStart

kernel32!RtlCompareMemoryStub

KB 2505256 FIX: Poor performance when worktables that are marked for deferred drop are cleaned up in SQL Server 2008 R2

Stack 15

ntdll!ZwReadFile

kernel32!ReadFile

sqlservr!DiskReadAsync

sqlservr!FCB::AsyncRead

sqlservr!BackupIoRequest::StartDatabaseRead

sqlservr!BackupCopyMachine::CopyFileToBackupSet0

sqlservr!BackupCopyMachine::CopyFileToBackupSet

KB 960543 FIX: SQL Server 2005 or SQL Server 2008 may stop responding when you are performing a backup

Stack 16

sqlservr!Worker::ProfilingCPUTicks::ProfilingCpuTicksCallback

sqlservr!SOS_Scheduler::TaskTransition

sqlservr!SOS_Scheduler::Switch

sqlservr!SOS_Scheduler::SuspendNonPreemptive

sqlservr!SOS_Scheduler::Suspend

sqlservr!SOS_Task::Sleep

sqlservr!BTreeMgr::Seek

sqlservr!BTreeMgr::GetHPageIdWithKey

sqlservr!IndexPageManager::GetPageWithKey

sqlservr!GetRowForKeyValue

sqlservr!IndexRowScanner::EstablishInitialKeyOrderPosition

sqlservr!IndexDataSetSession::GetNextRowValuesInternal

sqlservr!RowsetNewSS::GetNextRows

sqlservr!CMEDScan::FGetRow

sqlservr!CMEDCatalogOwner::GetOwnerAliasIdFromSid

sqlservr!CMEDCatalogOwner::LookupPrimaryIdInCatalog

sqlservr!CMEDCacheEntryFactory::GetProxiedCacheEntryByAltKey

sqlservr!CMEDCatalogOwner::GetProxyOwnerBySID

sqlservr!CMEDProxyDatabase::GetOwnerBySID

sqlservr!GetDefaultSchemaIdCrossDb

sqlservr!GetCtxtSchemaId

sqlservr!CMEDAccess::GetMultiNameObject

sqlservr!CRangeObject::CImplName::FSameObject

sqlservr!CRangeObject::FCheckImplNames

sqlservr!CRangeObject::XretPostSchemaChecks

sqlservr!CRangeObject::XretSchemaChanged

sqlservr!CRangeTable::XretSchemaChanged

sqlservr!CEnvCollection::XretSchemaChanged

sqlservr!CXStmtQuery::XretSchemaChanged

sqlservr!CXStmtSelect::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExecProc::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

KB 2699013 FIX: SQL Server 2008 R2 or SQL Server 2008 stops responding and a "Non-yielding Scheduler" error is logged

Stack 17

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQueryScan::GetRow

sqlservr!CXStmtQuery::ErsqExecuteQuery

sqlservr!CXStmtCondWithQuery::XretExecute

sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn

sqlservr!CMsqlExecContext::ExecuteStmts<1,0>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CXStmtDML::FExecTrigger

sqlservr!CXStmtDML::FExecAllTriggers

sqlservr!CXStmtDML::XretDMLExecute

sqlservr!CXStmtDML::XretExecute

sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn

sqlservr!CMsqlExecContext::ExecuteStmts<0,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtPrepQuery::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

KB 967169 FIX: When you run an UPDATE statement against a table that has a FOR UPDATE trigger that joins the DELETED and INSERTED tables, the query takes a long time to finish

Stack 18

msvcr80!memcpy

BackupString::vswcatf

BackupString::swcatf

BackupHistory::GenerateBackupDetails

sqlservr!BackupHistory::GenerateBackupSet

KB 917971 FIX: You may receive more than 100,000 page faults when you try to back up a SQL Server 2005 database that contains hundreds of files and file groups.

Stack 20

mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!Smux::InternalClose
sqlservr!Smux::ReadDone

"Non-yielding Scheduler" error and SQL Server 2008 or SQL Server 2008 R2 stops responding intermittently in Windows Server 2008 or in Windows Server 2008 R2

Stack 21

mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!Smux::InternalClose
sqlservr!Smux::ReadDone

"Non-yielding Scheduler" error and SQL Server 2008 or SQL Server 2008 R2 stops responding intermittently in Windows Server 2008 or in Windows Server 2008 R2

 

Related blogs: 

SQL Server Exception , EXCEPTION_ACCESS_VIOLATION and SQL Server Assertion

How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps ……

SQL Server Latch & Debugging latch time out

How to Analyze "Deadlocked Schedulers" Dumps

 

If you liked this post, do like us on Facebook at https://www.facebook.com/mssqlwiki and join our Facebook group to get answers for all your SQL Server related questions.

 

Thank you,

Karthick P.K |My Facebook Page |My Site| Blog space| Twitter

Disclaimer:

The views expressed on this website/blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights

138 Responses to “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”

  1. Pavel Chromy said

    Hi, our stack doesn’t match with any of the stack :
    Call Site
    sqlservr!SpinlockBase::SpinToAcquire
    sqlservr!SOS_Event::Signal
    sqlservr!BPool::Steal
    sqlservr!SQLSinglePageAllocator::AllocatePages
    sqlservr!MemoryNode::AllocatePagesInternal
    sqlservr!MemoryClerkInternal::AllocatePages
    sqlservr!CMemThread::PbGetNewPages
    sqlservr!CVarPageMgr::PviNewVarPage
    sqlservr!CVarPageMgr::PbAllocate
    sqlservr!CMemThread::Alloc
    sqlservr!operator new
    sqlservr!SNI_Conn::InitObject
    sqlservr!SNIAcceptDoneWithReturnCode
    sqlservr!SNIAcceptDoneWrapper
    sqlservr!SNIAcceptDoneRouter
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    Can you help me ?
    Thanks
    Pavel Chromy

    • Will look at this in few minutes. I am driving now

      Sent from my Windows Phone ________________________________

    • Can you paste the following info
      1. SQL Server version and windows version?
      2. Amont of physical server memory on this server?
      3.What is sQL Server max server memory and Max dop?
      4. Are we suing SQLClr in this server?
      5. How frequently are we getting this dump and errorlog message similar to Thread CPU Used: kernel n ms, user n ms

      • Pavel Chromy said

        Sending additional info :

        Microsoft SQL Server 2005 version 9.00.5296.05
        Computer type is AT/AT COMPATIBLE.
        Bios Version is DELL – 1
        Phoenix ROM BIOS PLUS Version 1.10 2.2.3
        40 Unknown CPU 9., 2 Mhz processor (s).
        Windows NT 6.1 Build 7601 CSD Service Pack 1.
        Memory Total Physical = 262130 MB

        sp_configure :
        name minimum maximum config_value run_value
        ———————————– ———– ———– ———— ———–
        Ad Hoc Distributed Queries 0 1 0 0
        affinity I/O mask -2147483648 2147483647 0 0
        affinity mask -2147483648 2147483647 0 0
        affinity64 I/O mask -2147483648 2147483647 0 0
        affinity64 mask -2147483648 2147483647 0 0
        Agent XPs 0 1 1 1
        allow updates 0 1 0 0
        awe enabled 0 1 0 0
        blocked process threshold 0 86400 5 5
        c2 audit mode 0 1 0 0
        clr enabled 0 1 0 0
        cost threshold for parallelism 0 32767 5 5
        cross db ownership chaining 0 1 0 0
        cursor threshold -1 2147483647 -1 -1
        Database Mail XPs 0 1 1 1
        default full-text language 0 2147483647 1033 1033
        default language 0 9999 0 0
        default trace enabled 0 1 1 1
        disallow results from triggers 0 1 0 0
        fill factor (%) 0 100 0 0
        ft crawl bandwidth (max) 0 32767 100 100
        ft crawl bandwidth (min) 0 32767 0 0
        ft notify bandwidth (max) 0 32767 100 100
        ft notify bandwidth (min) 0 32767 0 0
        index create memory (KB) 704 2147483647 0 0
        in-doubt xact resolution 0 2 0 0
        lightweight pooling 0 1 0 0
        locks 5000 2147483647 0 0
        max degree of parallelism 0 64 4 4
        max full-text crawl range 0 256 4 4
        max server memory (MB) 16 2147483647 2147483647 2147483647
        max text repl size (B) 0 2147483647 65536 65536
        max worker threads 128 32767 0 0
        media retention 0 365 0 0
        min memory per query (KB) 512 2147483647 1024 1024
        min server memory (MB) 0 2147483647 16 16
        nested triggers 0 1 1 1
        network packet size (B) 512 32767 4096 4096
        Ole Automation Procedures 0 1 0 0
        open objects 0 2147483647 0 0
        PH timeout (s) 1 3600 60 60
        precompute rank 0 1 0 0
        priority boost 0 1 1 1
        query governor cost limit 0 2147483647 0 0
        query wait (s) -1 2147483647 -1 -1
        recovery interval (min) 0 32767 0 0
        remote access 0 1 1 1
        remote admin connections 0 1 0 0
        remote login timeout (s) 0 2147483647 20 20
        remote proc trans 0 1 0 0
        remote query timeout (s) 0 2147483647 10800 10800
        Replication XPs 0 1 0 0
        scan for startup procs 0 1 0 0
        server trigger recursion 0 1 1 1
        set working set size 0 1 0 0
        show advanced options 0 1 1 1
        SMO and DMO XPs 0 1 1 1
        SQL Mail XPs 0 1 0 0
        transform noise words 0 1 0 0
        two digit year cutoff 1753 9999 2049 2049
        user connections 0 32767 0 0
        user options 0 32767 0 0
        Web Assistant Procedures 0 1 0 0
        xp_cmdshell 0 1 0 0

        SQLClr do not use.

        Over the past 14 days are few stiff SQL Server (Non-yielding IOCP Listener) – in the log then appears “IO Completion Listener (0×830) Worker 0x00000000112DC1C0 Appears to be non-yielding on Node 0th Approx CPU Used: kernel 187 ms, user 924898 ms, Interval: 925116th “processors working at 100% and the server is not responding.

        Thank you for any idea
        Pavel

        • Limit your SQL Server “Max server Memory” (Start with 200,000 MB . We can increase or reduce based on OS and other applications need).
          Most likely since all OS physical memory has been used and OS is struggling with intensive paging SQL Server is affected.
          So if we will limit SQL Server for amount of memory it will resolve this issue.
          Let me know the output of below command as well !findstack sqlservr!BPool::ReplenishFreeList.
          We will get this dump with this stack even if the lazywritter is not able to keep up to the speed of updates/Inserts because of I/O bottlenecks.

          • Pavel Chromy said

            Thank you for the advice, but unfortunately SQL server froze again. Max server memory I set to 150,000 megabytes. In the dump file is only “Non-yielding IOCP Listener”.

            • Did you get “Non-yielding IOCP Listener” dump after you set max server memory to 150000? how much was available physical memory in server when you got the dump?
              Check if stack is same in new dump ,Also run !findstack sqlservr!BPool::ReplenishFreeList and paste the output

              • Pavel Chromy said

                Yes, max memory has already been set to 150.000 MB. Physical memory was available 95.500 MB.
                Sending listing :
                Call Site
                ntdll!NtWaitForSingleObject
                KERNELBASE!WaitForSingleObjectEx
                sqlservr!CDmpDump::DumpInternal
                sqlservr!CDmpDump::Dump
                sqlservr!CImageHelper::DoMiniDump
                sqlservr!stackTrace
                sqlservr!SQL_SOSNonYieldIOCPCallback
                sqlservr!SOS_OS::ExecuteNonYieldIOCPCallbacks
                sqlservr!alloca_probe
                sqlservr!SchedulerMonitor::Run
                sqlservr!SchedulerMonitor::EntryPoint
                sqlservr!SOS_Task::Param::Execute
                sqlservr!SOS_Scheduler::RunTask
                sqlservr!SOS_Scheduler::ProcessTasks
                sqlservr!SchedulerManager::WorkerEntryPoint
                sqlservr!SystemThread::RunWorker
                sqlservr!SystemThreadDispatcher::ProcessWorker
                sqlservr!SchedulerManager::ThreadEntryPoint
                msvcr80!endthreadex
                msvcr80!endthreadex
                kernel32!BaseThreadInitThunk
                ntdll!RtlUserThreadStart

                • This look like scheduler monitor stack. You need to use .cxr sqlservr!g_copiedStackInfo+0X20 and then use kC to get the Non-Yield stack

                  • Pavel Chromy said

                    Sorry, here is the correct listing :
                    Call Site
                    sqlservr!SpinlockBase::SpinToAcquire
                    sqlservr!SOS_Event::Signal
                    sqlservr!alloca_probe
                    sqlservr!SQLSinglePageAllocator::AllocatePages
                    sqlservr!MemoryNode::AllocatePagesInternal
                    sqlservr!MemoryClerkInternal::AllocatePages
                    sqlservr!CMemThread::PbGetNewPages
                    sqlservr!CMemFixed::Alloc
                    sqlservr!CMemThread::Alloc
                    sqlservr!operator new
                    sqlservr!LockManager::AllocateLockOwnerBlocks
                    sqlservr!LockManager::GetLockOwnerBlocks
                    sqlservr!alloca_probe
                    sqlservr!XactWorkspaceImp::GetSharedDBLockFromLockManager
                    sqlservr!XactWorkspaceImp::GetDBLock
                    sqlservr!lockdb
                    sqlservr!CMEDUserStoreCallbacks::StartClockRun
                    sqlservr!ClockAlgorithm::MoveHand
                    sqlservr!ClockAlgorithm::ProcessTick
                    sqlservr!SOS_UserStore::Notify
                    sqlservr!ResourceMonitor::NotifyMemoryConsumers
                    sqlservr!ResourceMonitor::ResourceMonitorTask
                    sqlservr!SetupResourceMonitorTaskContext
                    sqlservr!SOS_Task::Param::Execute
                    sqlservr!SOS_Scheduler::RunTask
                    sqlservr!SOS_Scheduler::ProcessTasks
                    sqlservr!SchedulerManager::WorkerEntryPoint
                    sqlservr!SystemThread::RunWorker
                    sqlservr!SystemThreadDispatcher::ProcessWorker
                    sqlservr!SchedulerManager::ThreadEntryPoint
                    msvcr80!endthreadex
                    msvcr80!endthreadex
                    kernel32!BaseThreadInitThunk
                    ntdll!RtlUserThreadStart

                    We’re pretty desperate, do you think it would help to re-install Windows or SQL Server ?

  2. Pavel Chromy said

    The error log contains only::

    2012-10-29 04:25:41.63 Server Using ‘dbghelp.dll’ version ’4.0.5′
    2012-10-29 04:25:41.65 Server **Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
    2012-10-29 04:25:41.65 Server ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.txt
    2012-10-29 04:25:41.65 Server * *******************************************************************************
    2012-10-29 04:25:41.65 Server *
    2012-10-29 04:25:41.65 Server * BEGIN STACK DUMP:
    2012-10-29 04:25:41.65 Server * 10/29/12 04:25:41 spid 0
    2012-10-29 04:25:41.65 Server *
    2012-10-29 04:25:41.65 Server * Non-yielding IOCP Listener
    2012-10-29 04:25:41.65 Server *
    2012-10-29 04:25:41.65 Server * *******************************************************************************
    2012-10-29 04:25:41.65 Server * ——————————————————————————-
    2012-10-29 04:25:41.65 Server * Short Stack Dump
    2012-10-29 04:25:41.70 Server Stack Signature for the dump is 0×0000000000000207
    2012-10-29 04:25:44.80 Server External dump process return code 0×20000001.

    The dump file contains only:
    =====================================================================
    BugCheck Dump
    =====================================================================

    This file is generated by Microsoft SQL Server
    version 9.00.5296.05
    upon detection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is AT/AT COMPATIBLE.
    Bios Version is DELL – 1
    Phoenix ROM BIOS PLUS Version 1.10 2.7.7
    Current time is 04:25:41 10/29/12.
    40 Unknown CPU 9., 2 Mhz processor (s).
    Windows NT 6.1 Build 7600 CSD .

    Memory
    MemoryLoad = 90%
    Total Physical = 262130 MB
    Available Physical = 24467 MB
    Total Page File = 264176 MB
    Available Page File = 25893 MB
    Total Virtual = 8388607 MB
    Available Virtual = 8124449 MB
    **Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
    ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.txt
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 10/29/12 04:25:41 spid 0
    *
    * Non-yielding IOCP Listener
    *
    * *******************************************************************************
    * ——————————————————————————-
    * Short Stack Dump

    Lock pages is enabled. Yesterday we reinstalled the entire server (windows and SQL server – we apply the latest SP and hotfix) and after 3 hours the server again froze – unfortunately, in this state he can not run anything.

  3. Hi Karthick – I have stack dump but that is not matching anuy one of above listed dumps. Below is the dump that I got it, can you please help on this.

    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqlservr!CDmpDump::DumpInternal
    sqlservr!CDmpDump::DumpFilter
    sqlservr!CDmpDump::Dump
    msvcr80!__C_specific_handler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!RtlRaiseException
    KERNELBASE!RaiseException
    sqlservr!CDmpDump::Dump
    sqlservr!SQLDumperLibraryInvoke
    sqlservr!CImageHelper::DoMiniDump
    sqlservr!stackTrace
    sqlservr!stackTraceCallBack
    sqlservr!ex_terminator
    sqlservr!SOS_SEHTranslator
    msvcr80!_CallSETranslator
    msvcr80!FindHandlerForForeignException
    msvcr80!FindHandler
    msvcr80!__InternalCxxFrameHandler
    msvcr80!__CxxFrameHandler
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!KiUserExceptionDispatcher
    sqlservr!CSlotPageMgr::PbAllocate
    sqlservr!CMemObj::Alloc
    sqlservr!CMemThread::Alloc
    sqlservr!CCriticalSectionSOS::Initialize
    sqlservr!Tcp::FInit
    sqlservr!Tcp::AcceptConnection
    sqlservr!Tcp::AcceptDone
    sqlservr!SNIAcceptDoneWithReturnCode
    sqlservr!SNIAcceptDoneWrapper
    sqlservr!SNIAcceptDoneRouter
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  4. Scott McCormick said

    This is the error log a few hours before the dump you asked for in a Facebook post (https://www.facebook.com/mssqlwiki/posts/335361023227918)

    I’m assuming you don’t actually need usernames or IP addresses.

    2012-11-14 12:45:39.54 Logon Login succeeded for user ‘***\****’. Connection: trusted. [CLIENT: ***.***.***.***]
    2012-11-14 12:54:53.92 Server Using ‘dbghelp.dll’ version ’4.0.5′
    2012-11-14 12:54:56.81 Server **Dump thread – spid = 0, PSS = 0×00000000, EC = 0×00000000
    2012-11-14 12:54:56.81 Server ***Stack Dump being sent to ***\MSSQL.1\MSSQL\LOG\SQLDump0002.txt
    2012-11-14 12:54:56.81 Server * *******************************************************************************
    2012-11-14 12:54:56.81 Server *
    2012-11-14 12:54:56.81 Server * BEGIN STACK DUMP:
    2012-11-14 12:54:56.81 Server * 11/14/12 12:54:56 spid 0
    2012-11-14 12:54:56.81 Server *
    2012-11-14 12:54:56.81 Server * Non-yielding IOCP Listener
    2012-11-14 12:54:56.81 Server *
    2012-11-14 12:54:56.81 Server * *******************************************************************************
    2012-11-14 12:54:56.81 Server * ——————————————————————————-
    2012-11-14 12:54:56.81 Server * Short Stack Dump
    2012-11-14 12:55:00.36 Server Stack Signature for the dump is 0x0000036D
    2012-11-14 12:55:13.73 Logon Error: 18456, Severity: 14, State: 8.
    2012-11-14 12:55:20.67 Logon Login succeeded for user ‘***\****’. Connection: trusted. [CLIENT: ***.***.***.***]
    2012-11-14 13:03:04.04 Server Timeout waiting for external dump process 68908.

    2012-11-14 13:03:04.04 Server IO Completion Listener (0x5b0) Worker 0x008980E8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    ntdll!KiFastSystemCallRet
    ntdll!NtRequestWaitReplyPort
    secur32!CallSPM
    secur32!SecpAcceptSecurityContext
    secur32!LsaAcceptSecurityContext
    secur32!AcceptSecurityContext
    sqlservr!Ssl::Handshake
    sqlservr!Ssl::HandshakeReadDone
    sqlservr!CryptoBase::ReadDone
    sqlservr!SNIReadDone
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadStart

  5. Steve said

    Hi Karthick,

    I have a stack which is not listed. Can you help me with that?

    ntdll!KiFastSystemCallRet
    ntdll!ZwClose
    kernel32!CloseHandle
    sqlservr!CTraceFileIoProvider::OpenNextFile
    sqlservr!CTraceWriteRequest::InitForFileTrace
    sqlservr!CTraceFileIoProvider::InitializeWriteRequest
    sqlservr!CTrace::WriteRecord
    sqlservr!CTraceController::ProduceRecord
    sqlservr!CTraceController::ProduceRecordWithTextBase
    sqlservr!AuditEventBase
    sqlservr!AuditEventNoCheck
    sqlservr!AuditEvent
    sqlservr!DbccOperation::CleanupDbccOperation
    sqlservr!CStmtDbcc::XretExecute
    sqlservr!CExecuteStatement::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_endthreadex
    msvcr80!_endthreadex
    kernel32!BaseThreadStart

    Happend once.

    SQL Version: Microsoft SQL Server 2005 – 9.00.3080.00 (Intel X86)
    Sep 6 2009 01:43:32 Copyright (c) 1988-2005 Microsoft Corporation
    Enterprise Edition on Windows NT 5.2 (Build 3790: Service Pack 2)

    Windows Server 2003 Enterprise
    Build 379.srv03_sp2_gdr.120503-0334 SP2

    Physical Memory: 15.727.764 KB
    SQL Server 10GB

    Thanks

    • When profiler tace is collected to network location or to a disk which is very slow, we might end with this.
      1. Make sure you allways collect server side trace instead of client side traces when required.
      2. Dont collect multiple trace files to gether.

  6. [...] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… [...]

  7. Ewan Gibb said

    Hi,

    I’ve been through the steps in this blog and have the following stack trace:

    sqldk!CVariableInfo::PbGetBlock
    sqldk!TVarPageMgr::PbAllocate
    sqldk!CMemObj::Alloc
    sqlmin!BTreeRowFactory::AllocateNewPageContext
    sqlmin!SplitPage
    sqlmin!IndexDataSetSession::InsertSmallRecord
    sqlmin!IndexDataSetSession::InsertRowInternal
    sqlmin!DatasetSession::InsertRow
    sqlmin!RowsetNewSS::InsertRow
    sqlmin!CValRowMult::SetDataX
    sqltses!CEsExec::GeneralEval
    sqlmin!CQScanUpdateNew::GetRow
    sqlmin!CQScanNLJoinNew::GetRowHelper
    sqlmin!CQScanAssertNew::GetRow
    sqlmin!CQueryScan::GetRow
    sqllang!CXStmtQuery::ErsqExecuteQuery
    sqllang!CXStmtDML::XretDMLExecute
    sqllang!CXStmtDML::XretExecute
    sqllang!CMsqlExecContext::ExecuteStmts
    sqllang!CMsqlExecContext::FExecute
    sqllang!CSQLSource::Execute
    sqllang!ExecuteSql
    sqllang!CSpecProc::ExecuteSpecial
    sqllang!CSpecProc::Execute
    sqllang!process_request
    sqllang!process_commands
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    This is on a 64-bit Windows Server 2008 R2 Standard Edition server, 32gb ram, running SQL Server 2012 SP1.

    Microsoft SQL Server 2012 (SP1) – 11.0.3321.0 (X64)
    Nov 14 2012 19:41:38
    Copyright (c) Microsoft Corporation
    Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1)

    Any help would be much appreciated.

    Thanks!

    • Ewan,
      1. In SQLServer errolog immediately after this dump you will find message for “Approx CPU Used” can you share that line please. I am intrested in kernel and user time,
      2. Share your max server memory and min server memory config.
      3. Have you enabled LPIM for SQLServer startup account?

      • Ewan Gibb said

        Karthik,

        Thank you so much for the prompt response. I haven’t noticed it until today as I lost the link to this page. We are still having the problem only now it is happening with a higher regularity.

        To answer your questions:

        1 – I hope this is the log entry you are after:

        Process 0:0:0 (0x1fd0) Worker 0x00000007839F6160 appears to be non-yielding on Scheduler 2. Thread creation time: 13006905344342. Approx Thread CPU Used: kernel 0 ms, user 70465 ms. Process Utilization 6%. System Idle 92%. Interval: 70465 ms.

        2 – Minimum server memory (in MB): 0
        Maximum server memory (in MB): 2147483647

        3 – LPIM is not enabled.

        Kind regards.

        • How much RAM do we have on this system?

          • Ewan Gibb said

            The server has 32gb RAM.

            • Cap the MaxServermemory of SQL Server after considering the memory requirements by OS, other applications drivers, Non-bpool etc. If it is dedicated SQL Server start with 26GB. Let us increase or decrease on need later.

              • Ewan Gibb said

                Thanks! It is a dedicated SQL Server, so I have set it to 26gb. I will see how it goes today and monitor it closely.

                • Ewan Gibb said

                  Things looked fine for almost 8 hours, then the non yielding-scheduler message struck again. The stack trace is slightly different this time:

                  sqldk!CVariableInfo::PbGetBlock
                  sqldk!TVarPageMgr::PbAllocate
                  sqldk!CMemObj::Alloc
                  sqlmin!BTreeRowFactory::AllocateNewBTreeRow
                  sqlmin!IndexDataSetSession::InsertSmallRecord
                  sqlmin!IndexDataSetSession::InsertRowInternal
                  sqlmin!DatasetSession::InsertRow
                  sqlmin!RowsetNewSS::InsertRow
                  sqlmin!CValRowMult::SetDataX
                  sqltses!CEsExec::GeneralEval
                  sqlmin!CQScanUpdateNew::GetRow
                  sqlmin!CQScanNLJoinNew::GetRowHelper
                  sqlmin!CQScanAssertNew::GetRow
                  sqlmin!CQueryScan::GetRow
                  sqllang!CXStmtQuery::ErsqExecuteQuery
                  sqllang!CXStmtDML::XretDMLExecute
                  sqllang!CXStmtDML::XretExecute
                  sqllang!CMsqlExecContext::ExecuteStmts
                  sqllang!CMsqlExecContext::FExecute
                  sqllang!CSQLSource::Execute
                  sqllang!ExecuteSql
                  sqllang!CSpecProc::ExecuteSpecial
                  sqllang!CSpecProc::Execute
                  sqllang!process_request
                  sqllang!process_commands
                  sqldk!SOS_Task::Param::Execute
                  sqldk!SOS_Scheduler::RunTask
                  sqldk!SOS_Scheduler::ProcessTasks
                  sqldk!SchedulerManager::WorkerEntryPoint
                  sqldk!SystemThread::RunWorker
                  sqldk!SystemThreadDispatcher::ProcessWorker
                  sqldk!SchedulerManager::ThreadEntryPoint
                  kernel32!BaseThreadInitThunk
                  ntdll!RtlUserThreadStart

                  The entry in the log this time reads:

                  “Process 0:0:0 (0x1d80) Worker 0x00000004BBBEA160 appears to be non-yielding on Scheduler 19. Thread creation time: 13007047217492. Approx Thread CPU Used: kernel 0 ms, user 71339 ms. Process Utilization 7%. System Idle 92%. Interval: 71338 ms.”

                  Entries such as the above are logged at regular intervals until the SQL Server service is restarted.

                  In addition there are multiple examples of this:

                  “A time-out occurred while waiting for buffer latch — type 2, bp 000000002D049000, page 1:3704, stat 0xb, database id: 34, allocation unit Id: 20700475686912/302175452397568, task 0x00000008EE12B498 : 0, waittime 300 seconds, flags 0x3a, owning task 0x0000000482C42558. Not continuing to wait.”

                  If I run DBCC Opentran against all databases, the same database always has an open transaction. Using DBCC INPUTBUFFER(xxx) I can see that this is also the same each time.

                  Are there any other diagnostics I can perform?

                  The server was updated to SQL Server 2012 SP1 CU2 earlier in the week to see if that would help.

                  • 1. Again there is delay in memory allocation. ADD LPIM privilege to startup account of SQL Server. SQL Server uses different mechanism to allocate memory when we have LPIM, most probably it should resolve your issue.
                    2. Did Available memory on the system dropped very low when you got this dump? If so you may have to further reduce max server memory.
                    If issue doesn’t get resolve after enabling LPIM for startup account of SQL Server and restarting , share the dump and error log using your FTP share.I will review it for you. If there are security restrictions you may have to open a case with Microsoft support and can share the case number with me.

                    For the latch time out dump ““A time-out occurred while waiting for buffer latch” check if you have I/O request taking longer than %d seconds error message in SQL Error log? Follow the steps in http://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ once if you get the dump.

  8. Bian said

    Hi Karthik,

    Can you also help me on this stack?

    ntdll!ZwClose
    KERNELBASE!CloseHandle
    kernel32!CloseHandleImplementation
    sqlservr!CTraceFileIoProvider::OpenNextFile
    sqlservr!CTraceWriteRequest::InitForFileTrace
    sqlservr!CTraceFileIoProvider::InitializeWriteRequest
    sqlservr!CTrace::WriteRecord
    sqlservr!CTraceController::ProduceRecord
    sqlservr!CTraceDDL::TraceDDL
    sqlservr!CStatsDef::CreateStats
    sqlservr!CStmtCreateStats::XretExecute
    sqlservr!CXStmtDDL::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!CSQLSource::SeExecute
    sqlservr!CExecAutoCreateStats::FExecute
    sqlservr!COptContext::FInstantiateGuessedStats
    sqlservr!COptContext::PexprSimplify
    sqlservr!COptContext::PcxteOptimizeQuery
    sqlservr!CQuery::Optimize
    sqlservr!CQuery::PqoBuild
    sqlservr!CStmtQuery::InitQuery
    sqlservr!CStmtDML::InitNormal
    sqlservr!CStmtDML::Init
    sqlservr!CCompPlan::FCompileStep
    sqlservr!CSQLSource::FCompile
    sqlservr!CSQLSource::FCompWrapper
    sqlservr!CSQLSource::Transform
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex

    Thanks much!

    • Bian said

      Sorry Karthik. The actual stack is this. I followed your instruction and got this;

      ntdll!ZwClose
      KERNELBASE!CloseHandle
      kernel32!CloseHandleImplementation
      sqlservr!CTraceFileIoProvider::OpenNextFile
      sqlservr!CTraceWriteRequest::InitForFileTrace
      sqlservr!CTraceFileIoProvider::InitializeWriteRequest
      sqlservr!CTrace::WriteRecord
      sqlservr!CTraceController::ProduceRecord
      sqlservr!CTraceDDL::CommitOrUndo
      sqlservr!XdesRMReadWrite::CommitAllVirtualLogRecs
      sqlservr!XdesRMReadWrite::PreDestruct
      sqlservr!XdesRMFull::~XdesRMFull
      sqlservr!XdesRMFull::`scalar deleting destructor’
      sqlservr!XactRM::SinglePhaseCommit
      sqlservr!XactRM::CommitInternal
      sqlservr!FullXactImp::Commit
      sqlservr!CMsqlXactInternalBase::CommitInternal
      sqlservr!CMsqlXactInternalReadWrite::Commit
      sqlservr!CMsqlXactImp::Commit
      sqlservr!CAutoMsqlXact::CommitNestedXact
      sqlservr!COptContext::FInstantiateGuessedStats
      sqlservr!COptContext::PexprSimplify
      sqlservr!COptContext::PcxteOptimizeQuery
      sqlservr!CQuery::Optimize
      sqlservr!CQuery::PqoBuild
      sqlservr!CStmtQuery::InitQuery
      sqlservr!CStmtDML::InitNormal
      sqlservr!CStmtDML::Init
      sqlservr!CCompPlan::FCompileStep
      sqlservr!CSQLSource::FCompile
      sqlservr!CSQLSource::FCompWrapper
      sqlservr!CSQLSource::Transform
      sqlservr!CSQLSource::Execute
      sqlservr!process_request
      sqlservr!process_commands
      sqlservr!SOS_Task::Param::Execute
      sqlservr!SOS_Scheduler::RunTask
      sqlservr!SOS_Scheduler::ProcessTasks
      sqlservr!SchedulerManager::WorkerEntryPoint
      sqlservr!SystemThread::RunWorker
      sqlservr!SystemThreadDispatcher::ProcessWorker
      *** WARNING: Unable to verify timestamp for msvcr80.dll
      sqlservr!SchedulerManager::ThreadEntryPoint
      msvcr80!_callthreadstartex

      thanks!!

      • sqlservr!CTraceFileIoProvider::OpenNextFile you are running a profiler trace and it looks like you have set the maximum file size , So SQL engine is trying to create a new file.
        Have you configured to capture profiler trace on local drive or network share? If local system check the speed of destination drive. If remote system (Network share) change it to local drive.

        • bian said

          ah, that makes sense. we’ll change the destination to a local drive and see if that helps.

          I really appreicate for your help but here’s one more. can you also help me on this as well?

          Child-SP RetAddr Call Site
          00000081`683fd0e8 000007fe`fda84bf6 ntdll!NtCreateFile+0xa
          00000081`683fd0f0 00000000`76c718ed KERNELBASE!CreateFileW+0x2cd
          *** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlservr.exe –
          00000081`683fd250 00000000`01f72617 kernel32!CreateFileWImplementation+0x7d
          00000081`683fd2b0 00000000`01f7513f sqlservr!GetIUMSForMsxml+0x8055f3
          00000081`683fd340 00000000`00c53aa0 sqlservr!GetIUMSForMsxml+0x80811b
          00000081`683fd390 00000000`00c54a5a sqlservr+0xbd3aa0
          00000081`683fd3e0 00000000`00c5547b sqlservr+0xbd4a5a
          00000081`683fd420 00000000`00c550e1 sqlservr+0xbd547b
          00000081`683fd4f0 00000000`01989cc5 sqlservr+0xbd50e1
          00000081`683fd580 00000000`000c67a4 sqlservr!GetIUMSForMsxml+0x21cca1
          00000081`683fd5d0 00000000`001027b4 sqlservr+0x467a4
          00000081`683fd600 00000000`00146525 sqlservr+0x827b4
          00000081`683fd660 00000000`001460db sqlservr+0xc6525
          00000081`683fd690 00000000`00101e05 sqlservr+0xc60db
          00000081`683fd730 00000000`001021d8 sqlservr+0x81e05
          00000081`683fd760 00000000`0014cef5 sqlservr+0x821d8
          00000081`683fd8f0 00000000`015b7c97 sqlservr+0xccef5
          00000081`683fd990 00000000`015b7f0b sqlservr!_____SQL______Process______Available+0x713d47
          00000081`683fda80 00000000`01527c45 sqlservr!_____SQL______Process______Available+0x713fbb
          00000081`683fdab0 00000000`006eacba sqlservr!_____SQL______Process______Available+0x683cf5
          00000081`683fea60 00000000`000ec3bc sqlservr+0x66acba
          00000081`683febe0 00000000`000ed3ab sqlservr+0x6c3bc
          00000081`683fed10 00000000`0012a4c2 sqlservr+0x6d3ab
          00000081`683ff370 00000000`0008f280 sqlservr+0xaa4c2
          00000081`683ff580 00000000`0008ee3e sqlservr+0xf280
          00000081`683ff690 00000000`0008eb3b sqlservr+0xee3e
          00000081`683ff6f0 00000000`001b5caa sqlservr+0xeb3b
          00000081`683ff760 00000000`001b5e25 sqlservr+0x135caa
          00000081`683ff840 00000000`001b567d sqlservr+0x135e25
          00000081`683ff880 00000000`001b6586 sqlservr+0x13567d
          00000081`683ff930 00000000`74b137d7 sqlservr+0×136586
          00000081`683ff9d0 00000000`74b13894 msvcr80!_callthreadstartex+0×17
          00000081`683ffa00 00000000`76c7652d msvcr80!_threadstartex+0×84
          00000081`683ffa30 00000000`7736c521 kernel32!BaseThreadInitThunk+0xd
          00000081`683ffa60 00000000`00000000 ntdll!RtlUserThreadStart+0×1

          Again, thank you for your help.

          • bian said

            Also, is there a way to get sqlservr.pdb for SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64)?

          • Hello Bian,
            Second stack which you posted us Access violation dump or Non-Yielding scheduler dump?

            • Brian Kang said

              that is Non-Yielding scheduler dump as well.

              thanks!

              • Hello Brian, Do you have any antivirus which traces CreateFile and cause delay in file creation. Both the stacks are pointing that the thread are waiting on CreateFileWImplementation. I couldnt find the cause for second dump from stack because symbols are not aligned. Please share the memory dump and errorlog file. I will look in to it. Thanks > Date: Fri, 22 Feb 2013 14:52:45 +0000 > To: karthick_pk83@hotmail.com >

                • Hello Brian,

                  I doubt if I can locate the public symbols for your build any time soon. So I would suggest to share the dump (As long as there is no security constraint).

                  Thanks
                  Karthick

                  ——————————————————————————–

                  Subject: Re: [MSSQLWIKI] Comment: “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”
                  Date: Sun, 24 Feb 2013 11:40:37 -0800

                  Hi Karthick,

                  Thanks a lot for your help on this, first of all.

                  Well, for the second dump, you’re right. We don’t have a correct symbols for it, which our server is Microsoft SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64) running on 2-node clustering. The size of dump file is about 3 GB (original size is 33 GB so I zipped it and split them into 4 files). This is because we had to enable filtered dump since the mini dump with default setting didn’t contain anything.

                  If you have a shared folder that I can drop the dump files with you. However, if you can share the correct symbols for our server version with me, I would analyzed it and then share the output with you. I’m sure this will save lots of your time and effort. Is this something possible?

                  thanks again,

                  Brian

              • Brian Kang said

                Hi Karthick,

                Thanks a lot for your help on this, first of all.

                Well, for the second dump, you’re right. We don’t have a correct symbols for it, which our server is Microsoft SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64) running on 2-node clustering. The size of dump file is about 3 GB (original size is 33 GB so I zipped it and split them into 4 files). This is because we had to enable filtered dump since the mini dump with default setting didn’t contain anything.

                If you have a shared folder that I can drop the dump files with you. However, if you can share the correct symbols for our server version with me, I would analyzed it and then share the output with you. I’m sure this will save lots of your time and effort. Is this something possible?

                thanks again, Brian

                From: karthick pk Sent: Saturday, February 23, 2013 10:28 PM To: comment+rw812t0753pis_xm_l4uxebl0@comment.wordpress.com ; bsk64@hotmail.com Subject: RE: [MSSQLWIKI] Comment: “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”

                Hello Brian, Do you have any antivirus which traces CreateFile and cause delay in file creation. Both the stacks are pointing that the thread are waiting on CreateFileWImplementation. I couldnt find the cause for second dump from stack because symbols are not aligned. Please share the memory dump and errorlog file. I will look in to it. Thanks

                > Date: Fri, 22 Feb 2013 14:52:45 +0000 > To: karthick_pk83@hotmail.com >

        • Brian Kang said

          Thanks Karthick.

          I posted another question but wonder if you can help me to get sqlservr.pdb for SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64)?

          Thanks much!! -Brian

  9. MAdler said

    Hi- i hope you can help me – this is the stack i am getting (seems much shorter then the rest)
    Sqlserver 2005 6bit latest SP (0.5)

    0:021> kc 100
    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqlservr!SystemThread::Suspend
    sqlservr!SOS_Scheduler::SuspendPreemptive
    sqlservr!SOS_Event::Wait
    sqlservr!alloca_probe
    sqlservr!CMemThread::Alloc
    sqlservr!operator new
    sqlservr!SNI_Conn::InitObject
    sqlservr!SNIAcceptDoneWithReturnCode
    sqlservr!SNIAcceptDoneRouter
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    • Brian Kang said

      Hi Karthick,

      No, we don’t see any “Non-yielding IOCP Listener” messages at all. just flood of these errors,

      2013-02-20 18:12:57.63 Server Process 0:0:0 (0x28ac) Worker 0x000000129E0781A0 appears to be non-yielding on Scheduler 35. Thread creation time: 13005864565060. Approx Thread CPU Used: kernel 0 ms, user 0 ms. Process Utilization 17%. System Idle 98%. Interval: 70358 ms. 2013-02-20 18:12:57.65 spid39s Database mirroring is inactive for database ‘XXXX’. This is an informational message only. No user action is required.

      we’re experiencing lots of cxpacket waits so this could be due to a parallelism issue?

      Thanks, Brian

      • Hello Brian,

        Excessive parallelism can cause deadlocked scheduler but not non-Yielding scheduler.
        Remember SQL Server will generate dump only once when it detects non-yielding condition. Once the dump is generated warnings will be printed on error log for successive occurrence’s but dumps will not be generated. In our case it has detected the condition and printed the warning in SQL Server error log but has not generated the dump so ideally it would have generated dump in past.

        You can enabled tracefalg 1262 to generate dump on every non-Yield condition. But it may generate dumps continously and cause performance problem. So if you see the above error continoulsy

        1. Enable the traceflag
        2. Wait for first dump
        3. Disable the trace
        4. Paste the Non-yield stack.

        Did you change the min-server memory to default?
        Also let me know the verision of mscorwks which is loaded in SQLServer.

    • Hello Madler,

      Stack which you pasted is current or copied stack? From stack it looks like there is a memory pressure on the system. Can you share the SQL-Error log and output of two queries in http://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/

      • madler said

        It was the stack trace off of the memdump that sql created at the time of the error.
        We hade sql set to default max memory – we limited sql to maintain 4 GB for the operating system and the errors seem to have gone away – so i guess the issue you dexcribed in that link was indeed what was happending.

        here are the results of the queries:
        RunTime Notification_Time Notification_type MemoryUtilization % Node Id Process_Indicator System_Indicator type state reserved Effect type state reserved Effect type state reserved Effect SQL_ReservedMemory_KB SQL_CommittedMemory_KB SQL_AWEMemory SinglePagesMemory MultiplePagesMemory TotalPhysicalMemory_KB AvailablePhysicalMemory_KB TotalPageFile_KB AvailablePageFile_KB TotalVirtualAddressSpace_KB AvailableVirtualAddressSpace_KB Record Id Type Record Time Current Time
        2013-03-04 10:50:35.310 2013-03-01 10:55:45.683 RESOURCE_MEMPHYSICAL_HIGH 100 1 NULL NULL APPLY_LOWPM EFFECT_OFF 0 0 APPLY_HIGHPM EFFECT_ON 0 0 REVERT_HIGHPM EFFECT_OFF 0 0 33683712 118416 0 5056 14048 33541144 28767956 67080432 32551328 8589934464 8556024928 0 RING_BUFFER_RESOURCE_MONITOR 6984298 265873923
        and:

        runtime Notification_time ProcessUtilization SystemIdle % UserModeTime KernelModeTime PageFaults WorkingSetDelta MemoryUtilization (%workingset) Record Time Current Time
        2013-03-04 10:57:19.680 2013-03-04 06:41:27.380 0 99 0 5148033 68 -16 98 250922562 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:42:27.383 5 91 28392182 526971378 2339 36 98 250982565 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:43:27.387 0 92 10920070 37908243 68 0 98 251042568 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:44:27.390 6 88 27144174 622755992 2325 -8 98 251102572 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:45:27.393 0 98 7176046 47580305 153 20 98 251162575 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:46:27.397 4 94 25116161 442106834 2293 36 98 251222579 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:47:27.403 20 78 13104084 1945332470 584 -404 98 251282583 266274861
        2013-03-04 10:57:19.680 2013-03-04 06:48:27.417 12 86 28080180 1192783646 2221 -72 98 251342599 266274861

  10. Kile said

    Hi – I couldn’t find anything that matched the stack trace that I received. Any help is greatly appreciated.

    Microsoft SQL Server 2012 – 11.0.2100.60 (X64)

    0:525> kc 100
    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqldk!SOS_Scheduler::SwitchContext
    sqldk!SOS_Scheduler::SuspendNonPreemptive
    sqldk!EventInternal::Wait
    sqldk!EventInternal::WaitAllowPrematureWakeup
    sqlmin!CXPacketList::RemoveHead
    sqlmin!CXPipe::ReceivePacket
    sqlmin!CXTransLocal::ReceiveBuffers
    sqlmin!CQScanExchangeNew::GetRowFromProducer
    sqlmin!CQScanExchangeNew::GetRowHelper
    sqlmin!CQScanRangePrefetchDelayNew::GetRow
    sqlmin!CQScanNLJoinTrivialNew::GetRow
    sqlmin!CQScanXProducerNew::GetRowHelper
    sqlmin!CQScanXProducerNew::GetRow
    sqlmin!FnProducerOpen
    sqlmin!FnProducerThread
    sqlmin!SubprocEntrypoint
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  11. Donny said

    Hi, I could not match my stack trace also. Thanks for any help/insight you may be able to provide!

    This was in error log:
    2013-03-14 12:45:51.480 Server Using ‘dbghelp.dll’ version ’4.0.5′
    2013-03-14 12:45:51.630 Server **Dump thread – spid = 0, EC = 0×0000000000000000
    2013-03-14 12:45:51.630 Server ***Stack Dump being sent to J:\MSSQL11.NAPSQL24\MSSQL\LOG\SQLDump0002.txt
    2013-03-14 12:45:51.630 Server *
    2013-03-14 12:45:51.630 Server * BEGIN STACK DUMP:
    2013-03-14 12:45:51.630 Server * 03/14/13 12:45:51 spid 50876
    2013-03-14 12:45:51.630 Server *
    2013-03-14 12:45:51.630 Server * Non-yielding IOCP Listener
    2013-03-14 12:45:51.630 Server *
    2013-03-14 12:45:51.630 Server * Short Stack Dump
    2013-03-14 12:45:51.690 Server Stack Signature for the dump is 0x00000000000003E1
    2013-03-14 12:46:16.270 Server External dump process return code 0×20000001. External dump process returned no errors.
    2013-03-14 12:46:16.270 Server IO Completion Listener (0x15ec0) Worker 0x00000080F9280160 appears to be non-yielding on Node 1. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15085.
    2013-03-14 12:46:41.800 Server IO Completion Listener (0x15ec0) Worker 0x00000080F9280160 appears to be non-yielding on Node 1. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 50280.
    2013-03-14 12:46:44.690 Server IO Completion Listener (0xdd8c) Worker 0x00000080FBF00160 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 40982.

    This is the debug session:

    0:042> .cxr sqlmin!g_copiedStackInfo+0X20
    rax=0000000000000000 rbx=0000000000000000 rcx=00000079a773f498
    rdx=00000000ce11f238 rsi=00000000ffffffff rdi=0000000000000b1c
    rip=0000000077af135a rsp=00000000ce11ee40 rbp=00000000ce11f018
    r8=00000080faf20040 r9=0000000000000000 r10=0000000000000000
    r11=00000000ce11f1b8 r12=0000000000000000 r13=0000000000000000
    r14=00000080fb024130 r15=00000000ffffffff
    iopl=0 nv up ei pl zr na po nc
    cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
    ntdll!ZwWaitForSingleObject+0xa:
    00000000`77af135a c3 ret
    0:042> kc 100
    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqldk!SystemThread::Suspend
    sqldk!SOS_Scheduler::SuspendPreemptive
    sqldk!EventInternal::Wait
    sqldk!SOS_UnfairMutexPair::LongWait
    sqldk!SOS_UnfairMutexPair::AcquirePair
    sqldk!CMemThread::Alloc
    sqldk!SOS_Node::EnqueueTaskDirectInternal
    sqllang!SOS_Node::EnqueueTaskDirect
    sqllang!SNIAcceptDoneRouter
    sqldk!SOS_Node::ListenOnIOCompletionPort
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart
    0:042> lmvm sqlservr
    start end module name
    00000000`ff080000 00000000`ff0be000 sqlservr (pdb symbols) c:\websymbols\sqlservr.pdb\5F808D3337994C2AB03A56D03A08B9192\sqlservr.pdb
    Loaded symbol image file: sqlservr.exe
    Mapped memory image file: c:\Websymbols\sqlservr.exe\4F36496E3e000\sqlservr.exe
    Image path: C:\Program Files\Microsoft SQL Server\MSSQL11.NAPSQL24\MSSQL\Binn\sqlservr.exe
    Image name: sqlservr.exe
    Timestamp: Sat Feb 11 05:56:46 2012 (4F36496E)
    CheckSum: 00034130
    ImageSize: 0003E000
    File version: 2011.110.2100.60
    Product version: 11.0.2100.60
    File flags: 0 (Mask 3F)
    File OS: 40000 NT Base
    File type: 1.0 App
    File date: 00000000.00000000
    Translations: 0409.04b0
    CompanyName: Microsoft Corporation
    ProductName: Microsoft SQL Server
    InternalName: SQLSERVR
    OriginalFilename: SQLSERVR.EXE
    ProductVersion: 11.0.2100.60
    FileVersion: 2011.0110.2100.060 ((SQL11_RTM).120210-1917 )
    FileDescription: SQL Server Windows NT – 64 Bit
    LegalCopyright: Microsoft Corp. All rights reserved.
    LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
    Comments: SQL
    0:042> !findstack sqlservr!SQL_SOSNonYieldSchedulerCallback
    0:042> !findstack sqlservr!SQL_SOSNonYield
    Thread 006, 1 frame(s) match
    * 13 00000000861eea10 000007fedf76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0×184

    Thread 042, 1 frame(s) match
    * 07 00000000ce13ee70 000007fedf76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0xeb

    0:042> ~[006]s
    ntdll!ZwAllocateVirtualMemory+0xa:
    00000000`77af149a c3 ret
    0:006> kP
    Child-SP RetAddr Call Site
    00000000`861edb48 000007fe`fe261935 ntdll!ZwAllocateVirtualMemory+0xa
    00000000`861edb50 000007fe`df6d7275 KERNELBASE!VirtualAlloc+0×45
    00000000`861edb90 000007fe`df6ca47b sqldk!MemoryNode::VirtualAllocNativeNoCheck+0xa0
    00000000`861edbf0 000007fe`df6ca1b4 sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality+0×256
    00000000`861edd10 000007fe`df6c9e39 sqldk!SOS_MemoryBlockAllocator::AllocateBlock+0x5bd
    00000000`861ede00 000007fe`df6ca816 sqldk!SOS_MemoryWorkSpace::AllocatePage+0×273
    00000000`861edfb0 000007fe`df6cac61 sqldk!MemoryNode::AllocatePagesInternal+0×187
    00000000`861ee040 000007fe`df6cae86 sqldk!MemoryClerkInternal::AllocatePages+0xb1
    00000000`861ee370 000007fe`df6cae2d sqldk!CVarPageMgr::PviNewVarPage+0×36
    00000000`861ee3e0 000007fe`df6c0287 sqldk!CVarPageMgr::PbAllocate+0x16f
    00000000`861ee420 000007fe`d4c878ee sqldk!CMemObj::Alloc+0×47
    00000000`861ee450 00000000`ff098349 sqllang!CErrorReportingManager::CwchFormatAndPrint+0×472
    00000000`861ee910 00000000`ff093504 sqlservr!scierrlog+0×79
    00000000`861ee9b0 000007fe`df76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0×184
    00000000`861eea20 000007fe`df704c8c sqldk!SOS_OS::ExecuteNonYieldIOCPCallbacks+0xde
    00000000`861eecc0 000007fe`df6c8389 sqldk!SchedulerMonitor::CheckIoCompletionHangs+0x1d0
    00000000`861eee00 000007fe`df79bf89 sqldk!SchedulerMonitor::Run+0×175
    00000000`861eef30 000007fe`df6c3170 sqldk!SchedulerMonitor::EntryPoint+0×9
    00000000`861eef60 000007fe`df6c2f64 sqldk!SOS_Task::Param::Execute+0x21e
    00000000`861ef560 000007fe`df6c2d66 sqldk!SOS_Scheduler::RunTask+0xa8
    0:006> kC
    Call Site
    ntdll!ZwAllocateVirtualMemory
    KERNELBASE!VirtualAlloc
    sqldk!MemoryNode::VirtualAllocNativeNoCheck
    sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality
    sqldk!SOS_MemoryBlockAllocator::AllocateBlock
    sqldk!SOS_MemoryWorkSpace::AllocatePage
    sqldk!MemoryNode::AllocatePagesInternal
    sqldk!MemoryClerkInternal::AllocatePages
    sqldk!CVarPageMgr::PviNewVarPage
    sqldk!CVarPageMgr::PbAllocate
    sqldk!CMemObj::Alloc
    sqllang!CErrorReportingManager::CwchFormatAndPrint
    sqlservr!scierrlog
    sqlservr!SQL_SOSNonYieldIOCPCallback
    sqldk!SOS_OS::ExecuteNonYieldIOCPCallbacks
    sqldk!SchedulerMonitor::CheckIoCompletionHangs
    sqldk!SchedulerMonitor::Run
    sqldk!SchedulerMonitor::EntryPoint
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    0:006> ~[042]s
    ntdll!ZwWaitForSingleObject+0xa:
    00000000`77af135a c3 ret
    0:066> kP
    Child-SP RetAddr Call Site
    00000000`d501dc18 000007fe`fe2610dc ntdll!ZwWaitForSingleObject+0xa
    00000000`d501dc20 000007fe`d5995f51 KERNELBASE!WaitForSingleObjectEx+0×79
    00000000`d501dcc0 000007fe`d59964cf sqllang!FRunCommunicationsManager+0xf1
    00000000`d501de90 000007fe`df6c3170 sqllang!CommunicationsManager+0x3bf
    00000000`d501f0f0 000007fe`df6c2f64 sqldk!SOS_Task::Param::Execute+0x21e
    00000000`d501f6f0 000007fe`df6c2d66 sqldk!SOS_Scheduler::RunTask+0xa8
    00000000`d501f760 000007fe`df6de22f sqldk!SOS_Scheduler::ProcessTasks+0×299
    00000000`d501f7e0 000007fe`df6de5c0 sqldk!SchedulerManager::WorkerEntryPoint+0×261
    00000000`d501f880 000007fe`df6dd9cb sqldk!SystemThread::RunWorker+0x8f
    00000000`d501f8b0 000007fe`df6de059 sqldk!SystemThreadDispatcher::ProcessWorker+0x3c8
    00000000`d501f960 00000000`7799652d sqldk!SchedulerManager::ThreadEntryPoint+0×236
    00000000`d501fa00 00000000`77acc521 kernel32!BaseThreadInitThunk+0xd
    00000000`d501fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
    0:066> kC
    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqllang!FRunCommunicationsManager
    sqllang!CommunicationsManager
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  12. ccpjeff said

    Just a thank you for the great resource… currently troubleshooting some non-yielding errors…

    It has hit us once every 2 weeks for the last few months and only caused crash dumps twice, and never causes SQL to die or anything like that. It typically results in multiple ‘client unable to reuse a session’ errors, and some errors in the front end application.

    I used your info to dig through the dump, and found this:

    mswsock!_GSHandlerCheck_SEH
    mswsock!WSPCloseSocket
    ws2_32!closesocket
    sqlservr!Tcp::FCloseRefHandle
    sqlservr!Tcp::Close

    Which references several of your included dumps and KB articles. Checking those, it shows that this should of been fixed in SQL 2008 SP2 CU2. Our server is @ SP2 CU5 and most of our clients as well, but it’s possible a new client/service/application is not @ the correct level.

    I’m verifying the native client versions on those new machines and hoping they are outdated! If so, this should be an easy fix!

    but again, thanks for the great resource!

    Cheers,

    jeff

  13. KK said

    I got 2 SQL 2005 instances and they both gave non-yielding IOCP Listener error. The server has 16GB of physical memory but the both SQL instances are limited to Min of 2GB & Max of 4GB memory. Below the info from one of the instances dumpfile.

    0:008> KC 100
    Call Site
    sqlservr!MemoryNode::AllocatePagesInternal
    sqlservr!MemoryClerkInternal::AllocatePages
    sqlservr!CVarPageMgr::PviNewVarPage
    sqlservr!CVarPageMgr::PbAllocate
    sqlservr!CMemObj::Alloc
    sqlservr!operator new
    sqlservr!operator new[]
    sqlservr!CErrorReportingManager::SendErrorToErrLog
    sqlservr!CErrorReportingManager::CwchFormatAndPrint
    sqlservr!scierrlog
    sqlservr!alloca_probe
    sqlservr!Np::PrepareForNextAccept
    sqlservr!SNIAcceptDoneWrapper
    sqlservr!SNIAcceptDoneRouter
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    0×0
    0×0
    0×0
    0×0
    msvcr80!_threadstartex
    msvcr80!_threadstartex
    msvcr80!__initialmbcinfo
    0×0

    =====================================================================
    BugCheck Dump
    =====================================================================

    This file is generated by Microsoft SQL Server
    version 9.00.5000.00
    upon detection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is AT/AT COMPATIBLE.
    Current time is 20:25:03 04/29/13.
    4 Unknown CPU 9., 2660 Mhz processor (s).
    Windows NT 5.2 Build 3790 CSD Service Pack 2.

    Memory
    MemoryLoad = 98%
    Total Physical = 16383 MB
    Available Physical = 211 MB
    Total Page File = 19864 MB
    Available Page File = 3580 MB
    Total Virtual = 8388607 MB
    Available Virtual = 8371825 MB
    **Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
    ***Stack Dump being sent to E:\MSSQL.1\MSSQL\LOG\SQLDump0008.txt
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 04/29/13 20:26:07 spid 0
    *
    * Non-yielding IOCP Listener
    *
    * *******************************************************************************
    * ——————————————————————————-
    * Short Stack Dump

    Can you advise if this is related to disk i/o performance or memory ?

  14. Danni said

    Hi Karthick,

    thank you for that nice Tutorial.
    I couldn’t find any stack that matches. Maybe you can help me..

    Errorlog:
    2013-04-30 18:30:23.83 Server **Dump thread – spid = 0, EC = 0×00000000
    2013-04-30 18:30:23.83 Server ***Stack Dump being sent to c:\Programme\Microsoft SQL Server\MSSQL10_50.DESIGO\MSSQL\LOG\SQLDump0007.txt
    2013-04-30 18:30:23.83 Server * *******************************************************************************
    2013-04-30 18:30:23.83 Server *
    2013-04-30 18:30:23.83 Server * BEGIN STACK DUMP:
    2013-04-30 18:30:23.83 Server * 04/30/13 18:30:23 spid 3688
    2013-04-30 18:30:23.83 Server *
    2013-04-30 18:30:23.83 Server * Non-yielding IOCP Listener
    2013-04-30 18:30:23.83 Server *
    2013-04-30 18:30:23.83 Server * *******************************************************************************
    2013-04-30 18:30:23.83 Server * ——————————————————————————-
    2013-04-30 18:30:23.83 Server * Short Stack Dump
    2013-04-30 18:30:23.84 Server Stack Signature for the dump is 0×00000180
    2013-04-30 18:30:25.66 Server External dump process return code 0×20000001.
    External dump process returned no errors.

    2013-04-30 18:30:25.66 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-04-30 20:03:35.68 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-04-30 20:03:45.71 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25046.
    2013-04-30 21:02:40.75 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-04-30 23:29:15.79 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 00:29:50.81 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 00:30:00.84 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
    2013-05-01 02:02:00.87 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 02:02:10.92 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
    2013-05-01 03:30:00.95 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 03:30:10.98 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
    2013-05-01 13:29:13.14 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 14:29:23.18 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 16:01:28.20 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 22:01:53.23 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 22:02:03.27 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25046.
    2013-05-01 23:01:13.29 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-01 23:01:23.32 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
    2013-05-02 08:30:18.34 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-02 09:30:38.39 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-02 10:29:53.42 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-02 11:31:13.46 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-02 13:03:48.49 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
    2013-05-02 14:03:23.51 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.

    Debugging:
    0:004> .cxr sqlservr!g_copiedStackInfo+0X20
    eax=03b6f8e0 ebx=7c94845c ecx=00000246 edx=0000001b esi=03b6f90c edi=03d8e988
    eip=00000000 esp=00000000 ebp=00000023 iopl=0 nv up di pl nz na po nc
    cs=0000 ss=0010 ds=b34e es=dc08 fs=0000 gs=0000 efl=00000000
    00000000 ?? ???
    0:004> kc 100

    WARNING: Frame IP not in any known module. Following frames may be wrong.
    0×0
    0:004> kc 100

    WARNING: Frame IP not in any known module. Following frames may be wrong.
    0×0
    0:004> .cxr
    Resetting default scope
    0:004> .cxr sqlservr!g_copiedStackInfo+0X00c
    eax=03d8e988 ebx=00000000 ecx=0000b34e edx=7c94dc08 esi=00000000 edi=00bc8048
    eip=7c94845c esp=03b6f8e0 ebp=03b6f90c iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
    ntdll!KiFastSystemCallRet:
    7c94845c c3 ret
    0:004> kc 100

    ntdll!KiFastSystemCallRet
    ntdll!NtRemoveIoCompletion
    kernel32!GetQueuedCompletionStatus
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadStart

    Thank you for your Help

    • Hello Danni,
      This is 32bit SQLServer or 64-Bit SQLServer?

    • Also can you do ~~[0xe84]s and then kc100 and share the output

      • Danni said

        Here it is:

        0:004> ~~[0xe84]s
        eax=053d3ff0 ebx=00000000 ecx=00000001 edx=00001fb0 esi=000002e8 edi=00000000
        eip=7c94845c esp=3361f16c ebp=3361f1e0 iopl=0 nv up ei pl zr na pe nc
        cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
        ntdll!KiFastSystemCallRet:
        7c94845c c3 ret
        0:013> kc 100

        ntdll!KiFastSystemCallRet
        ntdll!NtSignalAndWaitForSingleObject
        kernel32!SignalObjectAndWait
        sqlservr!SOS_Scheduler::Switch
        sqlservr!SOS_Scheduler::SwitchContext
        sqlservr!SOS_Scheduler::SuspendNonPreemptive
        sqlservr!SOS_Scheduler::Suspend
        sqlservr!EventInternal<Spinlock >::Wait
        sqlservr!ResQueueBase::Dequeue
        sqlservr!ResQueue::Dequeue
        sqlservr!systemTaskThread
        sqlservr!SOS_Task::Param::Execute
        sqlservr!SOS_Scheduler::RunTask
        sqlservr!SOS_Scheduler::ProcessTasks
        sqlservr!SchedulerManager::WorkerEntryPoint
        sqlservr!SystemThread::RunWorker
        sqlservr!SystemThreadDispatcher::ProcessWorker
        sqlservr!SchedulerManager::ThreadEntryPoint
        msvcr80!_callthreadstartex
        msvcr80!_threadstartex
        kernel32!BaseThreadStart

        • Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.

          From the above warming non-yielding thread is not consuming CPU at all (kernel 0 ms, user 0 ms).

          I think garbage collector is suspending all the other threads.

          type ~*kc 100 and search for mscorwks!SVR::GCHeap

          If you find atleast one garbage collector thread apply Apply the .Net fix mentioned in http://support.microsoft.com/kb/2504603

          If you dont find any garbage collector thread let me know.

  15. Adam Angelini said

    Do you knwo what cause this one? SQL2012 sp1 enterprise x64 bit

    Call Site
    ntdll!ZwWaitForSingleObject
    ntdll!RtlpWaitOnCriticalSection
    ntdll!RtlEnterCriticalSection
    ws2_32!closesocket
    sqllang!Tcp::FCloseTransport
    sqllang!SNI::detail::Transport::FCloseRefHandle
    sqllang!Tcp::Close
    sqllang!Smux::InternalClose
    sqllang!Smux::ReadDone
    sqllang!SNIReadDone
    sqldk!SOS_Node::ListenOnIOCompletionPort
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    • Hello Adam,

      which windows version are you using?

      You need to apply the windows fix mentioned in below KB. (windows fix in KB is applicabel for SQL2012 as well)

      http://support.microsoft.com/kb/2491214/en-US

      {
      Hotfix for Windows Server 2008 R2
      976932 Information about Service Pack 1 for Windows 7 and for Windows Server 2008 R2
      Hotfix for Windows Server 2008
      2398202 An application may stop responding in Windows Server 2008 or in Windows Vista when the application closes a socket connection or shuts down
      }

  16. tilak said

    Hi Karthick,

    Can you please let me know the cause of this. SQL version is 10.0.2531

    Call Site
    ntdll!NtWriteFile
    KERNELBASE!WriteFile
    kernel32!WriteFileImplementation
    sqlservr!DiskWriteAsync
    sqlservr!CTraceIoBuf::IncrementalFlush
    sqlservr!CTraceFileIoProvider::IncrementalFlush
    sqlservr!CTraceController::IncrementalFlush
    sqlservr!CTraceManagementTask::Invoke
    sqlservr!CTraceBatchTask::Task
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    0×0
    0×0

    Thanks,
    Tilak M

  17. Raka said

    Hi,
    could you please help me to fix this error

    Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’, id 000000008082EEF0, type 2, Task 0x000000000568B948 : 0, waittime 600, flags 0×4018, owning task 0×0000000000000000. Continuing to wait.
    2013-06-30 13:30:08.55 spid54 Time out occurred while waiting for buffer latch — type 4, bp 0000000084FDD740, page 1:412488, stat 0xc0000b, database id: 2, allocation unit id: 6488064, task 0x00000002E970D048 : 0, waittime 6900, flags 0×1000000039, owning task 0x00000000056A4E08. Continuing to wait.

    • Hello Raka,

      This is latch timeout dump and you should be following http://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ and get the stack of owning task. Steps documented in above blog are very simple.

      You have posted two errors

      Error 1 Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’

      1. Check if there is any “The operating system returned error” mentioned in error log. 2. Check if data file is auto growing or I/O request taking longer than 60 seconds warning

      Error 2 Time out occurred while waiting for buffer latch — type 4, bp 0000000084FDD740, page 1:412488, stat 0xc0000b, database id: 2, allocation unit id: 6488064, task 0x00000002E970D048 : 0, waittime 6900, flags 0×1000000039, owning task 0x00000000056A4E08. If you look at above warning carefully we are waiting to occur latch on PFS page 1:412488 of tempdb (every 8088th page is PFS page). Did the error message disappear after some time?

      We can follow below steps to reduce contention on PFS page but if you find continuous time out on same pages (1:412488) in error log you have to find the stack of thread which is owning the latch on page 1:412488 follow http://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ in that case and share the stack else follow below steps.

      1.Increase the number of Tempdb data files to 8, size them equally and place them in different physical disk and then if contention continues, increase the number of data files by multiples of 4 (up to the number of logical processors) until the contention is reduced to acceptable levels or make changes to the workload/code.

      2. Enable trace flag 1118 Related KB: http://support.microsoft.com/kb/2154845

      Thanks Karthick

  18. Raka said

    Hi Karthick,

    Thanks for reply.
    we are facing performance issue and for that time iam able to see these error in sql server logs.

    Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’

    Timeout occurred while waiting for latch: class ‘ACCESS_METHODS_HOBT_COUNT’, id 000000008082EEF0, type 2, Task 0x000000000568B948 : 0, waittime 600, flags 0×4018, owning task 0×0000000000000000. Continuing to wait.
    2013-06-30 13:30:08.55 spid54 Time out occurred while waiting for buffer latch — type 4, bp 0000000084FDD740, page 1:412488, stat 0xc0000b, database id: 2, allocation unit id: 6488064, task 0x00000002E970D048 : 0, waittime 6900, flags 0×1000000039, owning task 0x00000000056A4E08. Continuing to wait.

    no there is not any operating system related error in logs.

    i have followed the steps suggested by you and found below logs but unable to find reason behind these errors.
    could you please help me to expalin these error and how to resolve these.

    Thanks,
    Raka

    157 Id: a78.3378 Suspend: 0 Teb: 000007ff`fff2e000 Unfrozen
    # Child-SP RetAddr Call Site
    00 00000000`42f57a58 00000000`77a44327 ntdll!NtSignalAndWaitForSingleObject+0xa
    01 00000000`42f57a60 00000000`00194b08 kernel32!SignalObjectAndWait+0×117
    02 00000000`42f57b20 00000000`00192d76 sqlservr!SOS_Scheduler::Switch+0×181
    03 00000000`42f58270 00000000`00192700 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xca
    04 00000000`42f582b0 00000000`001929ac sqlservr!SOS_Scheduler::Suspend+0x2d
    05 00000000`42f582e0 00000000`001f7e46 sqlservr!EventInternal<Spinlock >::Wait+0x1a8
    06 00000000`42f58330 00000000`001f7a29 sqlservr!LatchBase::Suspend+0×599
    07 00000000`42f58fa0 00000000`001b0689 sqlservr!LatchBase::AcquireInternal+0x1ff
    08 00000000`42f59040 00000000`001b1792 sqlservr!BUF::AcquireLatch+0x8d
    09 00000000`42f59360 00000000`001b166c sqlservr!BPool::Get+0xc7
    0a 00000000`42f593c0 00000000`0074d0ea sqlservr!PageRef::Fix+0xbc
    0b 00000000`42f59420 00000000`0074cb53 sqlservr!DeAllocMgr::DropAllocUnitInternal+0×642
    0c 00000000`42f59790 00000000`0066dd5c sqlservr!DropAllocationUnit+0x18b
    0d 00000000`42f59850 00000000`0027c167 sqlservr!DeAllocateIndexNew+0×539
    0e 00000000`42f59aa0 00000000`001f6e20 sqlservr!SETruncateWorktableRowset+0×777
    0f 00000000`42f59eb0 00000000`00614673 sqlservr!SETruncateRowset+0×144
    10 00000000`42f5d6f0 00000000`0060db49 sqlservr!RowsetWorkTableSS::Truncate+0×43
    11 00000000`42f5d770 00000000`0060e5bb sqlservr!SqlProfTruncateWorktableRowset+0×50
    12 00000000`42f5d7b0 00000000`01a8e82b sqlservr!CQScanSpoolNew::ChangeSpoolState+0xbc
    13 00000000`42f5d7e0 00000000`0024f7db sqlservr!CQScanSpoolNew::Dormant+0x5f
    06 0000000

    • Hello Raka,
      Which command did you run?
      your owning task is “owning task 0x00000000056A4E08″ from error.
      so you should run
      ~*e .echo ThreadId:; ?? @$tid; r? @$t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? @$t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; s -d @$t1 @$t2 0x00000000056A4E08

      and do kL only on threads which as reference to owning task.

  19. halabj said

    we got performace issue fo about 40 minutes, and created this:

    ntdll!KiFastSystemCallRet
    ntdll!ZwWaitForSingleObject
    kernel32!WaitForSingleObjectEx
    kernel32!WaitForSingleObject
    sqlservr!SOS_Scheduler::SwitchContext
    sqlservr!SOS_Scheduler::SuspendNonPreemptive
    sqlservr!SOS_Scheduler::Suspend
    sqlservr!SOS_Event::Wait
    sqlservr!LatchBase::Suspend
    sqlservr!LatchBase::AcquireInternal
    sqlservr!CSbReentrantAutoLatch::Acquire
    sqlservr!CSbTransmissionStoreNotifications::SteppedOnEntryCommon
    sqlservr!ClockHand::Move
    sqlservr!ClockAlgorithm::MoveHand
    sqlservr!ClockAlgorithm::ProcessTick
    sqlservr!SOS_CacheStore::Notify
    sqlservr!ResourceMonitor::NotifyMemoryConsumers
    sqlservr!ResourceMonitor::ResourceMonitorTask
    sqlservr!SetupResourceMonitorTaskContext
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadStart

    • Brian Kang said

      Hi Karthik, We just got below dump and followed your instruction but i can’t find a matcing case so wonder if you can help us why we got this. Thanks much as always!! -Brian 2013-07-15 18:55:24.64 Server **Dump thread – spid = 0, EC = 0×0000000000000000 2013-07-15 18:55:24.95 Server ***Stack Dump being sent to E:\Microsoft SQL Server\MSSQL11.INT\MSSQL\LOG\SQLDump0001.txt 2013-07-15 18:55:24.97 Server * ******************************************************************************* 2013-07-15 18:55:24.97 Server * 2013-07-15 18:55:24.97 Server * BEGIN STACK DUMP: 2013-07-15 18:55:24.97 Server * 07/15/13 18:55:24 spid 2080 2013-07-15 18:55:24.97 Server * 2013-07-15 18:55:24.97 Server * Non-yielding Resource Monitor 2013-07-15 18:55:24.97 Server * 2013-07-15 18:55:24.97 Server * ******************************************************************************* 2013-07-15 18:55:24.97 Server * ——————————————————————————- 2013-07-15 18:55:24.97 Server * Short Stack Dump 2013-07-15 18:55:29.84 Server Stack Signature for the dump is 0×0000000000000268 2013-07-15 18:59:11.79 Server Timeout waiting for external dump process 9612. 2013-07-15 18:59:11.93 Server Resource Monitor (0×858) Worker 0x000000047F688160 appears to be non-yielding on Node 0. Memory freed: 6640 KB. Last wait: MISCELLANEOUS. Approx CPU Used: kernel 15 ms, user 0 ms, Interval: 60423.

      0:005> .cxr sqlmin!g_copiedStackInfo+0X20 rax=000007ff83773480 rbx=00000003ac59c060 rcx=000000046b078f80 rdx=000000023ac91ec0 rsi=0000000000000000 rdi=0000000000000000 rip=000007ff837aaa36 rsp=000000000684e950 rbp=0000000000000000 r8=000000000684ea30 r9=0000000000000000 r10=000000043688afe0 r11=000000000684ea20 r12=0000000000000000 r13=7fffffffffffffff r14=000000000000004f r15=7fffffffffffffff iopl=0 nv up ei pl nz na pe nc cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00010202 sqlmin!RowsetNewSS::ReleaseAccessor+0×6: 000007ff`837aaa36 4883baa800000000 cmp qword ptr [rdx+0xa8],0×0 ds:00000002`3ac91f68=???????????????? 0:005> kc 100 Child-SP RetAddr Call Site 00000000`0684e950 000007ff`837ab187 sqlmin!RowsetNewSS::ReleaseAccessor+0×6 00000000`0684e980 000007ff`837aa9c8 sqlmin!CAccessorRelease::ReleaseAccessor+0x10a 00000000`0684ea30 000007ff`837ab0b7 sqlmin!CAccessorRelease::`scalar deleting destructor’+0×18 00000000`0684ea60 000007ff`837aad33 sqlmin!CAccessorRelease::Release+0×17 00000000`0684ea90 000007ff`837aabce sqlmin!CQueryExecContext::~CQueryExecContext+0x23a 00000000`0684ece0 000007ff`837ab627 sqlmin!CQueryExecContext::`scalar deleting destructor’+0xe 00000000`0684ed10 000007ff`837ab81b sqlmin!CQueryScan::ShutdownQueryExecContext+0×76 00000000`0684ed40 000007ff`816cbe61 sqlmin!CQueryScan::DestroyQuery+0x1e2 00000000`0684ed90 000007ff`81bb1369 sqllang!CExecuteStatement::Destroy+0×61 00000000`0684edc0 000007ff`8247e135 sqllang!CMetaStmtSet::DeleteXstmtsInPartition+0×74 00000000`0684ee10 000007ff`8247fdd0 sqllang!CMetaStmtSet::StepOnStatements+0×106 00000000`0684ee70 000007ff`82452ad8 sqllang!CCompPlan::StepOnStatements+0×30 ^ Extra character error in ‘kc 100′

      Date: Tue, 16 Jul 2013 18:06:20 +0000 To: bsk64@hotmail.com

      • Hello Brian,

        I reviewed the error log and memory dump which you send me. Non-yielding Resource Monitor dump is just a side effect of memory pressure on the system.

        When there was non-yielding dump just 10% of overall memory used by SQL Server was in RAM and remaining was in page file. Look at the error message in SQL Server just before the dump.

        Total memory used by SQL Server is 12080092 KB (12GB) and only 10 % of this memory is from RAM (Working set (KB): 1246168). So SQL Server has started responding to the memory pressure and releasing the memory back to OS, at the same time windows working set manger have started trimming the working set (Page out)all the processes including the working of SQL Server process. When the critical portions of SQL Server memory is paged it takes long time for SQL Server to respond to memory pressure and scale down its memory usage. I also see garbage collector is kicked and running (you are using CLR). When Garbage collector is kicked off during memory pressure all other threads in the process are suspended. So if Garbage collector is taking a long time resource monitor thread appears stuck and hence the non-yielding errors and dumps are generated. Look at error immediately after the dump resource monitor thread has got only 15ms of CPU time in interval of 60.5 seconds.

        { 2013-07-15 18:54:10.86 spid13s A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 333 seconds. Working set (KB): 1246168, committed (KB): 12080092, memory utilization: 10%.

        Resource Monitor (0×858) Worker 0x000000047F688160 appears to be non-yielding on Node 0. Memory freed: 6640 KB. Last wait: MISCELLANEOUS. Approx CPU Used: kernel 15 ms, user 0 ms, Interval: 60423. }

        How to fix this ?

        Look at the SQLDump0001.txt. I have copied a small portion for your reference. Total memory on this server is 32 GB and just 700 Mb is free. SQL Server is consuming around 1GB in RAM .Identify who is consuming remaining memory and fix that process. { MemoryLoad = 97% Total Physical = 32767 MB Available Physical = 773 MB }

        1. You can follow the steps in http://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/ 2. Since this is a VMware box I suspect that this issue could be because of VMware ballooning. Make sure you disable ballooning. More details in http://mssqlwiki.com/2013/03/31/sql-server-and-vmware-ballooning/ 3. You can temporarily use Locked pages in memory till you identify the process/Driver which is consuming most of the memory in the server making SQL Server and windows to starve for memory.

    • Hello Halabj,
      I know your issue existed in SQLServer2005 SP2 and got fixed in Cumulative update package 13 for SQL Server 2005 Service Pack 2.
      Are you on SQL2005 with out latest service pack? Apply latest SP for 2005 and you will not face it.

      @Brian Kang You have replied to Halabj comment instead of new comment. so you are getting this mail. Ignore this I will look at your stack in a while.

  20. Jose Ramon Vela Martin said

    Hello I have one SQL stack, is similar like you but is not the same, Could anybody help me?
    ç

    0:005> kC
    *** Stack trace for last set context – .thread/.cxr resets it
    Call Site
    ntdll!ZwWaitForSingleObject
    kernel32!WaitForSingleObjectEx
    sqlservr!SystemThread::Suspend
    sqlservr!SOS_Scheduler::SuspendPreemptive
    sqlservr!SOS_Event::Wait
    sqlservr!LatchBase::Suspend
    sqlservr!LatchBase::AcquireInternal
    sqlservr!CAutoLatch::GetAccess
    sqlservr!BPool::ReplenishFreeList
    sqlservr!BPool::Steal
    sqlservr!SQLSinglePageAllocator::AllocatePages
    sqlservr!MemoryNode::AllocatePagesInternal
    sqlservr!MemoryClerkInternal::AllocatePages
    sqlservr!CMemThread::PbGetNewPages
    sqlservr!CMemFixed::Alloc
    sqlservr!CMemThread::Alloc
    sqlservr!operator new
    sqlservr!SOS_Node::EnqueueTaskDirectInternal
    sqlservr!SOS_Node::EnqueueTaskDirect
    sqlservr!TDSSNIClient::AcceptCommon

    ection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is AT/AT COMPATIBLE.
    Current time is 09:10:11 08/25/13.
    2 Unknown CPU 9., 2800 Mhz processor (s).
    Windows NT 5.2 Build 3790 CSD Service Pack 2.

    Memory
    MemoryLoad = 95%
    Total Physical = 4095 MB
    Available Physical = 184 MB
    Total Page File = 9974 MB
    Available Page File = 4497 MB
    Total Virtual = 8388607 MB
    Available Virtual = 8384006 MB
    **Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
    ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0083.txt
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 08/25/13 09:10:11 spid 0
    *
    * Non-yielding IOCP Listener
    *
    * *******************************************************************************
    * ——————————————————————————-
    * Short Stack Dump

    ate 25/08/2013 8:30:34
    Log SQL Server (Archive #1 – 25/08/2013 10:05:00)

    Source spid2s

    Message
    CACHESTORE_STACKFRAMES (Total)
    VM Reserved = 0 KB
    VM Committed = 0 KB
    AWE Allocated = 0 KB
    SM Reserved = 0 KB
    SM Committed = 0 KB
    SinglePage Allocator = 0 KB
    MultiPage Allocator = 8 KB

    • Jose Ramon Vela Martin said

      More info—Please does anyone know this issue???

      0:006> kP
      Child-SP RetAddr Call Site
      00000000`047cf018 00000000`77d6d03b ntdll!ZwWaitForMultipleObjects+0xa
      00000000`047cf020 00000000`77d6bb41 kernel32!WaitForMultipleObjectsEx+0x1cf
      00000000`047cf140 00000000`014f6b64 kernel32!WaitForMultipleObjects+0×11
      00000000`047cf180 00000000`02b449d7 sqlservr!ResourceMonitor::CheckIndicators+0×184
      00000000`047cf230 00000000`02b49a51 sqlservr!ResourceMonitor::ResourceMonitorTask+0×127
      00000000`047cf360 00000000`015c483e sqlservr!SetupResourceMonitorTaskContext+0×481
      00000000`047cfaf0 00000000`015c4ad9 sqlservr!SOS_Task::Param::Execute+0xee
      00000000`047cfc00 00000000`015bb254 sqlservr!SOS_Scheduler::RunTask+0xc9
      00000000`047cfc90 00000000`014d4687 sqlservr!SOS_Scheduler::ProcessTasks+0xb4
      00000000`047cfd00 00000000`017be7b9 sqlservr!SchedulerManager::WorkerEntryPoint+0xe7
      00000000`047cfda0 00000000`017be585 sqlservr!SystemThread::RunWorker+0×59
      00000000`047cfde0 00000000`017eade8 sqlservr!SystemThreadDispatcher::ProcessWorker+0xb5
      00000000`047cfe80 00000000`781337d7 sqlservr!SchedulerManager::ThreadEntryPoint+0×128
      00000000`047cff20 000007ff`fffa9498 msvcr80!_callthreadstartex(void)+0×17 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 348]
      00000000`047cff28 00000000`005732f0 0x7ff`fffa9498
      00000000`047cff30 00000000`005732f0 0x5732f0
      00000000`047cff38 00000000`005adce0 0x5732f0
      00000000`047cff40 00000000`78133810 0x5adce0
      00000000`047cff48 00000000`78133894 msvcr80!_threadstartex(void) [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 260]
      00000000`047cff50 00000000`781e95c0 msvcr80!_threadstartex(
      void * ptd = 0×00000000`77d6b6e0)+0×84 [f:\dd\vctools\crt_bld\self_64_amd64\crt\src\threadex.c @ 326]

  21. […] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… […]

  22. […] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… […]

  23. Mudit said

    Hi Karthick,
    I am facing similar problem, occurrences of bugcheck dump in sql server logs. I asked same question in msdn forum where you mentioned using winbdg tool.
    Thank you,

    Here is the Stack file:

    sqltses!CEsCompValSeg::DestroyExecValSeg
    sqltses!CEsExec::`vector deleting destructor’
    sqltses!CEsRuntime::DestroyPesx
    sqlmin!RowsetNewSS::~RowsetNewSS
    sqlmin!RowsetNewSS::`vector deleting destructor’
    sqlmin!RowsetNewSS::Release
    sqlmin!CQueryExecContext::~CQueryExecContext
    sqlmin!CQueryScan::ShutdownQueryExecContext
    sqlmin!CQueryScan::DestroyQuery
    sqllang!CExecuteStatement::Destroy
    sqllang!CMetaStmtSet::DeleteXstmtsInPartition
    sqllang!CMetaStmtSet::StepOnStatements
    sqllang!CCompPlan::StepOnStatements
    sqllang!CCacheStoreCPCallback::SteppedOnEntry
    sqldk!CacheClockHand::MoveInternal
    sqldk!CacheClockHand::Move
    sqldk!TClockAlgorithm<CacheClockHand >::StartExternalPressure
    sqldk!ResourceMonitor::NotifyMemoryConsumers
    sqldk!ResourceMonitor::ResourceMonitorTask
    sqlservr!SetupResourceMonitorTaskContext
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    And here is the bugcheck dump file:

    Current time is 18:16:11 01/12/14.
    =====================================================================
    BugCheck Dump
    =====================================================================

    This file is generated by Microsoft SQL Server
    version 11.0.3393.0
    upon detection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is AMD Opteron(TM) Processor 6276 .
    Bios Version is INTEL – 6040000
    PhoenixBIOS 4.0 Release 6.0
    1 X64 level 8664, 14 Mhz processor (s).
    Windows NT 6.2 Build 9200 CSD .

    Memory
    MemoryLoad = 75%
    Total Physical = 4095 MB
    Available Physical = 1023 MB
    Total Page File = 6319 MB
    Available Page File = 1725 MB
    Total Virtual = 8388607 MB
    Available Virtual = 8378657 MB
    **Dump thread – spid = 0, EC = 0×0000000000000000
    ***Stack Dump being sent to E:\MSSQL\Data\MSSQL11.MSSQLSERVER\MSSQL\LOG\SQLDump0013.txt
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 01/12/14 18:16:11 spid 4088
    *
    * Non-yielding Resource Monitor
    *
    * *******************************************************************************
    * ——————————————————————————-
    * Short Stack Dump

    • Hello Mudit,
      Mail the dump to me and sql server error log. I will let you know the next steps. Basically your resource monitor thread is stuck trying to clear the procedure cache.

      • Mudit said

        Hi Karthick,
        I sent an email to karthick_pk83 at hotmaildotcom. Thanks for looking into this issue.
        I am running dbcc freeproccache on this server. Trying to see, after clear cache, how long it take for resource monitor messages to reappear.
        Thanks,

  24. m60freeman said

    I am working with a SQL Server 2005 instance. I installed the debugger and followed the instructions, but I get this:

    Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
    Copyright (c) Microsoft Corporation. All rights reserved.

    Loading Dump File [C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.mdmp]
    User Mini Dump File: Only registers, stack and portions of memory are available

    Comment: ‘Stack Trace’
    Comment: ‘Deadlocked Schedulers’
    Symbol search path is: *** Invalid ***
    ****************************************************************************
    * Symbol loading may be unreliable without a symbol search path. *
    * Use .symfix to have the debugger choose a symbol path. *
    * After setting your symbol path, use .reload to refresh symbol locations. *
    ****************************************************************************
    Executable search path is:
    Windows 7 Version 7601 (Service Pack 1) UP Free x86 compatible
    Product: Server, suite: TerminalServer SingleUserTS
    Machine Name:
    Debug session time: Thu Jan 16 15:06:51.000 2014 (GMT-5)
    System Uptime: 1 days 11:40:53.029
    Process Uptime: 1 days 11:41:45.000
    ……………………………………………………….
    …………………………………………..
    Loading unloaded module list
    ….
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (610.708): Unknown exception – code 00000000 (first/second chance not available)
    eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=0301fc28 edi=00000000
    eip=7737fd91 esp=0301fbe4 ebp=0301fc4c iopl=0 nv up ei pl zr na pe nc
    cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for ntdll.dll –
    ntdll!NtDelayExecution+0×15:
    7737fd91 83c404 add esp,4
    0:004> .sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;
    Symbol search path is: srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols
    Expanded Symbol search path is: srv*c:\websymbols*http://msdl.microsoft.com/download/symbols
    0:004> .reload /f
    .

    Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
    Run !sym noisy before .reload to track down problems loading symbols.

    …………………………*** ERROR: Symbol file could not be found. Defaulted to export symbols for instapi.dll –
    ..*** ERROR: Module load completed but symbols could not be loaded for sqlevn70.rll
    ………………*** ERROR: Module load completed but symbols could not be loaded for comres.dll
    ……*** ERROR: Symbol file could not be found. Defaulted to export symbols for msfte.dll –
    …….
    …………..*** ERROR: Module load completed but symbols could not be loaded for sqlnclir.rll
    …*** ERROR: Symbol file could not be found. Defaulted to export symbols for msftepxy.dll –
    …….*** ERROR: Module load completed but symbols could not be loaded for odbcint.dll
    .*** ERROR: Module load completed but symbols could not be loaded for XPStar90.RLL
    ..*** ERROR: Module load completed but symbols could not be loaded for xplog70.rll
    ….*** ERROR: Module load completed but symbols could not be loaded for oledb32r.dll
    …*** WARNING: Unable to verify checksum for _IBProvider_v2_i.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for _IBProvider_v2_i.dll –
    .*** WARNING: Unable to verify checksum for gds32.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for gds32.dll –
    ..*** WARNING: Unable to verify checksum for cc3250mt.dll
    *** ERROR: Symbol file could not be found. Defaulted to export symbols for cc3250mt.dll –
    ………….
    Loading unloaded module list
    ….
    0:004> .cxr sqlservr!g_copiedStackInfo+0X20
    Unable to read context, Win32 error 0n30
    0:004> kc 100

    ntdll!NtDelayExecution
    KERNELBASE!SleepEx
    KERNELBASE!Sleep
    sqlservr!SchedulerMonitor::Run
    sqlservr!SchedulerMonitor::EntryPoint
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_endthreadex
    msvcr80!_endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!__RtlUserThreadStart
    ntdll!_RtlUserThreadStart

    • m60freeman said

      I tried lmvm sqlservr and got this result:

      start end module name
      01000000 02c27000 sqlservr (pdb symbols) C:\Program Files (x86)\Debugging Tools for Windows (x86)\sym\sqlservr.pdb\C6512984CD7C4067AB9E7B96653A666F2\sqlservr.pdb
      Loaded symbol image file: sqlservr.exe
      Mapped memory image file: C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe
      Image path: C:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\Binn\sqlservr.exe
      Image name: sqlservr.exe
      Timestamp: Fri Dec 10 17:23:31 2010 (4D02A863)
      CheckSum: 01BFD679
      ImageSize: 01C27000
      File version: 2005.90.5000.0
      Product version: 9.0.5000.0
      File flags: 0 (Mask 3F)
      File OS: 40000 NT Base
      File type: 1.0 App
      File date: 00000000.00000000
      Translations: 0409.04e4
      CompanyName: Microsoft Corporation
      ProductName: Microsoft SQL Server
      InternalName: SQLSERVR
      OriginalFilename: SQLSERVR.EXE
      ProductVersion: 9.00.5000.00
      FileVersion: 2005.090.5000.00
      FileDescription: SQL Server Windows NT
      LegalCopyright: © Microsoft Corp. All rights reserved.
      LegalTrademarks: Microsoft® is a registered trademark of Microsoft Corporation. Windows(TM) is a trademark of Microsoft Corporation
      Comments: NT INTEL X86

      I then tried !findstack sqlservr!SQL_SOSNonYieldSchedulerCallback, but it didn’t return a result.

      • Hello Mark,

        I looked at the dump. You can ignore this dump if it is one time occurrence. I believe you haven’t changed the default max worker threads. If you see this dump again or if it has happened many times before then please let me know.

        Thanks Karthick

        • m60freeman said

          Thank you very much for taking the time to look at the files I sent. I have not changed the default max worker threads. Would doing so help to avoid a future such problem with this instance?

          Do you have any suggestions as to what I can do to be able to debug these dumps so I do not get the “Unable to read context, Win32 error 0n30″ error? I’d like to be able to analyze future dumps without bothering you.

          Thanks, Mark

  25. Lina said

    Hi,

    Could you please take a look to my stack:

    msxmlsql!assignRO
    msxmlsql!ChildNodeSet::advance
    msxmlsql!AdvanceNodeSet::nextElement
    msxmlsql!FilterMatchNodeSet::getIndex
    msxmlsql!FilterMatcher::matches
    msxmlsql!FilterQuery::matches
    msxmlsql!ChildQuery::matches
    msxmlsql!DescendantSelector::selectNextDescendant
    msxmlsql!DescendantNodeSet::advance
    msxmlsql!AdvanceNodeSet::nextElement
    msxmlsql!XSLPatternWrapper::nextNode
    sqlservr!CXMLRowset::FGetNextRow
    sqlservr!CXMLRowset::GetNextRows
    sqlservr!CQScanRmtScanNew::GetRowHelper
    sqlservr!CQScanCountStarNew::GetRowHelper
    sqlservr!CQScanStreamAggregateNew::GetCalculatedRow
    sqlservr!CQScanStreamAggregateNew::Open
    sqlservr!CQueryScan::Uncache
    sqlservr!CXStmtQuery::SetupQueryScanAndExpression
    sqlservr!CXStmtQuery::InitForExecute
    sqlservr!CXStmtQuery::ErsqExecuteQuery
    sqlservr!CXStmtSelect::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    I have Microsoft SQL Server 2008R2 SP2.
    Cumulative update package 1 has been installed.

    Thanks!

    • Hello Polina,

      I think SQL server is Iterating through the Nodes of the XML document and the document is very large. How frequent the issue is? Can you reduce the size of XML document? Share the dump with me I will check If I can pull out the XML document Thanks Karthick

      • Lina said

        Thanks Karthick!

        Yes you are right.
        We have a stored procedure which parses xml files. Files come from third party and we can’t affect the size.
        In normal situation there is no problem.
        Problem comes only if procedure execution is terminated. In our case it happens from application side by time out.
        I got same result when terminate procedure execution in Management Studio.
        Problem is not so critical for us, because we can control it by increasing time out.
        But we want to find better solution.

        I’ll send dump files by email.

        BR. Polina

        • Hello Polina,

          I looked at the dumps and for me it looks like some sort of flaw in SQL. Let me confirm my understanding, I see oc_Set_Account_Statement_To_DB is executing. Do we get this dump when you kill the SP execution?

          I would need to see the repro to confirm.

          Which region are you in?

          Thanks

          Karthick

          • Lina said

            Thank you Karthick.
            Yes, error can be reproduced by killing oc_Set_Account_Statement_To_DB stored procedure execution.
            I am from Finland.

            BR, Polina

  26. m60freeman said

    Karthick,

    I have a stack dump from a Non-yielding Resource Monitor (Resource Monitor (0x5e4) Worker 0x7FFFC0E8 appears to be non-yielding on Node 0) on a SQL Server 2005 (32-bit) SP4 system. Using the debugger, I was able to get this from the dump:

    sqlservr!CMxcContainer::UlDelete
    sqlservr!CCacheStoreCPCallback::SteppedOnEntry
    sqlservr!ClockHand::Move
    sqlservr!ClockAlgorithm::MoveHand
    sqlservr!ClockAlgorithm::ProcessTick
    sqlservr!SOS_CacheStore::Notify
    sqlservr!ResourceMonitor::NotifyMemoryConsumers
    sqlservr!ResourceMonitor::ResourceMonitorTask
    sqlservr!SetupResourceMonitorTaskContext
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_endthreadex
    msvcr80!_endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!__RtlUserThreadStart
    ntdll!_RtlUserThreadStart

    This doesn’t match any of the stacks in your original post. What do you think might be causing it?

    Thanks,
    Mark

  27. m60freeman said

    Karthick,

    I have yet another stack dump. Really, I’m not trying to set a record. :-)

    This one is from a Non-yielding Resource Monitor (Resource Monitor (0×564) Worker 0x000000008004E1A0 appears to be non-yielding on Node 0) on a SQL Server 2008R2 (64-bit) SP2 system. (My other ones were from 2005 32-bit instances.) Using the debugger, I was able to get this from the dump:

    Call Site
    sqlservr!CExecuteStatement::Destroy
    sqlservr!CMetaStmtSet::DeleteXstmtsInPartition
    sqlservr!CMetaStmtSet::StepOnStatements
    sqlservr!CCompPlan::StepOnStatements
    sqlservr!CCacheStoreCPCallback::SteppedOnEntry
    sqlservr!CacheClockHand::MoveInternal
    sqlservr!CacheClockHand::Move
    sqlservr!TClockAlgorithm::ProcessTick
    sqlservr!SOS_CacheStore::Notify
    sqlservr!ResourceMonitor::NotifyMemoryConsumers
    sqlservr!ResourceMonitor::ResourceMonitorTask
    sqlservr!SetupResourceMonitorTaskContext
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    This doesn’t match any of the stacks in your original post. What do you think might be causing it?

    Thanks,
    Mark

    • Hello Mark,

      If you look at the stack carefully, you will notice resource monitor thread is trying to clear the cache (Scaling down the memory usage) and could not complete the operation in expected time (Crossed 60 seconds without yielding), so certainly there should have been memory pressure on the system when this has happened, but that doesn’t mean thread can execute for long time without yielding. We have to look at the error log when the issue occurred , you will notice Kernel time, user time, process utilization next to ” appears to be non-yielding ” that will help us to understand if thread really did something (or) got affected because of some performance problem on OS. We also had some known issues with thread not yielding when there is large cache. Send me all the dumps, complete SQL Server error log and SQLDump00*.txt files. I will look at them and let you know the next steps.

      Thanks Karthick

      • Hello Mark,

        Great you provided the logs.

        Previous analysis stand same. From the stack you would have noticed that the resource monitor is scaling down the memory usage of SQL Server. This certainly happens when the available memory on the system drops.
        Look at the error log just above the dump . You will see the below error message before the dump.
        A significant part of sql server process memory has been paged out. This may result in a performance degradation. Duration: 0 seconds. Working set (KB): 107520, committed (KB): 240840, memory utilization: 44%.”

        SQL Server is using only 240 MB, Just 107 Mb in RAM and remaining is paged out. Now you have to identify which other process is taking all the memory in the server, causing SQL Server to scale down its memory usage, making windows to move SQL Server working set to page file and fix that process. SQL Server is just a Victim in this scenario.
        Check if vmware balloming is taking the memory from server Follow: http://mssqlwiki.com/2013/03/31/sql-server-and-vmware-ballooning/ If it is not then we have to identify the culprit process (other process which is eating the memory on the sever) we have to follow http://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/

  28. Hugo Tremblay said

    I`m getting this one:
    0:018> kc 100
    Call Site
    ntdll!ZwWaitForSingleObject
    KERNELBASE!WaitForSingleObjectEx
    sqldk!SystemThread::Suspend
    sqldk!SOS_Scheduler::SuspendPreemptive
    sqldk!UnfairRecursiveMutexInternal::LongWait
    sqldk!TAutoMutex<UnfairRecursiveMutexInternal,4294967295>::GetAccess
    sqldk!PhysicalPageCache::AllocatePhysicalPages
    sqldk!MemoryNode::AllocateUserPhysicalPages
    sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality
    sqldk!SOS_MemoryBlockAllocator::AllocateBlock
    sqldk!SOS_MemoryWorkSpace::AllocatePage
    sqldk!MemoryNode::AllocatePagesInternal
    sqldk!MemoryClerkInternal::AllocatePages
    sqldk!CMemThread::PbGetNewPages
    sqldk!CSlotPageMgr::PbAllocate
    sqldk!CMemThread::Alloc
    sqllang
    sqllang
    sqllang
    sqllang
    sqllang
    sqllang
    sqllang
    sqllang
    sqldk!SOS_Node::ListenOnIOCompletionPort
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    This is version 11.0.3393

    My errors are alternating between latch timeouts and non-yielding (the one I wrote about is a non-yielding)

    Thanks in advance if you find something

  29. I have the following dump from a non-yielding scheduler. This is on 10.0.5785. Immediately prior to this, there were several log entries regarding delayed I/O requests to tempdb.

    Call Site
    ntdll!NtClose
    KERNELBASE!CloseHandle
    kernel32!CloseHandleImplementation
    sqlservr!CDiskDev::Close
    sqlservr!CDiskDev::~CDiskDev
    sqlservr!CDiskDev::`scalar deleting destructor’
    sqlservr!CStreamDev::Release
    sqlservr!CStreamBuf::~CStreamBuf
    sqlservr!CStreamBuf::`scalar deleting destructor’
    sqlservr!CStreamBuf::Release
    sqlservr!CImportStream::~CImportStream
    sqlservr!CImportStream::`scalar deleting destructor’
    sqlservr!CAutoImportStream::~CAutoImportStream
    sqlservr!CImpImportProvider::GetRowsetPtr
    msvcr80!_CallSettingFrame
    msvcr80!__FrameUnwindToState
    msvcr80!__FrameUnwindToEmptyState
    msvcr80!__InternalCxxFrameHandler
    msvcr80!__CxxFrameHandler
    sqlservr!_GSHandlerCheck_EH
    ntdll!RtlpExecuteHandlerForUnwind
    ntdll!RtlUnwindEx
    kernel32!RtlUnwindExStub
    msvcr80!_UnwindNestedFrames
    msvcr80!FindHandler
    msvcr80!__InternalCxxFrameHandler
    msvcr80!__CxxFrameHandler
    sqlservr!_GSHandlerCheck_EH
    ntdll!RtlpExecuteHandlerForException
    ntdll!RtlDispatchException
    ntdll!RtlRaiseException
    KERNELBASE!RaiseException
    msvcr80!_CxxThrowException
    sqlservr!Worker::SetExceptionInfo
    sqlservr!ex_raise2
    sqlservr!ex_raise
    sqlservr!CStreamBuf::CheckIOError
    sqlservr!CImportStream::SwitchBuffer
    sqlservr!CImpImportProvider::GetRowsetPtr
    sqlservr!CreateInprocProvider
    sqlservr!CBcpImport::CreateInprocProvider
    sqlservr!CBcpImport::InitForOpenRowset
    sqlservr!CreateBulkRowset
    sqlservr!COledbRangeStreamBulk::DoNormalize
    sqlservr!COledbRange::Normalize
    sqlservr!alloca_probe
    sqlservr!CAlgTableMetadata::Bind
    sqlservr!CRelOp_Get::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_QuerySpec::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_Insert::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_DMLQuery::BindTree
    sqlservr!COptExpr::BindTree
    sqlservr!CRelOp_Query::FAlgebrizeQuery
    sqlservr!CStmtBulkInsert::FBind
    sqlservr!CProchdr::FNormalizeStep
    sqlservr!CSQLSource::FCompile
    sqlservr!CSQLSource::FCompWrapper
    sqlservr!CSQLSource::Transform
    sqlservr!CSQLSource::Execute
    sqlservr!ExecuteSql
    sqlservr!CSpecProc::ExecuteSpecial
    sqlservr!CXProc::Execute
    sqlservr!alloca_probe
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!CStmtExecProc::XretLocalExec
    sqlservr!CStmtExecProc::XretExecExecute
    sqlservr!CXStmtExecProc::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  30. Hi Karthick-

    I am getting a non-yeilding IOCP dump in my SQL 2012 server 11.0.3128

    I didn’t find the dump listed, but here is the stack please let me know if you have any advice.

    Kind regards,

    -Nathan

    Call Site
    ntdll!ZwClose
    mswsock!SockCloseSocket
    mswsock!WSPCloseSocket
    ws2_32!closesocket
    sqllang!Tcp::FCloseTransport
    sqllang!SNI::detail::Transport::FCloseRefHandle
    sqllang!Tcp::Close
    sqllang!Smux::InternalClose
    sqllang!Smux::ReadDone
    sqllang!SNIReadDone
    sqldk!SOS_Node::ListenOnIOCompletionPort
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  31. B Jones said

    The stack we received recently along with Non-yielding IOCP Listener is similar, but not quite the same as others I have seen here.

    sqlservr!BPool::Steal
    sqlservr!SQLSinglePageAllocator::AllocatePages
    sqlservr!MemoryNode::AllocatePagesInternal
    sqlservr!MemoryClerkInternal::AllocatePages
    sqlservr!CMemThread::PbGetNewPages
    sqlservr!CVarPageMgr::PviNewVarPage
    sqlservr!CVarPageMgr::PbAllocate
    sqlservr!CMemThread::Alloc
    sqlservr!operator new
    sqlservr!SNI_Conn::InitObject
    sqlservr!SNIAcceptDoneWithReturnCode
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadStart

    SQL 2005 Standard (64-bit)
    version 9.00.5000.00

    Computer type is AT/AT COMPATIBLE.
    Current time is 05:40:56 05/06/14.
    4 Unknown CPU 9., 2328 Mhz processor (s).
    Windows NT 5.2 Build 3790 CSD Service Pack 2.

    Memory
    MemoryLoad = 88%
    Total Physical = 32762 MB
    Available Physical = 3654 MB
    Total Page File = 81450 MB
    Available Page File = 47144 MB
    Total Virtual = 8388607 MB
    Available Virtual = 8355267 MB
    **Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 05/06/14 05:40:56 spid 0
    *
    * Non-yielding IOCP Listener
    *
    * *******************************************************************************

    This is not recurring often. It did happen approximately 5 months ago. The only difference between the stack from that time and the most recent is the previous stack included as the first line:
    sqlservr!BPool::ReplenishFreeList

    Note, we had not limited max memory, but based on many references, including much of what is here, I have today placed a max memory limit of 30715.

    If you have a solid idea, that would be more than fantastic!

  32. Kaitlyn said

    I don’t see a match for my call stack here. I can’t seem to figure it out, is it memory pressure?

    from SQLDump0001.txt:
    Current time is 01:07:38 06/05/14.
    =====================================================================
    BugCheck Dump
    =====================================================================

    This file is generated by Microsoft SQL Server
    version 11.0.3379.0
    upon detection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz.
    Bios Version is INTEL – 6040000
    PhoenixBIOS 4.0 Release 6.0
    4 X64 level 8664, 14 Mhz processor (s).
    Windows NT 6.2 Build 9200 CSD .

    Memory
    MemoryLoad = 97%
    Total Physical = 32767 MB
    Available Physical = 791 MB
    Total Page File = 37631 MB
    Available Page File = 5777 MB
    Total Virtual = 134217727 MB
    Available Virtual = 134111055 MB
    **Dump thread – spid = 0, EC = 0×0000000000000000
    ***Stack Dump being sent to J:\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\LOG\SQLDump0001.txt
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 06/05/14 01:07:38 spid 1432
    *
    * Non-yielding Resource Monitor
    *
    * *******************************************************************************
    * ——————————————————————————-
    * Short Stack Dump

    Call Site
    sqlmin!CQueryExecContext::FinalCleanup
    sqlmin!CQueryScan::DestroyQuery
    sqllang!CExecuteStatement::Destroy
    sqllang!CMetaStmtSet::DeleteXstmtsInPartition
    sqllang!CMetaStmtSet::StepOnStatements
    sqllang!CCompPlan::StepOnStatements
    sqllang!CCacheStoreCPCallback::SteppedOnEntry
    sqldk!CacheClockHand::MoveInternal
    sqldk!CacheClockHand::Move
    sqldk!TClockAlgorithm<CacheClockHand >::StartExternalPressure
    sqldk!ResourceMonitor::NotifyMemoryConsumers
    sqldk!ResourceMonitor::ResourceMonitorTask
    sqlservr!SetupResourceMonitorTaskContext
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    We are on Windows 2012 R2 with SQL Server 2012 SP CU6 I believe. We have set max server memory at 28 GB. There’s 32gb available.
    Thanks for any help or guidance on this!

    Kaitlyn

  33. Luke said

    Hi there,

    I’m seeing this in the stack:

    0:046> .cxr sqlservr!g_copiedStackInfo+0X20
    rax=0000000000000002 rbx=0000000000000000 rcx=000000008018c240
    rdx=0000000000000e2c rsi=000000002a8c5a80 rdi=0000000000000001
    rip=00007ff87e41af5a rsp=00000000069ceb30 rbp=00000000069cebc1
    r8=0000000000000000 r9=00000000069d0450 r10=000000008050f320
    r11=00000000013c4af0 r12=00000000069cef90 r13=00007ff87bb62940
    r14=00000000069cf0e9 r15=00000000069cefa8
    iopl=0 nv up ei pl zr na po nc
    cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
    ntdll!NtQueryAttributesFile+0xa:
    00007ff8`7e41af5a c3 ret
    0:046> KC 100
    Call Site
    ntdll!NtQueryAttributesFile
    ntdll!RtlDoesFileExists_UstrEx
    ntdll!RtlDosSearchPath_Ustr
    KERNELBASE!BasepLoadLibraryAsDataFileInternal
    KERNELBASE!LoadLibraryExW
    KERNELBASE!GetFileVersionInfoSizeExW
    sqlservr!CCLRHost::CheckSqlAccessVersion
    sqlservr!CCLRHost::InitProductSpecific
    sqlservr!CCLRHostBase::InitRunTime
    sqlservr!SysClrTypes::GetAssemblyFullPathName
    sqlservr!SystemAssemblyFileBuffer::GetSystemAssemblyFileBuffer
    sqlservr!CMEDProxyAssemblyFile::GetSourceBytes
    sqlservr!CAssemblyMetaInfo::LoadAssemblyFromDatabase
    sqlservr!GetAssemblyProperty
    sqlservr!SsvarAsmPropWstrStr
    sqlservr!CEs::GeneralEval
    sqlservr!CXStmtQuery::ErsqExecuteQuery
    sqlservr!CXStmtSelect::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

    Other info:

    =====================================================================
    BugCheck Dump
    =====================================================================

    This file is generated by Microsoft SQL Server
    version 10.50.4305.0
    upon detection of fatal unexpected error. Please return this file,
    the query or program that produced the bugcheck, the database and
    the error log, and any other pertinent information with a Service Request.

    Computer type is Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz.
    Bios Version is INTEL – 6040000
    PhoenixBIOS 4.0 Release 6.0
    4 X64 level 8664, 2 Mhz processor (s).
    Windows NT 6.2 Build 9200 CSD .

    Memory
    MemoryLoad = 92%
    Total Physical = 16383 MB
    Available Physical = 1270 MB
    Total Page File = 33815 MB
    Available Page File = 18096 MB
    Total Virtual = 134217727 MB
    Available Virtual = 134200178 MB
    ***Unable to get thread context for spid 0
    * *******************************************************************************
    *
    * BEGIN STACK DUMP:
    * 06/17/14 02:12:31 spid 1372
    *
    * Non-yielding Scheduler
    *
    * *******************************************************************************

    Any ideas as I’m having trouble finding a solution???

    Many thanks,

    Luke

    • How many times this issue has occurred.

      • Luke said

        I’ve had the server go non responsive a few times in the last few weeks (not good as it’s production!). I’d have to get more logs off the box tomorrow to find out if it’s a recurring issue.

      • Luke said

        I’ve checked the other logs and I do have another non-yielding scheduler dump, but for reasons unknown the dump file is 0kb so I can’t give any more information sadly

  34. Oliver Jen said

    Hi Karthik,

    Have you seen this one before?

    Call Site
    ntdll!ExpInterlockedPopEntrySListResume16
    sqldk!TMemoryBlockAllocator::GetNewActiveBlockFromPartialListInt
    sqldk!TMemoryBlockAllocator::GetNewActiveBlockFromPartialList
    sqldk!SOS_MemoryBlockAllocator::AllocateBlockInt
    sqldk!SOS_MemoryBlockAllocator::AllocateBlock
    sqldk!SOS_MemoryWorkSpace::AllocatePage
    sqldk!MemoryClerkInternal::AllocateReservedPages
    sqlmin!BPool::Steal
    sqlmin!PbufGetGrantedBufs
    sqlmin!soGetBuf
    sqlmin!soInitBobMgr
    sqlmin!addrun
    sqlmin!soAllocRecBuf
    sqlmin!RowsetSorted::InsertRow
    sqlmin!RowsetChangeSort::InsertRow
    sqlmin!CValRowNoHrow::SetDataX
    sqltses!CEsExec::GeneralEval
    sqlmin!CQScanSortNew::PushRow
    sqlmin!CQScanSortNew::BuildSortTable
    sqlmin!CQScanSortNew::OpenHelper
    sqlmin!CQScanMergeJoinNew::Open
    sqlmin!CQueryScan::StartupQuery
    sqllang!CXStmtQuery::SetupQueryScanAndExpression
    sqllang!CXStmtQuery::InitForExecute
    sqllang!CXStmtQuery::ErsqExecuteQuery
    sqllang!CXStmtSelect::XretExecute
    sqllang!CMsqlExecContext::ExecuteStmts
    sqllang!CMsqlExecContext::FExecute
    sqllang!CSQLSource::Execute
    sqllang!InternalPrepExec
    sqllang!SpPrepExec
    sqllang!CSpecProc::ExecuteSpecial
    sqllang!CSpecProc::Execute
    sqllang!process_request
    sqllang!process_commands
    sqldk!SOS_Task::Param::Execute
    sqldk!SOS_Scheduler::RunTask
    sqldk!SOS_Scheduler::ProcessTasks
    sqldk!SchedulerManager::WorkerEntryPoint
    sqldk!SystemThread::RunWorker
    sqldk!SystemThreadDispatcher::ProcessWorker
    sqldk!SchedulerManager::ThreadEntryPoint
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  35. Nishtha said

    Call Site
    sqlservr!AccessorCache::Destroy
    sqlservr!CEnvCollection::CleanupAccessorCache
    sqlservr!CStmtQuery::~CStmtQuery
    sqlservr!CStmtSelect::`vector deleting destructor’
    sqlservr!CStatement::Destroy
    sqlservr!CStatement::Release
    sqlservr!CPlanSkeleton::~CPlanSkeleton
    sqlservr!CPlanSkeleton::`scalar deleting destructor’
    sqlservr!CPlanSkeleton::Release
    sqlservr!CCompPlan::~CCompPlan
    sqlservr!CCompPlan::`vector deleting destructor’
    sqlservr!CCacheObject::DeleteObject
    sqlservr!CCacheObject::Release
    sqlservr!SOS_CacheStore::RemoveDescriptor
    sqlservr!SOS_CacheStore::CacheEntryDescriptor::Destroy
    sqlservr!ClockHand::Move
    sqlservr!ClockAlgorithm::MoveHand
    sqlservr!ClockAlgorithm::ProcessTick
    sqlservr!SOS_CacheStore::Notify
    sqlservr!ResourceMonitor::NotifyMemoryConsumers
    sqlservr!ResourceMonitor::ResourceMonitorTask
    sqlservr!SetupResourceMonitorTaskContext
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    0×0
    0×0
    0×0
    0×0
    msvcr80!_threadstartex
    msvcr80!_threadstartex
    msvcr80!__initialmbcinfo
    0×0

  36. Michael Tang said

    Hi

    Can you please look at my dump output?
    SQL server 2005 SP4

    ntdll!ZwWaitForSingleObject
    ntdll!RtlpWaitOnCriticalSection
    ntdll!RtlEnterCriticalSection
    swi_ifslsp_64!GetLspGuid
    swi_ifslsp_64!GetLspGuid
    ws2_32!closesocket
    sqlservr!Tcp::FCloseRefHandle
    sqlservr!Tcp::Close
    sqlservr!SNIClose
    sqlservr!CNetConnection::CloseNetlibEP
    sqlservr!CNetConnection::KillConnection
    sqlservr!CPhysicalConnection::NotifySessionLogout
    sqlservr!CSession::LogoutSession
    sqlservr!kill_session
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

  37. Shivam bhatia said

    hi Karthick,

    got the below dump and sql server became unresponsive after that …..

    can you please help me with this.

    00000034`fdfe69c8 000007fe`fd0810dc ntdll!ZwWaitForSingleObject+0xa
    00000034`fdfe69d0 00000000`0068ceea KERNELBASE!WaitForSingleObjectEx+0×79
    00000034`fdfe6a70 00000000`00674485 sqlservr!SOS_Scheduler::SwitchContext+0x72f
    00000034`fdfe6d30 00000000`00673db6 sqlservr!SOS_Scheduler::SuspendNonPreemptive+0xc5
    00000034`fdfe6d70 00000000`00bba45d sqlservr!EventInternal<Spinlock >::Wait+0×436
    00000034`fdfe7360 00000000`00bc32af sqlservr!LockOwner::Sleep+0xbd
    00000034`fdfe73d0 00000000`006960c0 sqlservr!lck_lockInternal+0xe8d
    00000034`fdfe8140 00000000`0154c562 sqlservr!BTreeRow::AcquireLock+0×478
    00000034`fdfe82d0 00000000`00696227 sqlservr!alloca_probe+0x1366a6
    00000034`fdfe8330 00000000`006a8f3d sqlservr!IndexDataSetSession::GetNextRowValuesInternal+0x77a
    00000034`fdfe89f0 00000000`006c66a2 sqlservr!RowsetNewSS::FetchNextRow+0x15d
    00000034`fdfe8c10 00000000`006b510c sqlservr!CQScanRangeNew::GetRow+0x6b6
    00000034`fdfe92d0 00000000`006c7378 sqlservr!CQScanStreamAggregateNew::GetRowHelper+0×244
    00000034`fdfe9630 00000000`007abb0a sqlservr!CQScanFilterNew::GetRow+0×28
    00000034`fdfe9660 00000000`007596d5 sqlservr!CQScanNLJoinNew::GetRowHelper+0x218e
    00000034`fdfe9be0 00000000`007589fc sqlservr!CQueryScan::GetRow+0×65
    00000034`fdfe9c10 00000000`00bf0bd1 sqlservr!CXStmtQuery::ErsqExecuteQuery+0×357
    00000034`fdfecc80 00000000`023c9243 sqlservr!CXStmtCondWithQuery::XretExecute+0xe4
    00000034`fdfecfb0 00000000`023ccf7a sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn+0×13
    00000034`fdfecfe0 00000000`017540f0 sqlservr!CMsqlExecContext::ExecuteStmts+0xbca
    00000034`fdfedf90 00000000`00774629 sqlservr!alloca_probe+0x3d87fc
    00000034`fdfee110 00000000`022b84fd sqlservr!CSQLSource::Execute+0×319
    00000034`fdfee240 00000000`022c19d4 sqlservr!ExecuteSql+0x72d
    00000034`fdfeece0 00000000`02d06461 sqlservr!CSpecProc::ExecuteSpecial+0×234
    00000034`fdfeee00 00000000`00778d3f sqlservr!CSpecProc::Execute+0x1f1
    00000034`fdfeef70 00000000`0072fb2a sqlservr!process_request+0×371
    00000034`fdfef230 00000000`0067cc1b sqlservr!process_commands+0x1ba
    00000034`fdfef430 00000000`0067dffa sqlservr!SOS_Task::Param::Execute+0x11b
    00000034`fdfef550 00000000`0067cd69 sqlservr!SOS_Scheduler::RunTask+0xca
    00000034`fdfef5e0 00000000`00b49940 sqlservr!SOS_Scheduler::ProcessTasks+0×99
    00000034`fdfef650 00000000`00b48c30 sqlservr!SchedulerManager::WorkerEntryPoint+0×110
    00000034`fdfef710 00000000`00b47426 sqlservr!SystemThread::RunWorker+0×60
    00000034`fdfef740 00000000`00b49c4f sqlservr!SystemThreadDispatcher::ProcessWorker+0×130
    00000034`fdfef7d0 00000000`74ac37d7 sqlservr!SchedulerManager::ThreadEntryPoint+0x12f
    00000034`fdfef860 00000000`74ac3894 msvcr80!endthreadex+0×47
    00000034`fdfef890 00000000`76d6652d msvcr80!endthreadex+0×104
    00000034`fdfef8c0 00000000`76e9c541 kernel32!BaseThreadInitThunk+0xd
    00000034`fdfef8f0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

  38. Scott said

    Hello –

    Just got a Non-Yielding IOCP Listener stack dump that appears to have caused a failover on a cluster.

    Would you please look at it and give me any advice?

    Windows Server 2003 R2 EE
    SQL Server 2005 EE (9.00.2153)
    Server Ram: 31.9 GB
    SQL Server Max Server Memory: 28000
    MaxDOP: 0
    CLR objects : Many in several databases.

    sqlservr!BPool::BulkUnmap
    sqlservr!BPool::Map
    sqlservr!BPool::Steal
    sqlservr!SQLSinglePageAllocator::AllocatePages
    sqlservr!MemoryNode::AllocatePagesInternal
    sqlservr!MemoryClerkInternal::AllocatePages
    sqlservr!IMemObj::PbGetNewPages
    sqlservr!CMemThread::PbGetNewPages
    sqlservr!CVarPageMgr::PviNewVarPage
    sqlservr!CVarPageMgr::PbAllocate
    sqlservr!CMemObj::Alloc
    sqlservr!CMemThread::Alloc
    sqlservr!Tcp::AcceptConnection
    sqlservr!Tcp::AcceptDone
    sqlservr!SNIAcceptDoneWithReturnCode
    sqlservr!SNIAcceptDone
    sqlservr!SOS_Node::ListenOnIOCompletionPort
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadStart

    Thank you,
    Scott

  39. Tilak said

    Hi Karthick,

    My SQL version is 10.50.4000 and i see the below stack

    Call Site
    ntdll!NtClose
    KERNELBASE!CloseHandle
    kernel32!CloseHandleImplementation
    sqlservr!CTraceFileIoProvider::OpenNextFile
    sqlservr!CTraceWriteRequest::InitForFileTrace
    sqlservr!CTraceFileIoProvider::InitializeWriteRequest
    sqlservr!CTrace::WriteRecord
    sqlservr!CTraceController::ProduceRecord
    sqlservr!CTraceController::ProduceRecordWithTextBase
    sqlservr!CSECAuditBase::PostTraceEvent
    sqlservr!CSecRuleAudit::AuditEvent
    sqlservr!_RuleAudit::AuditEvent
    sqlservr!ISECManager::CheckPermRuleAuditOnly
    sqlservr!DbccThreadContext::Cleanup
    sqlservr!DbccThreadContext::`scalar deleting destructor’
    sqlservr!ThreadContextFactory::UninstallContext
    sqlservr!AutoDbccThreadContext::~AutoDbccThreadContext
    sqlservr!CStmtDbcc::XretExecute
    sqlservr!CMsqlExecContext::ExecuteStmts
    sqlservr!CMsqlExecContext::FExecute
    sqlservr!CSQLSource::Execute
    sqlservr!process_request
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    0×0
    0×0
    0×0

    Can you please suggest what this means.

    Thanks.
    Tilak M

  40. Saravnaa said

    Hi Karthick,

    Below is the output of dump file which I got.
    Please provide your suggestions on this,

    ntdll!NtWaitForSingleObject
    ntdll!RtlAcquireResourceExclusive
    schannel!SpDeleteUserModeContext
    sspicli!LsaDeleteSecurityContext
    sspicli!DeleteSecurityContext
    sqlservr!Ssl::~Ssl
    sqlservr!Ssl::`vector deleting destructor’
    sqlservr!CryptoBase::Release
    sqlservr!SNI_Conn::Release
    sqlservr!SNIPacketRelease
    sqlservr!CNetConnection::Destroy
    sqlservr!ProtocolHelpers::DestroyOrEnqueueIfIocp
    sqlservr!CNetConnection::Release
    sqlservr!CPhysicalConnection::NotifySessionLogout
    sqlservr!CSession::LogoutSession
    sqlservr!CSession::RemoveBatch
    sqlservr!process_close_connection
    sqlservr!process_commands
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!endthreadex
    msvcr80!endthreadex
    kernel32!BaseThreadInitThunk
    ntdll!RtlUserThreadStart

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 2,107 other followers

%d bloggers like this: