Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes

Posted by Karthick P.K on August 21, 2012

Do you see below errors in SQL error along with dumps and stuck?

Non-yielding IOCP Listener

* BEGIN STACK DUMP:
* 05/06/12 03:54:59 spid 0
* Non-yielding IOCP Listener

Non-yielding Scheduler
* BEGIN STACK DUMP:
* 04/16/12 10:09:58 spid 6256
* Non-yielding Scheduler

Non-yielding Resource Monitor

* BEGIN STACK DUMP

* 01/22/09 19:11:16 spid 0

* Non-yielding Resource Monitor

External dump process returned no errors.
Date Time Server Process 0:0:0 (0x31e8) Worker 0x000000016F41d140 appears to be non-yielding on Scheduler 4. Thread creation time: 12010668087858. Approx Thread CPU Used: kernel 2 ms, user 60516 ms. Process Utilization 11%. System Idle 83%. Interval: 71227 ms.

Refer “How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps” for analyzing the Non-yielding Scheduler, Non-yielding IOCP Listener and Non-yielding Resource Monitor Dumps.

If you are interested in just finding a quick resolution follow the below steps to get the Non-Yield stack from the dump and check if it is matching with any existing known issues in SQL Server.

To analyze the dump download and Install Windows Debugger from This link

Step 1:

Open Windbg

step 2:

Choose File menu –> select Open crash dump –>Select the Dump file (SQLDump000#.mdmp)

Step 3:

on command window type
.sympath srv*c:\Websymbols*http://msdl.microsoft.com/download/symbols;

Step 4:

Type .reload /f and hit enter. This will force debugger to immediately load all the symbols.

Step 5:

Type .cxr sqlservr!g_copiedStackInfo+0X20 for SQL Server2005 and SQL Server2008/2008R2 (or) .cxr sqlmin!g_copiedStackInfo+0X20 for SQL Server2012.

Type kc 100 and look at the stack to see if it matches with the stack of any of known issues in SQL Server listed below.

If kc 100 doesn’t display any stack and throws “WARNING: Frame IP not in any known module. Following frames may be wrong” type .cxr to reset to default scope and try .cxr sqlservr!g_copiedStackInfo+0X00c (In 32-Bit (X86) SQL server valid offset for context is 0X00c Look at This blog to see how we identified the offset)

Note:If your stack doesn’t match with any of the stack listed below then paste the stack in comments session of this blog (or) In This face book group.We will try to find the cause for you. If you don’t get any prompt reply from the community, you may need to open a support ticket with Microsoft.

Stack 1

sqlservr!COptExpr::DetachPointersIntoMemo

sqlservr!COptContext::PcxteOptimizeQuery

sqlservr!CQuery::Optimize

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtDML::InitNormal

sqlservr!CStmtDML::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

sqlservr!CSQLSource::Execute

sqlservr!ExecuteSql

sqlservr!CSpecProc::ExecuteSpecial

sqlservr!CXProc::Execute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExecProc::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB :2344600:FIX: "Non-yielding Scheduler" error may occur when you use the CONTAINSTABLE function together with many OR and AND predicates in SQL Server 2008 or in SQL Server 2008 R2

Stack 2

sqlservr!TMatchPattern

sqlservr!FMatchStrTxt

sqlservr!I8CharindexStrBhI8

sqlservr!CEs::GeneralEval4

sqlservr!CXStmtCond::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,0>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands 0x12a

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

2633357 FIX: "Non-yielding Scheduler" error might occur when you run a query that uses the CHARINDEX function in SQL Server 2008 R2

Stack 3

sqlservr!CItvlVal::Copy

sqlservr!CConstraintItvl::PcnstrItvlUnion

sqlservr!CConstraintProp::FBuildItvlFromOr

sqlservr!CConstraintProp::FBuildItvlFromPexpr

sqlservr!CConstraintProp::FAndItvlConstraint

sqlservr!CConstraintProp::AndNewConstraint

sqlservr!CConstraintProp::PcnstrDeriveSelect

sqlservr!CLogOp_Select::PcnstrDerive

sqlservr!CLogOpArg::PcnstrDeriveHandler

sqlservr!CLogOpArg::DeriveGroupProperties

sqlservr!COpArg::DeriveNormalizedGroupProperties

sqlservr!COptExpr::DeriveGroupProperties

sqlservr!COptExpr::DeriveGroupProperties 0xc6

sqlservr!COptExpr::DeriveGroupProperties

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtDML::InitNormal

sqlservr!CStmtDML::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

KB: 982376 FIX: A non-yielding scheduler error or an error 8623 occurs when you run a query that contains a large IN clause in SQL Server 2005,SQL Server 2008, or SQL Server 2008 R2

Stack 4

sqlservr!COptExpr::AdjustParallelPlan

sqlservr!COptContext::PcxteOptimizeQuery

sqlservr!CQuery::Optimize

sqlservr!CQuery::PqoBuild

sqlservr!CStmtQuery::InitQuery

sqlservr!CStmtSelect::Init

sqlservr!CCompPlan::FCompileStep

sqlservr!CSQLSource::FCompile

sqlservr!CSQLSource::FCompWrapper

sqlservr!CSQLSource::Transform

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB: 943060 FIX: A query that has many outer joins takes a long time to compile in SQL Server 2005

Stack 5

sqlservr!CXid::GetBlockingTask

sqlservr!SNode::SearchForDeadlock

sqlservr!DeadlockMonitor::SearchForDeadlock

sqlservr!DeadlockMonitor::SearchAndResolve

sqlservr!DeadlockMonitor::SearchTaskAndResolve

sqlservr!DeadlockMonitor::WorkLoop

sqlservr!lockMonitor

sqlservr!lockMonitorThread

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

956854 Cumulative update package 10 for SQL Server 2005 Service Pack 2

Stack 6

ntdll!ZwQueryAttributesFile

ntdll!RtlDoesFileExists_UstrEx

ntdll!LdrpSearchPath

ntdll!LdrpCheckForLoadedDll

ntdll!LdrpLoadDll

ntdll!LdrLoadDll

kernel32!LoadLibraryExW

mswsock!SockLoadHelperDll

mswsock!SockGetTdiName

mswsock!SockSocket

mswsock!WSPSocket

ws2_32!WSASocketW

ws2_32!WSASocketA

sqlservr!CreateSocket

sqlservr!AcceptObject::AsyncAccept

sqlservr!Tcp::AcceptDone

sqlservr!SNIAcceptDoneWithReturnCode

sqlservr!SNIAcceptDoneWrapper

sqlservr!SNIAcceptDoneRouter

sqlservr!SOS_Node::ListenOnIOCompletionPort

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB 2711549 FIX: An error message is logged when you start SQL Server 2008 R2 or when a client sends a request to SQL Server 2008 R2

Stack 7

ntdll!ZwOpenKey

advapi32!LocalBaseRegOpenKey

advapi32!RegOpenKeyExW

sqlservr!COledbConnect::GetProviderOptions

sqlservr!COledbConnect::SetClsidFromProvider

sqlservr!COledbConnect::Init

sqlservr!CStmtExecProc::XretRemoteExec

sqlservr!CRemoteProcExecLevel::Execute

sqlservr!CStmtExecProc::XretWrapRemoteExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExec::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

KB2468047 FIX: Error code 17883 or "Non-yielding Scheduler" error may occur when you use the OPENQUERY function on SQL Server 2005

Stack 8

ntdll!ZwQueryVirtualMemory

psapi!QueryWorkingSetEx

sqlservr!BPool::Shrink

sqlservr!BPool::ReleaseAwayBufs

sqlservr!BPool::LazyWriter

sqlservr!lazywriter

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

967908 Cumulative update package 13 for SQL Server 2005 Service Pack 2 or 970279 Cumulative update package 4 for SQL Server 2005 Service Pack 3

Stack 9

sqlservr!LatchBase::ReleaseInternal

sqlservr!XVB::GetRecord

sqlservr!RowsetVersionScan::GetData

sqlservr!CQScanRowsetNew::GetRowWithPrefetch

sqlservr!CQScanRowsetNew::GetRow

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRow

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQScanNLJoinNew::GetRow

sqlservr!CQueryScan::GetRow

sqlservr!CXStmtQuery::InitForExecute

sqlservr!CXStmtQuery::ErsqExecuteQuery

sqlservr!CXStmtCondWithQuery::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CXStmtDML::FExecTrigger

sqlservr!CXStmtDML::FExecAllTriggers

sqlservr!CXStmtDML::XretDMLExecute

sqlservr!CXStmtDML::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<0,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtPrepQuery::XretExecute

sqlservr!CExecuteStatement::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecStr::XretExecStrExecute

sqlservr!CXStmtExecStr::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB : 949595 FIX: Error message when you run a query that uses a join condition in SQL Server 2005: "Non-yielding Scheduler"

Stack 10

sqlservr!SQLServerLogIter::LookupScanCache

sqlservr!SQLServerLogIterForward::GetNextBlock

sqlservr!SQLServerLogIterForward::GetNext

sqlservr!LsMgr::GetEndOfLog

sqlservr!LsMgr::ProcessInternalRollForward

sqlservr!LsWorkRequest::Execute

sqlservr!LsWorker::ThreadRoutine

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadStart

KB 970044 FIX: Error message when you try to break database mirroring between two servers that are running SQL Server 2008: "Non-yielding Scheduler"

Stack 11

sqlservr!CLinkedMap

sqlservr!CCheckReadersAndWriters::Release

sqlservr!CMainIlb::~CMainIlb

sqlservr!CBlobHandleFactoryMain::ReleaseILockBytes

sqlservr!CMainIlb::Release

sqlservr!CTraceRpcBinaryStream::~CTraceRpcBinaryStream

sqlservr!CTraceTvpData::~CTraceTvpData

sqlservr!CRpcTraceHelper::CleanUpTraceTvpData

sqlservr!CRpcTraceHelper::TracePostExec

sqlservr!CRPCExecEnv::OnExecFinish

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!endthreadex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2520808 FIX: Non-yielding scheduler error when you run a query that uses a TVP in SQL Server 2008 or in SQL Server 2008 R2 if SQL Profiler or SQL Server Extended Events is used

Stack 12

ntdll!ZwFreeVirtualMemory

KERNELBASE!VirtualFree

sqlservr!MemoryNode::VirtualFree

sqlservr!ReservedMemBlock::FreeMemory

sqlservr!MultiPageAllocator::FreePagesInternal

sqlservr!MultiPageAllocator::FreePages

sqlservr!MemoryNode::FreePagesInternal

sqlservr!MemoryClerkInternal::FreePagesInline

sqlservr!CVarPageMgr::Release

sqlservr!CMemObj::Free

sqlservr!CMemThread<CMemObj>::Free

sqlservr!LockBytesSS::~LockBytesSS

sqlservr!LockBytesHolder::`scalar deleting destructor’

sqlservr!LockBytesHolder::DestroyCallback

sqlservr!CacheLbss

sqlservr!LockBytesSS::Release

sqlservr!CQueryIlb::~CQueryIlb

sqlservr!CBlobHandleFactoryMain::ReleaseILockBytes

sqlservr!CMainIlb::Release

sqlservr!CTraceRpcBinaryStream::~CTraceRpcBinaryStream

sqlservr!CTraceTvpData::~CTraceTvpData

sqlservr!CRpcTraceHelper::CleanUpTraceTvpData

sqlservr!CRpcTraceHelper::TracePostExec

sqlservr!CRPCExecEnv::OnExecFinish

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!endthreadex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2520808 FIX: Non-yielding scheduler error when you run a query that uses a TVP in SQL Server 2008 or in SQL Server 2008 R2 if SQL Profiler or SQL Server Extended Events is used

Stack 13

sqlservr!CompareStringWEnglishNoCase

sqlservr!CTypeInfo::ICompW

sqlservr!CDefaultCollation::ICompW

sqlservr!CDependElem::ICompare

sqlservr!CDependList::Find

sqlservr!CDependList::Insert

sqlservr!CDependList::Concat

sqlservr!CDependList::CollectDependencies

sqlservr!FillSysdepends

sqlservr!CProchdr::CreateProc

sqlservr!CSQLSource::PerformPphFakeExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

ntdll!RtlUserThreadStart

KB 2306162 FIX: Poor performance and some occasional non-yielding scheduler errors occur when you create a complex view that references a large amount of nested views or tables in SQL Server 2008 or in SQL Server 2008 R2

Stack 14

sqlservr!BaseSharedHoBt::GetHoBtId

sqlservr!HoBtFactory::GetDeferredDropCacheHobt

sqlservr!DropDeferredWorkTables

sqlservr!GhostRecordCleanupTask

sqlservr!CGhostCleanupTask::ProcessTskPkt

sqlservr!TaskReqPktTimer::ExecuteTask

sqlservr!OnDemandTaskContext::ProcessTskPkt

sqlservr!SystemTaskContext::ExecuteFunc

sqlservr!SystemTaskEntryPoint

sqlservr!OnDemandTaskContext::FuncEntryPoint

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SchedulerManager::FiberEntryPoint

kernel32!BaseFiberStart

kernel32!RtlCompareMemoryStub

KB 2505256 FIX: Poor performance when worktables that are marked for deferred drop are cleaned up in SQL Server 2008 R2

Stack 15

ntdll!ZwReadFile

kernel32!ReadFile

sqlservr!DiskReadAsync

sqlservr!FCB::AsyncRead

sqlservr!BackupIoRequest::StartDatabaseRead

sqlservr!BackupCopyMachine::CopyFileToBackupSet0

sqlservr!BackupCopyMachine::CopyFileToBackupSet

KB 960543 FIX: SQL Server 2005 or SQL Server 2008 may stop responding when you are performing a backup

Stack 16

sqlservr!Worker::ProfilingCPUTicks::ProfilingCpuTicksCallback

sqlservr!SOS_Scheduler::TaskTransition

sqlservr!SOS_Scheduler::Switch

sqlservr!SOS_Scheduler::SuspendNonPreemptive

sqlservr!SOS_Scheduler::Suspend

sqlservr!SOS_Task::Sleep

sqlservr!BTreeMgr::Seek

sqlservr!BTreeMgr::GetHPageIdWithKey

sqlservr!IndexPageManager::GetPageWithKey

sqlservr!GetRowForKeyValue

sqlservr!IndexRowScanner::EstablishInitialKeyOrderPosition

sqlservr!IndexDataSetSession::GetNextRowValuesInternal

sqlservr!RowsetNewSS::GetNextRows

sqlservr!CMEDScan::FGetRow

sqlservr!CMEDCatalogOwner::GetOwnerAliasIdFromSid

sqlservr!CMEDCatalogOwner::LookupPrimaryIdInCatalog

sqlservr!CMEDCacheEntryFactory::GetProxiedCacheEntryByAltKey

sqlservr!CMEDCatalogOwner::GetProxyOwnerBySID

sqlservr!CMEDProxyDatabase::GetOwnerBySID

sqlservr!GetDefaultSchemaIdCrossDb

sqlservr!GetCtxtSchemaId

sqlservr!CMEDAccess::GetMultiNameObject

sqlservr!CRangeObject::CImplName::FSameObject

sqlservr!CRangeObject::FCheckImplNames

sqlservr!CRangeObject::XretPostSchemaChecks

sqlservr!CRangeObject::XretSchemaChanged

sqlservr!CRangeTable::XretSchemaChanged

sqlservr!CEnvCollection::XretSchemaChanged

sqlservr!CXStmtQuery::XretSchemaChanged

sqlservr!CXStmtSelect::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtExecProc::XretLocalExec

sqlservr!CStmtExecProc::XretExecExecute

sqlservr!CXStmtExecProc::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

KB 2699013 FIX: SQL Server 2008 R2 or SQL Server 2008 stops responding and a "Non-yielding Scheduler" error is logged

Stack 17

sqlservr!CQScanNLJoinNew::GetRowHelper

sqlservr!CQueryScan::GetRow

sqlservr!CXStmtQuery::ErsqExecuteQuery

sqlservr!CXStmtCondWithQuery::XretExecute

sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn

sqlservr!CMsqlExecContext::ExecuteStmts<1,0>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CXStmtDML::FExecTrigger

sqlservr!CXStmtDML::FExecAllTriggers

sqlservr!CXStmtDML::XretDMLExecute

sqlservr!CXStmtDML::XretExecute

sqlservr!CExecStmtLoopVars::ExecuteXStmtAndSetXretReturn

sqlservr!CMsqlExecContext::ExecuteStmts<0,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!CStmtPrepQuery::XretExecute

sqlservr!CMsqlExecContext::ExecuteStmts<1,1>

sqlservr!CMsqlExecContext::FExecute

sqlservr!CSQLSource::Execute

sqlservr!process_request

sqlservr!process_commands

sqlservr!SOS_Task::Param::Execute

sqlservr!SOS_Scheduler::RunTask

sqlservr!SOS_Scheduler::ProcessTasks

sqlservr!SchedulerManager::WorkerEntryPoint

sqlservr!SystemThread::RunWorker

sqlservr!SystemThreadDispatcher::ProcessWorker

sqlservr!SchedulerManager::ThreadEntryPoint

msvcr80!_callthreadstartex

msvcr80!_threadstartex

kernel32!BaseThreadInitThunk

KB 967169 FIX: When you run an UPDATE statement against a table that has a FOR UPDATE trigger that joins the DELETED and INSERTED tables, the query takes a long time to finish

Stack 18

msvcr80!memcpy

BackupString::vswcatf

BackupString::swcatf

BackupHistory::GenerateBackupDetails

sqlservr!BackupHistory::GenerateBackupSet

KB 917971 FIX: You may receive more than 100,000 page faults when you try to back up a SQL Server 2005 database that contains hundreds of files and file groups.

Stack 20

mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!Smux::InternalClose
sqlservr!Smux::ReadDone

"Non-yielding Scheduler" error and SQL Server 2008 or SQL Server 2008 R2 stops responding intermittently in Windows Server 2008 or in Windows Server 2008 R2

Stack 21

mswsock!SockCloseSocket
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close
sqlservr!Smux::InternalClose
sqlservr!Smux::ReadDone

"Non-yielding Scheduler" error and SQL Server 2008 or SQL Server 2008 R2 stops responding intermittently in Windows Server 2008 or in Windows Server 2008 R2

How to analyze Non-Yielding scheduler or Non-yielding IOCP Listener dumps ……

SQL Server Latch & Debugging latch time out

How to Analyze "Deadlocked Schedulers" Dumps

If you liked this post, do like us on Facebook at
https://www.facebook.com/mssqlwiki
and join our Facebook group to get answers for all your SQL Server related questions.

Thank you,

Karthick P.K |My Facebook Page |My Site| Blog space| Twitter

Disclaimer:

The views expressed on this website/blog are mine alone and do not reflect the views of my company or anyone else. All postings on this blog are provided “AS IS” with no warranties, and confers no rights

This entry was posted on August 21, 2012 at 9:08 AM and is filed under Debugging, Performance, SQL General, SQL Server Tools. Tagged: 17883, 17884, 17887, 17888, Configuration, Debugging, error 17883 non yielding, External dump process returned no errors, External dump process returned no errors.DoMiniDump () encountered error, Interval:, IO Completion Listener () Worker appears to be non-yielding on Node . Approx CPU Used: kernel ms, memory dump, non-yielding, Non-yielding IOCP Listener, non-yielding resource monitor, Non-yielding Scheduler, Process 0:0:0 ( ) Worker appears to be non-yielding on Scheduler, Process :0:0 () Worker appears to be non-yielding on Scheduler n. Thread creation time: . Approx Thread CPU Used: kernel 0 ms, SQL General, sql scheduler, SQL Server 2005, SQL Server 2008, SQL Server Engine. Non-yielding Scheduler, SQLServer dump, SQLServer mdmp, Stack Dump, user 0 ms. Process Utilization 0%. System Idle %. Interval: ms., user ms. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

77 Responses to “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”

Pavel Chromy said

October 27, 2012 at 6:53 AM
Hi, our stack doesn’t match with any of the stack :
Call Site
sqlservr!SpinlockBase::SpinToAcquire
sqlservr!SOS_Event::Signal
sqlservr!BPool::Steal
sqlservr!SQLSinglePageAllocator::AllocatePages
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CMemThread::PbGetNewPages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemThread::Alloc
sqlservr!operator new
sqlservr!SNI_Conn::InitObject
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Can you help me ?
Thanks
Pavel Chromy

Reply
- Karthick P.K said
  
  October 27, 2012 at 7:29 AM
  Will look at this in few minutes. I am driving now
  
  Sent from my Windows Phone ________________________________
  
  Reply
- Karthick P.K said
  
  October 27, 2012 at 8:50 AM
  Can you paste the following info
  1. SQL Server version and windows version?
  2. Amont of physical server memory on this server?
  3.What is sQL Server max server memory and Max dop?
  4. Are we suing SQLClr in this server?
  5. How frequently are we getting this dump and errorlog message similar to Thread CPU Used: kernel n ms, user n ms
  
  Reply
  - Pavel Chromy said
    
    October 27, 2012 at 9:17 AM
    Sending additional info :
    
    Microsoft SQL Server 2005 version 9.00.5296.05
    Computer type is AT/AT COMPATIBLE.
    Bios Version is DELL – 1
    Phoenix ROM BIOS PLUS Version 1.10 2.2.3
    40 Unknown CPU 9., 2 Mhz processor (s).
    Windows NT 6.1 Build 7601 CSD Service Pack 1.
    Memory Total Physical = 262130 MB
    
    sp_configure :
    name minimum maximum config_value run_value
    ———————————– ———– ———– ———— ———–
    Ad Hoc Distributed Queries 0 1 0 0
    affinity I/O mask -2147483648 2147483647 0 0
    affinity mask -2147483648 2147483647 0 0
    affinity64 I/O mask -2147483648 2147483647 0 0
    affinity64 mask -2147483648 2147483647 0 0
    Agent XPs 0 1 1 1
    allow updates 0 1 0 0
    awe enabled 0 1 0 0
    blocked process threshold 0 86400 5 5
    c2 audit mode 0 1 0 0
    clr enabled 0 1 0 0
    cost threshold for parallelism 0 32767 5 5
    cross db ownership chaining 0 1 0 0
    cursor threshold -1 2147483647 -1 -1
    Database Mail XPs 0 1 1 1
    default full-text language 0 2147483647 1033 1033
    default language 0 9999 0 0
    default trace enabled 0 1 1 1
    disallow results from triggers 0 1 0 0
    fill factor (%) 0 100 0 0
    ft crawl bandwidth (max) 0 32767 100 100
    ft crawl bandwidth (min) 0 32767 0 0
    ft notify bandwidth (max) 0 32767 100 100
    ft notify bandwidth (min) 0 32767 0 0
    index create memory (KB) 704 2147483647 0 0
    in-doubt xact resolution 0 2 0 0
    lightweight pooling 0 1 0 0
    locks 5000 2147483647 0 0
    max degree of parallelism 0 64 4 4
    max full-text crawl range 0 256 4 4
    max server memory (MB) 16 2147483647 2147483647 2147483647
    max text repl size (B) 0 2147483647 65536 65536
    max worker threads 128 32767 0 0
    media retention 0 365 0 0
    min memory per query (KB) 512 2147483647 1024 1024
    min server memory (MB) 0 2147483647 16 16
    nested triggers 0 1 1 1
    network packet size (B) 512 32767 4096 4096
    Ole Automation Procedures 0 1 0 0
    open objects 0 2147483647 0 0
    PH timeout (s) 1 3600 60 60
    precompute rank 0 1 0 0
    priority boost 0 1 1 1
    query governor cost limit 0 2147483647 0 0
    query wait (s) -1 2147483647 -1 -1
    recovery interval (min) 0 32767 0 0
    remote access 0 1 1 1
    remote admin connections 0 1 0 0
    remote login timeout (s) 0 2147483647 20 20
    remote proc trans 0 1 0 0
    remote query timeout (s) 0 2147483647 10800 10800
    Replication XPs 0 1 0 0
    scan for startup procs 0 1 0 0
    server trigger recursion 0 1 1 1
    set working set size 0 1 0 0
    show advanced options 0 1 1 1
    SMO and DMO XPs 0 1 1 1
    SQL Mail XPs 0 1 0 0
    transform noise words 0 1 0 0
    two digit year cutoff 1753 9999 2049 2049
    user connections 0 32767 0 0
    user options 0 32767 0 0
    Web Assistant Procedures 0 1 0 0
    xp_cmdshell 0 1 0 0
    
    SQLClr do not use.
    
    Over the past 14 days are few stiff SQL Server (Non-yielding IOCP Listener) – in the log then appears “IO Completion Listener (0×830) Worker 0x00000000112DC1C0 Appears to be non-yielding on Node 0th Approx CPU Used: kernel 187 ms, user 924898 ms, Interval: 925116th “processors working at 100% and the server is not responding.
    
    Thank you for any idea
    Pavel
    
    Reply
    - Karthick P.K said
      
      October 27, 2012 at 12:45 PM
      Limit your SQL Server “Max server Memory” (Start with 200,000 MB . We can increase or reduce based on OS and other applications need).
      Most likely since all OS physical memory has been used and OS is struggling with intensive paging SQL Server is affected.
      So if we will limit SQL Server for amount of memory it will resolve this issue.
      Let me know the output of below command as well !findstack sqlservr!BPool::ReplenishFreeList.
      We will get this dump with this stack even if the lazywritter is not able to keep up to the speed of updates/Inserts because of I/O bottlenecks.
      
      Reply
      - Pavel Chromy said
        
        October 28, 2012 at 9:18 AM
        Thank you for the advice, but unfortunately SQL server froze again. Max server memory I set to 150,000 megabytes. In the dump file is only “Non-yielding IOCP Listener”.
        
        Reply
        
        Karthick P.K said
        
        October 28, 2012 at 11:05 AM
        Did you get “Non-yielding IOCP Listener” dump after you set max server memory to 150000? how much was available physical memory in server when you got the dump?
        Check if stack is same in new dump ,Also run !findstack sqlservr!BPool::ReplenishFreeList and paste the output
        
        Reply
        
        Pavel Chromy said
        
        October 28, 2012 at 11:41 AM
        Yes, max memory has already been set to 150.000 MB. Physical memory was available 95.500 MB.
        Sending listing :
        Call Site
        ntdll!NtWaitForSingleObject
        KERNELBASE!WaitForSingleObjectEx
        sqlservr!CDmpDump::DumpInternal
        sqlservr!CDmpDump::Dump
        sqlservr!CImageHelper::DoMiniDump
        sqlservr!stackTrace
        sqlservr!SQL_SOSNonYieldIOCPCallback
        sqlservr!SOS_OS::ExecuteNonYieldIOCPCallbacks
        sqlservr!alloca_probe
        sqlservr!SchedulerMonitor::Run
        sqlservr!SchedulerMonitor::EntryPoint
        sqlservr!SOS_Task::Param::Execute
        sqlservr!SOS_Scheduler::RunTask
        sqlservr!SOS_Scheduler::ProcessTasks
        sqlservr!SchedulerManager::WorkerEntryPoint
        sqlservr!SystemThread::RunWorker
        sqlservr!SystemThreadDispatcher::ProcessWorker
        sqlservr!SchedulerManager::ThreadEntryPoint
        msvcr80!endthreadex
        msvcr80!endthreadex
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart
        
        Reply
        
        Karthick P.K said
        
        October 28, 2012 at 12:25 PM
        This look like scheduler monitor stack. You need to use .cxr sqlservr!g_copiedStackInfo+0X20 and then use kC to get the Non-Yield stack
        
        Reply
        
        Pavel Chromy said
        
        October 28, 2012 at 12:56 PM
        Sorry, here is the correct listing :
        Call Site
        sqlservr!SpinlockBase::SpinToAcquire
        sqlservr!SOS_Event::Signal
        sqlservr!alloca_probe
        sqlservr!SQLSinglePageAllocator::AllocatePages
        sqlservr!MemoryNode::AllocatePagesInternal
        sqlservr!MemoryClerkInternal::AllocatePages
        sqlservr!CMemThread::PbGetNewPages
        sqlservr!CMemFixed::Alloc
        sqlservr!CMemThread::Alloc
        sqlservr!operator new
        sqlservr!LockManager::AllocateLockOwnerBlocks
        sqlservr!LockManager::GetLockOwnerBlocks
        sqlservr!alloca_probe
        sqlservr!XactWorkspaceImp::GetSharedDBLockFromLockManager
        sqlservr!XactWorkspaceImp::GetDBLock
        sqlservr!lockdb
        sqlservr!CMEDUserStoreCallbacks::StartClockRun
        sqlservr!ClockAlgorithm::MoveHand
        sqlservr!ClockAlgorithm::ProcessTick
        sqlservr!SOS_UserStore::Notify
        sqlservr!ResourceMonitor::NotifyMemoryConsumers
        sqlservr!ResourceMonitor::ResourceMonitorTask
        sqlservr!SetupResourceMonitorTaskContext
        sqlservr!SOS_Task::Param::Execute
        sqlservr!SOS_Scheduler::RunTask
        sqlservr!SOS_Scheduler::ProcessTasks
        sqlservr!SchedulerManager::WorkerEntryPoint
        sqlservr!SystemThread::RunWorker
        sqlservr!SystemThreadDispatcher::ProcessWorker
        sqlservr!SchedulerManager::ThreadEntryPoint
        msvcr80!endthreadex
        msvcr80!endthreadex
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart
        
        We’re pretty desperate, do you think it would help to re-install Windows or SQL Server ?
        
        Reply
        
        Karthick P.K said
        
        October 28, 2012 at 2:06 PM
        There is still a memory pressure on this server.Not sure what is causing memory pressure? By any chance do you see OOM errors or dbcc memory status output in SQL Server error log?
        
        1. Enable Lock pages in memory for startup account of SQL Server and restart.
        2. Are you very sure available physical memory was 95GB when the dump occurred?
        3. I would say apply the latest SP and hotfix to windows instead of reinstalling windows. (http://mssqlwiki.com/sqlwiki/sql-performance/windows-2008-and-windows-2008-r2-known-issues-related-to-working-set-memory/)
        4. If you have not restarted SQL Server after the dump please run two SQL Scripts mentioned in http://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/ and share the output.
        5. Also share the output of sys.dm_os_memory_clerks
Pavel Chromy said

October 29, 2012 at 6:05 AM
The error log contains only::
…
2012-10-29 04:25:41.63 Server Using ‘dbghelp.dll’ version ’4.0.5′
2012-10-29 04:25:41.65 Server **Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
2012-10-29 04:25:41.65 Server ***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.txt
2012-10-29 04:25:41.65 Server * *******************************************************************************
2012-10-29 04:25:41.65 Server *
2012-10-29 04:25:41.65 Server * BEGIN STACK DUMP:
2012-10-29 04:25:41.65 Server * 10/29/12 04:25:41 spid 0
2012-10-29 04:25:41.65 Server *
2012-10-29 04:25:41.65 Server * Non-yielding IOCP Listener
2012-10-29 04:25:41.65 Server *
2012-10-29 04:25:41.65 Server * *******************************************************************************
2012-10-29 04:25:41.65 Server * ——————————————————————————-
2012-10-29 04:25:41.65 Server * Short Stack Dump
2012-10-29 04:25:41.70 Server Stack Signature for the dump is 0×0000000000000207
2012-10-29 04:25:44.80 Server External dump process return code 0×20000001.
…
The dump file contains only:
=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 9.00.5296.05
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is AT/AT COMPATIBLE.
Bios Version is DELL – 1
Phoenix ROM BIOS PLUS Version 1.10 2.7.7
Current time is 04:25:41 10/29/12.
40 Unknown CPU 9., 2 Mhz processor (s).
Windows NT 6.1 Build 7600 CSD .

Memory
MemoryLoad = 90%
Total Physical = 262130 MB
Available Physical = 24467 MB
Total Page File = 264176 MB
Available Page File = 25893 MB
Total Virtual = 8388607 MB
Available Virtual = 8124449 MB
**Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0001.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 10/29/12 04:25:41 spid 0
*
* Non-yielding IOCP Listener
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Lock pages is enabled. Yesterday we reinstalled the entire server (windows and SQL server – we apply the latest SP and hotfix) and after 3 hours the server again froze – unfortunately, in this state he can not run anything.

Reply
- Karthick P.K said
  
  October 29, 2012 at 2:58 PM
  Just noticing you have standard edition did you enable trace flag 845 ? while you enabled lock pages in memory
  
  Reply
- Karthick P.K said
  
  October 29, 2012 at 3:45 PM
  Also disable priority boost for SQL Server.
  
  Reply
  - Pavel Chromy said
    
    October 30, 2012 at 10:43 AM
    I did everything you wrote (Add traceflag, Enable lockpages in memory,Restart SQL Server and remove priority boost) and from that time’s running server great !!!
    Thank you.
    
    Reply
In Depth SQL said

October 31, 2012 at 1:38 PM
Hi Karthick – I have stack dump but that is not matching anuy one of above listed dumps. Below is the dump that I got it, can you please help on this.

Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqlservr!CDmpDump::DumpInternal
sqlservr!CDmpDump::DumpFilter
sqlservr!CDmpDump::Dump
msvcr80!__C_specific_handler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!RtlRaiseException
KERNELBASE!RaiseException
sqlservr!CDmpDump::Dump
sqlservr!SQLDumperLibraryInvoke
sqlservr!CImageHelper::DoMiniDump
sqlservr!stackTrace
sqlservr!stackTraceCallBack
sqlservr!ex_terminator
sqlservr!SOS_SEHTranslator
msvcr80!_CallSETranslator
msvcr80!FindHandlerForForeignException
msvcr80!FindHandler
msvcr80!__InternalCxxFrameHandler
msvcr80!__CxxFrameHandler
ntdll!RtlpExecuteHandlerForException
ntdll!RtlDispatchException
ntdll!KiUserExceptionDispatcher
sqlservr!CSlotPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!CMemThread::Alloc
sqlservr!CCriticalSectionSOS::Initialize
sqlservr!Tcp::FInit
sqlservr!Tcp::AcceptConnection
sqlservr!Tcp::AcceptDone
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  October 31, 2012 at 2:05 PM
  Hello InDepth,
  
  This is Exception stack
  {
  ntdll!KiUserExceptionDispatcher
  sqlservr!CSlotPageMgr::PbAllocate
  }
  
  Please follow http://mssqlwiki.com/2012/10/16/sql-server-exception_access_violation-and-sql-server-assertion/
  
  Any ways looking at your stack, This issue is fixed in
  Cumulative Update package 6 for SQL Server 2008 R2
  Cumulative update package 13 for SQL Server 2008 Service Pack 1
  Cumulative update package 3 for SQL Server 2008 Service Pack 2
  
  Thanks
  
  Reply
Scott McCormick said

November 16, 2012 at 8:21 PM
This is the error log a few hours before the dump you asked for in a Facebook post (https://www.facebook.com/mssqlwiki/posts/335361023227918)

I’m assuming you don’t actually need usernames or IP addresses.

2012-11-14 12:45:39.54 Logon Login succeeded for user ‘***\****’. Connection: trusted. [CLIENT: ***.***.***.***]
2012-11-14 12:54:53.92 Server Using ‘dbghelp.dll’ version ’4.0.5′
2012-11-14 12:54:56.81 Server **Dump thread – spid = 0, PSS = 0×00000000, EC = 0×00000000
2012-11-14 12:54:56.81 Server ***Stack Dump being sent to ***\MSSQL.1\MSSQL\LOG\SQLDump0002.txt
2012-11-14 12:54:56.81 Server * *******************************************************************************
2012-11-14 12:54:56.81 Server *
2012-11-14 12:54:56.81 Server * BEGIN STACK DUMP:
2012-11-14 12:54:56.81 Server * 11/14/12 12:54:56 spid 0
2012-11-14 12:54:56.81 Server *
2012-11-14 12:54:56.81 Server * Non-yielding IOCP Listener
2012-11-14 12:54:56.81 Server *
2012-11-14 12:54:56.81 Server * *******************************************************************************
2012-11-14 12:54:56.81 Server * ——————————————————————————-
2012-11-14 12:54:56.81 Server * Short Stack Dump
2012-11-14 12:55:00.36 Server Stack Signature for the dump is 0x0000036D
2012-11-14 12:55:13.73 Logon Error: 18456, Severity: 14, State: 8.
2012-11-14 12:55:20.67 Logon Login succeeded for user ‘***\****’. Connection: trusted. [CLIENT: ***.***.***.***]
2012-11-14 13:03:04.04 Server Timeout waiting for external dump process 68908.

2012-11-14 13:03:04.04 Server IO Completion Listener (0x5b0) Worker 0x008980E8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
ntdll!KiFastSystemCallRet
ntdll!NtRequestWaitReplyPort
secur32!CallSPM
secur32!SecpAcceptSecurityContext
secur32!LsaAcceptSecurityContext
secur32!AcceptSecurityContext
sqlservr!Ssl::Handshake
sqlservr!Ssl::HandshakeReadDone
sqlservr!CryptoBase::ReadDone
sqlservr!SNIReadDone
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadStart

Reply
- Karthick P.K said
  
  November 19, 2012 at 5:21 AM
  Please update your SQLServer to latest service pack. Looks like you are hitting a known issue mentioned in http://support.microsoft.com/kb/929376
  
  Reply
Steve said

November 26, 2012 at 8:46 PM
Hi Karthick,

I have a stack which is not listed. Can you help me with that?

ntdll!KiFastSystemCallRet
ntdll!ZwClose
kernel32!CloseHandle
sqlservr!CTraceFileIoProvider::OpenNextFile
sqlservr!CTraceWriteRequest::InitForFileTrace
sqlservr!CTraceFileIoProvider::InitializeWriteRequest
sqlservr!CTrace::WriteRecord
sqlservr!CTraceController::ProduceRecord
sqlservr!CTraceController::ProduceRecordWithTextBase
sqlservr!AuditEventBase
sqlservr!AuditEventNoCheck
sqlservr!AuditEvent
sqlservr!DbccOperation::CleanupDbccOperation
sqlservr!CStmtDbcc::XretExecute
sqlservr!CExecuteStatement::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_endthreadex
msvcr80!_endthreadex
kernel32!BaseThreadStart

Happend once.

SQL Version: Microsoft SQL Server 2005 – 9.00.3080.00 (Intel X86)
Sep 6 2009 01:43:32 Copyright (c) 1988-2005 Microsoft Corporation
Enterprise Edition on Windows NT 5.2 (Build 3790: Service Pack 2)

Windows Server 2003 Enterprise
Build 379.srv03_sp2_gdr.120503-0334 SP2

Physical Memory: 15.727.764 KB
SQL Server 10GB

Thanks

Reply
- Karthick P.K said
  
  November 27, 2012 at 2:12 AM
  When profiler tace is collected to network location or to a disk which is very slow, we might end with this.
  1. Make sure you allways collect server side trace instead of client side traces when required.
  2. Dont collect multiple trace files to gether.
  
  Reply
How to Analyze "Deadlocked Schedulers" Dumps? « MSSQLWIKI said

January 17, 2013 at 8:12 AM
[...] Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues an… [...]

Reply
Ewan Gibb said

February 14, 2013 at 5:09 PM
Hi,

I’ve been through the steps in this blog and have the following stack trace:

sqldk!CVariableInfo::PbGetBlock
sqldk!TVarPageMgr::PbAllocate
sqldk!CMemObj::Alloc
sqlmin!BTreeRowFactory::AllocateNewPageContext
sqlmin!SplitPage
sqlmin!IndexDataSetSession::InsertSmallRecord
sqlmin!IndexDataSetSession::InsertRowInternal
sqlmin!DatasetSession::InsertRow
sqlmin!RowsetNewSS::InsertRow
sqlmin!CValRowMult::SetDataX
sqltses!CEsExec::GeneralEval
sqlmin!CQScanUpdateNew::GetRow
sqlmin!CQScanNLJoinNew::GetRowHelper
sqlmin!CQScanAssertNew::GetRow
sqlmin!CQueryScan::GetRow
sqllang!CXStmtQuery::ErsqExecuteQuery
sqllang!CXStmtDML::XretDMLExecute
sqllang!CXStmtDML::XretExecute
sqllang!CMsqlExecContext::ExecuteStmts
sqllang!CMsqlExecContext::FExecute
sqllang!CSQLSource::Execute
sqllang!ExecuteSql
sqllang!CSpecProc::ExecuteSpecial
sqllang!CSpecProc::Execute
sqllang!process_request
sqllang!process_commands
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

This is on a 64-bit Windows Server 2008 R2 Standard Edition server, 32gb ram, running SQL Server 2012 SP1.

Microsoft SQL Server 2012 (SP1) – 11.0.3321.0 (X64)
Nov 14 2012 19:41:38
Copyright (c) Microsoft Corporation
Standard Edition (64-bit) on Windows NT 6.1 (Build 7601: Service Pack 1)

Any help would be much appreciated.

Thanks!

Reply
- Karthick P.K said
  
  February 15, 2013 at 4:07 AM
  Ewan,
  1. In SQLServer errolog immediately after this dump you will find message for “Approx CPU Used” can you share that line please. I am intrested in kernel and user time,
  2. Share your max server memory and min server memory config.
  3. Have you enabled LPIM for SQLServer startup account?
  
  Reply
  - Ewan Gibb said
    
    March 5, 2013 at 1:33 PM
    Karthik,
    
    Thank you so much for the prompt response. I haven’t noticed it until today as I lost the link to this page. We are still having the problem only now it is happening with a higher regularity.
    
    To answer your questions:
    
    1 – I hope this is the log entry you are after:
    
    Process 0:0:0 (0x1fd0) Worker 0x00000007839F6160 appears to be non-yielding on Scheduler 2. Thread creation time: 13006905344342. Approx Thread CPU Used: kernel 0 ms, user 70465 ms. Process Utilization 6%. System Idle 92%. Interval: 70465 ms.
    
    2 – Minimum server memory (in MB): 0
    Maximum server memory (in MB): 2147483647
    
    3 – LPIM is not enabled.
    
    Kind regards.
    
    Reply
    - Karthick P.K said
      
      March 5, 2013 at 2:20 PM
      How much RAM do we have on this system?
      
      Reply
      - Ewan Gibb said
        
        March 5, 2013 at 4:18 PM
        The server has 32gb RAM.
        
        Reply
        
        Karthick P.K said
        
        March 6, 2013 at 5:57 AM
        Cap the MaxServermemory of SQL Server after considering the memory requirements by OS, other applications drivers, Non-bpool etc. If it is dedicated SQL Server start with 26GB. Let us increase or decrease on need later.
        
        Reply
        
        Ewan Gibb said
        
        March 6, 2013 at 8:11 AM
        Thanks! It is a dedicated SQL Server, so I have set it to 26gb. I will see how it goes today and monitor it closely.
        
        Reply
        
        Ewan Gibb said
        
        March 6, 2013 at 4:04 PM
        Things looked fine for almost 8 hours, then the non yielding-scheduler message struck again. The stack trace is slightly different this time:
        
        sqldk!CVariableInfo::PbGetBlock
        sqldk!TVarPageMgr::PbAllocate
        sqldk!CMemObj::Alloc
        sqlmin!BTreeRowFactory::AllocateNewBTreeRow
        sqlmin!IndexDataSetSession::InsertSmallRecord
        sqlmin!IndexDataSetSession::InsertRowInternal
        sqlmin!DatasetSession::InsertRow
        sqlmin!RowsetNewSS::InsertRow
        sqlmin!CValRowMult::SetDataX
        sqltses!CEsExec::GeneralEval
        sqlmin!CQScanUpdateNew::GetRow
        sqlmin!CQScanNLJoinNew::GetRowHelper
        sqlmin!CQScanAssertNew::GetRow
        sqlmin!CQueryScan::GetRow
        sqllang!CXStmtQuery::ErsqExecuteQuery
        sqllang!CXStmtDML::XretDMLExecute
        sqllang!CXStmtDML::XretExecute
        sqllang!CMsqlExecContext::ExecuteStmts
        sqllang!CMsqlExecContext::FExecute
        sqllang!CSQLSource::Execute
        sqllang!ExecuteSql
        sqllang!CSpecProc::ExecuteSpecial
        sqllang!CSpecProc::Execute
        sqllang!process_request
        sqllang!process_commands
        sqldk!SOS_Task::Param::Execute
        sqldk!SOS_Scheduler::RunTask
        sqldk!SOS_Scheduler::ProcessTasks
        sqldk!SchedulerManager::WorkerEntryPoint
        sqldk!SystemThread::RunWorker
        sqldk!SystemThreadDispatcher::ProcessWorker
        sqldk!SchedulerManager::ThreadEntryPoint
        kernel32!BaseThreadInitThunk
        ntdll!RtlUserThreadStart
        
        The entry in the log this time reads:
        
        “Process 0:0:0 (0x1d80) Worker 0x00000004BBBEA160 appears to be non-yielding on Scheduler 19. Thread creation time: 13007047217492. Approx Thread CPU Used: kernel 0 ms, user 71339 ms. Process Utilization 7%. System Idle 92%. Interval: 71338 ms.”
        
        Entries such as the above are logged at regular intervals until the SQL Server service is restarted.
        
        In addition there are multiple examples of this:
        
        “A time-out occurred while waiting for buffer latch — type 2, bp 000000002D049000, page 1:3704, stat 0xb, database id: 34, allocation unit Id: 20700475686912/302175452397568, task 0x00000008EE12B498 : 0, waittime 300 seconds, flags 0x3a, owning task 0x0000000482C42558. Not continuing to wait.”
        
        If I run DBCC Opentran against all databases, the same database always has an open transaction. Using DBCC INPUTBUFFER(xxx) I can see that this is also the same each time.
        
        Are there any other diagnostics I can perform?
        
        The server was updated to SQL Server 2012 SP1 CU2 earlier in the week to see if that would help.
        
        Reply
        
        Karthick P.K said
        
        March 7, 2013 at 5:59 AM
        1. Again there is delay in memory allocation. ADD LPIM privilege to startup account of SQL Server. SQL Server uses different mechanism to allocate memory when we have LPIM, most probably it should resolve your issue.
        2. Did Available memory on the system dropped very low when you got this dump? If so you may have to further reduce max server memory.
        If issue doesn’t get resolve after enabling LPIM for startup account of SQL Server and restarting , share the dump and error log using your FTP share.I will review it for you. If there are security restrictions you may have to open a case with Microsoft support and can share the case number with me.
        
        For the latch time out dump ““A time-out occurred while waiting for buffer latch” check if you have I/O request taking longer than %d seconds error message in SQL Error log? Follow the steps in http://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/ once if you get the dump.
        
        Reply
        
        Ewan Gibb said
        
        March 7, 2013 at 9:05 AM
        I’ve granted LPIM for the SQL Server service account. Memory levels were fine all day yesterday. Didn’t breach 27gb in total. SQL Server sat at around 25gb.
        
        I will continue to monitor.
        
        Many thanks!
        
        Karthick P.K said
        
        March 7, 2013 at 9:51 AM
        Yep , LPIM should solve it. If by any chance you get dump again share the dump.
        
        Ewan Gibb said
        
        March 12, 2013 at 1:34 PM
        The configuration was set at the end of last week; I’ve been monitoring the server for 2 working days now, and the error has not arisen again. I will continue to monitor.
        
        Thanks for your help Karthick!
        
        Karthick P.K said
        
        March 12, 2013 at 4:17 PM
        Thanks Ewan😊 . Continue monitoring and and let me know how it goes. It can help other SQL users who read your comment. Sent from my Windows Phone ________________________________
        
        Karthick P.K said
        
        June 1, 2013 at 2:15 AM
        Hello Ewan,
        Fix for our issue is released today. http://support.microsoft.com/kb/2846378 has details of our issue and you can apply CU4 from http://support.microsoft.com/kb/2833645
        to fix it.
Bian said

February 20, 2013 at 6:39 PM
Hi Karthik,

Can you also help me on this stack?

ntdll!ZwClose
KERNELBASE!CloseHandle
kernel32!CloseHandleImplementation
sqlservr!CTraceFileIoProvider::OpenNextFile
sqlservr!CTraceWriteRequest::InitForFileTrace
sqlservr!CTraceFileIoProvider::InitializeWriteRequest
sqlservr!CTrace::WriteRecord
sqlservr!CTraceController::ProduceRecord
sqlservr!CTraceDDL::TraceDDL
sqlservr!CStatsDef::CreateStats
sqlservr!CStmtCreateStats::XretExecute
sqlservr!CXStmtDDL::XretExecute
sqlservr!CMsqlExecContext::ExecuteStmts
sqlservr!CMsqlExecContext::FExecute
sqlservr!CSQLSource::Execute
sqlservr!CSQLSource::SeExecute
sqlservr!CExecAutoCreateStats::FExecute
sqlservr!COptContext::FInstantiateGuessedStats
sqlservr!COptContext::PexprSimplify
sqlservr!COptContext::PcxteOptimizeQuery
sqlservr!CQuery::Optimize
sqlservr!CQuery::PqoBuild
sqlservr!CStmtQuery::InitQuery
sqlservr!CStmtDML::InitNormal
sqlservr!CStmtDML::Init
sqlservr!CCompPlan::FCompileStep
sqlservr!CSQLSource::FCompile
sqlservr!CSQLSource::FCompWrapper
sqlservr!CSQLSource::Transform
sqlservr!CSQLSource::Execute
sqlservr!process_request
sqlservr!process_commands
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex

Thanks much!

Reply
- Bian said
  
  February 20, 2013 at 7:34 PM
  Sorry Karthik. The actual stack is this. I followed your instruction and got this;
  
  ntdll!ZwClose
  KERNELBASE!CloseHandle
  kernel32!CloseHandleImplementation
  sqlservr!CTraceFileIoProvider::OpenNextFile
  sqlservr!CTraceWriteRequest::InitForFileTrace
  sqlservr!CTraceFileIoProvider::InitializeWriteRequest
  sqlservr!CTrace::WriteRecord
  sqlservr!CTraceController::ProduceRecord
  sqlservr!CTraceDDL::CommitOrUndo
  sqlservr!XdesRMReadWrite::CommitAllVirtualLogRecs
  sqlservr!XdesRMReadWrite::PreDestruct
  sqlservr!XdesRMFull::~XdesRMFull
  sqlservr!XdesRMFull::`scalar deleting destructor’
  sqlservr!XactRM::SinglePhaseCommit
  sqlservr!XactRM::CommitInternal
  sqlservr!FullXactImp::Commit
  sqlservr!CMsqlXactInternalBase::CommitInternal
  sqlservr!CMsqlXactInternalReadWrite::Commit
  sqlservr!CMsqlXactImp::Commit
  sqlservr!CAutoMsqlXact::CommitNestedXact
  sqlservr!COptContext::FInstantiateGuessedStats
  sqlservr!COptContext::PexprSimplify
  sqlservr!COptContext::PcxteOptimizeQuery
  sqlservr!CQuery::Optimize
  sqlservr!CQuery::PqoBuild
  sqlservr!CStmtQuery::InitQuery
  sqlservr!CStmtDML::InitNormal
  sqlservr!CStmtDML::Init
  sqlservr!CCompPlan::FCompileStep
  sqlservr!CSQLSource::FCompile
  sqlservr!CSQLSource::FCompWrapper
  sqlservr!CSQLSource::Transform
  sqlservr!CSQLSource::Execute
  sqlservr!process_request
  sqlservr!process_commands
  sqlservr!SOS_Task::Param::Execute
  sqlservr!SOS_Scheduler::RunTask
  sqlservr!SOS_Scheduler::ProcessTasks
  sqlservr!SchedulerManager::WorkerEntryPoint
  sqlservr!SystemThread::RunWorker
  sqlservr!SystemThreadDispatcher::ProcessWorker
  *** WARNING: Unable to verify timestamp for msvcr80.dll
  sqlservr!SchedulerManager::ThreadEntryPoint
  msvcr80!_callthreadstartex
  
  thanks!!
  
  Reply
  - Karthick P.K said
    
    February 21, 2013 at 3:00 PM
    sqlservr!CTraceFileIoProvider::OpenNextFile you are running a profiler trace and it looks like you have set the maximum file size , So SQL engine is trying to create a new file.
    Have you configured to capture profiler trace on local drive or network share? If local system check the speed of destination drive. If remote system (Network share) change it to local drive.
    
    Reply
    - bian said
      
      February 21, 2013 at 3:13 PM
      ah, that makes sense. we’ll change the destination to a local drive and see if that helps.
      
      I really appreicate for your help but here’s one more. can you also help me on this as well?
      
      Child-SP RetAddr Call Site
      00000081`683fd0e8 000007fe`fda84bf6 ntdll!NtCreateFile+0xa
      00000081`683fd0f0 00000000`76c718ed KERNELBASE!CreateFileW+0x2cd
      *** ERROR: Symbol file could not be found. Defaulted to export symbols for sqlservr.exe –
      00000081`683fd250 00000000`01f72617 kernel32!CreateFileWImplementation+0x7d
      00000081`683fd2b0 00000000`01f7513f sqlservr!GetIUMSForMsxml+0x8055f3
      00000081`683fd340 00000000`00c53aa0 sqlservr!GetIUMSForMsxml+0x80811b
      00000081`683fd390 00000000`00c54a5a sqlservr+0xbd3aa0
      00000081`683fd3e0 00000000`00c5547b sqlservr+0xbd4a5a
      00000081`683fd420 00000000`00c550e1 sqlservr+0xbd547b
      00000081`683fd4f0 00000000`01989cc5 sqlservr+0xbd50e1
      00000081`683fd580 00000000`000c67a4 sqlservr!GetIUMSForMsxml+0x21cca1
      00000081`683fd5d0 00000000`001027b4 sqlservr+0x467a4
      00000081`683fd600 00000000`00146525 sqlservr+0x827b4
      00000081`683fd660 00000000`001460db sqlservr+0xc6525
      00000081`683fd690 00000000`00101e05 sqlservr+0xc60db
      00000081`683fd730 00000000`001021d8 sqlservr+0x81e05
      00000081`683fd760 00000000`0014cef5 sqlservr+0x821d8
      00000081`683fd8f0 00000000`015b7c97 sqlservr+0xccef5
      00000081`683fd990 00000000`015b7f0b sqlservr!_____SQL______Process______Available+0x713d47
      00000081`683fda80 00000000`01527c45 sqlservr!_____SQL______Process______Available+0x713fbb
      00000081`683fdab0 00000000`006eacba sqlservr!_____SQL______Process______Available+0x683cf5
      00000081`683fea60 00000000`000ec3bc sqlservr+0x66acba
      00000081`683febe0 00000000`000ed3ab sqlservr+0x6c3bc
      00000081`683fed10 00000000`0012a4c2 sqlservr+0x6d3ab
      00000081`683ff370 00000000`0008f280 sqlservr+0xaa4c2
      00000081`683ff580 00000000`0008ee3e sqlservr+0xf280
      00000081`683ff690 00000000`0008eb3b sqlservr+0xee3e
      00000081`683ff6f0 00000000`001b5caa sqlservr+0xeb3b
      00000081`683ff760 00000000`001b5e25 sqlservr+0x135caa
      00000081`683ff840 00000000`001b567d sqlservr+0x135e25
      00000081`683ff880 00000000`001b6586 sqlservr+0x13567d
      00000081`683ff930 00000000`74b137d7 sqlservr+0×136586
      00000081`683ff9d0 00000000`74b13894 msvcr80!_callthreadstartex+0×17
      00000081`683ffa00 00000000`76c7652d msvcr80!_threadstartex+0×84
      00000081`683ffa30 00000000`7736c521 kernel32!BaseThreadInitThunk+0xd
      00000081`683ffa60 00000000`00000000 ntdll!RtlUserThreadStart+0×1
      
      Again, thank you for your help.
      
      Reply
      - bian said
        
        February 22, 2013 at 11:42 AM
        Also, is there a way to get sqlservr.pdb for SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64)?
        
        Reply
      - Karthick P.K said
        
        February 22, 2013 at 1:33 PM
        Hello Bian,
        Second stack which you posted us Access violation dump or Non-Yielding scheduler dump?
        
        Reply
        
        Brian Kang said
        
        February 22, 2013 at 2:52 PM
        that is Non-Yielding scheduler dump as well.
        
        thanks!
        
        Reply
        
        Karthick P.K said
        
        February 24, 2013 at 6:28 AM
        Hello Brian, Do you have any antivirus which traces CreateFile and cause delay in file creation. Both the stacks are pointing that the thread are waiting on CreateFileWImplementation. I couldnt find the cause for second dump from stack because symbols are not aligned. Please share the memory dump and errorlog file. I will look in to it. Thanks > Date: Fri, 22 Feb 2013 14:52:45 +0000 > To: karthick_pk83@hotmail.com >
        
        Reply
        
        Karthick P.K said
        
        February 25, 2013 at 4:02 AM
        Hello Brian,
        
        I doubt if I can locate the public symbols for your build any time soon. So I would suggest to share the dump (As long as there is no security constraint).
        
        Thanks
        Karthick
        
        ——————————————————————————–
        
        Subject: Re: [MSSQLWIKI] Comment: “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”
        Date: Sun, 24 Feb 2013 11:40:37 -0800
        
        Hi Karthick,
        
        Thanks a lot for your help on this, first of all.
        
        Well, for the second dump, you’re right. We don’t have a correct symbols for it, which our server is Microsoft SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64) running on 2-node clustering. The size of dump file is about 3 GB (original size is 33 GB so I zipped it and split them into 4 files). This is because we had to enable filtered dump since the mini dump with default setting didn’t contain anything.
        
        If you have a shared folder that I can drop the dump files with you. However, if you can share the correct symbols for our server version with me, I would analyzed it and then share the output with you. I’m sure this will save lots of your time and effort. Is this something possible?
        
        thanks again,
        
        Brian
        
        Reply
        
        Brian Kang said
        
        February 25, 2013 at 6:09 PM
        ah, unfortunately, we can’t share the dump/log to the public. what do you think, is there any other way? thanks,Brian Date: Mon, 25 Feb 2013 04:02:43 +0000 To: bsk64@hotmail.com
        
        Reply
        
        Karthick P.K said
        
        February 26, 2013 at 1:07 AM
        I think the only option u have is contact “Microsoft SQL Server support”
        
        Brian Kang said
        
        February 26, 2013 at 11:56 AM
        yeah, I guess that’s the only option for us.
        
        thanks for your help and suggestion, we’ll open up a case for this.
        
        Brian Kang said
        
        February 24, 2013 at 7:42 PM
        Hi Karthick,
        
        Thanks a lot for your help on this, first of all.
        
        Well, for the second dump, you’re right. We don’t have a correct symbols for it, which our server is Microsoft SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64) running on 2-node clustering. The size of dump file is about 3 GB (original size is 33 GB so I zipped it and split them into 4 files). This is because we had to enable filtered dump since the mini dump with default setting didn’t contain anything.
        
        If you have a shared folder that I can drop the dump files with you. However, if you can share the correct symbols for our server version with me, I would analyzed it and then share the output with you. I’m sure this will save lots of your time and effort. Is this something possible?
        
        thanks again, Brian
        
        From: karthick pk Sent: Saturday, February 23, 2013 10:28 PM To: comment+rw812t0753pis_xm_l4uxebl0@comment.wordpress.com ; bsk64@hotmail.com Subject: RE: [MSSQLWIKI] Comment: “Non-yielding IOCP Listener, Non-yielding Scheduler and non-yielding resource monitor known issues and fixes”
        
        Hello Brian, Do you have any antivirus which traces CreateFile and cause delay in file creation. Both the stacks are pointing that the thread are waiting on CreateFileWImplementation. I couldnt find the cause for second dump from stack because symbols are not aligned. Please share the memory dump and errorlog file. I will look in to it. Thanks
        
        > Date: Fri, 22 Feb 2013 14:52:45 +0000 > To: karthick_pk83@hotmail.com >
        
        Reply
    - Brian Kang said
      
      February 22, 2013 at 11:44 AM
      Thanks Karthick.
      
      I posted another question but wonder if you can help me to get sqlservr.pdb for SQL Server 2008 R2 (SP2) – 10.50.4270.0 (X64)?
      
      Thanks much!! -Brian
      
      Reply
MAdler said

March 1, 2013 at 6:19 PM
Hi- i hope you can help me – this is the stack i am getting (seems much shorter then the rest)
Sqlserver 2005 6bit latest SP (0.5)

0:021> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqlservr!SystemThread::Suspend
sqlservr!SOS_Scheduler::SuspendPreemptive
sqlservr!SOS_Event::Wait
sqlservr!alloca_probe
sqlservr!CMemThread::Alloc
sqlservr!operator new
sqlservr!SNI_Conn::InitObject
sqlservr!SNIAcceptDoneWithReturnCode
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
msvcr80!endthreadex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Brian Kang said
  
  March 4, 2013 at 5:01 AM
  Hi Karthick,
  
  No, we don’t see any “Non-yielding IOCP Listener” messages at all. just flood of these errors,
  
  2013-02-20 18:12:57.63 Server Process 0:0:0 (0x28ac) Worker 0x000000129E0781A0 appears to be non-yielding on Scheduler 35. Thread creation time: 13005864565060. Approx Thread CPU Used: kernel 0 ms, user 0 ms. Process Utilization 17%. System Idle 98%. Interval: 70358 ms. 2013-02-20 18:12:57.65 spid39s Database mirroring is inactive for database ‘XXXX’. This is an informational message only. No user action is required.
  
  we’re experiencing lots of cxpacket waits so this could be due to a parallelism issue?
  
  Thanks, Brian
  
  Reply
  - Karthick P.K said
    
    March 4, 2013 at 11:51 AM
    Hello Brian,
    
    Excessive parallelism can cause deadlocked scheduler but not non-Yielding scheduler.
    Remember SQL Server will generate dump only once when it detects non-yielding condition. Once the dump is generated warnings will be printed on error log for successive occurrence’s but dumps will not be generated. In our case it has detected the condition and printed the warning in SQL Server error log but has not generated the dump so ideally it would have generated dump in past.
    
    You can enabled tracefalg 1262 to generate dump on every non-Yield condition. But it may generate dumps continously and cause performance problem. So if you see the above error continoulsy
    
    1. Enable the traceflag
    2. Wait for first dump
    3. Disable the trace
    4. Paste the Non-yield stack.
    
    Did you change the min-server memory to default?
    Also let me know the verision of mscorwks which is loaded in SQLServer.
    
    Reply
- Karthick P.K said
  
  March 4, 2013 at 9:20 AM
  Hello Madler,
  
  Stack which you pasted is current or copied stack? From stack it looks like there is a memory pressure on the system. Can you share the SQL-Error log and output of two queries in http://mssqlwiki.com/2012/06/27/a-significant-part-of-sql-server-process-memory-has-been-paged-out/
  
  Reply
  - madler said
    
    March 4, 2013 at 4:01 PM
    It was the stack trace off of the memdump that sql created at the time of the error.
    We hade sql set to default max memory – we limited sql to maintain 4 GB for the operating system and the errors seem to have gone away – so i guess the issue you dexcribed in that link was indeed what was happending.
    
    here are the results of the queries:
    RunTime Notification_Time Notification_type MemoryUtilization % Node Id Process_Indicator System_Indicator type state reserved Effect type state reserved Effect type state reserved Effect SQL_ReservedMemory_KB SQL_CommittedMemory_KB SQL_AWEMemory SinglePagesMemory MultiplePagesMemory TotalPhysicalMemory_KB AvailablePhysicalMemory_KB TotalPageFile_KB AvailablePageFile_KB TotalVirtualAddressSpace_KB AvailableVirtualAddressSpace_KB Record Id Type Record Time Current Time
    2013-03-04 10:50:35.310 2013-03-01 10:55:45.683 RESOURCE_MEMPHYSICAL_HIGH 100 1 NULL NULL APPLY_LOWPM EFFECT_OFF 0 0 APPLY_HIGHPM EFFECT_ON 0 0 REVERT_HIGHPM EFFECT_OFF 0 0 33683712 118416 0 5056 14048 33541144 28767956 67080432 32551328 8589934464 8556024928 0 RING_BUFFER_RESOURCE_MONITOR 6984298 265873923
    and:
    
    runtime Notification_time ProcessUtilization SystemIdle % UserModeTime KernelModeTime PageFaults WorkingSetDelta MemoryUtilization (%workingset) Record Time Current Time
    2013-03-04 10:57:19.680 2013-03-04 06:41:27.380 0 99 0 5148033 68 -16 98 250922562 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:42:27.383 5 91 28392182 526971378 2339 36 98 250982565 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:43:27.387 0 92 10920070 37908243 68 0 98 251042568 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:44:27.390 6 88 27144174 622755992 2325 -8 98 251102572 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:45:27.393 0 98 7176046 47580305 153 20 98 251162575 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:46:27.397 4 94 25116161 442106834 2293 36 98 251222579 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:47:27.403 20 78 13104084 1945332470 584 -404 98 251282583 266274861
    2013-03-04 10:57:19.680 2013-03-04 06:48:27.417 12 86 28080180 1192783646 2221 -72 98 251342599 266274861
    
    Reply
Kile said

March 13, 2013 at 6:02 PM
Hi – I couldn’t find anything that matched the stack trace that I received. Any help is greatly appreciated.

Microsoft SQL Server 2012 – 11.0.2100.60 (X64)

0:525> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqldk!SOS_Scheduler::SwitchContext
sqldk!SOS_Scheduler::SuspendNonPreemptive
sqldk!EventInternal::Wait
sqldk!EventInternal::WaitAllowPrematureWakeup
sqlmin!CXPacketList::RemoveHead
sqlmin!CXPipe::ReceivePacket
sqlmin!CXTransLocal::ReceiveBuffers
sqlmin!CQScanExchangeNew::GetRowFromProducer
sqlmin!CQScanExchangeNew::GetRowHelper
sqlmin!CQScanRangePrefetchDelayNew::GetRow
sqlmin!CQScanNLJoinTrivialNew::GetRow
sqlmin!CQScanXProducerNew::GetRowHelper
sqlmin!CQScanXProducerNew::GetRow
sqlmin!FnProducerOpen
sqlmin!FnProducerThread
sqlmin!SubprocEntrypoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  March 14, 2013 at 4:03 AM
  Hello Kile,
  Is this copied stack . I mean did u do .cxr sqlmin!g_copiedStackInfo+0X20 and kc 100?
  
  Reply
Donny said

March 15, 2013 at 3:58 PM
Hi, I could not match my stack trace also. Thanks for any help/insight you may be able to provide!

This was in error log:
2013-03-14 12:45:51.480 Server Using ‘dbghelp.dll’ version ’4.0.5′
2013-03-14 12:45:51.630 Server **Dump thread – spid = 0, EC = 0×0000000000000000
2013-03-14 12:45:51.630 Server ***Stack Dump being sent to J:\MSSQL11.NAPSQL24\MSSQL\LOG\SQLDump0002.txt
2013-03-14 12:45:51.630 Server *
2013-03-14 12:45:51.630 Server * BEGIN STACK DUMP:
2013-03-14 12:45:51.630 Server * 03/14/13 12:45:51 spid 50876
2013-03-14 12:45:51.630 Server *
2013-03-14 12:45:51.630 Server * Non-yielding IOCP Listener
2013-03-14 12:45:51.630 Server *
2013-03-14 12:45:51.630 Server * Short Stack Dump
2013-03-14 12:45:51.690 Server Stack Signature for the dump is 0x00000000000003E1
2013-03-14 12:46:16.270 Server External dump process return code 0×20000001. External dump process returned no errors.
2013-03-14 12:46:16.270 Server IO Completion Listener (0x15ec0) Worker 0x00000080F9280160 appears to be non-yielding on Node 1. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15085.
2013-03-14 12:46:41.800 Server IO Completion Listener (0x15ec0) Worker 0x00000080F9280160 appears to be non-yielding on Node 1. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 50280.
2013-03-14 12:46:44.690 Server IO Completion Listener (0xdd8c) Worker 0x00000080FBF00160 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 40982.

This is the debug session:

0:042> .cxr sqlmin!g_copiedStackInfo+0X20
rax=0000000000000000 rbx=0000000000000000 rcx=00000079a773f498
rdx=00000000ce11f238 rsi=00000000ffffffff rdi=0000000000000b1c
rip=0000000077af135a rsp=00000000ce11ee40 rbp=00000000ce11f018
r8=00000080faf20040 r9=0000000000000000 r10=0000000000000000
r11=00000000ce11f1b8 r12=0000000000000000 r13=0000000000000000
r14=00000080fb024130 r15=00000000ffffffff
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
ntdll!ZwWaitForSingleObject+0xa:
00000000`77af135a c3 ret
0:042> kc 100
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqldk!SystemThread::Suspend
sqldk!SOS_Scheduler::SuspendPreemptive
sqldk!EventInternal::Wait
sqldk!SOS_UnfairMutexPair::LongWait
sqldk!SOS_UnfairMutexPair::AcquirePair
sqldk!CMemThread::Alloc
sqldk!SOS_Node::EnqueueTaskDirectInternal
sqllang!SOS_Node::EnqueueTaskDirect
sqllang!SNIAcceptDoneRouter
sqldk!SOS_Node::ListenOnIOCompletionPort
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
0:042> lmvm sqlservr
start end module name
00000000`ff080000 00000000`ff0be000 sqlservr (pdb symbols) c:\websymbols\sqlservr.pdb\5F808D3337994C2AB03A56D03A08B9192\sqlservr.pdb
Loaded symbol image file: sqlservr.exe
Mapped memory image file: c:\Websymbols\sqlservr.exe\4F36496E3e000\sqlservr.exe
Image path: C:\Program Files\Microsoft SQL Server\MSSQL11.NAPSQL24\MSSQL\Binn\sqlservr.exe
Image name: sqlservr.exe
Timestamp: Sat Feb 11 05:56:46 2012 (4F36496E)
CheckSum: 00034130
ImageSize: 0003E000
File version: 2011.110.2100.60
Product version: 11.0.2100.60
File flags: 0 (Mask 3F)
File OS: 40000 NT Base
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft SQL Server
InternalName: SQLSERVR
OriginalFilename: SQLSERVR.EXE
ProductVersion: 11.0.2100.60
FileVersion: 2011.0110.2100.060 ((SQL11_RTM).120210-1917 )
FileDescription: SQL Server Windows NT – 64 Bit
LegalCopyright: Microsoft Corp. All rights reserved.
LegalTrademarks: Microsoft SQL Server is a registered trademark of Microsoft Corporation.
Comments: SQL
0:042> !findstack sqlservr!SQL_SOSNonYieldSchedulerCallback
0:042> !findstack sqlservr!SQL_SOSNonYield
Thread 006, 1 frame(s) match
* 13 00000000861eea10 000007fedf76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0×184

Thread 042, 1 frame(s) match
* 07 00000000ce13ee70 000007fedf76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0xeb

0:042> ~[006]s
ntdll!ZwAllocateVirtualMemory+0xa:
00000000`77af149a c3 ret
0:006> kP
Child-SP RetAddr Call Site
00000000`861edb48 000007fe`fe261935 ntdll!ZwAllocateVirtualMemory+0xa
00000000`861edb50 000007fe`df6d7275 KERNELBASE!VirtualAlloc+0×45
00000000`861edb90 000007fe`df6ca47b sqldk!MemoryNode::VirtualAllocNativeNoCheck+0xa0
00000000`861edbf0 000007fe`df6ca1b4 sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality+0×256
00000000`861edd10 000007fe`df6c9e39 sqldk!SOS_MemoryBlockAllocator::AllocateBlock+0x5bd
00000000`861ede00 000007fe`df6ca816 sqldk!SOS_MemoryWorkSpace::AllocatePage+0×273
00000000`861edfb0 000007fe`df6cac61 sqldk!MemoryNode::AllocatePagesInternal+0×187
00000000`861ee040 000007fe`df6cae86 sqldk!MemoryClerkInternal::AllocatePages+0xb1
00000000`861ee370 000007fe`df6cae2d sqldk!CVarPageMgr::PviNewVarPage+0×36
00000000`861ee3e0 000007fe`df6c0287 sqldk!CVarPageMgr::PbAllocate+0x16f
00000000`861ee420 000007fe`d4c878ee sqldk!CMemObj::Alloc+0×47
00000000`861ee450 00000000`ff098349 sqllang!CErrorReportingManager::CwchFormatAndPrint+0×472
00000000`861ee910 00000000`ff093504 sqlservr!scierrlog+0×79
00000000`861ee9b0 000007fe`df76246e sqlservr!SQL_SOSNonYieldIOCPCallback+0×184
00000000`861eea20 000007fe`df704c8c sqldk!SOS_OS::ExecuteNonYieldIOCPCallbacks+0xde
00000000`861eecc0 000007fe`df6c8389 sqldk!SchedulerMonitor::CheckIoCompletionHangs+0x1d0
00000000`861eee00 000007fe`df79bf89 sqldk!SchedulerMonitor::Run+0×175
00000000`861eef30 000007fe`df6c3170 sqldk!SchedulerMonitor::EntryPoint+0×9
00000000`861eef60 000007fe`df6c2f64 sqldk!SOS_Task::Param::Execute+0x21e
00000000`861ef560 000007fe`df6c2d66 sqldk!SOS_Scheduler::RunTask+0xa8
0:006> kC
Call Site
ntdll!ZwAllocateVirtualMemory
KERNELBASE!VirtualAlloc
sqldk!MemoryNode::VirtualAllocNativeNoCheck
sqldk!SOS_MemoryBlockAllocator::CommitBlockAndCheckNumaLocality
sqldk!SOS_MemoryBlockAllocator::AllocateBlock
sqldk!SOS_MemoryWorkSpace::AllocatePage
sqldk!MemoryNode::AllocatePagesInternal
sqldk!MemoryClerkInternal::AllocatePages
sqldk!CVarPageMgr::PviNewVarPage
sqldk!CVarPageMgr::PbAllocate
sqldk!CMemObj::Alloc
sqllang!CErrorReportingManager::CwchFormatAndPrint
sqlservr!scierrlog
sqlservr!SQL_SOSNonYieldIOCPCallback
sqldk!SOS_OS::ExecuteNonYieldIOCPCallbacks
sqldk!SchedulerMonitor::CheckIoCompletionHangs
sqldk!SchedulerMonitor::Run
sqldk!SchedulerMonitor::EntryPoint
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
0:006> ~[042]s
ntdll!ZwWaitForSingleObject+0xa:
00000000`77af135a c3 ret
0:066> kP
Child-SP RetAddr Call Site
00000000`d501dc18 000007fe`fe2610dc ntdll!ZwWaitForSingleObject+0xa
00000000`d501dc20 000007fe`d5995f51 KERNELBASE!WaitForSingleObjectEx+0×79
00000000`d501dcc0 000007fe`d59964cf sqllang!FRunCommunicationsManager+0xf1
00000000`d501de90 000007fe`df6c3170 sqllang!CommunicationsManager+0x3bf
00000000`d501f0f0 000007fe`df6c2f64 sqldk!SOS_Task::Param::Execute+0x21e
00000000`d501f6f0 000007fe`df6c2d66 sqldk!SOS_Scheduler::RunTask+0xa8
00000000`d501f760 000007fe`df6de22f sqldk!SOS_Scheduler::ProcessTasks+0×299
00000000`d501f7e0 000007fe`df6de5c0 sqldk!SchedulerManager::WorkerEntryPoint+0×261
00000000`d501f880 000007fe`df6dd9cb sqldk!SystemThread::RunWorker+0x8f
00000000`d501f8b0 000007fe`df6de059 sqldk!SystemThreadDispatcher::ProcessWorker+0x3c8
00000000`d501f960 00000000`7799652d sqldk!SchedulerManager::ThreadEntryPoint+0×236
00000000`d501fa00 00000000`77acc521 kernel32!BaseThreadInitThunk+0xd
00000000`d501fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
0:066> kC
Call Site
ntdll!ZwWaitForSingleObject
KERNELBASE!WaitForSingleObjectEx
sqllang!FRunCommunicationsManager
sqllang!CommunicationsManager
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  March 16, 2013 at 7:12 AM
  Hello Donny,
  How much RAM do we have on this server and how frequently you see non-yielding error in errorlog ?
  
  Reply
  - Donny said
    
    March 18, 2013 at 9:49 PM
    Physical memory on server: 524211 MB
    
    SQL memory setting:
    name minimum maximum config_value run_value
    max server memory (MB) 128 2147483647 409600 409600
    
    SSIS also runs on this server.
    Thanks for having a look!
    
    Reply
  - Donny said
    
    March 18, 2013 at 9:50 PM
    That was the first time we saw non-yielding on that server.
    
    Reply
    - Karthick P.K said
      
      April 1, 2013 at 1:44 PM
      Hello Donny,
      I missed your comment. Please apply the fix in http://support.microsoft.com/kb/2819662
      (Upgrade to 11.0.3344.0 or higher) .Also let me know if you have Lock pages enabled for startup account of SQLServer.
      
      Reply
ccpjeff said

March 25, 2013 at 1:22 PM
Just a thank you for the great resource… currently troubleshooting some non-yielding errors…

It has hit us once every 2 weeks for the last few months and only caused crash dumps twice, and never causes SQL to die or anything like that. It typically results in multiple ‘client unable to reuse a session’ errors, and some errors in the front end application.

I used your info to dig through the dump, and found this:

mswsock!_GSHandlerCheck_SEH
mswsock!WSPCloseSocket
ws2_32!closesocket
sqlservr!Tcp::FCloseRefHandle
sqlservr!Tcp::Close

Which references several of your included dumps and KB articles. Checking those, it shows that this should of been fixed in SQL 2008 SP2 CU2. Our server is @ SP2 CU5 and most of our clients as well, but it’s possible a new client/service/application is not @ the correct level.

I’m verifying the native client versions on those new machines and hoping they are outdated! If so, this should be an easy fix!

but again, thanks for the great resource!

Cheers,

jeff

Reply
- Karthick P.K said
  
  March 25, 2013 at 1:48 PM
  Remember to apply the windows fix along with SQL Fix mentioned in http://support.microsoft.com/kb/2491214?wa=wsignin1.0 If there is difficulties applying the windows fix you can use the workaround mentioned in same Kb.
  
  Reply
KK said

April 29, 2013 at 11:44 AM
I got 2 SQL 2005 instances and they both gave non-yielding IOCP Listener error. The server has 16GB of physical memory but the both SQL instances are limited to Min of 2GB & Max of 4GB memory. Below the info from one of the instances dumpfile.

0:008> KC 100
Call Site
sqlservr!MemoryNode::AllocatePagesInternal
sqlservr!MemoryClerkInternal::AllocatePages
sqlservr!CVarPageMgr::PviNewVarPage
sqlservr!CVarPageMgr::PbAllocate
sqlservr!CMemObj::Alloc
sqlservr!operator new
sqlservr!operator new[]
sqlservr!CErrorReportingManager::SendErrorToErrLog
sqlservr!CErrorReportingManager::CwchFormatAndPrint
sqlservr!scierrlog
sqlservr!alloca_probe
sqlservr!Np::PrepareForNextAccept
sqlservr!SNIAcceptDoneWrapper
sqlservr!SNIAcceptDoneRouter
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
0×0
0×0
0×0
0×0
msvcr80!_threadstartex
msvcr80!_threadstartex
msvcr80!__initialmbcinfo
0×0

=====================================================================
BugCheck Dump
=====================================================================

This file is generated by Microsoft SQL Server
version 9.00.5000.00
upon detection of fatal unexpected error. Please return this file,
the query or program that produced the bugcheck, the database and
the error log, and any other pertinent information with a Service Request.

Computer type is AT/AT COMPATIBLE.
Current time is 20:25:03 04/29/13.
4 Unknown CPU 9., 2660 Mhz processor (s).
Windows NT 5.2 Build 3790 CSD Service Pack 2.

Memory
MemoryLoad = 98%
Total Physical = 16383 MB
Available Physical = 211 MB
Total Page File = 19864 MB
Available Page File = 3580 MB
Total Virtual = 8388607 MB
Available Virtual = 8371825 MB
**Dump thread – spid = 0, PSS = 0×0000000000000000, EC = 0×0000000000000000
***Stack Dump being sent to E:\MSSQL.1\MSSQL\LOG\SQLDump0008.txt
* *******************************************************************************
*
* BEGIN STACK DUMP:
* 04/29/13 20:26:07 spid 0
*
* Non-yielding IOCP Listener
*
* *******************************************************************************
* ——————————————————————————-
* Short Stack Dump

Can you advise if this is related to disk i/o performance or memory ?

Reply
- Karthick P.K said
  
  April 30, 2013 at 5:14 AM
  Hello KK,
  This is side effect of memory pressure on the box. What other applications are running in this server ? You have to control the memory usage by other applications (or) increase physical memory in the server and max server memory. Review your error logs and let me know if you see any of the memory errors mentioned in http://mssqlwiki.com/sqlwiki/sql-performance/troubleshooting-sql-server-memory/
  
  Reply
Danni said

May 7, 2013 at 1:24 PM
Hi Karthick,

thank you for that nice Tutorial.
I couldn’t find any stack that matches. Maybe you can help me..

Errorlog:
2013-04-30 18:30:23.83 Server **Dump thread – spid = 0, EC = 0×00000000
2013-04-30 18:30:23.83 Server ***Stack Dump being sent to c:\Programme\Microsoft SQL Server\MSSQL10_50.DESIGO\MSSQL\LOG\SQLDump0007.txt
2013-04-30 18:30:23.83 Server * *******************************************************************************
2013-04-30 18:30:23.83 Server *
2013-04-30 18:30:23.83 Server * BEGIN STACK DUMP:
2013-04-30 18:30:23.83 Server * 04/30/13 18:30:23 spid 3688
2013-04-30 18:30:23.83 Server *
2013-04-30 18:30:23.83 Server * Non-yielding IOCP Listener
2013-04-30 18:30:23.83 Server *
2013-04-30 18:30:23.83 Server * *******************************************************************************
2013-04-30 18:30:23.83 Server * ——————————————————————————-
2013-04-30 18:30:23.83 Server * Short Stack Dump
2013-04-30 18:30:23.84 Server Stack Signature for the dump is 0×00000180
2013-04-30 18:30:25.66 Server External dump process return code 0×20000001.
External dump process returned no errors.

2013-04-30 18:30:25.66 Server 2013-04-30 20:03:35.68 Server 2013-04-30 20:03:45.71 Server 2013-04-30 21:02:40.75 Server 2013-04-30 23:29:15.79 Server 2013-05-01 00:29:50.81 Server 2013-05-01 00:30:00.84 Server 2013-05-01 02:02:00.87 Server 2013-05-01 02:02:10.92 Server 2013-05-01 03:30:00.95 Server 2013-05-01 03:30:10.98 Server 2013-05-01 13:29:13.14 Server 2013-05-01 14:29:23.18 Server 2013-05-01 16:01:28.20 Server 2013-05-01 22:01:53.23 Server 2013-05-01 22:02:03.27 Server 2013-05-01 23:01:13.29 Server 2013-05-01 23:01:23.32 Server 2013-05-02 08:30:18.34 Server 2013-05-02 09:30:38.39 Server 2013-05-02 10:29:53.42 Server 2013-05-02 11:31:13.46 Server 2013-05-02 13:03:48.49 Server 2013-05-02 14:03:23.51 Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25046.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25046.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 25031.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.

Debugging:
0:004> .cxr sqlservr!g_copiedStackInfo+0X20
eax=03b6f8e0 ebx=7c94845c ecx=00000246 edx=0000001b esi=03b6f90c edi=03d8e988
eip=00000000 esp=00000000 ebp=00000023 iopl=0 nv up di pl nz na po nc
cs=0000 ss=0010 ds=b34e es=dc08 fs=0000 gs=0000 efl=00000000
00000000 ?? ???
0:004> kc 100

WARNING: Frame IP not in any known module. Following frames may be wrong.
0×0
0:004> kc 100

WARNING: Frame IP not in any known module. Following frames may be wrong.
0×0
0:004> .cxr
Resetting default scope
0:004> .cxr sqlservr!g_copiedStackInfo+0X00c
eax=03d8e988 ebx=00000000 ecx=0000b34e edx=7c94dc08 esi=00000000 edi=00bc8048
eip=7c94845c esp=03b6f8e0 ebp=03b6f90c iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
ntdll!KiFastSystemCallRet:
7c94845c c3 ret
0:004> kc 100

ntdll!KiFastSystemCallRet
ntdll!NtRemoveIoCompletion
kernel32!GetQueuedCompletionStatus
sqlservr!SOS_Node::ListenOnIOCompletionPort
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!_callthreadstartex
msvcr80!_threadstartex
kernel32!BaseThreadStart

Thank you for your Help

Reply
- Karthick P.K said
  
  May 7, 2013 at 3:12 PM
  Hello Danni,
  This is 32bit SQLServer or 64-Bit SQLServer?
  
  Reply
  - Danni said
    
    May 7, 2013 at 5:40 PM
    It’s 32-Bit
    
    Reply
    - Karthick P.K said
      
      May 7, 2013 at 11:41 PM
      Ok. In debugger rype thebelow commands andshare the output
      ~~[0xe84]s
      kc100
      
      Reply
- Karthick P.K said
  
  May 7, 2013 at 3:14 PM
  Also can you do ~~[0xe84]s and then kc100 and share the output
  
  Reply
  - Danni said
    
    May 8, 2013 at 6:04 AM
    Here it is:
    
    0:004> ~~[0xe84]s
    eax=053d3ff0 ebx=00000000 ecx=00000001 edx=00001fb0 esi=000002e8 edi=00000000
    eip=7c94845c esp=3361f16c ebp=3361f1e0 iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
    ntdll!KiFastSystemCallRet:
    7c94845c c3 ret
    0:013> kc 100
    
    ntdll!KiFastSystemCallRet
    ntdll!NtSignalAndWaitForSingleObject
    kernel32!SignalObjectAndWait
    sqlservr!SOS_Scheduler::Switch
    sqlservr!SOS_Scheduler::SwitchContext
    sqlservr!SOS_Scheduler::SuspendNonPreemptive
    sqlservr!SOS_Scheduler::Suspend
    sqlservr!EventInternal<Spinlock >::Wait
    sqlservr!ResQueueBase::Dequeue
    sqlservr!ResQueue::Dequeue
    sqlservr!systemTaskThread
    sqlservr!SOS_Task::Param::Execute
    sqlservr!SOS_Scheduler::RunTask
    sqlservr!SOS_Scheduler::ProcessTasks
    sqlservr!SchedulerManager::WorkerEntryPoint
    sqlservr!SystemThread::RunWorker
    sqlservr!SystemThreadDispatcher::ProcessWorker
    sqlservr!SchedulerManager::ThreadEntryPoint
    msvcr80!_callthreadstartex
    msvcr80!_threadstartex
    kernel32!BaseThreadStart
    
    Reply
    - Karthick P.K said
      
      May 10, 2013 at 6:45 AM
      Server IO Completion Listener (0xe84) Worker 0x00FE80D8 appears to be non-yielding on Node 0. Approx CPU Used: kernel 0 ms, user 0 ms, Interval: 15000.
      
      From the above warming non-yielding thread is not consuming CPU at all (kernel 0 ms, user 0 ms).
      
      I think garbage collector is suspending all the other threads.
      
      type ~*kc 100 and search for mscorwks!SVR::GCHeap
      
      If you find atleast one garbage collector thread apply Apply the .Net fix mentioned in http://support.microsoft.com/kb/2504603
      
      If you dont find any garbage collector thread let me know.
      
      Reply
Adam Angelini said

May 16, 2013 at 7:42 PM
Do you knwo what cause this one? SQL2012 sp1 enterprise x64 bit

Call Site
ntdll!ZwWaitForSingleObject
ntdll!RtlpWaitOnCriticalSection
ntdll!RtlEnterCriticalSection
ws2_32!closesocket
sqllang!Tcp::FCloseTransport
sqllang!SNI::detail::Transport::FCloseRefHandle
sqllang!Tcp::Close
sqllang!Smux::InternalClose
sqllang!Smux::ReadDone
sqllang!SNIReadDone
sqldk!SOS_Node::ListenOnIOCompletionPort
sqldk!SOS_Task::Param::Execute
sqldk!SOS_Scheduler::RunTask
sqldk!SOS_Scheduler::ProcessTasks
sqldk!SchedulerManager::WorkerEntryPoint
sqldk!SystemThread::RunWorker
sqldk!SystemThreadDispatcher::ProcessWorker
sqldk!SchedulerManager::ThreadEntryPoint
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart

Reply
- Karthick P.K said
  
  May 17, 2013 at 5:49 AM
  Hello Adam,
  
  which windows version are you using?
  
  You need to apply the windows fix mentioned in below KB. (windows fix in KB is applicabel for SQL2012 as well)
  
  http://support.microsoft.com/kb/2491214/en-US
  
  {
  Hotfix for Windows Server 2008 R2
  976932 Information about Service Pack 1 for Windows 7 and for Windows Server 2008 R2
  Hotfix for Windows Server 2008
  2398202 An application may stop responding in Windows Server 2008 or in Windows Vista when the application closes a socket connection or shuts down
  }
  
  Reply
tilak said

May 31, 2013 at 8:57 AM
Hi Karthick,

Can you please let me know the cause of this. SQL version is 10.0.2531

Call Site
ntdll!NtWriteFile
KERNELBASE!WriteFile
kernel32!WriteFileImplementation
sqlservr!DiskWriteAsync
sqlservr!CTraceIoBuf::IncrementalFlush
sqlservr!CTraceFileIoProvider::IncrementalFlush
sqlservr!CTraceController::IncrementalFlush
sqlservr!CTraceManagementTask::Invoke
sqlservr!CTraceBatchTask::Task
sqlservr!SOS_Task::Param::Execute
sqlservr!SOS_Scheduler::RunTask
sqlservr!SOS_Scheduler::ProcessTasks
sqlservr!SchedulerManager::WorkerEntryPoint
sqlservr!SystemThread::RunWorker
sqlservr!SystemThreadDispatcher::ProcessWorker
sqlservr!SchedulerManager::ThreadEntryPoint
msvcr80!endthreadex
0×0
0×0

Thanks,
Tilak M

Reply
- Karthick P.K said
  
  May 31, 2013 at 9:14 AM
  It is because of the profiler trace you are running.
  
  Reply

MSSQLWIKI

Karthick P.K on SQL Server

Karthick P.K

Categories

Archives

Tags

Facebook

Do you have a question in SQL Server or stuck in SQL Server issue? Click here to join our facebook group and post your questions to SQL Server experts

Email Subscription

SQLWiki

feedburner email subscription

Subscribe

Visits

SQL Server Blogs

Author

Other SQL blogs

SQLSERVERSCRIBBLES.COM