> ## Content Index > Fetch the complete content index at: https://devinside.cloud/llms.txt > Use this file to discover other available public pages before exploring further. # ASP.NET Waits with Low CPU: Find the Queue Before Changing the Pool - URL: https://devinside.cloud/asp-net-waits-with-low-cpu-find-the-queue-before-changing-the-pool/ - Published: 2025-11-20T11:30:00.000Z - Updated: 2026-11-12T17:08:26.000Z - Description: Eight controlled HTTP requests reveal three different waits. Actual stacks, pool counters and permit ownership explain why low CPU is not a diagnosis. - Author: Marcin Borecki - Tags: Engineering, .NET and performance, #eng15, #devinside-review START WITH THE MECHANISM Interactive companion and laboratory update: 12 November 2026\. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates. THE FULL INVESTIGATION ## ASP.NET Waits with Low CPU: Find the Queue Before Changing the Pool Editorial context: 20 November 2025 · Prepared and updated: 12 November 2026 Dates and technology context This is a retrospective editorial timeline. Topic context: 20 November 2025\. First publicly available: 12 November 2026\. Current text, visuals and laboratory revisions were prepared on 12 November 2026. The .NET/ASP.NET11 server, eight-request gate experiment, managed stack dump and numerical observations are 2026-11-12 additions. No .NET11 dump or measurement may be placed before GA. A service can be slow while its CPU graph looks reassuring. Requests remain unfinished, a dependency appears quiet and someone proposes increasing the ThreadPool minimum. Sometimes the thread pool is the relevant boundary. Sometimes the service is deliberately awaiting a result. Sometimes requests are waiting for admission to a scarce resource. Those situations can produce similar latency and very different repairs. The useful question is not whether async is generally faster than blocking. It is which resource currently owns progress. Is a worker thread parked inside a synchronous wait? Is an incomplete task allowing the thread to leave while the operation remains pending? Is a bounded resource gate holding later requests outside the dependency? A latency number alone cannot identify the owner. This article uses an executed ASP.NET Core laboratory to separate those mechanisms. Eight HTTP requests encounter one of three controlled boundaries: a synchronous event wait, an asynchronous task wait or an asynchronous semaphore with two permits. A controller releases the boundary only after the requests have arrived. The experiment records actual HTTP results, ThreadPool observations, resource-gate counters and a managed stack dump. The measurements were collected on 12 November 2026 using SDK 11.0.100 and .NET/ASP.NET Core 11.0.0\. This is a bounded mechanism demonstration on a shared machine, not a load test designed to exhaust it. All requests complete with checked results. The evidence does not establish production starvation, a database connection-pool defect or a service-level percentile, and the distinction matters to the conclusions. ## Define the same result before changing the wait Every request supplies an integer identifier from zero through seven. The endpoint must return that identifier and the value id times three plus one. The runner checks the complete set of eight response pairs, so returning the right number of responses with the wrong values would fail. The request body does almost no useful computation. Its purpose is to make the wait visible without adding a large CPU workload. All three variants implement the same result contract and wait for a controlled release. That lets the investigation focus on how progress is suspended rather than comparing unrelated business algorithms. The client submits eight requests concurrently from a separate Python process. It does not keep adding more traffic while they wait. This is a finite batch, not a constant-rate load generator. There is no claim about sustainable requests per second, saturation throughput or queue growth under an unbounded arrival stream. The server binds to a dynamically assigned loopback port. The runner owns the child process, knows its exact address and terminates it within a bounded protocol. It never attaches to another service or uses an existing database. Each variant starts in a fresh process so that its gate state and request counters are independent of the previous case. The host reports sixty-four logical processors, a minimum worker-thread setting of sixty-four and a maximum of 32,767\. The laboratory does not change those settings. Eight requests are intentionally too small to demonstrate general ThreadPool exhaustion on this host. That limitation is a feature of the controlled scope, not a reason to manufacture more load until a desired failure appears. The observed result can still be useful. A blocked thread can be identified without exhausting every thread. A semaphore queue can be identified without creating a production incident. Understanding those mechanisms gives a better starting point for a real investigation than assuming that every low-CPU delay is starvation. ## Three gates create three different ownership stories 01 / See the idea ### Similar latency can hide different owners ![Blocking occupies request threads, async suspends operations and the semaphore limits admission to two holders.](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v1.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v1.svg) All variants wait for the same external release. Their response contract stays fixed while ownership differs. Interpretation and evidence - Keep the response contract fixed. - Separate operation and thread lifetimes. - Admission is a separate queue. Conceptual explanation unless labelled measured; counts and bytes derived from saved lab evidence. The synchronous variant calls ManualResetEventSlim.Wait inside the request delegate. Until the controller sets the event, a worker thread remains in that wait. The request has entered the endpoint and has not completed its response. Its progress is tied to a physical thread parked in the synchronous waiting path. The asynchronous variant awaits an incomplete TaskCompletionSource task. The request operation remains unfinished, but the await allows its current executing thread to return rather than staying blocked until release. When the controller completes the task, the continuation can run and produce the response. The semaphore variant first awaits a permit from a SemaphoreSlim initialized with two permits. The first two requests acquire permits and then await the same incomplete task gate. The remaining six requests await admission. All eight operations are pending, but they do not all own the same resource: two hold permits and six are outside the permit boundary. The actual branch point in the executed endpoint is small: ```csharp if(mode=="blocking")syncGate.Wait(stop.Token); else if(mode=="async")await asyncGate.Task.WaitAsync(stop.Token); ``` The semaphore branch makes its ownership explicit: ```csharp Interlocked.Increment(ref waiting); try{await permits.WaitAsync(stop.Token);}finally{Interlocked.Decrement(ref waiting);} acquired=Stopwatch.GetElapsedTime(arrived).TotalMilliseconds;Interlocked.Increment(ref holders); try{await asyncGate.Task.WaitAsync(stop.Token);}finally{Interlocked.Decrement(ref holders);permits.Release();} ``` This is a resource-gate model. It is not an implementation of SqlClient's connection pool, and the experiment does not open database connections. The shape is useful because it isolates the distinction between waiting for admission and waiting while holding a resource. Translating the diagnosis to a database requires the actual provider's pool and connection evidence. An asynchronous semaphore wait does not eliminate scarcity. It changes how a caller waits for access. If two resources remain occupied, later operations still cannot proceed merely because they are awaiting efficiently. This is why replacing a blocking wait with await can improve thread usage without making the dependency itself faster or increasing its capacity. [Runtime SemaphoreSlim source, v11.0.0](https://github.com/dotnet/runtime/blob/v11.0.0/src/libraries/System.Private.CoreLib/src/System/Threading/SemaphoreSlim.cs?ref=devinside.cloud) ## Let an independent controller release the experiment The server uses a small dedicated controller thread. It waits for all eight requests to enter, writes a readiness record and then waits for the runner's release command. This thread is part of the laboratory protocol, not a proposed production architecture. Using an independent controller prevents the release mechanism from depending on the same worker availability being investigated. If the test scheduled its only release action through a pool whose threads were all deliberately blocked, it could accidentally turn a bounded observation into a deadlock or a misleading scheduler test. A second dedicated thread samples runtime and application state approximately every ten milliseconds. It records ThreadPool.ThreadCount, PendingWorkItemCount, CompletedWorkItemCount and available worker capacity, alongside entered and finished requests, semaphore holders and semaphore waiters. It also records elapsed wall time and process CPU time. Sampling intervals are observations, not a promise of exact periodic execution. Thread scheduling and the operating system can delay a sample. The report preserves the actual timestamps, so the analysis uses the recorded interval rather than assuming that twenty samples always represent exactly two hundred milliseconds. The finite protocol also has an escape route. The service uses a timeout, the controller eventually releases its gates, and the wrapper writes the release command in finally before terminating an unresponsive owned child. On normal completion, the sampler and controller are joined, the HTTP server stops and the semaphore returns to two available permits. These cleanup checks belong to correctness. A demonstration that leaves a sampler thread or held permit behind is not complete merely because it produced a chart. The recorded results confirm eight finished requests, restored permits and closed controller and sampler threads in every case. ## Use the stack to identify the blocking boundary 02 / A concrete example ### The stack identifies the blocking location ![Eight handler stacks contain the synchronous gate while other runtime waits are explicitly excluded from that count.](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v2.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v2.svg) The request source location distinguishes application blocking from expected infrastructure waits. Dump collection extends the held interval, so its latency is not compared with no-dump runs. Interpretation and evidence - Use stack context. - Do not diagnose by keyword count. - Separate diagnostics from timing. Conceptual explanation unless labelled measured; counts and bytes derived from saved lab evidence. The functional diagnostic run captures a mini dump while the synchronous gate is held. The analyzer runs clrstack for all managed threads and records ThreadPool information. The raw stack report is retained with the dump rather than replaced by a hand-drawn call sequence. Eight request stacks pass through ManualResetEventSlim.Wait and the request delegate at the line containing the synchronous gate. This is direct evidence of the controlled blocking mechanism. Other threads also contain waits, including runtime infrastructure and the application's entry-point wait, so counting every occurrence of the word Wait would overstate the number of blocked request handlers. The source-location context is what makes the distinction useful. A stack containing a wait is not automatically defective. A runtime I/O poller waits as part of its job. A controller waiting for a test command is expected. The relevant observation is the request handler occupying a worker thread while waiting on the deliberately closed gate. The dump adds diagnostic cost. In this run, the blocking case stayed held for much longer while collection and analysis completed. Its functional-run response times therefore are not comparable with the other functional cases. The separate timing series omits dump collection entirely. This separation prevents an easy mistake: a chart could show the diagnostic blocking run taking roughly nine seconds and the other variants taking a fraction of a second, then attribute the difference to blocking. Much of that difference would be the investigation itself. The article uses the dump to identify the stack and uses the independent no-dump series for bounded timing observations. Microsoft's starvation guidance combines runtime counters with stack or trace evidence rather than treating low CPU alone as proof. The same discipline applies here, although this laboratory deliberately stops short of creating starvation. [Microsoft: debugging ThreadPool starvation](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/debug-threadpool-starvation?ref=devinside.cloud) ## A small ThreadPool queue does not mean no one is waiting 03 / Watch the mechanism ### An asynchronous pool can still be full ![Two permits are held while six operations wait asynchronously for admission.](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v3.svg) An asynchronous pool can still be fullMeasured holder/waiter counts; conceptual movement. Semaphore model only; no actual SQL connection pool is exercisedAn asynchronous pool can still be fullMeasured holder/waiter counts; conceptual movement8 arrivalssame HTTP contract2 permits2 holders wait6 waitersoutside admissionRelease dependency gate → holders return permitsSemaphore model only; no actual SQL connection pool is exercised Play animation[Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v3.svg) Static view. Play to see how the mechanism unfolds. The moving marker illustrates resumed progress after release. The observed ThreadPool queue can stay small while this application resource queue is real. Interpretation and evidence - Await does not create capacity. - Measure acquisition and hold time. - Model evidence is not provider evidence. Conceptual explanation unless labelled measured; counts and bytes derived from saved lab evidence. In the no-dump measurements, the maximum sampled PendingWorkItemCount is between zero and three across the nine processes. At the same time, the application has eight entered and unfinished requests during the held interval. Those observations are compatible because a pending ThreadPool work item is not the same thing as every unfinished application operation. A request waiting asynchronously on an incomplete task is represented by asynchronous state and a future continuation. It need not occupy a work item that is currently ready for a worker to execute. A semaphore waiter can likewise remain outside the ready-to-run pool queue while waiting for a permit. The sampled queue count is also a momentary observation. Short-lived items can appear and disappear between samples. A low maximum in this finite test does not prove that the queue was always empty, nor does it establish the behavior of a sustained production workload. It tells us what the chosen sampler actually observed. The tagged implementation sums tracked local and global work queues. [Runtime ThreadPool queue source, v11.0.0](https://github.com/dotnet/runtime/blob/v11.0.0/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs?ref=devinside.cloud) ThreadPool thread count is another distinct dimension. The recorded maxima range from seventeen to twenty-one. This is the number of pool threads reported during the process lifecycle, not the number of threads actively executing useful request code and not a count of the eight requests themselves. The available-thread value is easy to misread too. It is capacity relative to the pool's configured limits, not a guarantee that a newly arriving request will immediately receive a newly created thread. In this host configuration the reported maximum is large, but that number should not be interpreted as a preallocated inventory of idle operating-system threads. [Microsoft: GetAvailableThreads](https://learn.microsoft.com/en-us/dotnet/api/system.threading.threadpool.getavailablethreads?ref=devinside.cloud) These distinctions explain why an application-level gate counter is valuable. In the semaphore case, the explicit counters show two holders and six waiters. The ThreadPool queue does not have to grow for the resource queue to be real. If a dashboard exposes only pool metrics, it can miss the boundary controlling progress. ## Read low CPU as a clue, not a diagnosis The measured held intervals last roughly 194 to 210 milliseconds. Most report no increase in the process CPU-time counter across the selected samples; one async interval reports 15.625 milliseconds. These are coarse observations from a short window, not precise estimates of CPU utilization to several decimal places. The sampler, runtime housekeeping and any remaining application work can contribute to process CPU time. The counter does not isolate one request or one gate. A zero delta at this resolution does not prove that absolutely no instructions executed. It is consistent with a waiting-dominated interval and insufficient for a fine-grained CPU cost comparison. The important point is causal. Both the synchronous wait and asynchronous wait can show low CPU while requests remain unfinished. Low CPU does not tell us whether threads are being occupied. The stack shows the blocking request path; the async code and gate state show a different waiting mechanism. Likewise, the semaphore case can be quiet because the permit holders are waiting on the controlled dependency gate. The six remaining operations cannot enter, even though the machine has available CPU. Increasing processor capacity would not open that gate. Increasing the semaphore limit would change the admission policy, but it would not make the dependency complete. For a production service, low CPU narrows the investigation only when combined with other facts. Are requests queued before the handler? Are threads blocked inside a synchronous library? Are connections checked out and not returned? Are remote dependencies responding slowly? Is a cancellation path waiting for work that never observes its token? Each explanation requires evidence at its own boundary. This laboratory provides three concrete examples of those distinctions without claiming to cover every cause. It does not model network packet loss, DNS stalls, database locks or connection establishment. Naming those exclusions helps prevent a small correct experiment from becoming an overbroad troubleshooting rule. ## Compare the no-dump observations with their protocol attached 04 / A concrete example ### Three no-dump observations per variant ![The median handler durations for each of three processes are shown separately for all variants.](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v4.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v4.svg) These similar durations follow the shared gate protocol. They do not rank the capacity of blocking and asynchronous services under sustained load. Interpretation and evidence - Report the gate duration. - Keep process observations visible. - Do not manufacture tail percentiles. Conceptual explanation unless labelled measured; counts and bytes derived from saved lab evidence. The timing series contains three fresh processes per variant, with the order reversed in alternating rounds. Each process handles eight requests. The runner waits for the held-ready signal, waits another two hundred milliseconds and releases the gate. All seventy-two HTTP responses match the expected identifiers and values. | Variant | Per-process median handler times, ms | Peak permit holders | Peak permit waiters | | ----------------- | ------------------------------------ | ------------------- | ------------------- | | Synchronous gate | 220.46, 219.95, 208.72 | 0 | 0 | | Asynchronous gate | 205.90, 219.30, 215.95 | 0 | 0 | | Two-permit gate | 220.26, 219.15, 217.81 | 2 | 6 | The similar handler durations are expected: the external protocol deliberately keeps all variants waiting for approximately the same release interval. The experiment is not designed to make await finish before an unreleased gate. Its value is that different resource ownership can exist behind similar elapsed times. The client-side duration includes additional HTTP and lifecycle costs. In the first synchronous process, the median client observation is 578 milliseconds, while the median handler duration is 220.46 milliseconds. Other client medians are around 297 to 313 milliseconds. Those boundaries should not be collapsed or used to claim a stable synchronous penalty from three short processes. The semaphore's maximum observed permit wait is approximately 218 to 220 milliseconds in the three runs. That aligns with six requests waiting outside the two occupied permits until release allows the holders to finish. It is a measured resource-admission delay in this model, not a database connection acquisition measurement. The timing table is intentionally not labelled p99\. Eight requests in a process and three processes per variant do not support a production tail-latency claim. The full request observations remain in the evidence pack, and the table reports its aggregation explicitly rather than giving a statistical label that suggests a larger study. There is also no inferred throughput capacity. The runner submits a finite batch, holds it and then releases it. It does not measure an open-loop arrival rate or a sustained equilibrium. Treating the batch completion time as the service's maximum throughput would assign business meaning to an artificial control interval. ## The right repair changes the owner of the wait 05 / See the idea ### Find the queue that owns progress ![Ready ThreadPool work, suspended operations and resource waiters are separate diagnostic categories.](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v5.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG15-v5.svg) The repair depends on the responsible boundary. Increasing one pool may leave the true wait unchanged or move pressure downstream. Interpretation and evidence - Name the blocking resource. - Pair counters with causal evidence. - Change one mechanism at a time. Conceptual explanation unless labelled measured; counts and bytes derived from saved lab evidence. If a real request blocks on work that has an asynchronous API, awaiting that API can release the worker while preserving the operation. This addresses thread occupation. It does not guarantee shorter dependency latency or eliminate all allocations. The expected benefit must be evaluated at the resource boundary actually affected. If the bottleneck is a bounded pool, the next question is why its leases remain occupied. Perhaps operations hold a connection across unrelated work, perhaps the dependency is slow, or perhaps the configured limit intentionally protects a scarce backend. Increasing the limit without understanding those conditions can move the bottleneck downstream or increase contention. If the application is correctly awaiting a genuinely slow dependency, thread-pool tuning may be irrelevant. The useful repair could involve reducing dependency work, changing request shape, adding a deadline or limiting concurrency. The async wait is then a symptom of where progress depends on another component, not itself a defect. The controlled variants are therefore diagnostic alternatives, not a leaderboard. The synchronous gate demonstrates occupied request threads. The asynchronous gate demonstrates pending operations without that synchronous wait. The semaphore demonstrates an admission queue separate from ThreadPool readiness. A real repair chooses among those explanations using evidence, not a preference for one API name. Cancellation requires the same care. The fixture has a process timeout and release paths so that its owned resources close. It does not claim to test client-disconnect cancellation, a database driver's timeout semantics or every race between cancellation and permit acquisition. A production change must preserve the correct release behavior on those paths. The finally block around the semaphore is particularly important. A successful acquisition creates an obligation to release the permit. Forgetting that release could make later requests wait indefinitely while CPU stays low. That would be a resource ownership defect, even if every wait was written with await and the ThreadPool looked healthy. ## Do not convert a semaphore model into a database finding A connection pool has more behavior than a semaphore. It creates or reuses connections, handles failures and timeouts, applies provider-specific grouping rules and interacts with the database's own capacity. This laboratory exercises none of those provider paths. Its two permits represent scarce admission rights so that ownership can be observed independently of a real backend. That limited model is useful if it produces the right next question. In a database incident, record when the application begins acquiring a connection, when acquisition succeeds, how long the connection remains checked out and when it is disposed. Compare that interval with the command's execution time. A long endpoint does not establish that the SQL statement itself was slow. If acquisition is the delay, inspect actual pool configuration and the lifetime of existing leases. If command execution is the delay, inspect the database work and waits. If the connection is held during unrelated network calls or user interaction, the ownership interval may be larger than necessary. These are proposed diagnostic branches, not measured database results in this article. It is equally important not to assume that a full pool is wrongly sized. A limit can protect the database from excessive concurrency. Raising it may reduce acquisition waiting while increasing query contention or memory pressure. The relevant result is the completed business operation under the intended workload, not simply the disappearance of one queue. The semaphore's final count provides a clean regression signal in the fixture: after all eight requests complete, both permits must be available. A production connection pool needs different observables, but the underlying test idea remains useful. Verify that every successful acquisition has a matching lifetime end, including exceptional and cancelled paths. ## Give the sampler a budget and a blind-spot statement The observer in this experiment is small, but it is not free. It uses a dedicated thread, reads process and runtime counters, allocates sample records and appends them to a concurrent queue. Its activity contributes to the process being observed. The article does not subtract an invented observer cost from the CPU counter. The sampling thread exists outside the ThreadPool so that it can still record a held state without waiting behind the same pool work being investigated. That makes the demonstration easier to reason about, while also making it unlike a typical production telemetry pipeline. A production observer may use a different scheduler, buffering policy and export path. Ten-millisecond sampling is appropriate only within this short bounded example. The program stores the observations in memory and writes them after the run. Extending that behavior indefinitely would create another workload and another retention question. The timeout and explicit thread joins are therefore part of the design, not incidental housekeeping. There are transitions the sampler can miss. A request can enter and acquire a permit between two observations; a queued work item can be executed before the next read. The maximum observed value is a lower bound on an unobserved instantaneous maximum, not proof that a greater transient value never occurred. This limitation does not invalidate the recorded two-holder, six-waiter state. That state persists long enough to be observed repeatedly and follows the deliberately held gate. It does limit finer claims about exact scheduling order or every transient queue fluctuation. A trace or another instrument would be needed if those transitions became the question controlling the next repair. ## Make the next diagnostic question falsifiable A useful investigation ends each observation with a claim that another engineer can check. “Eight request stacks are inside this synchronous gate” is supported by the saved stack report. “Two operations hold permits and six are waiting” is supported by the explicit counters. “The service is starved” would require additional evidence that this small run does not provide. For starvation, look for the relevant combination of unavailable processing capacity, queued ready work and blocked or occupied workers over the affected interval. A transient queue value or a large thread count by itself is not enough. For a pool problem, capture acquisition and hold boundaries rather than inferring them from an endpoint's total duration. The same principle improves test design. Keep the response contract independent of the wait implementation. Verify all requests finish, values remain correct, permits return and background observers stop. Then add the diagnostic assertion that distinguishes the mechanism. A test that merely finishes eventually may miss a resource leak; a test that only counts threads may miss a wrong response. The code and raw results in this bundle preserve those layers. There is a functional run with a blocking stack dump and a separate no-dump measurement series. Every child is owned and bounded. The summaries retain actual sample timestamps and do not claim that the sampler captured every transition. This is enough to change the way a low-CPU incident is approached. Instead of increasing every limit, identify the queue that currently owns progress. Instead of treating all unfinished requests as queued ThreadPool work, separate ready work, suspended operations and resource waiters. Instead of reading a latency number as a mechanism, connect it to a stack or an explicit ownership boundary. The machine can be quiet for several reasons. The most useful observability is the evidence that tells those reasons apart. ## Source code and execution evidence [Download the complete lab, source record and raw execution results](https://devinside.cloud/content/files/2026/11/ENG15-lab.zip). This package supports the specific execution claims in this article. It is not a production certification.