Consider a checkout service that has just moved from .NET 10 to .NET 11. In the same week three tickets arrive. One order has been "in checkout" for minutes, yet CPU is idle and no thread stack mentions the checkout code. A timeout page told a customer the reservation failed, and the warehouse shipped it twice. And a profiler shows a fresh Task<int> at every level of a pricing cache that almost never waits for anything.
All three tickets are about the same few objects: the code the compiler or the JIT builds for an await, the heap object a suspended method becomes, the thread pool that runs what comes after the await, the tokens and timers that end waits, and the pooled sources that make some of it allocation-free. This guide draws the map of those parts, puts the measured numbers from six deep dives side by side, and tells you which deep dive answers the question you arrived with.
- An
awaiton an incomplete task saves the method's state on the heap and gives the thread back. In .NET 10 the C# compiler builds that state machine; in .NET 11 the JIT can build it instead (runtime async, a preview you opt into per project). - A suspended method has no frame on any thread. It is held only by whatever will complete it (a timer, a socket, a
TaskCompletionSource), and its code after theawaitneeds a free pool worker to run. - Timeouts end waits, tokens ask producers to stop, and neither can undo a commit. In one measurement every checkout variant answered within about 101 ms at p99, yet the naive ones committed 1,186 orders twice.
- Reuse (
ValueTask, pooled sources, cached tasks) removed 96 bytes per operation in the measured round trip, and it hands you a protocol to keep. Measure before you take it on.
One await, four places: the map
Every async question in .NET is a question about one of four places: the worker thread that runs your method, the heap object that holds it while it waits, the operation it waits for, and the thread pool's queue that brings it back. Most production surprises come from assuming the method is somewhere it is not. Where is an async method between the await and the line after it?
An await leaves the thread, waits on the heap and comes back through the pool queue
- An async method runs on a thread-pool worker and awaits something it does not control: a timer, a socket read or a
TaskCompletionSource. Four places matter from here on: the worker, the heap, the operation and the pool queue. - If the awaited task is already complete, the method simply continues on the same thread. Nothing is saved and nothing is queued. This is the cache-hit path, and it is where .NET 11 runtime async saves the most.
- If it is not complete, the method saves what it needs to resume on the heap and returns. The classic compiler boxes its whole state machine; the .NET 11 JIT saves only the locals still needed. The worker goes back to the pool.
- The awaited operation now references that saved state. A timer queue entry, a pending socket read or whoever holds the
TaskCompletionSourceis the only root that keeps the suspended method alive. - When the operation completes, the completing code either runs the continuation inline or queues it to the pool. A timer queues its callback as a high-priority work item, so the code after the
awaitstill needs a free worker. - A free worker dequeues the item, restores the
ExecutionContext(soAsyncLocalvalues andActivity.Currentcome back) and resumes the method after theawait, often on a different thread. - If the workers are blocked in
.Result, nobody runs the queued continuation that would unblock them. In the starvation run the queue held 8 to 22 items while the pool added about one thread a second.
Simplified to one suspension. Step 7 values come from the blocking run in the thread-pool starvation deep dive (.NET 11.0.0, a 4-processor pool, per-second samples). The rest follows Task, ThreadPoolWorkQueue and the runtime-async design notes in dotnet/runtime.
Each layer has its own deep dive, failure and tool; the sections below follow the figure. Everything except runtime async applies to .NET 10 as it does to .NET 11: where the deep dives compared the two versions, the behaviour matched.
Who writes the code for an await: the compiler, or in .NET 11 the JIT
Since C# 5, the compiler has turned every async method into a small stub plus a generated state machine. The stub fills in a struct and hands it to a builder; the real body lives in MoveNext, a switch over the await the method is parked at. Every local that crosses an await becomes a field, and so does each awaiter. On the first real suspension the builder boxes the struct onto the heap. The runtime executes all of this as ordinary code without knowing it is async. That is still the model for every async method you build on .NET 10, and for every method on .NET 11 unless you opt in.
.NET 11 adds a second implementation. With <Features>runtime-async=on</Features> in a project file, the compiler emits the method almost as you wrote it, flagged Async, with each await turned into a call to an AsyncHelpers.Await overload. The JIT then compiles an "async variant" with an extra continuation argument and an extra return value: either a null continuation and the result in a register, or a continuation object that holds exactly the locals still needed after the suspension. Only the thunk at the edge, where synchronous code calls in, creates a Task. The runtime-async specification describes the encoding and the code generation notes describe the calling convention. Microsoft still calls it a preview for your code, while its own libraries are already built with it.
The deep dive compiled one source file both ways and read the result. A method that the classic build turned into a 63-byte stub and a six-field struct became, in the runtime-async build, the method body itself; in the x64 listing a suspension stored a single local in the continuation. A three-level cache chain that completes synchronously allocated 216 bytes per call in the classic build and 72 bytes with runtime async: three tasks became one. The census of the framework is the clearest sign of where this is going: System.Private.CoreLib contained 77 compiler-generated async state-machine structs on .NET 10.0.15 and 5 on .NET 11.0.0, all five using PoolingAsyncValueTaskMethodBuilder. When your classic code awaits HttpClient on .NET 11, the library half of the chain already runs on the new model.
Semantics did not change: in the deep dive's 32 tests, exceptions, AsyncLocal flow and cancellation behaved identically in both builds. Tools that parse compiler-generated names (stack-trace cleaners, tests that look for AsyncStateMachineAttribute) do see something different. A suspended chain still allocates; what disappears is the per-level Task on paths that complete without waiting.
A suspended method is a heap object, held by whatever will complete it
Once a method has suspended, its frame is gone from the thread. What remains is a heap object: a boxed state machine in the classic model, a runtime continuation in the new one. It is linked to its caller's object through the awaited task's continuation slot, so the logical call chain (Main called Outer, which called Middle, which called Inner) exists only as a chain of heap objects until the dispatcher resumes them one at a time. A thread only ever borrows the head of that chain.
That has a direct diagnostic consequence. In the hung-checkout lab, dotnet-stack listed 15 threads and not one frame from the checkout code, while the order waited for a fraud reply that had already been dropped. A heap dump read with dumpasync --coalesce found it at once: PlaceOrderAsync in state 1, its second await, waiting on a Task that nobody would complete. The state number is something no thread stack can give you, because a parked method has no instruction pointer. Built with runtime async, the same hang produced no checkout frames in dumpasync at all: runtime-async methods have no state-machine boxes to find.

dumpasync --coalesce on the hung checkout finds PlaceOrderAsync in state 1, waiting on the fraud reply in NaiveReplyCorrelator.RequestAsync (.NET 11).Who keeps that heap object alive? Not the caller. The caller may have dropped its reference long ago, for example after a timeout. The object is reachable through the operation it waits on. In the timeout deep dive, the abandoned reservation was rooted through a Task.Delay timer in one of the per-processor timer queues; swap the delay for a socket read and the root is the pending I/O, swap it for a reply message and the root is whoever holds the TaskCompletionSource. If nothing holds it, the method is collected without ever running its finally. That is the second reason the map matters: an operation nobody owns can commit later, fault later as an unobserved exception, or vanish silently.
Nor can you read identity off a thread. In the async lab, 60 to 79 of 100 operations hopped threads on .NET 10, depending on the run, yet each kept its own trace ID, because Activity.Current flows with the ExecutionContext.
The code after an await needs a free worker, and blocking takes it away
When the awaited operation completes, the continuation has to run somewhere. If the awaiter captured a SynchronizationContext or a non-default TaskScheduler, it is posted there; ASP.NET Core installs no such context, so on a server it runs inline on the completing thread or as a pool work item. RunContinuationsAsynchronously on a TaskCompletionSource forces the queue. Inline goes further than most expect: in the ValueTask deep dive, the consumer started its next operation inside the producer's SetResult call.
The pool itself is a set of queues and a goal. ThreadPoolWorkQueue keeps a global queue, a high-priority queue and one local work-stealing queue per worker; dotnet.thread_pool.queue.length is the sum of them. The pool runs at most NumThreadsGoal threads, and a thread blocked in a wait still counts against that goal. The goal moves three ways: hill climbing as work completes, starvation detection on the gate thread every 500 ms (one more thread if nothing has been dequeued for too long), and blocking compensation that reacts immediately when a pool thread blocks in a Task wait. The starvation guide on Microsoft Learn describes the same climb.
Put a blocking wait on a request path and the map in the figure closes into a loop. The catalogue API in the starvation deep dive waited for a 50 ms pricing call with .Result on a 4-processor pool. The pricing answer arrived on a timer, which queued its callback; the callback needed a worker; every worker allowed to run was blocked waiting for exactly such a callback. The result was 7.4 requests per second against a ceiling of 160, p99 at 14.4 s, and CPU at 9.2 % of one core. The await version of the same endpoint served 158.2 requests per second with 11 threads. SetMinThreads(64) also reached 158.3, with 37 threads: it pays for the blocked threads up front instead of removing them.
Blocking hides in less obvious places too. A slow synchronous factory behind the default Lazy<T> mode holds one blocked thread per waiting caller for as long as it runs: in the Lazy deep dive, every one of 16 callers waited the whole 20 ms load (p50 20,607 µs). And once the pool is healthy, requests can still queue where the pool counter cannot see them: in the await run, 24 requests waited at the pricing gate, holding no thread, while dotnet.thread_pool.queue.length read 0. Async frees workers. It does not create dependency capacity.
A timeout ends a wait, a token asks politely, and a commit is final
Ending a wait and ending work are different operations, and .NET gives you separate tools for each. Task.WaitAsync(timeout) builds a small promise that listens to three inputs: the producer's completion, a one-shot timer and the caller's token. Whichever fires first completes the promise. The promise never held anything that could stop the producer, so when the timer wins, the caller gets a TimeoutException and the producer keeps running. Task.WhenAny(operation, Task.Delay(timeout)) has the same blind spot, plus three of its own: the outer task always completes successfully, .Result turns the real exception into an AggregateException, and the losing delay keeps its timer armed until it fires.
A CancellationToken is the only handle that reaches the producer, and it is cooperative: the producer must check it or pass it to an API that does. It also loses once the producer has passed its commit point. In the timeout deep dive's checkout, cancelling on timeout stopped the reservations that had not committed yet and nothing else. Over 6,000 orders per variant with the same 100 ms timeout, the naive WaitAsync and the WhenAny helper committed 1,186 orders twice, cancelling on timeout cut that to 599, and only an operation key plus an owner that observes every attempt brought it to 0. The latency was the same for all four, within a millisecond at p50, p95 and p99.
The lesson for the map is about ownership. After the caller's wait ends, something must still observe the producer's outcome, otherwise a late failure surfaces only as UnobservedTaskException (176 times in the naive runs) or not at all. A retry that may repeat a write needs an operation key created once per user intent, so the receiver can return the stored receipt instead of acting twice. And a request with several stages needs one shared deadline: three stages with a fresh 1 s timeout each have quietly promised the client 3 s.
Timers are the cost you can count. After 10,000 fast operations with a 30 s timeout, the WhenAny race left 10,000 armed timers and WaitAsync at most a few.
Reuse removes an allocation and hands you a protocol
Every async operation that completes asynchronously needs an object the caller can wait on. A Task<T> is allocated per operation, can be awaited any number of times and works with every combinator. A ValueTask<T> has three forms: an inline result (no allocation), a wrapped Task<T>, or an IValueTaskSource<T> plus a 16-bit token. The third form lets one long-lived object serve operation after operation, which is how sockets, channels and PoolingAsyncValueTaskMethodBuilder avoid per-operation allocations.
The price is a protocol. The token says which operation on the source you mean; reuse is the source moving on to a new token. ManualResetValueTaskSourceCore<T> validates tokens but records nothing about consumption: called twice with the same token, its GetResult returned 42 twice. The runtime's own pooled sources release in the consumer's GetResult, after the result has been read, and that is the only safe reset point. A wrapper in the ValueTask deep dive that set a flag one line after SetResult hung deterministically with inline continuations and 99 and 117 times in two runs of 2,000 trials with queued ones. That is why the ValueTask documentation forbids awaiting an instance twice: by the second read, the source may belong to someone else.
What does reuse buy? In the round-trip harness a new TaskCompletionSource<int> per operation allocated 96 bytes; the reusable source and a single-reader Channel<int> allocated none. The times told no consistent story, and the channel gave the same zero bytes with no custom protocol.
Reuse also hides in caching. Lazy<T> in its default mode caches a factory exception forever, so a two-second outage at startup can break an instance for the life of the process. Lazy<Task<T>> releases callers at once (p50 11.8 µs inside .Value against about 20 ms for every blocking variant) but caches a faulted task just the same, and a cached ValueTask breaks the await-once rule outright. The Lazy deep dive ends with a small async cache that shares an in-flight load, keeps a success and forgets a failure.
Where a waiting request is, and which record can see it
The first figure followed one await. In production you start from the other end: a request is slow or stuck, and you need to know where it is waiting before you choose a tool. The thread-pool and async-stack deep dives found four distinct places, each holding something different and each visible to a different record. Which record answers which question?
Four places a request can wait, and only one record says which request it is
- A request that is not running can wait in four places. Each holds something different, and each is visible to a different record. Start from the question you need answered, not from the tool you happen to know.
- Not started yet: its work item sits in the pool queue.
dotnet.thread_pool.queue.lengthcounts it; it holds no thread and no permit yet. - Blocked on a worker by
.ResultorWait(): a stack snapshot shows your frame under the wait. In the starvation run the pool grew from 4 to 31 threads in 20 s. - Suspended at an
await: no thread and no frame.dumpasyncwalks classic state-machine boxes and prints the exact await; on runtime-async code it showed none of the checkout. - Waiting at a
SemaphoreSlimgate: no runtime counter sees it. In the healthyawaitrun, 24 requests waited there while the pool queue read 0; only the gate's own meter showed them. - None of these records names the business request. A trace ID carried through
Activity, kept in an in-flight list with its stage, is the record that does.
Numbers from the thread-pool starvation deep dive (round 1, .NET 11.0.0, 4-processor pool) and the async thread-stack deep dive (.NET 11.0.0, classic and runtime-async builds of the same service).
Diagnostics have a cost of their own: a heap dump froze request handling for about a quarter of a second, and a CPU-sampling session roughly doubled p95. Choose the record by the question, then take it from an instance you can afford to pause.
The measured numbers side by side
Every number below is quoted from a published deep dive and its solution's results/ folder. All runs used .NET 11.0.0 Release builds on Windows Server 2022, an AMD EPYC 7502P with 64 logical processors, on a shared machine; the thread-pool runs limited the process to 4 processors with DOTNET_PROCESSOR_COUNT=4. Absolute times are this machine's; byte counts are exact. The tables are not comparable with each other: each answers its own question with its own harness.
What an await costs: classic against runtime async
| Workload (per call) | Classic | Runtime async |
|---|---|---|
| Completed Task chain, bytes | 216 B | 72 B |
| Completed Task chain, mean / p99 | 42.7 / 75.0 ns | 29.6 / 49.7 ns |
| Completed ValueTask chain, bytes / mean | 0 B / 26.4 ns | 0 B / 27.1 ns |
| Suspended chain incl. gate, bytes | 384 B | 384 B |
| Suspended chain incl. gate, mean / p99 | 251.8 / 313.2 ns | 206.9 / 255.3 ns |
Source: Runtime Async in .NET 11. Means from BenchmarkDotNet 0.15.8 (30 iterations); p99 from the batch harness, 2,000 batches of 1,000 calls per workload. The gate alone is 72 B and 19.1 ns in both builds.
What blocking costs the thread pool
| Run (32 users, 20 s) | Requests/s | p99 | Pool threads |
|---|---|---|---|
Blocking (.Result) | 7.4 | 14,418.9 ms | 4 to 31 |
Blocking, SetMinThreads(64) | 158.3 | 206.3 ms | 20 to 37 |
await | 158.2 | 204.7 ms | 5 to 11 |
Source: ASP.NET Waits with Low CPU, round 1 of 3. CPU stayed between 9.2 and 11.5 % of one core in all three configurations; rounds 2 and 3 told the same story.
What a timeout does to the work behind it
| Checkout (6,000 orders) | Click 1 p99 | Committed twice | Unobserved exceptions |
|---|---|---|---|
Naive WaitAsync | 101.5 ms | 1,186 | 176 |
WhenAny helper | 101.4 ms | 1,186 | 176 |
| Cancel on timeout | 101.3 ms | 599 | 0 |
| Keyed and owned | 101.5 ms | 0 | 0 |
Source: A Timeout Is Not Cancellation, TimeoutLab latency, 3 rounds of 2,000 orders per variant, 100 ms timeout. Per call when the operation wins (BenchmarkDotNet ShortRun): plain await 190 B, WaitAsync(30 s) 339 B, WhenAny with Task.Delay 548 B, owner plus WaitAsync 767 B.
What reuse buys per operation
| Variant | Inline p50 | Queued p50 | Bytes/op |
|---|---|---|---|
New TaskCompletionSource<int> | 259 ns | 2,804 ns | 96.0 |
| Reusable source | 263 ns | 1,408 ns | 0.0 |
| Packet receiver (64-byte packet) | 328 ns | 1,821 ns | 0.0 |
Channel<int>, single reader | 411 ns | 2,336 ns | 0.0 |
Source: Reusable ValueTask Sources, ReceiveBench roundtrip, 200 batches of 1,000 round trips, measured once. The queued medians changed order in a second recorded run; the bytes did not.
What a cold start costs with 16 callers
| Variant | Cheap p50 | Factory runs | Slow (20 ms) p50 |
|---|---|---|---|
Lazy, ExecutionAndPublication | 241.8 µs | 1 | 20,607 µs |
Lazy, PublicationOnly | 62.2 µs | 15.9 (cheap), 16 (slow) | 20,671 µs |
Double-checked Lock | 184.6 µs | 1 | 20,767 µs |
Lazy<Task<T>>, time in .Value | not run | 1 | 11.8 µs |
Source: Lazy<T> Under Contention, LazyBench contention, one run, 300 cheap or 12 slow trials. A warm read cost 1.6 to 3.1 ns in every variant: the mode only matters on the first call.
What a diagnostic costs a live service
| Diagnostic | Measured effect |
|---|---|
| Heap dump (96.7 MB) | write p50 378 ms, request stall max 274 ms |
| Full dump (200.7 MB) | write p50 358 ms, request stall max 263 ms |
| CPU-sampling session | p95 0.223 and 0.213 ms against 0.117 and 0.105 ms without a session (rounds 2 and 3) |
| Async profiler session | p95 0.112 and 0.111 ms in the same rounds |
Source: Async Work Is Not on the Thread Stack, 12 dumps per type and 3 rounds of 40,000 checkout requests per configuration.
Read the tables together and one pattern repeats: the expensive outcomes were never the nanoseconds. Runtime async saved 13 ns per three-level call; a blocking wait cost 95 % of the throughput; a timeout without an owner or a key cost duplicate orders at unchanged latency. Optimise allocations after the waits, the owners and the keys are right.
Read this next
- What exactly changes if I turn on runtime async in .NET 11? Runtime Async in .NET 11: Who Builds the State Machine Now? compiles one file both ways, shows the IL and the x64 the JIT produced, and ships a solution with 32 equivalence tests and both benchmark harnesses.
- A request is stuck and no thread stack shows it. Where is it? Async Work Is Not on the Thread Stack reproduces the hang, finds it with
dumpasyncand Parallel Stacks in Tasks view, decodes a .NET 11 async profiler record byte by byte and measures what a dump costs. - Latency is high, CPU is low, and someone wants to raise the minimum threads. ASP.NET Waits with Low CPU runs blocking, a raised minimum and
awaitside by side, opens the pool's thread-injection logic and adds a test that fails when a request path blocks. - My timeout fired but the work still happened. A Timeout Is Not Cancellation shows who still holds an abandoned operation, counts the timers each pattern leaves, and builds an owned attempt with an operation key that took duplicates from 1,186 to 0.
- Should I write my own
IValueTaskSource? Reusable ValueTask Sources: When Is It Safe to Call Reset? takes the reuse protocol apart, reproduces the inline re-entry hang under a debugger, and measures the source against aTaskand a channel. - Is
Lazy<T>orLazy<Task<T>>safe for shared initialization? Lazy<T> Under Contention runs every mode under a cold-start burst, shows a cached exception in the debugger, and ends with an async cache that forgets failures. - Where do pooled buffers fit across an await? ArrayPool Across await covers buffer leases and ownership transfer, the memory side of the reuse question.
- What does the JIT do with allocations once the code runs? Why .NET 11 Removes This Box, but Keeps That Allocation follows escape analysis from .NET 10 to .NET 11.
Decision guide for async code on .NET 10 and 11
| You see or need | First move | Watch for |
|---|---|---|
| Latency up, CPU low, pool threads climbing about one per second | Find the blocking wait in a stack snapshot and await end to end | SetMinThreads hides it until the next burst above the minimum |
| A request stuck with no frames on any thread | An in-flight list keyed by trace ID, then a Heap dump with dumpasync | Runtime-async code is invisible to dumpasync; a dump pauses requests |
| Bound how long a caller waits | WaitAsync(timeout, token) | The work keeps running and needs an owner that records its outcome |
| Stop work nobody needs any more | A token passed into the producer | Cooperative only, and too late after the commit point |
| A write the client may retry | An operation key and a receipt stored in one transaction | Create the key once per user intent, not per attempt |
| Many thin async layers that usually complete synchronously (.NET 11) | Try runtime-async=on in a branch and measure | A preview feature; tools that parse async frames; pooled builders are converted too |
| An allocation per operation in a long-lived producer and consumer | A Channel<T>, then a custom source only with a measurement | Reset only after the consumer's GetResult; await each ValueTask once |
| Shared asynchronous initialization that can fail | A small async cache that forgets faulted attempts | Lazy<T> and Lazy<Task<T>> both cache failures |
| Requests wait although the pool queue is empty | Meter the application gate; bound its queue | Async frees threads, not dependency capacity |
Checklist
- No
.Result,.Wait()orGetAwaiter().GetResult()on request paths, and a test asserts the path returns while its dependency is pending. - Dashboards show the pool queue, pool thread count and every application gate's waiting count; every request carries a trace ID visible in an in-flight view.
- Every started operation has an owner that observes its outcome after the caller's wait ends; no fire-and-forget on request paths.
- Every
TaskCompletionSourcewait is registered before its trigger and bounded by a timeout and the caller's token. - Every
CancellationTokenSourcewithCancelAfteror a linked parent is disposed; noWhenAny(operation, Task.Delay(...))helpers remain. - Retried writes carry an operation key created once per user intent; stages share one deadline.
- Each
ValueTaskis awaited once; a custom source exists only where an allocation measurement justified it. - Runtime async is tried per project on .NET 11 with the classic build kept as the control, under the same tests and load.
Further reading
- How async/await really works in C# (.NET Blog): the classic compiler transformation, builders and awaiters.
- Runtime async specification and runtime async code generation (dotnet/runtime): the
Asyncflag,AsyncHelpers.Awaitand the continuation calling convention. - What's new in the .NET 11 runtime: runtime async, the libraries built with it and the async profiler.
- ThreadPoolWorkQueue.cs (release/11.0): the global, high-priority and local queues and the dequeue order.
- Debug ThreadPool starvation: counters, stacks and the injection heuristics.
- Cancellation in managed threads: the cooperative model.
- ValueTask<TResult>: the consumption rules for source-backed instances.
- ConfigureAwait FAQ (.NET Blog): where continuations run, and why ASP.NET Core has no SynchronizationContext.




