Async in .NET 10 and 11 from the Inside: From await to Thread Pool

What the compiler and the .NET 11 JIT build for an await, where a suspended method lives, how the thread pool resumes it, and what timeouts, cancellation and reuse really do.

A signalman holds a brass token beside a track board and waiting train, suggesting suspended work and its next continuation.

Consider a checkout service that has just moved from .NET 10 to .NET 11. In the same week three tickets arrive. One order has been "in checkout" for minutes, yet CPU is idle and no thread stack mentions the checkout code. A timeout page told a customer the reservation failed, and the warehouse shipped it twice. And a profiler shows a fresh Task<int> at every level of a pricing cache that almost never waits for anything.

All three tickets are about the same few objects: the code the compiler or the JIT builds for an await, the heap object a suspended method becomes, the thread pool that runs what comes after the await, the tokens and timers that end waits, and the pooled sources that make some of it allocation-free. This guide draws the map of those parts, puts the measured numbers from six deep dives side by side, and tells you which deep dive answers the question you arrived with.

  • An await on an incomplete task saves the method's state on the heap and gives the thread back. In .NET 10 the C# compiler builds that state machine; in .NET 11 the JIT can build it instead (runtime async, a preview you opt into per project).
  • A suspended method has no frame on any thread. It is held only by whatever will complete it (a timer, a socket, a TaskCompletionSource), and its code after the await needs a free pool worker to run.
  • Timeouts end waits, tokens ask producers to stop, and neither can undo a commit. In one measurement every checkout variant answered within about 101 ms at p99, yet the naive ones committed 1,186 orders twice.
  • Reuse (ValueTask, pooled sources, cached tasks) removed 96 bytes per operation in the measured round trip, and it hands you a protocol to keep. Measure before you take it on.

One await, four places: the map

Every async question in .NET is a question about one of four places: the worker thread that runs your method, the heap object that holds it while it waits, the operation it waits for, and the thread pool's queue that brings it back. Most production surprises come from assuming the method is somewhere it is not. Where is an async method between the await and the line after it?

Data flow

An await leaves the thread, waits on the heap and comes back through the pool queue

WORKER THREAD HEAP AWAITED OPERATION POOL QUEUE your async method no saved state timer, socket or TCS work items suspend holds queues runs until it reaches an incomplete await task already complete: continue, nothing saved thread returned to the pool resumed after the await, AsyncLocal and Activity flow blocked in .Result: cannot run the queue classic: boxed state machine .NET 11 JIT: live locals only the only root that keeps the suspended method alive or run inline by the thread that completes the task the pool adds about one thread per second
  1. An async method runs on a thread-pool worker and awaits something it does not control: a timer, a socket read or a TaskCompletionSource. Four places matter from here on: the worker, the heap, the operation and the pool queue.
  2. If the awaited task is already complete, the method simply continues on the same thread. Nothing is saved and nothing is queued. This is the cache-hit path, and it is where .NET 11 runtime async saves the most.
  3. If it is not complete, the method saves what it needs to resume on the heap and returns. The classic compiler boxes its whole state machine; the .NET 11 JIT saves only the locals still needed. The worker goes back to the pool.
  4. The awaited operation now references that saved state. A timer queue entry, a pending socket read or whoever holds the TaskCompletionSource is the only root that keeps the suspended method alive.
  5. When the operation completes, the completing code either runs the continuation inline or queues it to the pool. A timer queues its callback as a high-priority work item, so the code after the await still needs a free worker.
  6. A free worker dequeues the item, restores the ExecutionContext (so AsyncLocal values and Activity.Current come back) and resumes the method after the await, often on a different thread.
  7. If the workers are blocked in .Result, nobody runs the queued continuation that would unblock them. In the starvation run the queue held 8 to 22 items while the pool added about one thread a second.

Simplified to one suspension. Step 7 values come from the blocking run in the thread-pool starvation deep dive (.NET 11.0.0, a 4-processor pool, per-second samples). The rest follows Task, ThreadPoolWorkQueue and the runtime-async design notes in dotnet/runtime.

Each layer has its own deep dive, failure and tool; the sections below follow the figure. Everything except runtime async applies to .NET 10 as it does to .NET 11: where the deep dives compared the two versions, the behaviour matched.

Who writes the code for an await: the compiler, or in .NET 11 the JIT

Since C# 5, the compiler has turned every async method into a small stub plus a generated state machine. The stub fills in a struct and hands it to a builder; the real body lives in MoveNext, a switch over the await the method is parked at. Every local that crosses an await becomes a field, and so does each awaiter. On the first real suspension the builder boxes the struct onto the heap. The runtime executes all of this as ordinary code without knowing it is async. That is still the model for every async method you build on .NET 10, and for every method on .NET 11 unless you opt in.

.NET 11 adds a second implementation. With <Features>runtime-async=on</Features> in a project file, the compiler emits the method almost as you wrote it, flagged Async, with each await turned into a call to an AsyncHelpers.Await overload. The JIT then compiles an "async variant" with an extra continuation argument and an extra return value: either a null continuation and the result in a register, or a continuation object that holds exactly the locals still needed after the suspension. Only the thunk at the edge, where synchronous code calls in, creates a Task. The runtime-async specification describes the encoding and the code generation notes describe the calling convention. Microsoft still calls it a preview for your code, while its own libraries are already built with it.

The deep dive compiled one source file both ways and read the result. A method that the classic build turned into a 63-byte stub and a six-field struct became, in the runtime-async build, the method body itself; in the x64 listing a suspension stored a single local in the continuation. A three-level cache chain that completes synchronously allocated 216 bytes per call in the classic build and 72 bytes with runtime async: three tasks became one. The census of the framework is the clearest sign of where this is going: System.Private.CoreLib contained 77 compiler-generated async state-machine structs on .NET 10.0.15 and 5 on .NET 11.0.0, all five using PoolingAsyncValueTaskMethodBuilder. When your classic code awaits HttpClient on .NET 11, the library half of the chain already runs on the new model.

Semantics did not change: in the deep dive's 32 tests, exceptions, AsyncLocal flow and cancellation behaved identically in both builds. Tools that parse compiler-generated names (stack-trace cleaners, tests that look for AsyncStateMachineAttribute) do see something different. A suspended chain still allocates; what disappears is the per-level Task on paths that complete without waiting.

A suspended method is a heap object, held by whatever will complete it

Once a method has suspended, its frame is gone from the thread. What remains is a heap object: a boxed state machine in the classic model, a runtime continuation in the new one. It is linked to its caller's object through the awaited task's continuation slot, so the logical call chain (Main called Outer, which called Middle, which called Inner) exists only as a chain of heap objects until the dispatcher resumes them one at a time. A thread only ever borrows the head of that chain.

That has a direct diagnostic consequence. In the hung-checkout lab, dotnet-stack listed 15 threads and not one frame from the checkout code, while the order waited for a fraud reply that had already been dropped. A heap dump read with dumpasync --coalesce found it at once: PlaceOrderAsync in state 1, its second await, waiting on a Task that nobody would complete. The state number is something no thread stack can give you, because a parked method has no instruction pointer. Built with runtime async, the same hang produced no checkout frames in dumpasync at all: runtime-async methods have no state-machine boxes to find.

Console output of dotnet-dump analyze with dumpasync: STACKS 2 is rooted in a Task of FraudReply and lists RequestAsync, ScoreAsync, AuthorizeAsync, PlaceOrderAsync in state 1 and the endpoint lambda.
dotnet-dump 10.0.750501From Async Work Is Not on the Thread Stack, async-thread-stack solution: dumpasync --coalesce on the hung checkout finds PlaceOrderAsync in state 1, waiting on the fraud reply in NaiveReplyCorrelator.RequestAsync (.NET 11).

Who keeps that heap object alive? Not the caller. The caller may have dropped its reference long ago, for example after a timeout. The object is reachable through the operation it waits on. In the timeout deep dive, the abandoned reservation was rooted through a Task.Delay timer in one of the per-processor timer queues; swap the delay for a socket read and the root is the pending I/O, swap it for a reply message and the root is whoever holds the TaskCompletionSource. If nothing holds it, the method is collected without ever running its finally. That is the second reason the map matters: an operation nobody owns can commit later, fault later as an unobserved exception, or vanish silently.

Nor can you read identity off a thread. In the async lab, 60 to 79 of 100 operations hopped threads on .NET 10, depending on the run, yet each kept its own trace ID, because Activity.Current flows with the ExecutionContext.

The code after an await needs a free worker, and blocking takes it away

When the awaited operation completes, the continuation has to run somewhere. If the awaiter captured a SynchronizationContext or a non-default TaskScheduler, it is posted there; ASP.NET Core installs no such context, so on a server it runs inline on the completing thread or as a pool work item. RunContinuationsAsynchronously on a TaskCompletionSource forces the queue. Inline goes further than most expect: in the ValueTask deep dive, the consumer started its next operation inside the producer's SetResult call.

The pool itself is a set of queues and a goal. ThreadPoolWorkQueue keeps a global queue, a high-priority queue and one local work-stealing queue per worker; dotnet.thread_pool.queue.length is the sum of them. The pool runs at most NumThreadsGoal threads, and a thread blocked in a wait still counts against that goal. The goal moves three ways: hill climbing as work completes, starvation detection on the gate thread every 500 ms (one more thread if nothing has been dequeued for too long), and blocking compensation that reacts immediately when a pool thread blocks in a Task wait. The starvation guide on Microsoft Learn describes the same climb.

Put a blocking wait on a request path and the map in the figure closes into a loop. The catalogue API in the starvation deep dive waited for a 50 ms pricing call with .Result on a 4-processor pool. The pricing answer arrived on a timer, which queued its callback; the callback needed a worker; every worker allowed to run was blocked waiting for exactly such a callback. The result was 7.4 requests per second against a ceiling of 160, p99 at 14.4 s, and CPU at 9.2 % of one core. The await version of the same endpoint served 158.2 requests per second with 11 threads. SetMinThreads(64) also reached 158.3, with 37 threads: it pays for the blocked threads up front instead of removing them.

dotnet-counters 10.0.750501, .NET 11.0.0From ASP.NET Waits with Low CPU, threadpool-starvation solution: the blocking route under 32 users with a 4-processor pool. The queue holds 10 to 20 work items while the pool adds threads about one at a time, from 14 to 26 in 15 s.

Blocking hides in less obvious places too. A slow synchronous factory behind the default Lazy<T> mode holds one blocked thread per waiting caller for as long as it runs: in the Lazy deep dive, every one of 16 callers waited the whole 20 ms load (p50 20,607 µs). And once the pool is healthy, requests can still queue where the pool counter cannot see them: in the await run, 24 requests waited at the pricing gate, holding no thread, while dotnet.thread_pool.queue.length read 0. Async frees workers. It does not create dependency capacity.

A timeout ends a wait, a token asks politely, and a commit is final

Ending a wait and ending work are different operations, and .NET gives you separate tools for each. Task.WaitAsync(timeout) builds a small promise that listens to three inputs: the producer's completion, a one-shot timer and the caller's token. Whichever fires first completes the promise. The promise never held anything that could stop the producer, so when the timer wins, the caller gets a TimeoutException and the producer keeps running. Task.WhenAny(operation, Task.Delay(timeout)) has the same blind spot, plus three of its own: the outer task always completes successfully, .Result turns the real exception into an AggregateException, and the losing delay keeps its timer armed until it fires.

A CancellationToken is the only handle that reaches the producer, and it is cooperative: the producer must check it or pass it to an API that does. It also loses once the producer has passed its commit point. In the timeout deep dive's checkout, cancelling on timeout stopped the reservations that had not committed yet and nothing else. Over 6,000 orders per variant with the same 100 ms timeout, the naive WaitAsync and the WhenAny helper committed 1,186 orders twice, cancelling on timeout cut that to 599, and only an operation key plus an owner that observes every attempt brought it to 0. The latency was the same for all four, within a millisecond at p50, p95 and p99.

Checkout.Demo, .NET 11.0.0From A Timeout Is Not Cancellation, timeout-cancellation solution: click 1 gives up after 114 ms while the store commits at 236 ms. The retried click commits again, so order O-1001 is committed twice and stock falls from 5 to 1.

The lesson for the map is about ownership. After the caller's wait ends, something must still observe the producer's outcome, otherwise a late failure surfaces only as UnobservedTaskException (176 times in the naive runs) or not at all. A retry that may repeat a write needs an operation key created once per user intent, so the receiver can return the stored receipt instead of acting twice. And a request with several stages needs one shared deadline: three stages with a fresh 1 s timeout each have quietly promised the client 3 s.

Timers are the cost you can count. After 10,000 fast operations with a 30 s timeout, the WhenAny race left 10,000 armed timers and WaitAsync at most a few.

Reuse removes an allocation and hands you a protocol

Every async operation that completes asynchronously needs an object the caller can wait on. A Task<T> is allocated per operation, can be awaited any number of times and works with every combinator. A ValueTask<T> has three forms: an inline result (no allocation), a wrapped Task<T>, or an IValueTaskSource<T> plus a 16-bit token. The third form lets one long-lived object serve operation after operation, which is how sockets, channels and PoolingAsyncValueTaskMethodBuilder avoid per-operation allocations.

The price is a protocol. The token says which operation on the source you mean; reuse is the source moving on to a new token. ManualResetValueTaskSourceCore<T> validates tokens but records nothing about consumption: called twice with the same token, its GetResult returned 42 twice. The runtime's own pooled sources release in the consumer's GetResult, after the result has been read, and that is the only safe reset point. A wrapper in the ValueTask deep dive that set a flag one line after SetResult hung deterministically with inline continuations and 99 and 117 times in two runs of 2,000 trials with queued ones. That is why the ValueTask documentation forbids awaiting an instance twice: by the second read, the source may belong to someone else.

What does reuse buy? In the round-trip harness a new TaskCompletionSource<int> per operation allocated 96 bytes; the reusable source and a single-reader Channel<int> allocated none. The times told no consistent story, and the channel gave the same zero bytes with no custom protocol.

Reuse also hides in caching. Lazy<T> in its default mode caches a factory exception forever, so a two-second outage at startup can break an instance for the life of the process. Lazy<Task<T>> releases callers at once (p50 11.8 µs inside .Value against about 20 ms for every blocking variant) but caches a faulted task just the same, and a cached ValueTask breaks the await-once rule outright. The Lazy deep dive ends with a small async cache that shares an in-flight load, keeps a success and forgets a failure.

Where a waiting request is, and which record can see it

The first figure followed one await. In production you start from the other end: a request is slow or stuck, and you need to know where it is waiting before you choose a tool. The thread-pool and async-stack deep dives found four distinct places, each holding something different and each visible to a different record. Which record answers which question?

Architecture

Four places a request can wait, and only one record says which request it is

WHERE IT WAITS WHAT CAN SEE IT WHAT IT HOLDS not started: pool queue blocked in .Result suspended at an await waiting at a gate queue.length counter holds nothing yet queued behind blocked workers dotnet-stack frames holds a worker thread pool: 4 to 31 threads in 20 s dumpasync (classic) holds heap state, no thread runtime async: not visible only your own meter holds a request, no thread 24 waiting, queue.length 0 trace ID: which request, which stage, how long
  1. A request that is not running can wait in four places. Each holds something different, and each is visible to a different record. Start from the question you need answered, not from the tool you happen to know.
  2. Not started yet: its work item sits in the pool queue. dotnet.thread_pool.queue.length counts it; it holds no thread and no permit yet.
  3. Blocked on a worker by .Result or Wait(): a stack snapshot shows your frame under the wait. In the starvation run the pool grew from 4 to 31 threads in 20 s.
  4. Suspended at an await: no thread and no frame. dumpasync walks classic state-machine boxes and prints the exact await; on runtime-async code it showed none of the checkout.
  5. Waiting at a SemaphoreSlim gate: no runtime counter sees it. In the healthy await run, 24 requests waited there while the pool queue read 0; only the gate's own meter showed them.
  6. None of these records names the business request. A trace ID carried through Activity, kept in an in-flight list with its stage, is the record that does.

Numbers from the thread-pool starvation deep dive (round 1, .NET 11.0.0, 4-processor pool) and the async thread-stack deep dive (.NET 11.0.0, classic and runtime-async builds of the same service).

Diagnostics have a cost of their own: a heap dump froze request handling for about a quarter of a second, and a CPU-sampling session roughly doubled p95. Choose the record by the question, then take it from an instance you can afford to pause.

The measured numbers side by side

Every number below is quoted from a published deep dive and its solution's results/ folder. All runs used .NET 11.0.0 Release builds on Windows Server 2022, an AMD EPYC 7502P with 64 logical processors, on a shared machine; the thread-pool runs limited the process to 4 processors with DOTNET_PROCESSOR_COUNT=4. Absolute times are this machine's; byte counts are exact. The tables are not comparable with each other: each answers its own question with its own harness.

What an await costs: classic against runtime async

Workload (per call)ClassicRuntime async
Completed Task chain, bytes216 B72 B
Completed Task chain, mean / p9942.7 / 75.0 ns29.6 / 49.7 ns
Completed ValueTask chain, bytes / mean0 B / 26.4 ns0 B / 27.1 ns
Suspended chain incl. gate, bytes384 B384 B
Suspended chain incl. gate, mean / p99251.8 / 313.2 ns206.9 / 255.3 ns

Source: Runtime Async in .NET 11. Means from BenchmarkDotNet 0.15.8 (30 iterations); p99 from the batch harness, 2,000 batches of 1,000 calls per workload. The gate alone is 72 B and 19.1 ns in both builds.

What blocking costs the thread pool

Run (32 users, 20 s)Requests/sp99Pool threads
Blocking (.Result)7.414,418.9 ms4 to 31
Blocking, SetMinThreads(64)158.3206.3 ms20 to 37
await158.2204.7 ms5 to 11

Source: ASP.NET Waits with Low CPU, round 1 of 3. CPU stayed between 9.2 and 11.5 % of one core in all three configurations; rounds 2 and 3 told the same story.

What a timeout does to the work behind it

Checkout (6,000 orders)Click 1 p99Committed twiceUnobserved exceptions
Naive WaitAsync101.5 ms1,186176
WhenAny helper101.4 ms1,186176
Cancel on timeout101.3 ms5990
Keyed and owned101.5 ms00

Source: A Timeout Is Not Cancellation, TimeoutLab latency, 3 rounds of 2,000 orders per variant, 100 ms timeout. Per call when the operation wins (BenchmarkDotNet ShortRun): plain await 190 B, WaitAsync(30 s) 339 B, WhenAny with Task.Delay 548 B, owner plus WaitAsync 767 B.

What reuse buys per operation

VariantInline p50Queued p50Bytes/op
New TaskCompletionSource<int>259 ns2,804 ns96.0
Reusable source263 ns1,408 ns0.0
Packet receiver (64-byte packet)328 ns1,821 ns0.0
Channel<int>, single reader411 ns2,336 ns0.0

Source: Reusable ValueTask Sources, ReceiveBench roundtrip, 200 batches of 1,000 round trips, measured once. The queued medians changed order in a second recorded run; the bytes did not.

What a cold start costs with 16 callers

VariantCheap p50Factory runsSlow (20 ms) p50
Lazy, ExecutionAndPublication241.8 µs120,607 µs
Lazy, PublicationOnly62.2 µs15.9 (cheap), 16 (slow)20,671 µs
Double-checked Lock184.6 µs120,767 µs
Lazy<Task<T>>, time in .Valuenot run111.8 µs

Source: Lazy<T> Under Contention, LazyBench contention, one run, 300 cheap or 12 slow trials. A warm read cost 1.6 to 3.1 ns in every variant: the mode only matters on the first call.

What a diagnostic costs a live service

DiagnosticMeasured effect
Heap dump (96.7 MB)write p50 378 ms, request stall max 274 ms
Full dump (200.7 MB)write p50 358 ms, request stall max 263 ms
CPU-sampling sessionp95 0.223 and 0.213 ms against 0.117 and 0.105 ms without a session (rounds 2 and 3)
Async profiler sessionp95 0.112 and 0.111 ms in the same rounds

Source: Async Work Is Not on the Thread Stack, 12 dumps per type and 3 rounds of 40,000 checkout requests per configuration.

Read the tables together and one pattern repeats: the expensive outcomes were never the nanoseconds. Runtime async saved 13 ns per three-level call; a blocking wait cost 95 % of the throughput; a timeout without an owner or a key cost duplicate orders at unchanged latency. Optimise allocations after the waits, the owners and the keys are right.

Read this next

  • What exactly changes if I turn on runtime async in .NET 11? Runtime Async in .NET 11: Who Builds the State Machine Now? compiles one file both ways, shows the IL and the x64 the JIT produced, and ships a solution with 32 equivalence tests and both benchmark harnesses.
  • A request is stuck and no thread stack shows it. Where is it? Async Work Is Not on the Thread Stack reproduces the hang, finds it with dumpasync and Parallel Stacks in Tasks view, decodes a .NET 11 async profiler record byte by byte and measures what a dump costs.
  • Latency is high, CPU is low, and someone wants to raise the minimum threads. ASP.NET Waits with Low CPU runs blocking, a raised minimum and await side by side, opens the pool's thread-injection logic and adds a test that fails when a request path blocks.
  • My timeout fired but the work still happened. A Timeout Is Not Cancellation shows who still holds an abandoned operation, counts the timers each pattern leaves, and builds an owned attempt with an operation key that took duplicates from 1,186 to 0.
  • Should I write my own IValueTaskSource? Reusable ValueTask Sources: When Is It Safe to Call Reset? takes the reuse protocol apart, reproduces the inline re-entry hang under a debugger, and measures the source against a Task and a channel.
  • Is Lazy<T> or Lazy<Task<T>> safe for shared initialization? Lazy<T> Under Contention runs every mode under a cold-start burst, shows a cached exception in the debugger, and ends with an async cache that forgets failures.
  • Where do pooled buffers fit across an await? ArrayPool Across await covers buffer leases and ownership transfer, the memory side of the reuse question.
  • What does the JIT do with allocations once the code runs? Why .NET 11 Removes This Box, but Keeps That Allocation follows escape analysis from .NET 10 to .NET 11.

Decision guide for async code on .NET 10 and 11

You see or needFirst moveWatch for
Latency up, CPU low, pool threads climbing about one per secondFind the blocking wait in a stack snapshot and await end to endSetMinThreads hides it until the next burst above the minimum
A request stuck with no frames on any threadAn in-flight list keyed by trace ID, then a Heap dump with dumpasyncRuntime-async code is invisible to dumpasync; a dump pauses requests
Bound how long a caller waitsWaitAsync(timeout, token)The work keeps running and needs an owner that records its outcome
Stop work nobody needs any moreA token passed into the producerCooperative only, and too late after the commit point
A write the client may retryAn operation key and a receipt stored in one transactionCreate the key once per user intent, not per attempt
Many thin async layers that usually complete synchronously (.NET 11)Try runtime-async=on in a branch and measureA preview feature; tools that parse async frames; pooled builders are converted too
An allocation per operation in a long-lived producer and consumerA Channel<T>, then a custom source only with a measurementReset only after the consumer's GetResult; await each ValueTask once
Shared asynchronous initialization that can failA small async cache that forgets faulted attemptsLazy<T> and Lazy<Task<T>> both cache failures
Requests wait although the pool queue is emptyMeter the application gate; bound its queueAsync frees threads, not dependency capacity

Checklist

  • No .Result, .Wait() or GetAwaiter().GetResult() on request paths, and a test asserts the path returns while its dependency is pending.
  • Dashboards show the pool queue, pool thread count and every application gate's waiting count; every request carries a trace ID visible in an in-flight view.
  • Every started operation has an owner that observes its outcome after the caller's wait ends; no fire-and-forget on request paths.
  • Every TaskCompletionSource wait is registered before its trigger and bounded by a timeout and the caller's token.
  • Every CancellationTokenSource with CancelAfter or a linked parent is disposed; no WhenAny(operation, Task.Delay(...)) helpers remain.
  • Retried writes carry an operation key created once per user intent; stages share one deadline.
  • Each ValueTask is awaited once; a custom source exists only where an allocation measurement justified it.
  • Runtime async is tried per project on .NET 11 with the classic build kept as the control, under the same tests and load.

Further reading

Written by

Marcin Borecki

Software architect and entrepreneur, Microsoft Regional Director since 2014, with CloudExpert since 2011. I help teams design, measure and ship .NET, Azure, data and AI systems that hold up in production.

Facing this problem in your own system?

I run focused investigations and architecture reviews, and take hands-on roles as an accountable chief architect.

See how I can help