After the Restart: Durable Replay and External Effects in a .NET Agent

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

After the Restart: Durable Replay and External Effects in a .NET Agent

Dates and technology context

This is a retrospective editorial timeline. Topic context: 25 June 2026. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

Child exit17, reopened databases, one-effect/attempt observations, Agents.AI1.28.0 and timing remain12Nov2026 execution evidence. Preview label before3Apr2026.

Prepared on 12 November 2026. This reference implements an application-owned SQLite workflow ledger and a separate local receiver database. It tests a controlled process exit and reopen. It does not run Azure Durable Functions, simulate power loss or claim universal exactly-once execution.

The dangerous moment in a durable workflow is often the gap between an effect and the evidence that the caller has recorded it. The receiver may have committed the action. The caller may still report that it is executing. Restarting the caller restores its local state, but does not automatically reveal what happened elsewhere.

An agent does not remove this gap by explaining its intent more clearly. A model can propose an action, and a framework can arrange a tool call, while the application still needs to distinguish requested, attempted, confirmed and unknown outcomes.

This laboratory makes the gap concrete. A child process claims a durable job, writes one effect to a separate receiver database, and exits before the caller records completion. The parent opens the saved databases and reconciles the operation. The test requires one receiver effect and one execution attempt, even when reconciliation and event delivery are repeated.

The result is a narrow recovery mechanism, not a complete workflow engine. That distinction matters. The article uses replay to mean re-entering an application-owned operation with a stable identity and consulting its recorded state. It does not claim deterministic replay of arbitrary code, automatic re-execution of every activity, or the guarantees of a cloud orchestration service that was not used.

Restore the obligation before deciding the next action

The local job record contains an event identifier, object identifier, object version, expiry, payload, workflow version, state and attempt count. The receiver record contains the operation identifier and payload it accepted.

These records answer different questions. The job record describes what the caller intended and which transition it made. The receiver record establishes whether the controlled destination recorded the effect. Neither record can be replaced by a sentence in the conversation transcript.

After a restart, the first task is to load the obligation that survived. Was the job merely pending? Had execution been claimed? Was a result already confirmed? Was the outcome left unknown? The answer determines whether the application should execute, return a saved terminal state, reconcile or stop for an unresolved condition.

A blind instruction to retry everything that was not marked successful is inadequate. An executing job may already have caused an effect. Repeating it without a stable destination identity can create a second one.

The opposite shortcut is also wrong. Marking every interrupted job as failed may leave the user believing an action did not happen when it did. The application needs a representation of uncertainty that remains useful until evidence resolves it.

This is why the ledger is more than progress reporting. It preserves the question the system must answer after interruption.

Use two databases to expose the boundary

01 / See the idea

Two commits leave a real gap

Enlarge diagram

The fixture uses workflow.db for objects and jobs and receiver.db for effects. Both are real SQLite files created inside an owned temporary directory. Their separation is intentional: a local transaction in the workflow database cannot be presented as a transaction covering the receiver's commit.

The workflow store uses transactions for admission and execution claims. The receiver uses its own transaction and a primary key on the operation identifier. Microsoft.Data.Sqlite's transaction documentation explains the local atomicity and default isolation used here. It does not grant atomicity across two independent database files or a real external service. Microsoft.Data.Sqlite transactions.

The receiver's insertion is idempotent for the exact operation identity and payload. If the same identity already exists with a different payload, the fixture refuses the conflict. Reusing an operation identifier cannot silently redefine what that operation meant.

This receiver contract is stronger than many arbitrary external APIs. That strength is a declared fixture assumption, not a property inferred from HTTP or from the agent framework. A real integration must establish what its destination supports and how a caller can query the authoritative outcome.

The two-file design is therefore not an attempt to imitate all distributed systems. It isolates one important boundary with enough realism to execute commits, close processes and inspect saved state.

Claim execution before sending the effect

The application first admits an event as Pending if its scope, object version, expiry and workflow version are acceptable. Before an effect can be sent, Claim reloads the current object and rechecks the relevant conditions inside a transaction.

If permission was revoked, the job becomes Denied. If the object version changed, it becomes Superseded. If it expired while waiting, it becomes Expired. A rejected claim does not increment the attempt count.

For an allowed claim, the state becomes Executing and the attempt count increments. That transaction commits before ReceiverCommit is called. If the process then stops, the local record says that an attempt was authorized and may have reached the receiver.

The ordering is deliberate. Recording success before the effect would invent evidence. Recording nothing until after the effect would erase the identity needed for recovery. Executing is an obligation to determine the outcome, not a claim that the effect is already known.

The fixture also prevents another job for the same object from entering an active execution state while an Executing or Unknown operation remains. This is a conservative overlap policy. It may reduce availability, but it avoids silently treating an unresolved effect as harmless background work.

A production system could choose another policy with a stronger domain argument. The present article measures and tests the policy actually implemented rather than implying that one rule fits every workflow.

The child exits after the receiver commits

02 / Watch the mechanism

Exit after the effect, before its acknowledgement

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

The most important test starts a separate .NET process using the same compiled assembly. The child opens the fixture, claims the job and commits the receiver effect. It then terminates with a deliberate exit code before recording a caller-side acknowledgement:

if (store.Claim("lost", "blue", 100) != "Executing")
    throw new Exception("Child claim failed");
store.ReceiverCommit("lost");
Environment.Exit(17);

The receiver method completes its transaction and disposes its connection before the exit statement. The child is not interrupted in the middle of a database write. That is important to the interpretation: this is a process-exit test at a selected boundary, not a power-cut experiment or a test of every filesystem durability condition.

The parent waits for the child to exit and verifies the expected exit code. It then creates a new store instance over the saved files. The caller still says Executing, while the receiver contains one effect.

This is stronger evidence than setting a variable called crashed inside one continuous method. A real child process ended, and the next observer obtains state from the database files. It is also narrower than claiming survival of arbitrary machine failure. Both descriptions should remain visible.

The test uses synthetic payloads and no real customer action. Its only external effect is a row in the separately owned receiver fixture.

Reconciliation asks the destination instead of repeating the story

Reconcile loads the job's recorded state and payload. If the operation is Executing or Unknown, it reads the receiver by the same stable identifier. A matching receiver payload permits the caller to transition to Completed. An absent or nonmatching result leaves Unknown.

The decisive observation is not that the agent remembers issuing the command. It is that the receiver exposes the matching committed effect. The model is not invoked during reconciliation.

This separation prevents a subtle failure mode: an agent could generate a new plausible command after restart and lose the relationship to the original attempt. Even if the new command looks semantically similar, the destination may treat it as another operation. The application must preserve identity independently of conversational reconstruction.

Repeated reconciliation of a Completed job returns the saved terminal state. It does not issue another receiver write or increment the attempt count. Repeated delivery of the same event is likewise classified as Duplicate rather than admitted as a new job.

The receiver fixture's stable identity makes this behavior possible. If a real destination cannot reliably report an operation's state, the application needs a narrower automation promise and a defined human reconciliation path. It should not fabricate certainty from a timeout.

The absence of a receiver record is handled conservatively here. The fixture does not infer that no effect can ever appear later and automatically retry. It keeps the unresolved operation active until an explicit policy can resolve that question.

Unknown is a state with consequences

04 / See the idea

Unknown must change what can happen next

Enlarge diagram

An Unknown state is useful only if it changes what the system may do next. In this reference, it blocks another execution for the same object. The application cannot keep producing effects while its earlier obligation remains unresolved.

The correctness test creates an executing operation without a receiver effect and calls reconciliation. The result is Unknown. It then changes the source object to a newer version and admits a new event. When execution is claimed for that new job, the old uncertain operation causes the claim to return Overlap, with no new attempt recorded.

This behavior is intentionally strict. It demonstrates a possible operating cost of preserving uncertainty: new work can be delayed. That cost should not be hidden behind the comforting word reliable.

A more complete system might allow independent operations to proceed, support compensation or assign the unresolved case to an operator. Those choices depend on the meaning of the effects. They cannot be chosen merely because a general retry policy makes the code simpler.

The fixture does not include an administrative resolution interface. It records the condition and refuses to invent a safe next action. That is an honest limit of this demonstration, not an operational recommendation to leave Unknown jobs unattended forever.

The article's companion on incoming events explains how this execution boundary relates to pending work and scheduled overlap. Here the focus is the obligation that survives once an attempt has begun.

Keep model activity outside the recovery decision

The shared laboratory includes a real ChatClientAgent and FunctionInvokingChatClient, using Microsoft.Agents.AI 1.28.0 and Microsoft.Extensions.AI 11.0.0. A scripted provider exercises a read-only tool in a separate session-state test. Agent Framework package.

The recovery path does not call that provider. It reads the job and receiver records directly. This is a deliberate separation between interpreting a request and establishing a durable effect's state.

Calling a model after restart would not make the receiver's outcome more authoritative. The model could help explain a record to a person, but the record itself should remain the evidence. A generated summary is not a transaction log.

The same principle applies if the system later gains dynamic planning. A plan can select which capability to request, while the executor preserves the identity and state required to recover that capability's effect. Changing the planner should not redefine whether an already committed operation happened.

This article therefore does not claim to implement a durable agent framework engine. There is no replay of arbitrary framework callbacks, no persisted model conversation and no automatic restoration of every activity. The durable mechanism is the explicitly implemented application ledger and reconciliation routine.

That scope is enough to demonstrate a difficult boundary without borrowing guarantees from a platform whose runtime was never exercised.

Version the workflow as well as the object

The event envelope includes a workflow version. This fixture supports version one and rejects another version before execution. That is a simple compatibility gate, not a complete migration system.

The distinction from object version is important. Object version describes the business record the event concerns. Workflow version describes the interpretation and execution contract used to handle the event. Either can change independently.

If new code reads an old job, the application needs to know whether it still understands the saved state. Silently treating an unfamiliar state or payload as current can be more dangerous than refusing to proceed.

The fixture's conservative answer is to reject unsupported workflow versions. A production migration could instead translate state, route older jobs to compatible code or preserve version-specific handlers. Each option needs its own evidence about the transitions it changes.

This is another reason not to call database reopen the complete recovery story. Reopening bytes establishes that the files can be read. It does not establish that a new implementation interprets them correctly. The contract needs a version and a decision about compatibility.

The recorded test checks the implemented refusal, not a migration that has not been written.

The executed recovery observations

03 / See the idea

Reconciliation preserves one effect

Enlarge diagram

The shared correctness run completed twenty-seven named checks on .NET 10.0.15. The recovery-specific observations are particularly compact:

Boundary	Observed state
Child after receiver commit	Exited with the deliberate code 17
Caller opened afterward	Executing
Receiver opened afterward	One committed effect
First reconciliation	Completed
Repeated reconciliation	Completed, still one effect
Recorded attempts	One
Missing receiver evidence	Unknown, not invented failure or success

The parent also verifies duplicate delivery after restart and the blocking behavior of an unresolved older effect. These checks operate on actual database files and a real child process, with a bounded timeout for the child.

The evidence is retained in correctness.json. It is not a production incident report. No external payment, customer account or remote service was used. The databases are removed from the owned fixture directory after the parent completes, and the runner refuses to reuse an existing directory as its test workspace.

That cleanup is part of the reference's operating boundary. A test that leaves ambiguous processes or files behind would make its own lifecycle harder to assess. It would also complicate later runs by mixing old state with a supposedly fresh fixture.

The replay timing is not recovery latency

05 / See the idea

Completed replay cost is not recovery latency

Enlarge diagram

The integration owner executed three fresh processes for a small measurement. Each prepared one completed operation, then performed one hundred duplicate admissions and one hundred completed reconciliation reads.

The duplicate batches took 30.0120, 29.3391 and 29.5469 milliseconds. The completed reconciliation batches took 26.9909, 26.8502 and 27.0349 milliseconds. Their medians are 29.5469 and 26.9909 milliseconds per hundred calls.

These calls open local SQLite connections and inspect already established state. The reconciliation timing does not include the lost-response scenario, a process restart or waiting for an unavailable destination. It is a completed-state replay path, not a recovery-time objective.

There is no agent or model call in the measured interval. No broker transport, network delay or new receiver effect is included. The batch numbers should not appear in a presentation as the latency of a durable agent.

Only three process repetitions were run on a shared developer host. Task-owned measurements were serialized, while unrelated host activity was not controlled. The raw results preserve that limitation rather than presenting the medians as a broad statistical guarantee.

The useful evidence is that the local replay paths executed correctly under the stated workload and retained one effect. The implementation's more consequential recovery property comes from the controlled failure test, not from the stopwatch.

Replay has several meanings that should not be mixed

The word replay can describe redelivery of an input, repetition of an API call, restoration of an operation record or deterministic re-execution of an orchestration history. Those mechanisms have different requirements.

In this reference, redelivering an envelope checks its persisted identity. Calling reconciliation again checks the saved job and, when needed, the receiver's record. Neither operation reruns a model plan or re-executes arbitrary code from a captured history.

That distinction is important when comparing this application-owned mechanism with a durable orchestration platform. A platform may define restrictions on deterministic code, persistence of activity results and behavior during upgrades. The present lab does not implement or test those platform contracts.

The article therefore makes no claim that a model call can be placed inside any replayed function safely. A future orchestration integration would need to identify nondeterministic activities and preserve their results according to the selected engine's rules. Its tests would have to show what actually runs again after recovery.

The local ledger still contributes a useful principle: the identity and outcome of an external effect should not depend on regenerating a conversation. Even a capable engine cannot infer a destination's business state without an appropriate destination contract.

Choosing the smaller mechanism is not an argument against a workflow platform. It makes the current evidence precise enough to compare with the guarantees a platform would add. The right choice depends on the application's required operations, operating capacity and failure model.

Recovery needs an explanation an operator can use

An Unknown record is a beginning for an operating decision, not the end of the recovery design. Someone needs to know which operation is uncertain, which destination identity was used, what payload was authorized and which evidence has already been checked.

The fixture retains enough information to perform its local reconciliation. A production operator would need a suitable interface or procedure, access to the relevant records and authority to choose the next action. Those capabilities are not implemented by adding another state name to an enum.

The explanation should separate three things: what definitely happened, what the system attempted, and what remains unestablished. A generated summary can help present that information, but every consequential statement should resolve to a stored observation or an explicitly labelled inference.

For example, the controlled lost-response case can say that the receiver record exists with the expected payload and that the caller has now marked the job Completed. The absent-receiver case can say only that the current query did not find a matching effect. It should not automatically convert absence at one observation into proof that no delayed effect can arrive.

This is why the fixture holds the uncertain state. A real integration may offer stronger evidence of non-execution and therefore support safe retry. That stronger guarantee must come from the destination and its protocol, not from the caller's preference to finish.

The next operating step should be designed with the same care as the happy path. Otherwise the system is durable enough to preserve uncertainty but not useful enough to help anyone resolve it.

The failure point is part of the result

The child-process test chooses a precise point: the receiver commit has returned, the relevant connections have been disposed, and caller-side completion has not been persisted. This makes the expected records clear and the experiment repeatable.

A different failure point would answer a different question. Termination before the receiver write should leave no effect. Termination during a transaction would test the database's crash handling under the selected storage conditions. Loss of the machine or storage device would introduce another layer again.

The current successful result cannot be stretched to cover all of those conditions. A good recovery report identifies the injected failure point as carefully as it identifies the runtime version.

The parent also records the child exit code and enforces a bounded wait. A hung child is not treated as a successful crash simulation, and a process that exits before reaching the intended boundary cannot establish the same claim. The test's observed caller and receiver states supply the evidence that the selected boundary was actually crossed.

This discipline prevents a common form of demonstration inflation: a scenario is named crash recovery, one controlled restart passes, and the label gradually becomes a claim of complete durability. The experiment is more useful when its name and conclusion remain attached to the event it actually produced.

Prefer an explicit limited guarantee to an accidental broad one

The strongest objection is that this reference implements only a small portion of what a production durable workflow needs. That is true. It does not test arbitrary process termination during a transaction, power loss, storage corruption, multiple hosts, malicious administrators or a remote destination with weaker semantics.

It also does not automatically resolve every Unknown state. That requires a domain-specific policy and operational ownership. The fixture preserves the uncertainty so that the missing policy cannot be mistaken for a successful retry.

Those limits are not reasons to describe the example as a simulation when it uses real transactions and a real exited process. They are reasons to name exactly what the execution establishes.

The new value is the visible separation among local intent, attempted execution, destination evidence and caller confirmation. A developer can inspect the state before and after the gap, repeat the controlled sequence and see why blindly reissuing the action would answer the wrong question.

When adapting the pattern, begin with the destination's actual contract. Can it identify an operation stably? Can it report the effect? What happens if the same identity arrives with a different payload? Which uncertainty can the application resolve automatically, and which requires another authority?

Those questions define the recovery mechanism more honestly than the phrase exactly once. The agent can remain useful for interpreting work, while the application remains responsible for what happened after the tool call crossed the boundary.

Source code and execution evidence

Download the complete lab, source record and raw execution results. This package supports the specific execution claims in this article. It is not a production certification.