A Recovery Runbook Should Refuse the Wrong Next Step

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

A Recovery Runbook Should Refuse the Wrong Next Step

Dates and technology context

This is a retrospective editorial timeline. Topic context: 15 January 2026. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

Actual recovery exercise, decoy/epoch/replay/dependency cases and .NET11 harness remain12Nov2026 evidence.

A runbook can be perfectly readable and still be unsafe to execute. It may contain the right commands in an order that assumes the operator selected the right database, copied the right backup and restored every dependency. If those assumptions are wrong, a sequence of successful commands can move the application further from recovery while making the incident report look more complete.

The useful test of a runbook is therefore not only whether its happy path works. It is whether the procedure stops at a plausible wrong state, explains why it stopped and identifies the evidence required before the next consequential action. A runbook that accepts every reachable endpoint is not an operational contract; it is a list of optimistic instructions.

This article turns that idea into an executed SQL Server recovery exercise. The same small fixture is used for a backup/restore study and an application-cutover study, but the focus here is different. We deliberately supply a missing backup path, an old application epoch, an unreconciled target, a conflicting replay, a decoy database and the wrong external dependency content. Each must be rejected for a specific reason.

The exercise ran on 12 November 2026 against two disposable SQL Server 2022 instances, build 16.0.4315.2. The client used .NET 11.0.0 and Microsoft.Data.SqlClient 5.1.6. The runbook and its executable checks are local laboratory artifacts, not instructions that have been applied to an organization's production recovery system.

Define what the operator is allowed to conclude

01 / See the idea

The next step needs evidence

Enlarge diagram

The procedure's end condition is concrete. A target database must belong to the intended application, contain the expected transaction history, expose the expected application epoch and agree with an external template's content hash. A new transaction must then be written to the selected target and verified independently.

This end condition is stronger than database connectivity and narrower than complete disaster recovery. It does not claim that a web deployment, DNS, identity provider, encrypted backup key or cloud storage account has recovered. Those dependencies are outside the fixture. The runbook should state that boundary before the operator starts, not after a missing requirement causes confusion.

The initial database has one thousand transaction identifiers with deterministic amounts and hashes. The expected sum is known independently. A full backup captures that snapshot. A later transaction creates a deliberate gap between source and target. The external dependency is a synthetic text file whose expected hash lives in the database.

These inputs are small and predictable because the purpose is to test decision gates. The exercise does not need a large database to establish that a wrong identity marker must block readiness. It does not need thousands of failed retries to establish that the same transaction ID with a different payload must be rejected.

The operator's permitted conclusions follow the evidence. A backup hash identifies a file. VERIFYONLY establishes its documented verification result. RESTORE establishes that the target database was created from it. Exact ledger comparison establishes the selected data contract. The application probe establishes the declared dependency and epoch conditions. None silently substitutes for the others.

This separation makes the procedure reviewable. A colleague can ask which observation authorizes a step and find a specific answer. If the answer is merely that the preceding command returned zero, the runbook has probably hidden an assumption that deserves its own check.

Reserve names and resources before the first mutation

The wrapper uses two unique container names and checks that neither name is already occupied. It creates each container with a laboratory label, a two-CPU limit, a four-GiB memory limit and an ephemeral host port bound only to loopback. The database names are fixed within those exclusively owned instances.

This is an ownership guard. A cleanup command is safe here because it targets containers the wrapper created and verifies the expected label before removal. It is not a generic command to remove every SQL container or every database with a familiar prefix. The procedure preserves unrelated resources by design.

The source and target connection strings are generated only after the wrapper discovers the mapped ports. A random password exists in memory and child environment variables. The report redacts password and connection-string values rather than saving them as part of reproducibility. Reproduction needs the mechanism and versioned inputs, not a reusable secret.

The wrapper also refuses a stale fixture directory containing a previous preparation report. This prevents a new run from accidentally treating old evidence as the result of the current commands. The check run and dedicated measured run use separate directories, and their artifacts remain distinguishable.

These guards are not substitutes for authorization in a real recovery environment. They are the execution boundary of this authorized local exercise. A production runbook would need its own resource identifiers and change controls. The important transferable idea is to bind commands and cleanup to explicit ownership rather than to a loosely matched name.

Before moving on, the operator should be able to answer which instances are owned, where the backup will be written, which database may be created and which resources the finally block will remove. If those answers are ambiguous, the procedure is not yet concrete enough to run safely.

Gate the source artifact before interpreting its contents

The first deliberately wrong step points RESTORE VERIFYONLY at a file that does not exist. SQL Server returns error 3201 in the recorded execution. The harness records negative_missing_backup with rejected true and does not treat that result as a recoverable success.

The missing path is a controlled negative case. It tests whether the procedure distinguishes absence from a valid artifact. It does not test every storage permission issue, encrypted-backup problem or corrupted media set. The failure evidence should retain the actual error number rather than relabel it as a broader class of disaster.

The valid backup is then verified with CHECKSUM. Its file size and SHA-256 are retained by the wrapper, and the actual verification command is recorded. The hash ties the copied artifact to the report; SQL Server's command determines whether that artifact passes the chosen engine check.

VERIFYONLY is deliberately followed by an actual restore. Microsoft documents that verification does not restore the database and does not verify all internal data structure. The runbook therefore uses it as an early gate, not as the final recovery assertion. Microsoft: RESTORE VERIFYONLY

The operational rule is simple: do not skip from artifact presence to application readiness. Record the identity of the artifact, inspect it with the intended engine operation, then exercise the restore path. If a gate fails, preserve the failure and identify the missing prerequisite rather than continuing because later steps might happen to work.

This rule also improves incident communication. “The backup is present” and “the backup passed verification” are different updates. “The database restored” is another update. A team can make better decisions when those states are reported separately instead of compressed into an ambiguous statement that recovery is nearly done.

Restore into an explicit target rather than a remembered name

The executable obtains logical file names through FILELISTONLY and constructs MOVE clauses into new target paths. It restores RecoveryTarget on the second instance. It does not overwrite the source database or use a broad replacement instruction against an existing target.

The logical-file mapping is evidence of the actual backup contents. The procedure does not assume that a database name determines its internal file names. That matters when backups come from environments with historical naming, additional files or layouts that differ from the destination.

The target paths belong to the new container. Their isolation makes the exercise reversible: the source remains available, the target is distinct and cleanup removes only the owned containers. This does not establish that a production overwrite is safe; it demonstrates a recovery rehearsal without requiring one.

Microsoft's restore guidance describes moving a database to another location or instance and mapping logical files to destination paths. The runbook follows that mechanism while keeping the fixture's scope explicit. Microsoft: restore a database to a new location

After the command succeeds, the harness reads every ledger row and compares identifiers, amounts and hashes with the deterministic expectation. It records the count and independent sum. That is the evidence authorizing the next application-state check.

An operator should not infer that the correct database was restored merely because a database of the expected name now exists. The restore command, artifact identity, file mapping and application identity must agree. The later decoy test makes that requirement executable rather than leaving it as prose advice.

Refuse the database that answers but belongs elsewhere

The shared exercise creates a decoy database on the target instance. It contains an identity table and a ledger-shaped table, but its marker says WrongWorkload. The application probe connects successfully and can execute its queries. It then returns identity_mismatch.

This is a useful negative case because many weak readiness checks would pass it. A successful TCP connection, SQL login or SELECT 1 does not establish that the operator selected the intended application database. Even recognizable table names can be insufficient when environments share schema conventions.

The probe's identity check is deliberately ordered before readiness is granted. The wrong database is not repaired by changing its marker to match the expected value. In the runbook, the correct response is to stop and select the intended target. Editing evidence until it passes would defeat the purpose of the gate.

The decoy case does not exercise tenant isolation or authorization. The laboratory client has administrative access inside its own disposable instances. A production identity check should complement, not replace, permissions that prevent access to unrelated data. The present test establishes a configuration mistake detector for this controlled scope.

The expected outcome is recorded as a specific reason. That reason is useful to the next operator because it distinguishes wrong identity from missing dependency, stale epoch or missing transactions. A generic false result would still block the procedure but provide less information about the next safe action.

Good runbooks make the error path actionable without making it automatic. A failed identity check should explain what was observed and which input must be reviewed. It should not silently search other databases or guess a replacement target, because that would expand the procedure's authority beyond its declared resource boundary.

Refuse stale authority and unreconciled history separately

The controlled cutover changes the source's application epoch to zero and the target's epoch to two. A probe expecting epoch two against the old source returns epoch_mismatch. This detects a stale endpoint selection by the cooperating application adapter.

The marker is not an enforced database fence. The source remains writable to a client that ignores it. The runbook must say so, because otherwise an operator could infer that changing the marker prevents split-brain writes. The fixture has one controlled writer and does not implement a multi-client authority protocol.

The target presents a different failure. It has the expected epoch but only the thousand transactions from the backup. The source has committed transaction 1001 after the backup, so the target probe returns ledger_not_reconciled. Routing and history are therefore separate gates: the intended destination can still contain an insufficient recovery point.

The missing transaction has an ambiguous acknowledgement at the original client. The SQL batch commits and then delays before returning; the one-second command timeout occurs during that post-commit delay. A separate source read confirms the transaction exists. The runbook treats the timeout as uncertain knowledge, not as proof of rollback.

The next permitted step is reconciliation using the known operation identity and payload. It is not blind retry with a newly generated identifier. A new identifier could create a duplicate business effect if the original write already committed. The correct action depends on whether the target contains the same intended effect.

This distinction is one of the most valuable things a runbook can teach under pressure. “Retry the failed request” is underspecified when the failure concerns acknowledgement. The operator needs a stable identity, a source of truth for the intended payload and a rule for distinguishing absent, duplicate and conflicting state.

Make replay reject a plausible but wrong request

03 / Watch the mechanism

An existing ID is not always success

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

The replay implementation uses a primary key on transaction identity and compares the stored payload hash within a serializable transaction. If the identifier is absent, it inserts the intended row. If the identifier and hash already agree, it records a duplicate no-op. If the hash differs, it raises a conflict.

The exercise runs all three branches. Transaction 1001 with amount 7007 is applied to the target. Repeating that same request changes nothing. Reusing identifier 1001 with amount 999 is rejected as payload_conflict.

The conflict test is important because a superficially convenient runbook might treat any existing identifier as success. That would hide a disagreement about the operation's content. Conversely, a procedure that always inserts again would fail to preserve the identity boundary and could duplicate effects when the acknowledgement was lost.

The hash itself is not magic. The fixture defines a deterministic representation of two integers. A real payload requires a versioned canonicalization rule and a clear definition of which fields determine equivalence. The test does not establish that arbitrary serialized documents can be compared safely by copying this tiny representation.

The runbook's response to conflict is to stop and reconcile the disagreement. It must not overwrite the existing amount merely to make the replay complete. A conflicting history is a decision problem, and the available authority may differ from a normal duplicate retry.

After the accepted replay, the harness checks the target ledger exactly through identifier 1001 and verifies the independent sum. This turns replay from a command that was issued into an effect that was observed. Only that evidence authorizes the next readiness check.

Treat an external file as a consistency dependency

The restored database stores the hash of a synthetic template. Initially the target file is absent, and the probe reports dependency_missing. After the correct file is copied, the same probe passes. Later the file is replaced with incorrect content, and the probe reports dependency_hash_mismatch.

Those are two different wrong states. The first requires recovering missing content. The second requires establishing the correct content identity. A procedure that only checks File.Exists would accept the second state and could let the application use a template incompatible with the restored database.

The fixture uses a local file to model the dependency. It does not claim cloud blob recovery or remote configuration propagation. Its value is the explicit agreement rule between a database reference and external bytes. That rule can be tested before the application performs its next meaningful operation.

An operator should preserve the expected hash as part of the recovery evidence. Changing the database reference to match whatever file happens to be present would again turn the validation into a self-fulfilling check. The intended version comes from the recovery contract, not from the easiest artifact to find.

The runbook therefore restores the known source template, repeats the probe and only then proceeds. This is a small example of dependency closure: the application is not ready merely because its central database is online. The required external state must satisfy the same operation contract.

The test also keeps business content out of generic diagnostic logs. The synthetic template is harmless, but the pattern does not require dumping a real document into telemetry to compare its identity. A production procedure should choose appropriate artifact handling and access boundaries for its actual data.

Record the negative outcomes as acceptance evidence

02 / A concrete example

Negative tests make stop conditions concrete

Enlarge diagram

The measured sequence produced these distinct rejection results:

Deliberate wrong state	Observed result	Permitted next action
Missing backup path	SQL error 3201	Select and verify the owned backup artifact
Old source selected after epoch change	epoch_mismatch	Review the selected application endpoint
Restored target missing transaction 1001	ledger_not_reconciled	Reconcile the known operation identity
Same transaction ID, different amount	payload_conflict	Stop and resolve the conflicting intent
Decoy database selected	identity_mismatch	Select the intended application database
External template absent	dependency_missing	Restore the required template
External template has wrong bytes	dependency_hash_mismatch	Restore the expected content version

These are observed results, not hypothetical error messages invented for the runbook. The raw JSON records each probe and negative test alongside the successful steps. The valid path then reaches a passing target probe and a newly verified transaction 1002.

The first correct post-cutover write is later than the endpoint switch. In the dedicated measured sequence, the interval is 262.1158 milliseconds, but it includes the deliberate negative tests. That number is a property of this scripted drill, not a recovery objective or an estimate of operator reaction time.

The backup command, VERIFYONLY and actual restore are separately timed. Their measured durations are 82.6249, 28.6460 and 478.0350 milliseconds respectively. They belong to a tiny local fixture. Publishing those numbers without the row count, container placement and excluded work would make them misleading.

The stronger result is the decision coverage. The runbook demonstrates that it can reject several reachable-but-wrong states and eventually accept the intended one. It does not prove every failure mode, but it makes its selected gates executable and reviewable.

Close the exercise without erasing the evidence

04 / See the idea

Cleanup and evidence have different lifetimes

Enlarge diagram

The wrapper removes both owned containers in finally and records whether they are gone. The local synthetic backup, source reports, probe outcomes and command receipts remain available for review. Passwords and connection strings do not remain in those reports.

Cleanup is a separate completion condition. A successful restore test that leaves a database service running indefinitely has not fulfilled the bounded laboratory contract. Conversely, deleting every artifact immediately would make the result difficult to audit. The procedure preserves evidence while closing executable resources.

The original check run and dedicated measured run remain separate. Reusing a report is not presented as another execution, and a later chart reads from the saved result rather than triggering another recovery merely to refresh a number. This keeps the provenance understandable and avoids unnecessary work on the shared host.

The runbook also describes how to reproduce the fixture with pinned versions and an isolated output directory. The instructions are not a promise that another machine will produce identical timings. They provide the same logical inputs and checks so that differences can be investigated rather than hidden.

An operational procedure should end with the state of both the application and the tools used to recover it. Which endpoint is selected? Which transaction was last verified? Which dependency version is active? Which temporary resources remain? Which evidence is retained? Those answers are more useful than a final line that simply says completed.

Rehearse the handoff between people, not just commands

05 / A concrete example

The handoff names the verified state

Enlarge diagram

The laboratory is executed by one harness, but a real recovery can cross several owners. One person obtains the artifact, another restores the database and another decides whether application traffic may resume. The handoff should carry the evidence needed for the next decision, not merely a message that the previous task is done.

For the restore-to-application handoff, useful evidence includes the selected target identity, the backup hash, the exact validation boundary and the dependency version that passed. For the cutover handoff, include the ambiguous operation identity, replay outcome and first verified target write. These records let the next owner check the relevant promise without reconstructing the entire incident from chat history.

The negative tests provide useful rehearsal prompts. Ask the next operator what they would do when the database connects but its marker is wrong, or when replay finds a conflicting payload. The expected response should be a bounded review or reconciliation step. If the answer is to bypass the gate because the incident is urgent, the procedure has not established a usable decision boundary.

This exercise does not measure human response time or certify an operations team. It supplies concrete states and observations that can support such a rehearsal. The important distinction is between having a document available and having a shared understanding of what its success and stop conditions mean.

Improve the runbook at the first unsupported assumption

This exercise does not implement database-enforced fencing, recovery after permanent source loss, encrypted-backup key restoration or a production identity system. If one of those becomes part of the claimed scenario, the runbook needs a new prerequisite and an executable check at that boundary. Adding a paragraph saying to be careful would not close the gap.

The same principle prevents runbooks from becoming enormous generic checklists. Add detail where an unsupported assumption controls the next action. A procedure for this small unencrypted fixture does not become more reliable by listing every SQL Server feature. It becomes more reliable by proving that its chosen artifact, target, history and dependency are the intended ones.

The final standard is simple to state and demanding to execute: a runbook should help an operator know when not to proceed. Its happy path should produce a correct operation, and its negative path should preserve enough evidence to identify the next safe decision.

In this laboratory, that standard is met for the declared scope. Real SQL backup and restore commands run on isolated instances; exact transactions are checked; ambiguous acknowledgement is reconciled by identity; wrong database, epoch, history and dependency states are rejected; and the owned services are closed. The result is an operational procedure whose claims can be inspected, not just a document whose commands look familiar.

Source code and execution evidence

Download the complete lab, source record and raw execution results. This package supports the specific execution claims in this article. It is not a production certification.