> ## Content Index > Fetch the complete content index at: https://devinside.cloud/llms.txt > Use this file to discover other available public pages before exploring further. # A Verified Backup Is Not Yet a Recovered Application - URL: https://devinside.cloud/a-verified-backup-is-not-yet-a-recovered-application/ - Published: 2026-06-04T14:45:00.000Z - Updated: 2026-11-12T17:08:30.000Z - Description: A real checksummed backup is verified, restored and checked against exact transactions, then the application refuses readiness until its external dependency agrees. - Author: Marcin Borecki - Tags: Engineering, Data and production, #eng19, #devinside-review START WITH THE MECHANISM Interactive companion and laboratory update: 12 November 2026\. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates. THE FULL INVESTIGATION ## A Verified Backup Is Not Yet a Recovered Application Editorial context: 4 June 2026 · Prepared and updated: 12 November 2026 Dates and technology context This is a retrospective editorial timeline. Topic context: 4 June 2026\. First publicly available: 12 November 2026\. Current text, visuals and laboratory revisions were prepared on 12 November 2026. The 1,000-transaction fixture, checksummed backup, VERIFYONLY, restored rows, missing-template probe, timings and .NET11 client were executed 2026-11-12. A backup job finishes successfully. A verification command accepts the file. The database restores without error. The application still cannot perform the operation that matters. None of those observations is necessarily inconsistent. They describe different boundaries, and recovery fails when the team treats one boundary as proof of all the others. This article follows a real SQL Server recovery exercise with synthetic data. A source database contains one thousand known transactions and a reference to a small external template. We create a full backup with checksums, copy it to a second SQL Server container, run RESTORE VERIFYONLY, restore into a differently named database with explicit file moves, and compare every transaction with an independent expectation. The first application probe then fails because the template is missing. Only after the external dependency is restored does the probe report readiness. That final transition is the point of the experiment. The database backup is useful and internally consistent, but it is not a complete package of application responsibility. The execution took place on 12 November 2026\. Both SQL Server 2022 Developer Edition instances reported build 16.0.4315.2\. The client ran on .NET 11.0.0, built with SDK 11.0.100 and Microsoft.Data.SqlClient 5.1.6\. The instances were disposable, loopback-only containers with separate data directories. No production database, external storage account or existing service was touched. ## Define the recovery claim before collecting the backup 01 / See the idea ### Recovery is a chain of different proofs ![Backup verification, actual restore and application readiness are separate proof boundaries.](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v1.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v1.svg) The fixture restores known transactions and then checks an external template. The missing template can fail readiness even when the database is correct. Interpretation and evidence - Define the recovery operation. - Keep engine and application checks separate. - Include required external state. Actual fixture results where numeric; conceptual flow otherwise. Motion speed is not measured timing. The database fixture is intentionally small enough to inspect exactly. Transaction identifiers run from one through one thousand. Each transaction's amount is seven times its identifier. Each row also stores a hash of the identifier and amount representation. The expected amount sum is 3,503,500, calculated independently from the sequence rather than copied from a database query. A separate identity table declares which application owns the database and records an application epoch. Another table records the expected hash of an external template. The template itself is a synthetic local text file containing a fixed phrase. It is not stored in the database backup. These choices create several independent recovery questions. Did the restore produce the intended database? Are the expected transactions present with the right values? Does the application expect this epoch? Is the external template present, and does its content match the database's reference? A single successful connection cannot answer all of them. The exact-row check matters because count and sum can agree even when individual records are wrong. Two changed amounts can cancel each other in a total. Duplicate or substituted identifiers can preserve a row count. The harness therefore reads the ledger in identifier order and compares each identifier, amount and payload hash before checking the independent aggregate. The aggregate still has value. It provides a compact cross-check and a useful operator-visible summary. The mistake would be treating it as the only evidence of correctness. A good recovery proof combines an exact contract where feasible with summaries that are easy to review. For a large production database, comparing every business row may be impractical within the recovery window. That does not make a shallow health check sufficient. It means the team must choose validation coverage deliberately, based on the invariants and operations whose failure would invalidate readiness. This small fixture demonstrates the reasoning without claiming to solve every production validation strategy. ## Make the backup command's scope explicit The executed source database uses the simple recovery model. The exercise creates a full database backup, not a transaction-log backup chain or a point-in-time recovery sequence. The command includes COPY\_ONLY and CHECKSUM, and writes to an owned path inside the source container. ```sql BACKUP DATABASE RecoverySource TO DISK=N'/var/opt/mssql/data/recovery.bak' WITH INIT,COPY_ONLY,CHECKSUM; ``` INIT is acceptable here because the wrapper creates a fresh container and the destination is exclusively owned by the exercise. It is not a generic instruction to overwrite an organization's existing backup media. A production command needs an explicit media-retention and naming policy rather than inheriting a laboratory path. The backup is created after the initial thousand-row transaction commits. The later cutover exercise deliberately writes another transaction after this backup, but that transaction does not retroactively become part of the backup. The backup's logical recovery point is the state actually captured, not the state that happens to exist on the source when someone later inspects it. Microsoft's BACKUP documentation distinguishes full database backups from log backups and their applicable recovery models. The current experiment uses the former only. [Microsoft: BACKUP](https://learn.microsoft.com/en-us/sql/t-sql/statements/backup-transact-sql?view=sql-server-ver16&ref=devinside.cloud) The wrapper copies the resulting file from the source container to the local evidence directory and then into the target container. It records the file's byte size and SHA-256 hash. This gives the review a concrete artifact identity and a way to distinguish the copied file from another backup with a similar name. The hash is a provenance check for the file, not a replacement for SQL Server's own interpretation of the backup. A hash can establish that two byte sequences are equal while saying nothing about whether those bytes represent a recoverable database. The exercise uses both artifact identity and engine-level validation because they answer different questions. ## Understand what VERIFYONLY does and does not promise The target instance first runs RESTORE VERIFYONLY with CHECKSUM against the copied backup. The command succeeds. That is meaningful evidence that SQL Server can inspect this backup set and perform the checks supported by that operation. ```sql RESTORE VERIFYONLY FROM DISK=N'/var/opt/mssql/data/recovery.bak' WITH CHECKSUM; ``` It is not an actual restore. Microsoft explicitly documents that VERIFYONLY checks backup completeness and readability but does not verify the structure of all data contained in the backup volumes. A passing verification therefore should not be renamed an application recovery test. [Microsoft: RESTORE VERIFYONLY](https://learn.microsoft.com/en-us/sql/t-sql/statements/restore-statements-verifyonly-transact-sql?view=sql-server-ver16&ref=devinside.cloud) The distinction is operationally important. A verification job may run frequently and cheaply enough to provide useful early warning. An actual restore exercises additional responsibilities: target storage paths, database creation, file placement, recovery and subsequent access. Application validation adds another layer after those responsibilities succeed. The laboratory also tries a deliberately missing backup path. SQL Server rejects it with error number 3201 in the recorded run. That negative case checks that the runbook does not silently move forward when the required source artifact cannot be opened. It is not a test of every possible corruption or permission failure. No corruption is injected into the valid backup. The article therefore does not claim that the test proves detection of arbitrary page damage, malicious tampering or every incomplete-transfer pattern. The supported conclusion is narrower: the valid copied file passes the chosen verification, while the known missing source fails visibly. Keeping that boundary clear makes VERIFYONLY more useful, not less. It becomes one explicit gate with a defined consumer instead of a reassuring label expected to cover every later failure mode. The next gate is an actual restore into the isolated target. ## Discover logical files before choosing destination paths 02 / A concrete example ### Discover file identity before restoring elsewhere ![FILELISTONLY yields the source logical names, mapped to distinct RecoveryTarget files.](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v2.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v2.svg) The logical names come from this backup rather than being guessed from the destination database name. The restore creates a new target instead of replacing the source. Interpretation and evidence - Read the backup file list. - Map every relevant file deliberately. - Do not reuse this fixture’s names blindly. Actual fixture results where numeric; conceptual flow otherwise. Motion speed is not measured timing. The backup contains logical file names from the source database. The target database has a different name, and its physical files must be placed in distinct paths. The harness uses RESTORE FILELISTONLY to obtain the logical names and file types, then builds one MOVE clause for each returned file. The resulting command in the recorded execution is: ```sql RESTORE DATABASE RecoveryTarget FROM DISK=N'/var/opt/mssql/data/recovery.bak' WITH CHECKSUM,RECOVERY, MOVE N'RecoverySource' TO N'/var/opt/mssql/data/RecoveryTarget_0.mdf', MOVE N'RecoverySource_log' TO N'/var/opt/mssql/data/RecoveryTarget_1.ldf'; ``` Those logical names are actual values from this fixture's backup. They should not be copied into an unrelated recovery procedure as though every database uses them. The reproducible part is the discovery-and-mapping step. The target paths belong to the disposable target container and are not shared with the source. The command does not use REPLACE to overwrite an existing database. The wrapper creates a new target instance and the harness restores a new database there. This isolates the exercise from unrelated state and avoids making a destructive overwrite an accidental part of the example. Microsoft's restore guidance describes moving a database to a new location or instance and mapping logical files to physical paths. It also identifies prerequisites that are outside this fixture, such as the keys required for encrypted databases. The synthetic database here is not an encrypted-backup recovery test. [Microsoft: restore to a new location](https://learn.microsoft.com/en-us/sql/relational-databases/backup-restore/restore-a-database-to-a-new-location-sql-server?view=sql-server-ver16&ref=devinside.cloud) The actual restore succeeds, but the harness does not stop at the command's exit. It connects to RecoveryTarget on the second endpoint and performs the exact ledger comparison. This checks that the intended target is usable and contains the expected business-shaped state, rather than merely assuming that a successful administrative command implies the correct application dataset. ## Prove the database state against an independent expectation 05 / A concrete example ### Let exact data validate the restored snapshot ![Exact transaction fields and an independent aggregate both validate the restored thousand-row snapshot.](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v5.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v5.svg) The expected values come from the fixture rule rather than a source query certifying itself. The total is a cross-check, not the only evidence. Interpretation and evidence - Compare identity and content. - Use an independent expectation. - State the validation coverage. Actual fixture results where numeric; conceptual flow otherwise. Motion speed is not measured timing. After restoration, the target contains exactly the expected identifiers one through one thousand. Each amount equals identifier times seven, and each stored payload hash matches the independently computed representation. The sum is 3,503,500\. The result is recorded as an exact-ledger verification, not inferred from backup metadata. The check uses the same deterministic input rule as the seed generator, but it does not ask the source database to certify the target. That independence matters because a wrong source query could otherwise be copied into the validation and reproduce its own mistake. The fixture's mathematical definition is the shared contract. This is still a bounded test. It does not inspect every possible database object, permission, trigger, job or integration. It verifies the schema and data used by the selected application operation. Production recovery should enumerate its own additional requirements, including objects or services that live outside a user database backup. The identity marker provides another guard. A database can contain a table named Ledger without belonging to this application. The probe expects the exact application marker and epoch. A later negative case uses a decoy database with a different marker to demonstrate that mere connectivity and familiar table names are insufficient. The application probe is a C# adapter that executes real SQL queries and reads the external template file. It is not a deployed web service or an HTTP readiness endpoint in this laboratory. That scope is deliberate: the experiment establishes data and dependency readiness without claiming that routing, TLS, authentication or a production API has also recovered. An organization can expose a similar contract through its normal health or verification surface, but it should preserve the semantics. A shallow liveness endpoint and a business readiness probe are not interchangeable. The former may prove that a process responds; the latter must establish the responsibilities required for the next meaningful operation. ## Restore the database and watch the application remain unready 03 / Watch the mechanism ### A restored database can precede readiness ![The restored database passes exact checks but the application remains unready until the template is restored.](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v3.svg) A restored database can precede readinessExecuted dependency\_missing → matching content transition. Local synthetic file models an external dependency; not cloud restoreA restored database can precede readinessExecuted dependency\_missing → matching content transitionDB exact check passes1,000 known rowsTemplate absentready = falseCorrect bytes copiedready = trueLocal synthetic file models an external dependency; not cloud restore Play animation[Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v3.svg) Static view. Play to see how the mechanism unfolds. The database stores the expected content hash, not the content itself. The fixed labels preserve the failed and successful probe outcomes when motion stops. Interpretation and evidence - A reference is not its target. - Check content identity, not only existence. - Repeat the application probe. Actual fixture results where numeric; conceptual flow otherwise. Motion speed is not measured timing. The first target probe passes the database identity, epoch and ledger-count checks. It then looks for the target template file. That file does not yet exist, so the result is ready false with reason dependency\_missing. This failure is not caused by a bad SQL restore. It is the expected consequence of restoring only one member of the application's state set. The database contains the expected template hash, but a hash is not the template content. The external object has its own recovery responsibility. The wrapper has retained a separate synthetic source template. The recovery harness copies it into the target dependency location and repeats the probe. The content hash now agrees with the database reference, and the probe reports ready true for the thousand-transaction snapshot. The distinction between existence and identity matters here. A later negative test writes the wrong text into that same target path. The file exists, but the probe rejects it with dependency\_hash\_mismatch. A file-presence check alone would have accepted an incompatible dependency and produced a false readiness signal. This is a local-file model of an external blob or configuration dependency. It does not claim to have restored Azure Blob Storage, verified cloud object versions or exercised credentials against a remote service. The model isolates the consistency rule: the database's reference and the external content must agree. The recovery set in a real system can include encryption keys, configuration, object storage, identity mappings and service endpoints. The exact set depends on the operation being restored. The useful lesson is not to add every conceivable item to a checklist; it is to identify which external facts must be true for the chosen operation to be correct. ## Measure the stages without calling the result an RTO 04 / A concrete example ### Measure the stage you actually executed ![Backup, verification, restore and restore-to-ready intervals are displayed as separate measured values.](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v4.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/ENG19-v4.svg) The final interval includes validation and template restoration. Container startup and file transfer are outside the command timers. Interpretation and evidence - Label timer boundaries. - Keep dependency work visible. - Do not compare unlike intervals as throughput. Actual fixture results where numeric; conceptual flow otherwise. Motion speed is not measured timing. The dedicated measured run recorded the following stages: | Stage | Measured duration or interval | | ---------------------------------------------- | ----------------------------- | | Full backup command | 82.6249 ms | | VERIFYONLY command | 28.6460 ms | | Actual RESTORE command | 478.0350 ms | | Restore completion to application-ready record | 80.2591 ms | The last interval ends at the application-ready milestone recorded just after the passing probe. It includes exact ledger validation, a failed missing-template probe, copying the synthetic template and a successful repeated probe. The measured backup file contains 3,985,408 bytes. These are actual observations of this fixture, not extrapolations from another system. These durations describe one bounded local execution. They do not include an incident detection delay, a human decision, a production-size transfer, external credential recovery or client rollout. The two containers run on the same host and share its underlying resources. Their timings cannot establish a regional recovery-time objective. Even the stage timers have specific boundaries. The administrative-command timers begin before the client submits the command and end when it completes. Container startup and backup copying are separate wrapper work. The recovery process's monotonic timestamps can order its own steps, but the prepare process and recover process do not share one stopwatch origin. A useful report should therefore show what was measured rather than simply publishing a large label called recovery time. The actual restore duration helps explain this fixture. The interval from restore completion to a passing application probe exposes additional validation and dependency work. Neither number is a production guarantee. There is also only one dedicated measured recovery sequence. Repeating it many times would not automatically make it representative of a production database. Before expanding the matrix, identify which uncertainty matters: backup size, transfer path, storage throughput, encryption, dependency restoration or application verification. Measure that uncertainty with a workload that includes it. The correctness evidence is stronger than the timing generalization. We know which rows and dependency bytes were restored, which probes failed and which eventually passed. Those are concrete recovery facts. The timing values attach local cost to those facts without pretending to predict a disaster that was not simulated. ## Keep the snapshot boundary visible when the source moves on The companion cutover exercise writes transaction 1001 to the source after the backup has been created. Its acknowledgement intentionally times out after commit. The restored target still contains only the original thousand transactions. That does not make the restore incomplete relative to its backup; it makes the restored snapshot older than the source's later state. This distinction is essential when discussing recovery point. A valid restore can still omit later committed work. Backup verification cannot discover a transaction that was never in that backup, and restoring the external template does not reconcile the missing ledger entry. The cutover harness handles that later transaction through explicit, checked replay. That is an application-level reconciliation step described in the separate cutover article. It is not log shipping, an availability group or automatic database replication. The recovery article's initial readiness claim applies to the known thousand-transaction snapshot. A production plan must decide which later work can be lost, replayed or recovered through another mechanism. That decision cannot be inferred from a green backup job. It needs a declared recovery point and a method for identifying ambiguous or missing operations. The laboratory makes the boundary unusually visible because the identifiers are sequential and known. Real systems may need durable request identifiers, upstream receipts or a source-of-truth event stream. Those mechanisms should be designed before an incident, when their completeness can be tested rather than assumed under pressure. ## Ask which backup properties the exercise did not test The fixture is deliberately unencrypted and uses one data file and one log file. That makes the MOVE mapping easy to inspect. It does not prove that a production restore can find its encryption certificate, create every file on the correct volume or satisfy a more complex storage layout. Those are independent prerequisites that deserve their own recovery evidence when present. The source uses the simple recovery model, and the exercise does not replay transaction-log backups. There is no claim of restoring to an arbitrary time between backups. The later source transaction is reconciled at the application level, using an explicitly known identity and payload. That mechanism should not be confused with engine log recovery. The backup file is copied locally between two containers on one host. A remote transfer could introduce bandwidth constraints, object-store permissions, incomplete uploads or a separate retention lifecycle. The local hash and engine verification are useful controls, but the experiment does not measure those remote failure modes. The target's server-level configuration is also largely fresh. The client uses the laboratory's administrative credential and creates its own database. This does not verify a production application's least-privilege login mappings or scheduled jobs. A service that depends on those objects needs to include them in its readiness contract rather than assume the user database backup contains every operational dependency. These exclusions should influence the next drill, not merely decorate the report. If the real system is encrypted, test key recovery. If restore paths differ, test the actual file mapping. If log-chain recovery is required, test the chain and chosen recovery point. If a restricted application identity must work, run the final operation through that identity. The next experiment should close a named responsibility. The small current result remains valid without those extensions. It establishes the selected snapshot and external-file contract on the recorded engine build. Expanding the claim requires expanding the executed boundary, not changing the article's adjectives. ## Design the probe so that success means something stable The probe returns structured reasons rather than a single unexplained Boolean. That makes repeated execution useful during recovery. Missing content, wrong content, wrong database identity, stale epoch and unreconciled history lead to different next actions. An operator should not have to infer which prerequisite failed from a generic connection exception. The probe is intentionally read-oriented. It inspects database state and external bytes before the separate test write. A readiness check that performs an uncontrolled business mutation on every invocation can create its own reconciliation problem. If a production probe must write, the operation needs a deliberate identity, cleanup or idempotency contract. The fixture's final new transaction is such a deliberate operation. It uses a known identifier and amount and is followed by exact validation. It is not repeatedly generated by a background health check. This keeps the proof of write capability separate from routine observation of readiness. The quick probe's ledger-count condition is not sufficient by itself to detect every data error. The exact ledger validator supplies stronger evidence before the probe is trusted. That division is explicit because a convenient cheap probe should not be presented as a full integrity audit. In a long-running application, readiness can change after a successful check. A dependency can disappear, a configuration value can change or another writer can alter data. The recorded passing probe establishes a point-in-time observation. It does not create a permanent guarantee that future operations cannot fail. This is another reason to tie the recovery report to a concrete operation and timestamp. A statement that the application was ready after restoring the template is meaningful. A statement that the application is now safe forever would exceed what any one probe can establish. The probe contract should also be stable enough for an operator to interpret across runs. Renaming a reason, changing the expected transaction boundary or weakening a hash comparison changes the meaning of success. Those changes deserve review just as a schema migration would, because they redefine the evidence required to resume work. ## Treat the recovery package as a proof with limits The evidence package contains the backup artifact's size and hash, administrative commands, engine versions, exact ledger results, application probe outcomes and container cleanup receipts. The source code is pinned and the data generator is deterministic. Another reader can inspect which claim each artifact supports. The package also preserves negative outcomes. The missing backup source is rejected. The restored database initially lacks its external dependency. A wrong dependency value is rejected later in the shared exercise. These failures are part of the proof because they demonstrate that the acceptance path does not turn every reachable component into a success. There are clear exclusions. This is not an encrypted database, a multi-file enterprise schema, a point-in-time log-chain restore, a cross-version upgrade test or a production-scale recovery benchmark. It does not validate every server-level object or a complete web deployment. Those exclusions prevent the simple exercise from carrying claims it did not earn. The strongest conclusion is also the most useful: this particular backup was verified, actually restored into a distinct database on a second SQL Server instance, checked against exact expected transactions and combined with its required external content before the application probe passed. That is a more demanding standard than a successful backup job, and a more actionable standard than a generic statement that recovery should be tested. It identifies the recovery set, the expected operation and the evidence required to call the application ready. The database restore is an indispensable step in that path, but it is not the final step merely because it is the most visible command. ## Source code and execution evidence [Download the complete lab, source record and raw execution results](https://devinside.cloud/content/files/2026/11/ENG19-lab.zip). This package supports the specific execution claims in this article. It is not a production certification.