One Million Events, Three Failure Modes

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

One Million Events, Three Failure Modes

Dates and technology context

This is a retrospective editorial timeline. Topic context: 28 January 2026. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

The actual million-record SQLite companion, process restart and replay checks ran on 12 November2026 using .NET10.0.15. The January2026 Azure design remains a proposal, not a cloud execution report.

Execution update, 12 November 2026: the original cloud design below is preserved as a proposal. The dated appendix reports a separately implemented and tested local SQLite companion, including an actual million-record run. It does not claim execution of the original Azure stack.

A pipeline processes a million events and produces a reassuring throughput chart. Then a worker loses its connection after a database commit, a message returns to the queue, and the final aggregate changes. The chart did not lie. It measured something narrower than the team needed to know.

The number of events is an input to an experiment. It is not evidence of correctness, resilience or production scale. A million small, evenly distributed events can be easier to process than a much smaller workload containing large payloads, skewed keys and difficult recovery paths. The useful result is an account of what the system preserved when ordinary assumptions stopped holding.

This article specifies a laboratory with one million generated business events and three controlled failure scenarios. It has not been executed. There are no claimed customers, throughput measurements or recovery times. The design is deliberately bounded so a future run can produce evidence that a reader can inspect and, if necessary, contradict.

The proposed package baseline is Azure.Storage.Blobs 12.23.0, Azure.Messaging.ServiceBus 7.18.2, Microsoft.Data.SqlClient 5.2.2 and OpenTelemetry 1.11.2, with SDK 10.0.100 and runtime 10.0.0. The database reference is SQL Server 2022 RTM, build 16.0.1000.6. NuGet's Service Bus record and the accompanying metadata pack establish pre-context publication. This is an isolated historical fixture design, not a recommendation to deploy these old patches today. No package restore, database installation or runtime check is claimed.

Make the million events countable in more than one way

Generate a fixed set of unique event identities. Each event describes a synthetic adjustment to an account, with an integer amount in an explicitly named unit. Avoid floating-point arithmetic in the fixture so that rounding does not become an accidental fourth experiment.

Give the generator a recorded seed and a version. Store a manifest containing the number of unique events, the distribution of accounts, the range of payload sizes and hashes of the source files. Include skew deliberately: some accounts should receive more events than others. Otherwise the test may miss contention that appears when many workers update the same logical object.

Build the expected final state with an independent, simple implementation. For this fixture, that can be a single-process pass over the canonical input files, grouping by account and summing each unique event once. The oracle should not call the same production aggregation method, because a shared mistake would then make the pipeline agree with itself.

Record more than the grand total. Two incorrect accounts can cancel each other in an overall sum. Compare the set of event identities, the amount per account and a checksum over a canonical ordering of accepted records. Each comparison catches a different class of error. A correct total with missing provenance is still an incomplete result.

Keep durable data separate from transport

01 / See the idea

Bytes outlive their notifications

Enlarge diagram

The proposed architecture stores immutable source objects in Azure Blob Storage and sends small work notifications through Azure Service Bus. A .NET 10 worker reads the referenced input, validates it and writes accepted effects to SQL Server 2022. The exact runtime and client packages must be pinned when the fixture is built. The .NET 10 release predates this article's January 2026 contextual baseline.

The notification should identify the source object, its expected version or hash, and a deterministic unit of work. It should not be the sole surviving copy of the business event. Keeping the source artifact allows a replay to answer the same question against the same bytes, rather than whatever happens to be stored at an unchanged path later.

Transport identity and business identity should be distinct. A message can be retried or recreated while still referring to the same business event. Conversely, two genuinely different events can happen to have identical payloads. Deduplicating by payload alone would confuse repetition of content with repetition of intent.

Service Bus's settlement documentation describes the acknowledgements around message transfer and processing. The laboratory's application contract goes further: each unique business event should contribute one effect, regardless of how many transport attempts occur. That is a property to build and test, not a property to assume from the broker.

Establish the no-failure baseline without calling it the answer

Run the generated input once with the fault injector disabled. Fix worker concurrency, batch boundaries and database configuration. Record offered work, completed unique effects, queue age and resource usage. Save the final reconciliation before changing any setting.

The no-failure run checks that the fixture is coherent. It does not establish resilience. It also does not justify tuning immediately for maximum throughput. If the application cannot explain an event's transition from source to accepted effect, making that transition faster will only make the later investigation more difficult.

Use bounded concurrency from the start. A queue buffers a difference between arrival and processing, but it cannot make a permanently excessive arrival rate disappear. Microsoft's queue-based load-leveling pattern is a useful reference for that boundary. In this lab, queue age and remaining work are necessary companions to the completion rate.

Include a drain phase. Stop producing new notifications and wait according to a declared rule for the admitted work to reach a terminal state. “The producer has finished” is not the same event as “the result is reconciled.” Report those timestamps separately so a fast admission path cannot disguise a slow or incomplete processing path.

Failure one: the same business event arrives again

Inject duplicate notifications at recorded positions. Use the same business identity but vary the transport attempt identity. Test duplicates both while the original is in progress and after it has completed. The second case is usually easier; the first exposes concurrency mistakes.

A useful database design records the event's accepted identity and its business effect in the same transaction. The uniqueness boundary should be enforced by the database, not by a separate check followed by an insert. Two workers can both observe that a record is absent before either writes it.

This pseudocode expresses the intended contract without prescribing a particular data-access library:

begin database_transaction
    insert accepted_event(event_id, payload_hash)
        with unique constraint on event_id
    apply business_effect(event)
commit database_transaction
acknowledge transport_message

on duplicate_event_identity:
    rollback attempted_transaction
    verify stored_payload_hash_matches(event)
    acknowledge without applying the effect again

A matching identity with a different payload is not an ordinary duplicate. It is an integrity conflict. Quarantine it with the evidence needed to investigate rather than silently accepting whichever version arrived first. The lab should contain at least one deliberately conflicting event to verify that this distinction is observable.

Broker duplicate detection can reduce duplicate delivery within its configured scope, but it does not replace this application contract. Service Bus duplicate detection operates on identifiers within a history window. A replay outside that boundary or a failure after delivery still requires the application to reason about its own effects.

Failure two: success happens but the acknowledgement disappears

02 / A concrete example

A lost acknowledgement is not a lost effect

Enlarge diagram

Place a fault boundary immediately after the database transaction commits and before the transport message is acknowledged. Terminate the worker or interrupt that acknowledgement in the isolated fixture. On redelivery, the system must determine whether the effect already exists.

This is the uncomfortable interval that a happy-path demo rarely shows. The application cannot infer failure merely because its final call did not return successfully. Its durable state may already have changed. The recovery mechanism should ask the database about the event identity rather than reapplying the effect on the basis of an uncertain client outcome.

Record the committed event, the attempted acknowledgement and the later delivery as separate observations. The final business result should remain unchanged by the repeated transport attempt. The diagnostic record should reveal that a retry occurred. Erasing the retry from history would preserve the total while hiding the operational cost and the reason the worker repeated its work.

Try the inverse boundary too: terminate before commit. In that case the event should eventually receive its effect on a later attempt. These two closely spaced failures should lead to different recovery paths but the same final business invariant. A design that handles only one side of the boundary is not yet a satisfactory retry design.

Failure three: processing stops halfway through the source

Interrupt a worker in the middle of a source object. This scenario tests how the application defines progress. If it records only that the file was “started,” recovery may repeat all of it. If it records completion too early, recovery may omit the remainder. Neither outcome can be understood without a declared unit of work.

One option is to use independently identifiable records and accept harmless repeated processing during replay. Another is to checkpoint bounded segments. The second can reduce repeated reads, but it adds a checkpoint protocol and another consistency question. The first can be simpler, provided the idempotency boundary is correct and replay costs are acceptable.

Do not present checkpointing as automatically more sophisticated and therefore better. A small immutable file may be cheaper to replay completely than to manage a durable offset. A large compressed file may not support efficient random access. The source format and recovery objectives should decide the mechanism.

The lab should stop after a declared number of unsuccessful attempts and surface unresolved work. A poison record must not prevent unrelated records from reaching completion indefinitely. At the same time, moving a notification to a dead-letter path must not make it disappear from the reconciliation. A visible terminal failure is different from an accepted business effect.

Compare invariants before comparing speed

03 / Watch the mechanism

Reconcile every input identity

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

For every run, reconcile the expected unique event set with accepted effects, quarantined conflicts and explicitly unresolved items. The accounting should explain every input. If an item appears in two incompatible terminal categories, the result is not reconciled even if the totals happen to add up.

Then compare resources and time. Duplicate handling may preserve correctness while consuming substantial database and network capacity. A replay policy may be safe but exceed the recovery objective. The report should distinguish those results rather than declaring the entire system successful because the final sum is correct.

Use the same canonical input for the baseline and faulted runs. The fault schedule becomes an additional input, identified by its own seed and manifest. Record exactly when the injector acted, but keep that schedule separate from the operational evidence used by a reviewer to diagnose the run. This allows the lab to test whether the monitoring is useful as well as whether processing is correct.

The falsification criteria are concrete. An extra business effect disproves idempotent processing. A missing effect after the declared recovery interval disproves completeness for that scenario. A conflict silently treated as a duplicate disproves the integrity policy. A growing resource backlog after input stops disproves the proposed bounded recovery behaviour.

Know what this lab would still leave unknown

The fixture does not reproduce an arbitrary production workload. It has declared payloads, generated distributions and controlled failures. It cannot establish behaviour under every regional outage, every database plan or every future SDK version. Its value is that the conclusions can have clear limits.

A positive run would justify a narrower statement: this implementation preserved these invariants for this workload under these injected faults. The next experiment might increase skew, vary payload sizes, change worker concurrency or introduce a prolonged dependency outage. Each extension should answer a new question, rather than merely produce a larger number for a headline.

Publish the generator, independent oracle, run manifests, fault schedule, reconciliation queries and raw observations together. Label any implementation snippets as tested only after an actual run. Until then, the million events remain a specified workload and the three failure modes remain a protocol.

That distinction is the central lesson. Throughput describes how quickly the pipeline moved through a particular path. Recovery evidence describes whether the organisation could trust what emerged after the path broke. A useful data platform needs both, and a credible experiment should make it impossible to confuse one with the other.

Execution appendix, 12 November 2026: a durable local companion

The original sections specify a reference architecture and fault protocol. This appendix adds an executed SQLite companion on 12 November 2026. It does not claim that the earlier Azure Blob, Service Bus and SQL Server architecture has been deployed or measured. The companion isolates the durable acceptance boundary so its invariants can be examined without a broker or cloud account.

The executable uses SDK 10.0.400 and runtime 10.0.15 with Microsoft.Data.Sqlite 10.0.0 and the explicitly pinned SQLitePCLRaw.bundle_e_sqlite3 3.0.2. The package lockfile records the resolved graph. These are actual execution inputs for the appendix, not changes to the earlier historical baseline. Microsoft.Data.Sqlite package, SQLitePCLRaw bundle.

The small correctness suite passed 32 checks before the large run was requested. It verifies rollback, duplicate handling, conflicting source identity, separate output versions, exact event sets and account totals. It also launches a real child process, lets that child commit a checkpoint, and then reopens the database after the child exits deliberately.

The million-event run is separate from those small checks. Its actual observed volume and timing are reported below, rather than estimated from a smaller batch or inferred from the title. The test owner runs it in an exclusive measurement window and removes the owned database files after retaining the report.

Put acceptance, effect and lineage in the same transaction

01 / See the idea

One transaction owns the accepted batch

Enlarge diagram

The fixture has four related data structures. SourceEvent binds an event identity to its payload hash. Receipt records whether that event has been processed into a particular output version. AccountEffect holds the aggregate for each account and output version. Lineage connects a receipt to its transformation, source artifact and record position.

The two important uniqueness boundaries are visible in the schema:

CREATE TABLE SourceEvent(
    id INTEGER PRIMARY KEY,
    hash BLOB NOT NULL
);

CREATE TABLE Receipt(
    eventId INTEGER NOT NULL,
    outputVersion TEXT NOT NULL,
    PRIMARY KEY(eventId, outputVersion),
    FOREIGN KEY(eventId) REFERENCES SourceEvent(id)
);

For each record, the writer inserts the source identity if it is new. If the identity already exists, it compares the stored hash with the incoming payload hash. Only then does it try to insert the processing receipt. An existing matching receipt is a duplicate for that output version and skips the effect update.

For a new receipt, the aggregate update, lineage insertion and checkpoint update happen in the same transaction as source and receipt acceptance. An injected exception before commit rolls the whole batch back. The test checks that no source, receipt, lineage, aggregate or checkpoint remains from that failed first batch.

This is stronger than checking a receipt in application code and then writing the effect later. The database uniqueness constraint arbitrates whether a receipt is new, and the transaction keeps its related state consistent. SQLite's conflict handling can turn an insertion into a no-op at a specified uniqueness boundary; the program must still decide whether a conflicting payload is a duplicate or an integrity failure. SQLite UPSERT documentation.

The fixture uses one writer and one local database. It does not establish a distributed transaction or a concurrency strategy for multiple independent output stores. The value of the small design is that the accepted local invariant can be inspected directly.

Make the three failures produce distinct evidence

Before-commit failure leaves no accepted effect. The suite deliberately throws after processing the batch but before the transaction commits, then checks the empty durable state. Repeating the same input afterward accepts the records normally.

After-commit repetition has a different path. The stored source hash matches, and the receipt already exists. The writer reports duplicates rather than updating the aggregate again. The final event set and account totals remain unchanged, while the report shows that repeated input was processed.

Conflicting identity is neither of those cases. The test first presents a new record and then a changed payload under an existing identity inside the same batch. The conflict throws, and the preceding new record rolls back with the batch. The test verifies that the source count remains unchanged, rather than merely checking that an exception was thrown.

This distinction keeps a successful error message from hiding partial acceptance. A program can reject the second item while accidentally committing the first if it chooses different transaction boundaries. That behavior might be valid for another contract, but it is not the all-or-nothing batch contract tested here.

The local hash is computed from an explicit twenty-byte representation: event identity, account and integer amount. It identifies the fixture payload; it does not authenticate a producer or establish that the data is commercially valid. The domain is bounded to thirty-two accounts and amounts from minus eight through eight, making its arithmetic and expected results easy to inspect.

Reopen after a real process exit

02 / Watch the mechanism

A new process verifies the committed checkpoint

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

The restart scenario launches the compiled laboratory as a child process with a dedicated database path. The child writes ten events, commits the transaction and prints a marker confirming that the code reached the committed boundary. It then calls Environment.Exit(23).

The parent checks the exit code and marker, reopens the database, and verifies the first ten receipts, lineage rows, checkpoint and independently calculated account totals. It repeats the first ten events and expects ten duplicates. It then writes events eleven through twenty and verifies the complete accepted range.

This is an actual process-lifetime boundary. The successful result does not depend on the child's in-memory counters surviving. It also does not claim to reproduce machine power loss, a torn write or a storage controller that ignores flush requests. The database remains on the same running host, and the child exits only after the explicit commit.

The checkpoint in this fixture is a high-water mark for the ordered, contiguous generator. It is updated transactionally with accepted work. It is not a general cursor for out-of-order business events: a maximum identifier alone cannot prove that lower identifiers were not skipped. The exact-range verification is what makes the checkpoint meaningful for this declared input schedule.

That qualification matters when transferring the lesson to a broker. A production consumer with arbitrary delivery order needs a progress representation suited to that order. Copying the local maximum-ID checkpoint into such a system would preserve the code while changing the guarantee.

Generate a million records without retaining a million objects

03 / See the idea

An actual million-record run, not an extrapolation

Enlarge diagram

The large run allocates one reusable array with capacity for one thousand InputEvent values. The generator fills the next batch, the writer commits it, and the same array is reused. The client does not construct a million-element collection before starting the experiment.

The prepared SQL commands are also reused. Parameter values change per record, while the connection and command structures remain available across batches. This follows the practical SQLite pattern of using transactions and reusing parameterized commands rather than issuing unrelated ad hoc inserts. Microsoft's SQLite bulk-insert guidance.

The event-buffer bound is not a promise that process memory equals the array size. Hash results, boxed parameter values, SQLite caches, command internals and runtime bookkeeping still allocate. The report therefore separates the fixed client event capacity from measured managed allocations. Retained heap and native SQLite memory are not inferred from that allocation counter.

The writer uses DELETE journaling and FULL synchronous mode. The million-event phase consists of one thousand transactions of one thousand records. It is followed by one thousand repeated v1 records and one thousand records replayed into v2. The input bytes, transformation version and receiver semantics remain explicit throughout.

Reconcile identities and individual accounts before reading the stopwatch

Every phase is followed by verification outside its timed interval. The verifier checks receipt and lineage counts, the checkpoint and the complete ordered event-ID range. It reads identifiers sequentially from SQLite, so exact-set verification does not require another million-element client collection.

The arithmetic oracle is separate from the writer. It calculates thirty-two account totals using BigInteger from the deterministic generator formula. The writer's SQL aggregation is therefore not reused to produce the expected answer. The comparison includes every account, preventing opposite errors from cancelling in one grand total.

For v2, the transformation doubles the integer adjustment while keeping the same source identity and hash. That deliberately changes the derived output without pretending the source event is new. The verifier expects the v1 state to remain unchanged and v2 to have its own receipts and totals.

This oracle is independent of the SQL update algorithm, but not a proof of arbitrary business semantics. It verifies the specified synthetic adjustment rule. A real data product would need expected outcomes appropriate to its domain and a policy for rejected or unresolved records.

The exclusive run completed on 12 November 2026 at 14:10 UTC, with status passed. It used .NET 10.0.15 and SQLite 3.50.4 on Windows build 20348, with DELETE journaling and FULL synchronous mode. This was one sequential run, not a repeated statistical comparison or a service-level latency study.

Phase	Submitted records	Newly accepted	Duplicates	Transactions	Timed seconds	Managed bytes allocated
new-v1	1,000,000	1,000,000	0	1,000	12.183669	2,134,861,536
duplicate-v1	1,000	0	1,000	1	0.005602	1,353,280
new-v2	1,000	1,000	0	1	0.014496	2,689,200

The first phase actually accepted one million unique events into v1. Its observed rate was approximately 82,077 submitted records per second within the declared generation, hashing and transaction boundary. That number is not projected from a smaller test. It also is not a forecast for a broker-backed service: there was no network ingestion, cloud object read or production query workload in this path.

The duplicate phase left v1 with one million receipts and one million lineage rows. The v2 phase created one thousand separate receipts and lineage rows while retaining the global source count of one million. The smaller follow-up phases have different work and volumes, so their elapsed times should not be ranked as competing architectures.

All three phases passed exact event-range and independent per-account reconciliation. For v1 the expected receipt-ID sum was 500,000,500,000; for the first thousand v2 records it was 500,500. The sum is an additional summary, not the sole completeness check: the verifier also walks every ordered receipt identity and checks every account total.

The first phase allocated about 2.13 billion managed bytes cumulatively while using a reusable event buffer of one thousand records. These statements are compatible. Allocation measures work over time, not simultaneously retained memory. Hash results, parameter boxing and provider internals still allocate. The experiment did not collect a retained-heap or native-cache profile and therefore does not claim a process-wide memory ceiling or a zero-allocation ingestion path.

Reconciliation runs outside the timed phase, and the reported database state is captured before cleanup. After the report was written, connections were disposed and the owned database files removed. A later optimization should retain the same correctness gates and make its changed measurement boundary explicit rather than comparing a new, narrower stopwatch with this result.

What the executed companion adds, and what it leaves open

The appendix establishes more than the presence of a proposed test. It provides a compiled writer, durable uniqueness boundaries, an actual process restart, an independent oracle and an observed large input volume. The original failure questions now have a local implementation that a reader can run and challenge.

It still leaves the broker and cloud architecture untested. Service Bus delivery semantics, Blob conditional reads, SQL Server plans, network partitions and multiple deployment instances require their own experiments. The local result should inform those designs without claiming to substitute for them.

The useful next step is to preserve the same source and output-version invariants while replacing one boundary at a time. If a broker is introduced, inspect redelivery without changing what counts as a business event. If storage changes, verify the receipt and effect transaction. If throughput is optimized, keep exact event-set and account reconciliation as the acceptance gate.

That is how the million-event headline earns a precise meaning: an actual number of generated records crossed a particular implemented boundary, under recorded settings, with explicitly checked outputs. It does not become a shorthand for unmeasured production scale or universal resilience.

Historical versions and verification boundaries

These pins existed before the context date. Availability and static API checks do not establish a successful build or a production-safe patch level. Execution claims are limited to explicitly identified companion checks.

Component	Version	Available by	Source
OpenTelemetry	1.11.2	2025-03-04	Primary record
OpenTelemetry.Extensions.Hosting	1.11.2	2025-03-04	Primary record
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.11.2	2025-03-04	Primary record
Azure.Storage.Blobs	12.23.0	2024-11-12	Primary record
Azure.Messaging.ServiceBus	7.18.2	2024-10-08	Primary record
Microsoft.Data.SqlClient	5.2.2	2024-08-27	Primary record
.NET SDK	10.0.100	2025-11-11	Primary record
.NET runtime / ASP.NET Core shared framework	10.0.0	2025-11-11	Primary record
SQL Server	2022 / RTM 16.0.1000.6	2022-11-16	Primary record

Download the complete laboratory and execution records.