devinside.Marcin Borecki Ideas. Systems. Evidence.
← The notebook

Engineering

Bounded Batches: Moving Large Datasets Through an EF Core Service

The dangerous part of a large data job is often a small piece of code that looks beautifully simple: load the records, transform them, save the result. Each verb is reasonable. The problem is the missing boundary around “the records.”

Explore the mechanism ↓
Editorial ink and watercolor illustration for A checkpoint must follow the effect.

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

Bounded Batches: Moving Large Datasets Through an EF Core Service

Editorial context: 10 July 2025 · Prepared and updated: 12 November 2026
Dates and technology context

This is a retrospective editorial timeline. Topic context: 10 July 2025. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

The EF Core/SQL Server companion and its cancellation, receipt and timing evidence were executed on 12 November 2026. SQL Server build 16.0.4315.2 is a November 2026 build, distinct from the historical RTM baseline.

The dangerous part of a large data job is often a small piece of code that looks beautifully simple: load the records, transform them, save the result. Each verb is reasonable. The problem is the missing boundary around “the records.”

A pipeline that is comfortable with a development dataset may retain too much memory, hold a transaction too long or become impossible to resume when the dataset grows. Adding async does not answer those questions. Neither does replacing a loop with a more elegant LINQ expression. We need to decide how much work the application can own at one time and what happens when ownership is interrupted.

The original design below uses EF Core 9 and SQL Server 2022 as a reference baseline. Those small fragments remain illustrative. The companion later in this article adds actual EF Core 9.0.0 and SQL Server execution, performed on 12 November 2026 against a small synthetic dataset. Its dated evidence is separate from the July 2025 historical context; it is not a customer case study or a claim about production scale.

A batch is a promise about unfinished work

Suppose a service exports orders into another system. Calling a collection a “batch” does not make it bounded. A batch of a thousand records can still contain a thousand enormous documents. A thousand rows can also trigger many more queries if related data is loaded implicitly.

Define the bound in the dimension that creates risk. Record count is convenient, but payload bytes, retained object graph size, transaction duration and downstream capacity may be more important. A useful contract might limit the number of projected records, reject individually oversized payloads and stop admitting new work when a downstream queue is full.

The batch also defines a recovery unit. If a process disappears halfway through, which records are known to be complete? Which may have been delivered? Which must be retried? Until those questions have answers, batching is only a memory optimization with an unfinished correctness story.

EF's guidance distinguishes buffering an entire result from streaming it and recommends projecting only the fields an operation needs. Those are useful mechanisms, but the application still has to decide what completion means. EF Core efficient querying.

Separate selection from delivery

01 / See the idea

Selection is not delivery

A finite source is projected into a bounded page before delivery and durable progress.

A hypothetical export selects IDs 1 to 6 but sends only the fields the destination needs. Finite selection, bounded retained work and confirmed delivery are separate promises.

Interpretation and evidence
  • Project required fields only.
  • Bound bytes as well as row count.
  • A read page is not a delivered page.
conceptual/hypothetical, not measured

A common mistake is to let the shape of the database entity become the shape of every downstream operation. An export that needs an identifier, timestamp and amount should not materialize a customer's full history merely because the domain model can navigate to it.

Use an explicit projection. Apart from reducing transferred data, it documents the dependency between the source and the export. If a new field is added later, the change is visible in the pipeline contract rather than arriving silently through serialization of an entity graph.

The selection rule needs equal attention. Is this a snapshot export, an incremental feed or a best-effort scan? For a snapshot, a moving source can make successive queries inconsistent. For an incremental feed, a record's identifier may not be enough to discover later modifications. For a backfill, the source may need a stable cutoff that prevents the job from chasing new arrivals forever.

These are different products. Do not solve one and label it as all three. The following query is appropriate only for a declared, finite selection ordered by an immutable unique key. It does not implement change capture.

// Illustrative EF Core 9 fragment; not executed.
long lastCompletedId = checkpoint.LastCompletedId;
long upperId = checkpoint.UpperId;
const int pageSize = 500;

var page = await db.Orders
    .AsNoTracking()
    .Where(o => o.Id > lastCompletedId && o.Id <= upperId)
    .OrderBy(o => o.Id)
    .Select(o => new ExportOrder(o.Id, o.CreatedUtc, o.Total))
    .Take(pageSize)
    .ToListAsync(cancellationToken);

Here ExportOrder is an application DTO and checkpoint is an application-owned record. The number 500 is a starting hypothesis for a laboratory, not a recommended universal batch size. Even a bounded projection needs a rule for unexpectedly large values.

Pagination is a consistency decision

Offset pagination makes it easy to express “skip the first several thousand records.” It can be a poor fit for a long scan because the database must still deal with the skipped portion, and changes in the source can move rows between pages. Keyset pagination instead continues after the last observed ordering value. EF's documentation emphasizes that ordering must be fully unique. EF Core pagination.

An increasing identifier helps, but it is not a time machine. A high-water mark does not freeze every other column. If an order changes after its page was read, the export may represent different moments for different records. If identities are allocated before transactions commit, a transaction with a lower identifier might become visible after the scanner has passed it.

Therefore make the consistency assumption explicit. A quiescent source or a prepared snapshot dataset makes a keyset laboratory easy to reason about. A live production feed may need a database snapshot, a change-tracking mechanism, a monotonically ordered change log or reconciliation with an overlap window. Choose based on the required business semantics.

The question is not “Is keyset pagination good?” It is “Does this cursor identify every change this job is responsible for?” An excellent query plan cannot repair a cursor that expresses the wrong responsibility.

Keep transaction scope smaller than the story

It is tempting to wrap the complete export in one transaction because “all or nothing” sounds safest. That may hold locks or versions for the duration of the entire job and create a large rollback obligation. It also does not automatically make a remote HTTP call atomic with a database transaction.

EF normally applies a single SaveChanges call transactionally when the provider supports transactions. Explicit transactions can combine multiple local operations, but provider capabilities and retry behavior still matter. EF Core transaction behavior.

For an external destination, define an honest handoff. One option is to write an export intent and local state atomically, then let another worker deliver the intent with an idempotency key. Another is to make the destination idempotent and checkpoint only after it confirms receipt. Neither option eliminates every uncertainty; both make the uncertain interval visible.

Imagine a crash after the destination accepts a batch but before the checkpoint is committed. A restarted worker will resend it. If the destination interprets every resend as new business work, the design is incorrect regardless of its speed. If the destination recognizes the operation identity, the resend can be harmless. The trade-off is additional state and retention policy at the receiver.

A checkpoint must mean something durable

02 / A concrete example

The checkpoint must follow receipt

A hypothetical crash after receipt but before checkpoint causes replay with the same identity.

The destination accepts batch B7, then the worker crashes before saving its checkpoint. On restart B7 is resent, so the destination must recognize its identity rather than apply a second effect.

Interpretation and evidence
  • Receipt precedes progress.
  • A crash can create replay.
  • Stable identity makes replay inspectable.
conceptual/hypothetical, not measured

The most useful checkpoint is not “the loop reached this line.” It is evidence that the agreed recovery unit has completed. Store enough context to prevent a resumed job from accidentally becoming a different job: selection version, source cutoff, projection version and destination identity.

Do not advance the checkpoint as soon as data is read. That converts a crash during delivery into silent data loss. Do not update it from multiple workers without a concurrency rule. Two workers can both believe they own the next page and advance past each other's unfinished work.

For a single-worker laboratory, the control flow can remain deliberately plain:

load durable job definition and checkpoint
read the next bounded page
if page is empty: reconcile and finish
deliver page using stable operation identifiers
verify the destination acknowledgement
commit the new checkpoint with an expected version
repeat unless cancellation was requested

The expected version is a concurrency condition, not decoration. If another worker has changed the checkpoint, this worker must stop and reconcile ownership. “Last write wins” is not a safe default for progress that represents irreversible downstream work.

Bound concurrency as well as page size

Reading one page at a time does not help if every page launches unbounded background tasks. An application can have a bounded database query and an unbounded delivery queue. The retained payloads merely move to another collection.

A bounded channel can represent admission pressure. The producer waits when the channel is full, and a fixed set of consumers defines the maximum delivery concurrency. But capacity still needs a unit: a channel bounded by item count does not enforce a byte limit. Large individual records require a separate size policy.

// .NET 9 BCL fragment. Not executed for this manuscript.
var queue = Channel.CreateBounded<ExportOrder>(
    new BoundedChannelOptions(100)
    {
        FullMode = BoundedChannelFullMode.Wait,
        SingleWriter = true,
        SingleReader = true
    });

await queue.Writer.WriteAsync(order, cancellationToken);

This fragment illustrates backpressure, not a complete exporter. A full implementation must complete the writer, observe consumer failures and await owned tasks before reporting completion. It must also decide whether progress is checkpointed per item or per page. The choice affects both recovery cost and bookkeeping overhead.

Increasing consumers can improve utilization until the destination, database or network becomes the bottleneck. Beyond that point, concurrency may only add contention and queueing. Treat concurrency as an experimental variable, not a synonym for performance.

Cancellation is a boundary, not a time machine

03 / Watch the mechanism

Cancel admission, settle owned work

Cancellation stops new items while in-flight work becomes confirmed or explicitly uncertain.
Enlarge diagram

Static view. Play to see how the mechanism unfolds.

A hypothetical export has delivered A, is waiting for B, and has not admitted C. Cancel stops C; it cannot undo A or prove whether B committed.

Interpretation and evidence
  • Stop admitting C.
  • Keep the confirmed effect of A.
  • Reconcile B if its outcome is unknown.
conceptual/hypothetical, not measured

A cancellation request should stop new admissions promptly, but it cannot undo an operation that already committed elsewhere. The user interface and job status must distinguish “cancellation requested” from “all work stopped.”

Propagate cancellation to the database and transport APIs that support it. Still record the outcome of in-flight work. If the process does not know whether a remote call committed, retain an explicit uncertain state and reconcile it on restart. Do not manufacture certainty by labelling every interrupted operation “failed.”

Define how partial completion is exposed. A batch export can legitimately finish with some records delivered and the remainder pending, provided the contract says so and the checkpoint supports safe resumption. A financial settlement may require a different boundary. The same generic pipeline should not conceal that difference.

Cancellation testing must include awkward timing: before selection, after selection, during delivery, after remote acceptance and during checkpoint persistence. A happy-path cancellation button proves very little about recovery. The test should establish what the next worker sees, not merely whether the current task throws an exception.

Measure the shape of the work

Build a synthetic dataset with a declared seed and distribution. Include a typical row, an oversized row and a skewed customer with many more records than the median. State whether the data is static during the test. Those choices determine which conclusions the laboratory can support.

Vary one important boundary at a time: page size, projection width, consumer count or transaction grouping. Record rows and bytes delivered, elapsed time, database reads, execution plans, process memory, allocation rate, queue depth and destination latency. Record failed and retried work separately so that faster reprocessing cannot masquerade as useful throughput.

Reconciliation should use independent expected values. Count records, compare stable identifiers and verify a deterministic checksum over the projected fields. A checksum alone is not sufficient if the ordering or business semantics matter, but it can make accidental omissions and duplicates easier to detect.

A convincing performance result includes the recovery experiment. Kill the worker at a controlled boundary, resume it and demonstrate that the destination converges to the expected state. Measure the repeated work as well as the successful work. That is the cost of the chosen checkpoint interval.

Choose the smallest trustworthy unit

There are reasons not to use EF Core for every large transfer. Native bulk-loading facilities, database-side transformations or dedicated data movement services may fit better when the operation is mostly moving columns between systems. Application code becomes more valuable when each item requires domain decisions, external interaction or detailed workflow control.

The comparison should include operational complexity. A bulk operation with excellent throughput may have an unsuitable failure boundary. An application pipeline with modest throughput may be preferable if it can resume safely and explain every decision. Conversely, a complicated application loop is difficult to justify when a database operation can enforce the required invariants more directly.

Bounded batches are therefore a way of thinking about responsibility. At every stage, ask how much unfinished work exists, who owns it, how it becomes durable and what a replacement worker can know. Once those answers are clear, page size and concurrency become tunable parameters. Before that, they are just numbers attached to an uncertain system.

A database companion with an explicit execution boundary

The companion turns the design into a small application that uses the actual SQL Server provider for EF Core. It is not an in-memory replacement for SQL behavior. It builds with SDK 9.0.203, targets runtime 9.0.4 and pins Microsoft.EntityFrameworkCore.SqlServer to 9.0.0. A package lock preserves the restored dependency graph.

Build and execution produced different kinds of evidence. The initial Windows LocalDB attempts failed during engine startup. A later, separate SQL Server 2022 Linux container ran the application successfully. It reported Developer Edition, build 16.0.4315.2, with the laboratory database at compatibility level 150. The container was limited to two CPUs and 4 GiB of memory; the client remained a Windows x64 .NET process.

Provider-only SQL generation was checked before the database was available. That established translation of the query shape, not row materialization or transaction behavior. The subsequent container execution completed fifteen checks covering real queries, receipt rollback, replay, checkpoint ownership and cancellation of an observed lock wait.

The failed startup investigation also produced a concrete environmental observation. The Windows volume reported physical-sector values of 262,144 bytes. Microsoft's troubleshooting guidance describes SQL Server startup failures on volumes reporting physical sectors larger than the supported 4 KB boundary, including the misaligned-I/O warning seen in these instance logs. No registry setting, storage configuration or other user's database was changed to work around it. Microsoft's SQL Server sector-size troubleshooting guidance

The failed LocalDB attempts remain in the evidence pack. They are not performance results and were not replaced by an in-memory database pass. The successful container used an official SQL Server 2022 image pinned by its recorded digest, with a loopback-only ephemeral port and no persistent volume. A wrapper removed it in finally after each run, so the exercise did not depend on a long-lived shared database.

The historical design date remains July 2025. The companion's execution date is November 2026. Its later SQL Server patch and container configuration are recorded as contemporary test inputs, not retroactively attributed to the original design date. The .NET client used SDK 9.0.203 and runtime 9.0.4; the restored SQL client package was 5.1.6.

A finite fixture and an independent arithmetic oracle

The executed fixture contains 2,048 synthetic orders. Their identifiers run from 1 through 2,048. Each amount is an integer number of minor units defined by 3 * id - 2000; no customer record or monetary transaction is involved. The source is static during an export, making the selection contract finite and easy to inspect.

Payload width varies deliberately. Most rows contain 128 characters; every sixteenth row contains 2,048. The narrow query excludes that payload, while the wide query materializes it. Both send the same identifier and amount to the destination. The comparison therefore changes source materialization work without changing the destination's required result.

The oracle is not another call to the exporter's aggregation method. It uses closed-form arithmetic for the sum of identifiers and the sum of their squares. The expected count is 2,048, the expected amount sum is 2,198,528, and the expected identifier-weighted amount sum is 4,399,875,072. These are derived fixture expectations, not measured results.

The weighted sum matters because a correct grand total can conceal misplaced values. It is still not a cryptographic proof of complete identity equality, and a production reconciliation may need exact record comparison. In this deliberately simple fixture, the unique receipt key, count, sum and weighted sum provide complementary checks against common omissions and duplicate effects.

The fixture is intentionally much smaller than a production backfill. Its purpose is to make transaction and recovery behavior inspectable with bounded resource use. Increasing its size would be a separate performance experiment, not a prerequisite for finding a checkpoint that advances before delivery.

The compiled page loop keeps its promises separate

The following is the complete compiled export method. Its hooks exist to inject a controlled failure or cancellation at the first page boundary. They are part of the laboratory, not a proposed public API for arbitrary callbacks.

public static async Task<ExportOutcome> RunAsync(
    string connectionString, int upperId, int pageSize, bool wide,
    CommandLog log, CancellationToken token,
    Action? beforeFirstCommit = null, Action? afterFirstCommit = null)
{
    if (pageSize is < 1 or > 512)
        throw new ArgumentOutOfRangeException(nameof(pageSize));
    Progress progress = await Store.ReadProgress(connectionString);
    int selected = 0, pages = 0, peak = 0;
    long characters = 0;
    await using var db = new LabDb(connectionString, log);
    await db.Database.OpenConnectionAsync(token);
    while (true)
    {
        token.ThrowIfCancellationRequested();
        var query = db.Orders.AsNoTracking()
            .Where(o => o.Id > progress.LastId && o.Id <= upperId)
            .OrderBy(o => o.Id);
        List<ExportRow> page = wide
            ? await query.Select(o => new ExportRow(
                o.Id, o.CreatedUtc, o.AmountMinor, o.Payload))
                .Take(pageSize).ToListAsync(token)
            : await query.Select(o => new ExportRow(
                o.Id, o.CreatedUtc, o.AmountMinor, null))
                .Take(pageSize).ToListAsync(token);
        if (page.Count == 0) break;
        selected += page.Count;
        pages++;
        peak = Math.Max(peak, page.Count);
        characters += page.Sum(x => (long)(x.Payload?.Length ?? 0));
        await using (var transaction = await db.Database
            .BeginTransactionAsync(IsolationLevel.Serializable, token))
        {
            string rows = JsonSerializer.Serialize(
                page.Select(x => new { x.Id, x.AmountMinor }));
            await db.Database.ExecuteSqlRawAsync(
                Store.AcceptSql,
                new object[] { new SqlParameter("@rows", rows) }, token);
            if (pages == 1) beforeFirstCommit?.Invoke();
            token.ThrowIfCancellationRequested();
            await transaction.CommitAsync(token);
        }
        if (pages == 1) afterFirstCommit?.Invoke();
        await Store.Advance(
            connectionString, progress.Revision, page[^1].Id, token);
        progress = new Progress(page[^1].Id, progress.Revision + 1);
    }
    return new ExportOutcome(selected, pages, peak, characters);
}

This method was built and exercised against the container database. The surrounding source defines the DTOs, model mapping, SQL acceptance command and checkpoint update. Publishing only the loop without those dependencies would conceal the most consequential part of the design. The earlier illustrative channel snippet was not part of this executed companion.

The page-size guard bounds rows, not all memory. The fixture's maximum payload width gives this particular test another known bound. A general importer still needs a policy for unexpectedly large values and any additional serialization buffers.

The generated narrow SQL selects the identifier, creation time and amount, with TOP, an exclusive lower identifier and an inclusive upper identifier. The wide SQL additionally selects Payload. The provider generated these two shapes from the actual source. The executed reader commands confirm those query shapes. This companion does not capture actual execution plans or claim to isolate their engine CPU cost.

A receipt is the destination effect in this laboratory

01 / See the idea

Receipt commit and checkpoint remain separate

A SQL receipt transaction commits before a separately revision-checked checkpoint update.

An injected exception after receipt commit left 32 accepted rows and the old checkpoint. A fresh export invocation replayed the page and converged to the independently expected result without duplicate receipts.

Interpretation and evidence
  • A source read is not receipt.
  • Receipt commit precedes progress.
  • This local SQL test does not prove remote exactly-once delivery.
docker-check.json and compiled Export.cs

The destination is a SQL receipt table in the same laboratory database. A unique identifier allows repeated delivery of an identical record to become a no-op. A matching identifier with a different amount is an integrity conflict, not a second acceptable interpretation of the same event.

The acceptance command reads the serialized page using OPENJSON, checks existing receipt values and inserts only absent identifiers. It executes inside a serializable transaction with update/range-locking intent on the receipt lookup. The real SQL tests exercised the receipt behavior under the stated isolation configuration; an in-memory collection would not establish it. This first run did not test competing export workers. The configured database compatibility level was 150, and this acceptance command executed in that environment. This does not establish behavior under another compatibility level.

The checkpoint intentionally commits after the receipt transaction, through a separate command. That creates the uncertain interval we want to test. A failure after receipt commit leaves accepted data with an old checkpoint. The next run will select those records again and must recognize their existing effects.

The checkpoint command includes an expected revision. If another writer has already changed the revision, the update affects no row and reports an ownership conflict. This avoids silently accepting last-write-wins progress. It does not turn the whole exporter into a fully tested multi-worker scheduler. That broader contract is outside this first companion.

Using one SQL instance keeps the fixture inspectable, but it limits the inference. The receipt table models an acknowledged destination; it does not demonstrate atomic delivery to a remote HTTP service or broker. A later integration must test that destination's idempotency and reconciliation behavior directly.

Test the awkward boundaries before asking for throughput

The completed correctness run contains fifteen named checks. It first verifies the actual source row count, completes a narrow export and compares the destination with the independent oracle. It checks the peak page count, final checkpoint and the absence of repeated work after a fully completed checkpoint.

Then it injects a failure after the SQL write but before commit. The receipt transaction should roll back, and the checkpoint should remain unchanged. The cancellation case uses the same boundary with a real cancelled token, rather than assuming that catching a cancellation exception proves data rollback.

The next failure occurs after commit but before checkpoint advancement. The expected intermediate state is deliberately different: one page's receipts remain, while progress is still old. A fresh export run should replay the page without adding a second effect and then converge on the full expected result.

The suite also supplies a stale checkpoint revision and an existing receipt with conflicting payload. The first must refuse to advance progress. The second must reject the page without silently overwriting the original receipt or accepting unrelated rows from that failed page.

Finally, the database cancellation scenario holds a row lock from another connection. The EF query must reach an observed SQL lock wait before the cancellation token is triggered. This distinction prevents a token cancelled before execution from masquerading as cancellation of an active database request. The blocker uses a real value update and an exclusive table-lock hint in this test, with read-committed snapshot disabled. Its transaction is rolled back in a finally block, and the pending task is observed before resource closure.

All fifteen checks passed on the recorded container engine. The interrupted-export scenarios use injected exceptions and fresh export invocations in the same client process, not operating-system kills or a network partition. They establish rollback and replay at those controlled boundaries, not every possible crash or multi-worker interleaving.

What the executed cancellation actually reported

The application observed LCK_M_IS for the target session before requesting cancellation. The query then reported a SqlException containing the specific message “Operation cancelled by user.” Its task was faulted, not in the Canceled state. The runner recorded both SQL error numbers as zero. This is the observed behavior of the pinned SqlClient package and command path, not a general rule that every SQL exception means cancellation.

The test initially expected only OperationCanceledException, which was too narrow for the actual provider behavior. The revised assertion requires a requested token and that specific cancellation message before accepting the SQL exception as the intended outcome. It does not swallow arbitrary database errors. This message match belongs to the controlled English-language fixture; production classification requires a deliberate provider-specific policy.

The earlier blocker also failed to establish the required observed lock wait. Strengthening the fixture to a real update with an explicit exclusive lock made the intended precondition observable. These corrections are useful evidence: compiling the test did not prove that its setup created the state it claimed to examine.

Measurements of the complete export contract

02 / See the idea

Measured complete export, not query time alone

Measured median elapsed time and managed allocation for six page-size and projection combinations.

Each configuration exported the same 2,048 synthetic records and passed independent reconciliation. The timer includes source queries, receipt transactions and checkpoint writes; managed allocation excludes the separate SQL process.

Interpretation and evidence
  • Compare identical destination results.
  • Page size changes round-trip count.
  • Do not mistake client allocation for total system memory.
docker-measure.json and measurement-summary.json

The executed measurement compared page sizes of 32, 128 and 512 with narrow and wide projections. It repeated each combination three times and changed their order. Setup, seeding, ledger reset and final reconciliation stay outside the measured interval. A warm-up preceded collection to establish the EF model and exercise both projection shapes. It did not remove all observed variation, so the report retains every sample.

The timed interval includes source queries, materialization, serialization of receipt data, destination writes, transaction commit and separate checkpoint persistence. Reporting it as pure EF query time would be incorrect. The experiment measures this export contract as a whole.

Allocation uses a process-wide managed counter because asynchronous operations can continue on another thread. That counter excludes the SQL Server process and native client buffers. SQL work needs its own observations; an apparently small managed allocation total cannot establish that the database did little work.

The command interceptor records actual EF reader and non-query calls and preserves a bounded set of SQL examples. It does not count checkpoint commands executed directly through SqlClient, and it does not count every transaction protocol message as a separate SQL statement. The field names and interpretation must keep those omissions explicit.

All eighteen measured exports passed the independent count, sum and weighted-sum reconciliation. The following table shows the median and the full observed range of three complete exports per configuration. Managed MiB means allocated bytes divided by 1,048,576; it is not peak retained memory or total database memory.

Rows per page Projection Median ms Observed range ms Managed MiB, median EF reads / receipt commands
32 narrow 1477.4 1451.9–3403.6 2.96 65 / 64
32 wide 1446.9 1426.7–1572.8 5.10 65 / 64
128 narrow 706.4 705.3–752.7 1.38 17 / 16
128 wide 716.1 676.7–819.4 3.56 17 / 16
512 narrow 134.2 123.2–2119.5 1.02 5 / 4
512 wide 131.6 131.4–133.1 3.26 5 / 4

The command counts have a straightforward explanation. With 2,048 source rows, pages of 32 create 64 nonempty pages and one final empty read. Pages of 512 create four nonempty pages and one empty read. The receipt command count follows the number of nonempty pages. Each page also incurs a checkpoint update outside these EF counters and transaction work that the counters do not enumerate.

Larger pages therefore remove repeated round trips and commits in this implementation. The measured medians are consistent with that mechanism, but the timing ranges prevent a precise universal speedup claim. The first narrow 512-row sample took more than two seconds, while its other samples were much shorter. This run does not identify the cause of that outlier, and it would be wrong to silently exclude it because it is inconvenient.

The allocation comparison is easier to connect to the code. The wide projection materializes 507,904 payload characters over the export, while the narrow projection materializes none of that field. Every measured wide configuration allocated more managed memory than the corresponding narrow configuration. Both produced the same destination result. That is evidence for this fixture's unnecessary source materialization, not a statement that wide projections always dominate elapsed time.

In fact, the wide median is slightly lower than the narrow median in two page-size pairs. With three observations per configuration and a shared host, that is not a reason to claim that loading extra data makes the system faster. It is a reason to separate the clear allocation mechanism from a noisier timing comparison. The measured operation includes database writes and checkpoint persistence, so the cost of source materialization is only part of the total.

The page of 512 also owns more records at one time than the page of 32. The fixture bounds each payload, but the measured allocation total does not establish peak live memory. A workload with much larger rows or different destination behavior could prefer another limit. The result therefore informs the next experiment rather than prescribing a universal page size.

What this implementation proves, and what comes next

The companion demonstrates real EF queries, SQL receipt transactions and checkpoint behavior on the recorded engine. It also demonstrates why observing the actual provider matters: the first SQL attempt exposed an unquoted reserved table name, and the cancellation test revealed a provider exception shape that the initial expectation did not account for. Both problems were corrected before the accepted run.

The final evidence pack preserves the successful commands, engine identity, package lock, executed checks, captured SQL and all eighteen measurements. It also retains the failed environmental attempts rather than blending them into the result. The SQL container was removed after correctness and again after measurement; the earlier dedicated LocalDB instances remained stopped.

Several stronger claims remain outside this evidence. Injecting an exception is not the same as killing a machine during commit. A fresh exporter invocation is not proof of recovery from every process crash. A single worker and a revision check do not establish all concurrent-worker interleavings. A receipt table in the same SQL instance is not an HTTP receiver, and no remote exactly-once delivery guarantee follows from it.

The next useful test should follow an actual requirement. If the destination becomes a remote service, test ambiguous acknowledgements against that service's idempotency contract. If the source becomes live, address late commits and source versions before tuning throughput. If memory is the limiting factor, record peak live state with realistic payload distributions rather than reading an allocation total as a capacity guarantee.

Bounded batches remain a responsibility model. This implementation adds concrete evidence for a deliberately small part of that model, with enough source and observation to challenge it. The value is not that the largest tested page won a median. It is that every accepted result can be connected to the work admitted, the effect committed and the progress recorded afterward.

Historical versions and verification boundaries

These pins existed before the context date. Availability and static API checks do not establish a successful build or a production-safe patch level. Execution claims are limited to explicitly identified companion checks.

ComponentVersionAvailable bySource
.NET SDK9.0.1012024-12-03Primary record
.NET runtime9.0.02024-12-03Primary record
ASP.NET Core shared framework9.0.02024-12-03Primary record
C#12.02024-12-03Primary record
Microsoft.EntityFrameworkCore9.0.02024-11-12Primary record
Microsoft.EntityFrameworkCore.SqlServer9.0.02024-11-12Primary record
SQL Server 2022 RTM16.0.1000.62022-11-16Primary record

Download the complete laboratory and execution records.

THE AUTHOR

Marcin Borecki

Developer, entrepreneur and business angel. Writing about AI, software, data and the evidence behind technical decisions.

About the author →