Small Buffers, Large Files: An Azure Blob Transfer Lab

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

Small Buffers, Large Files: An Azure Blob Transfer Lab

Dates and technology context

This is a retrospective editorial timeline. Topic context: 24 July 2026. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

The Azure SDK/Azurite uploads ran on 12 November2026 using SDK9.0.203/runtime9.0.4 and Azure.Storage.Blobs12.23.0. Historical availability does not move the actual experiment into July2026.

A file transfer looks simple until it must be fast, bounded and recoverable at the same time. Reading the entire file into memory makes some operations convenient. Starting more transfers can improve utilisation. Retrying a failed request can repair a transient problem. Each choice becomes less convenient when files grow, requests overlap and the source changes during recovery.

The temptation is to search for the right buffer size. There is no useful answer without a workload, an environment and a failure contract. A configuration that performs well between nearby services can behave differently on a constrained connection. A configuration that maximises throughput for one file can exhaust memory when several users arrive together.

This article proposes a transfer laboratory rather than a speed claim. The fixture uses generated files, known hashes and a fixed set of transfer strategies. The broader cloud and interruption experiments remain proposed. A later section now reports a separately identified local SDK/Azurite run with real measurements and its narrower limits. The goal is to discover which resource becomes limiting, what a retry actually repeats and whether the destination still represents the intended source when the transfer ends.

The instrumentation baseline is OpenTelemetry 1.11.2 with its matching Hosting and OTLP exporter packages; the optional isolated benchmark uses BenchmarkDotNet 0.14.0. These are pinned historical inputs, not measured outcomes. The transfer protocol below is original pseudocode. Package publication and the presence of the relevant Blob client members were checked in the published package artifacts, but no dependency restoration, transfer or fault-injection run has been performed.

Define the transfer as a contract

The simplest contract is stronger than “the API returned success.” The destination should contain the intended bytes, belong to the intended source version and become available under an explicit publication rule. If an interrupted transfer leaves an incomplete object that another component treats as complete, success at the transport boundary is not enough.

Use a manifest for each test file. Record its size, generation rule, seed and cryptographic hash. Generate files outside the timed transfer interval unless generation is itself part of the workload being measured. Otherwise the benchmark may compare random-data generation rather than transfer strategies.

Include more than one file shape. A single large file tests a different scheduling problem from many smaller files of the same total size. A distribution with a long tail can reveal whether a few large objects hold resources while small objects wait. State the shapes explicitly rather than describing the dataset simply as “large.”

The proposed historical baseline is .NET 10 with SDK 10.0.100 and Azure.Storage.Blobs 12.23.0. Microsoft's release metadata and the NuGet publication record establish that the selected package predates the July 2026 context. These are reference pins for a future reproduction, not a claim that the dependency graph has been restored or that old package versions are the right production choice today.

Separate file size from memory commitment

01 / See the idea

Payload storage is not process memory

Enlarge diagram

A large object does not necessarily require a large in-memory representation. A streaming design can move a bounded portion at a time. But “streaming” is not, by itself, a memory guarantee. The source, client library, retry mechanism and destination can each buffer data.

Write down the parts of the memory budget that the application controls. A simple model might include a buffer per active subtransfer, queued file metadata and a small amount of application state. Then identify the parts that require measurement: runtime overhead, client-library buffering, network buffers and retained objects.

For illustration, eight simultaneously retained buffers of four mebibytes account for thirty-two mebibytes of payload storage. That is arithmetic, not a prediction of process memory. It excludes every other allocation and says nothing about whether the implementation retains one buffer or several per worker. The lab must inspect the actual memory behaviour rather than treating the simple model as a measured bound.

Microsoft documents transfer options that control initial transfer size, maximum subtransfer size and concurrency for the .NET client. Azure Storage transfer tuning. The important experiment is how those settings interact with this fixture. Copying a suggested value from documentation does not establish that the same setting is efficient in the target environment.

Compare strategies that preserve the same meaning

Choose a straightforward baseline: one file at a time, a bounded stream and a single destination publication step. Then compare controlled changes in subtransfer size and concurrency. Keep file contents, source location, destination configuration and validation rules constant.

A whole-file buffering approach can be included as a deliberately different strategy, provided the report states its memory requirement and refuses workloads that exceed the declared bound. It can be a useful comparison for small objects. Allowing it to exhaust the machine merely to prove that unbounded allocation is dangerous would add little knowledge.

If a server-side copy is considered, distinguish its boundary from an application-mediated download and upload. The application may schedule work without carrying every byte through its own process. That can be an excellent architecture for the right scenario, but its timing and failure semantics should not be reported as though they measured the same local streaming path.

The comparison should preserve the destination contract. A strategy that returns early while work continues elsewhere cannot be called faster without showing both admission time and verified completion time. A strategy that omits content verification cannot be compared fairly with one whose timed interval includes hashing the destination.

Make the experiment's clocks visible

02 / A concrete example

Three intervals, one user wait

Enlarge diagram

Record at least three intervals: waiting to begin, transferring and verifying completion. A user experiences their sum. An infrastructure engineer may need the transfer interval to understand bandwidth utilisation. Both views are useful as long as the report does not substitute one for the other.

Record the time spent draining admitted work after the last file has been scheduled. A concurrency setting can improve average completion while leaving a few large files at the end. Report the distribution of completion times and inspect the slowest cases rather than using only a total elapsed time.

Warm-up and connection reuse should be explicit. A cold first request includes work that repeated transfers may not. Run cold and warm scenarios separately if both matter. Alternating configurations helps reduce the chance that changing network conditions become confused with an application improvement.

The load generator must also be bounded. If it eagerly opens every source file or constructs an object for every chunk, memory pressure may originate outside the transfer implementation. Measure the generator and application separately where practical, and preserve the scheduling rule in the manifest.

Treat concurrency as a resource decision

More parallel work can hide waiting, but it can also compete for connections, CPU, memory and destination capacity. A throughput curve that rises and then flattens is an invitation to ask which resource has stopped benefiting from additional work.

Increase concurrency in a small, declared sequence. At each step, record verified bytes completed, process memory, CPU, failures, retries and file completion distribution. Stop increasing it when the acceptance conditions are violated. The purpose is to find a useful operating region, not to discover how dramatically the environment can fail.

Look at fairness across files. One scheduling strategy may allocate many workers to the first large object while small files wait. Another may distribute capacity more evenly but finish the total batch later. The right choice depends on whether the workflow values aggregate throughput, individual deadlines or predictable resource use.

Do not confuse a shared service limit with an application defect. Conversely, do not blame the service merely because the application experiences a timeout. The experiment should retain enough timing and retry information to determine whether work waited in the application, in a connection boundary or in a remote operation. Where the evidence cannot separate those intervals, say so.

Interrupt a transfer without changing the question

A retry is meaningful only if it continues to refer to the same intended data. If the source path now contains different bytes, resuming from an offset can produce a destination assembled from two versions. That failure can survive a simple length check.

Pin source identity using an immutable object or a checked version condition. Azure Blob Storage supports conditional operations based on ETags, as described in Microsoft's concurrency guidance. The lab should test the chosen condition rather than assume that a familiar filename identifies stable content.

The intended recovery protocol can be expressed without pretending that the following is an SDK method sequence:

source_identity = capture_source_version_and_hash()
destination = create_unpublished_destination()
for segment in bounded_transfer_plan:
    verify_source_identity_still_matches()
    transfer_or_retry_segment(segment, source_identity)
    record_segment_completion(segment)
verify_destination_length_and_content_hash()
publish_destination_reference()

Interrupt the transfer at a recorded point and verify which segments are repeated. Then deliberately change the source identity before recovery. The correct result for that second scenario may be to stop and start a new transfer, rather than attempt to be helpful by combining incompatible states.

Verify content independently of the happy path

03 / Watch the mechanism

Resume only the same source

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

A checksum is useful only when its scope and origin are clear. A hash computed from the same corrupted in-memory buffer used for upload can agree with the destination while failing to represent the intended source. Compare against the canonical manifest created before the timed transfer.

Validate file length and the complete content hash after the destination is readable. If the workflow also preserves metadata, test that separately. Correct bytes with the wrong content type, version label or routing metadata can still produce the wrong downstream behaviour.

Do not hide verification time. Report transfer-only and end-to-end completion where both are relevant. A design that performs a second full read for verification may have a different resource profile from one using a trusted checksum mechanism. The choice depends on the integrity requirement and the evidence available, not on making the throughput number look larger.

Introduce a deliberate corruption in a disposable destination to test the verifier itself. If the verification step cannot detect the corruption it was designed to catch, the rest of the benchmark's success labels are unreliable. Testing the oracle is often more valuable than adding another transfer configuration.

Make the result capable of rejecting the favourite setting

Before execution, write down what would invalidate the preferred design. A smaller buffer fails the objective if it reduces peak memory but creates unacceptable completion times. Higher concurrency fails if its extra throughput comes with excessive retries or poor tail behaviour. A resume mechanism fails if it cannot distinguish the original source from a changed one.

Retain all valid repetitions, not just the fastest. A best-case transfer may be interesting, but it is a weak basis for a service commitment. Report the environment and the range of observations, and separate an unexpected outlier from a known invalid run. Exclusions should have reasons that do not depend on whether the result is pleasing.

The strongest counterargument to this laboratory is that the client library already has sensible defaults. Often it does, and a team with a modest workload may gain little from tuning. The lab should be small enough to establish that conclusion efficiently. If defaults meet the resource and recovery requirements, the result can be to keep them and preserve the evidence.

Optimization earns its maintenance burden only when it changes a meaningful decision. A complicated custom transfer loop that is slightly faster in one fixture can be a worse choice than the supported client behaviour if the team cannot reliably maintain its recovery logic.

Publish a configuration, not a universal buffer size

The final evidence pack should contain source manifests, the generation method, package and runtime pins, transfer settings, raw observations, fault schedules and verification outcomes. Record the actual execution date. Until then, any result table remains intentionally empty.

A defensible conclusion names the workload and operating region in which a configuration met the requirements. It also names the changes that require another test: larger concurrent batches, a different network path, a new client version or a stronger integrity requirement.

Small buffers and large files are not inherently in conflict. The difficulty is making every layer respect a bounded contract while preserving the meaning of completion. A transfer laboratory becomes useful when it reveals those boundaries and leaves the team able to explain both the speed it observed and the failures it refused to hide.

Executed companion: the SDK meets a local Blob endpoint

The executable addition to this article uses the real Azure.Storage.Blobs client against a local Azurite Blob endpoint. It was prepared and run on 12 November 2026, with SDK 9.0.203, runtime 9.0.4 and Azure.Storage.Blobs 12.23.0. The client version was available before the historical context of this article. The run itself is contemporary and must not be described as a measurement made in July.

Azurite is a development emulator, not a small Azure region running on a laptop. Microsoft documents functional differences and explicitly does not provide a performance guarantee for it. The results here test a concrete SDK configuration and local integration boundary. They do not estimate a storage account's bandwidth, internet behavior or cloud service reliability. Microsoft's Azurite documentation.

The fixture creates a deterministic four-mebibyte payload. It uploads the same bytes through either a seekable MemoryStream or a wrapper that deliberately refuses seeking. After each upload, it downloads the complete result and compares its SHA-256 with the original payload. The source array is created before the measured interval.

That last detail is essential. The non-seekable wrapper hides an existing memory stream; it does not generate a four-mebibyte document incrementally from an unbounded source. This experiment therefore cannot prove constant-memory generation of arbitrarily large documents. It lets us examine the SDK's behavior when the interface does or does not expose seekability, while retaining a simple independent integrity oracle.

The local emulator runs in a disposable container with one CPU and a 512 MiB memory limit. Only its Blob endpoint is bound, on 127.0.0.1. The image is pinned by digest in the execution record. The wrapper starts the container, waits for the local endpoint, executes the fixture and removes the container in a finally path. No Azure account or customer data is involved.

Keep upload completion separate from integrity verification

The upload clock begins immediately before UploadAsync and ends when the awaited SDK call returns. The subsequent download and hash comparison are deliberately outside that interval. This makes the reported number an upload-call duration, not the duration of a complete upload-and-independent-verification workflow.

The distinction does not make verification optional. Every measured upload is still followed by the full hash check. A mismatch fails the run instead of being included as a fast successful transfer. The separation simply gives the number a precise meaning and keeps an additional download from masquerading as upload cost.

The code uses the same transfer parameters that a real Blob client accepts:

await blob.UploadAsync(source, new BlobUploadOptions
{
    TransferOptions = new StorageTransferOptions
    {
        InitialTransferSize = block,
        MaximumTransferSize = block,
        MaximumConcurrency = concurrency
    }
});

In this fixture, block is either 256 KiB or 1 MiB, and concurrency is either one or four. Both size settings are explicitly assigned so that a separate default initial-transfer threshold does not silently change the comparison. The maximum concurrency is a permitted level of parallelism, not a guarantee that every instant has that many active requests.

The SDK guidance distinguishes the initial transfer size, maximum subtransfer size and maximum concurrency. Those controls describe different parts of the transfer strategy. Setting one is not equivalent to setting all three. Azure Storage transfer-tuning guidance.

Automatic retries are disabled in this laboratory comparison. That makes the observed request count easier to interpret, but it is not advice to disable retries in a real application. A service exposed to transient faults needs a deliberate retry and idempotency policy. The local run did not exercise that policy or measure interruption recovery.

Five integration checks before the timed comparison

The correctness mode verifies two successful transfers, one from each source shape. Both must produce the exact expected content after a full download. It then obtains the blob's current ETag, overwrites the blob, and attempts a write conditioned on the old ETag. The expected HTTP status is 412, which makes the stale condition visible rather than treating the overwrite as an unconditional success.

A pre-cancelled upload supplies a token that is already cancelled. The fixture requires cancellation and verifies that the intended new blob does not exist afterward. Finally, a read of a deliberately absent blob must produce the expected 404 response. These checks passed against the local emulator.

The five cases are deliberately bounded. They do not prove behavior under every network interruption or every service version. The pre-cancelled case is not a test of cancellation halfway through a large multipart transfer. The ETag case is not a general proof of correct reconciliation after a lost acknowledgement. Those would be separate scenarios with their own conditions and evidence.

The SDK pipeline includes a small counter policy that increments when a request is sent. It records request attempts for the upload interval. It does not record payload contents or credentials. Because retries are disabled and each transfer is isolated in the test loop, the count can be compared across these particular configurations without silently including a retry policy.

A useful extension would distinguish request types and retry causes as separate fields. That extension is not needed to state what this counter actually measured: the number of SDK request sends before the upload call completed. The experiment should not pretend to have collected more detail than its instrumentation retained.

All eight configurations, without a cloud-performance ranking

Each combination of source shape, block size and concurrency was executed three times in the same laboratory process. The table reports the median upload-call time, the full observed range across those three repetitions, the stable request count and the median client-process managed allocation for that interval.

Source	Block size	Concurrency	Median upload	Observed range	Request sends	Median managed allocation
Seekable	256 KiB	1	256.998 ms	215.796–427.264 ms	17	278840 B
Seekable	256 KiB	4	247.306 ms	133.409–296.765 ms	17	312584 B
Seekable	1 MiB	1	128.258 ms	124.928–144.365 ms	5	89712 B
Seekable	1 MiB	4	232.650 ms	157.154–283.403 ms	5	1149488 B
Non-seekable wrapper	256 KiB	1	208.993 ms	194.060–212.268 ms	17	308056 B
Non-seekable wrapper	256 KiB	4	174.247 ms	140.485–289.728 ms	17	585104 B
Non-seekable wrapper	1 MiB	1	125.600 ms	122.908–126.731 ms	5	100368 B
Non-seekable wrapper	1 MiB	4	102.268 ms	97.496–113.178 ms	5	101312 B

The cleanest observation is the request count. Every 256 KiB configuration recorded 17 sends; every 1 MiB configuration recorded five. Within this fixture, changing the concurrency limit did not change that number. It changed the permitted execution pattern, not the amount of source data.

The timing table is less clean, and that is useful information. Increasing concurrency from one to four did not uniformly reduce time. In the seekable 1 MiB case, the median was higher with four. Several ranges are wide. The source variants also share one process and execute in a fixed loop order, so earlier transfers can affect connection pools, caches, JIT state and reusable storage encountered later.

These observations do not justify ranking seekable and non-seekable streams as inherently faster or slower. They show why a small local comparison needs its conditions attached. The emulator has its own CPU limit, storage path and request handling behavior. A production network and storage service can respond differently.

The allocation numbers need similar restraint. They are changes in the client process's managed-allocation counter around the upload call. They exclude the already-created source array and the verification download. They do not measure Azurite's memory, native allocations, process working set or peak simultaneously retained buffers. Reuse of storage across configurations can also change what is allocated during a particular interval.

The large allocation observed for one seekable concurrent configuration is retained in the table. It is a reason to investigate allocation sites and pool state if that configuration matters, not proof that seekability itself requires that amount of extra memory. The fixture records a symptom; attributing it to one internal SDK mechanism requires another level of evidence.

The shape of the source is part of the contract

Seekability tells a consumer which operations the source can support. It does not tell the consumer whether rereading is cheap, whether the underlying content can change, or whether the application is allowed to retain the source for the entire transfer. Those are additional properties of the surrounding system.

The wrapper used here delegates reads to a memory stream while throwing for seeking and length queries. It is intentionally simple enough to inspect. In a real source, a non-seekable stream might represent a pipe, a network response or generated data. Its backpressure and failure behavior could matter more than the local performance difference in our table.

That is why the proposed larger-file investigation should keep source production in view. If producing the next block performs expensive work or waits on another service, higher upload concurrency may not create more available data. If the application materializes the entire document before uploading, tuning the transfer buffer does not remove that upstream memory commitment.

Likewise, a download followed by a hash check is a useful laboratory oracle but may not be the right production verification workflow for every object. A real design has to decide which integrity guarantees are required, how they are provided and what action follows a mismatch. The existence of a successful local hash comparison does not choose those requirements for the product.

Reproduce the local result before extending the claim

Download the complete Blob-transfer companion and raw records. It contains the client, non-seekable wrapper, integration checks, transfer settings, image digest and bounded container runner. The standard development-storage connection string targets the local emulator; it is not an exported credential for an Azure account.

The runner's check mode executes the five integration cases. Its measure mode executes the 24 uploads and verifies every downloaded payload. A busy default Blob port is a reason to stop or deliberately reconfigure the isolated fixture, not to reuse an unrelated local service. The runner never mounts customer data into the emulator container.

For a real Azure experiment, preserve the same discipline while changing the environment deliberately: record region and endpoint, source behavior, retry policy, transfer parameters, object sizes and the verification boundary. Use an authorized test storage account and label any costs and results as that experiment's own observations. None of those cloud measurements has been performed by the local companion.

The important result is therefore a concrete engineering method. Configuration controls are separate. Correctness is checked independently of speed. Request counts, time and memory observations have different meanings. A configuration that looks attractive in one local run still has to earn its place under the conditions of the actual application.

Historical versions and verification boundaries

These pins existed before the context date. Availability and static API checks do not establish a successful build or a production-safe patch level. Execution claims are limited to explicitly identified companion checks.

Component	Version	Available by	Source
OpenTelemetry	1.11.2	2025-03-04	Primary record
OpenTelemetry.Extensions.Hosting	1.11.2	2025-03-04	Primary record
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.11.2	2025-03-04	Primary record
BenchmarkDotNet	0.14.0	2024-08-06	Primary record
Azure.Storage.Blobs	12.23.0	2024-11-12	Primary record
.NET SDK	10.0.100	2025-11-11	Primary record
.NET runtime / ASP.NET Core shared framework	10.0.0	2025-11-11	Primary record