Document Ingestion in .NET: Throughput, Versions and Retrieval Quality

Prepared on 12 November 2026. This executable uses a bounded .NET channel, local SQLite transactions and preserved real embeddings from the M08 laboratory. It processes plain text paragraphs, not PDFs, and makes no claim about a production Azure ingestion service.

A document import can finish quickly and still leave an unusable index. It may retain an obsolete fragment, duplicate a version after restart, separate a value from its scope, or attach an old embedding to changed text. The importer can report success while the next query receives evidence that no longer represents the source.

That is why ingestion needs two acceptance boundaries. One concerns the mechanics of moving and storing data. The other concerns what the resulting index is allowed to claim about the material it contains. Throughput is useful only inside those boundaries.

This laboratory implements a small, real pipeline. It accepts versioned documents, splits plain text at paragraph boundaries, hashes the resulting units and commits each source update atomically to SQLite. A bounded channel connects the producer and consumer. Cancellation is introduced at a controlled point, after which the same input is replayed to complete the import without adding duplicate chunks.

For retrieval, the experiment reuses the exact text and real sentence vectors already produced in the M08 local embedding laboratory. A changed paragraph without a matching verified embedding remains stored but is excluded from searchable-vector results. The program does not invent an embedding to make the update appear complete.

The result is deliberately smaller than a general document platform. Its value is that every stored fragment, version transition and quality check can be inspected without confusing successful import with trustworthy answering.

Define the source before defining the queue

01 / See the idea

Source identity is not content identity

Enlarge diagram

The input record contains a source identifier, tenant, text and integer version. These fields establish the identity of the document being imported. The source table stores its current version, content hash and tenant scope; the chunk table stores the units belonging to that accepted version.

A source identifier is not a content hash. Two distinct source records can contain identical text. Conversely, one source can change its contents over time. Keeping those identities separate lets the importer distinguish idempotent replay from a new revision.

The same version with different text is rejected as a conflict. The same version with a changed tenant is also rejected. Otherwise, a permission-related change could be ignored merely because the paragraph bytes remained the same. A caller must provide a new version when the source contract changes.

An older version arriving after a newer one is classified as stale. It does not replace the current chunks. This is an explicit latest-version import policy, not a universal event-history rule. An archival application might need to retain every version in a different schema.

The fixture's tenant is a controlled label. It is not derived from a real identity provider, and no production authorization guarantee is claimed. Its purpose is to preserve a scope field through import and use that field consistently when selecting retrieval candidates.

The data model therefore states what the pipeline considers the same source, the same revision and the same accessible material before any concurrency or performance setting is introduced.

A paragraph chunk is a declared parser choice

The parser normalizes line endings and splits on blank lines. It trims the resulting paragraphs and assigns each a stable position within the source version. It does not interpret PDF layout, extract tables, recognize footnotes or reconstruct headings from visual formatting.

That limitation is central to the result. A successful paragraph import should not be presented as evidence that arbitrary enterprise documents can be processed without losing meaning.

The original twenty synthetic documents each contain a short policy paragraph. Their complete text is the unit for which a real embedding was previously computed. A separate structural check creates a two-paragraph source from two existing paragraphs and verifies that both chunks appear, then that deleting the source removes both.

This is genuine chunk construction, but it does not evaluate all possible segmentation policies. It tests a clearly specified parser against input whose intended units are known. More complicated documents would require an independent expected-chunk manifest and tests for the relationships the parser must preserve.

The chunk ID combines source identity, version and ordinal position. Its text hash identifies the representation used for embedding lookup. Position alone would be insufficient across updates, while a hash alone would collapse identical text from different sources into one provenance identity.

The choice of chunk boundary is therefore part of the application contract. It determines which information can be retrieved together and which source/version relationship the result can explain.

Reuse an embedding only for the exact text it represents

02 / See the idea

Changed text cannot borrow an old vector

Enlarge diagram

The accepted M08 preprocessing artifact includes the original text and its normalized reference vector. The new laboratory builds a lookup from the text hash to that vector. Before use, it verifies the copied dataset, reference artifact and model manifest against their recorded hashes.

An imported paragraph receives a vector only when its exact normalized paragraph text matches a known representation. This is reuse of an actually computed embedding, not a fresh inference run and not a synthetic vector pretending to be semantic evidence.

The distinction becomes observable during update. A source advances to a new version with different text that has no matching embedding. The importer stores the new chunk but leaves its vector field absent. The retrieval reader selects only chunks with a vector, so the old representation cannot remain active under the new wording.

This is a conservative incomplete-index state. The source has been ingested, but its semantic representation has not been prepared. A production pipeline might enqueue a new embedding job and expose an indexing status. The fixture stops at preserving the distinction rather than fabricating completion.

The test later restores the explicit original source snapshot as a separate operation. It does not silently mutate the preserved M08 inputs. Those copies remain immutable evidence of the earlier embedding run.

This rule can prevent a particularly misleading failure: a search result carries new text and a current version label while its vector still describes an earlier paragraph. The apparent freshness of metadata would otherwise conceal stale retrieval behavior.

Bound the handoff, not just the final collection

The producer and consumer communicate through System.Threading.Channels. The channel has capacity two and uses the waiting policy when full. Each source document is also limited to 4,096 UTF-8 bytes in this fixture.

The two bounds answer different questions. Channel capacity limits how many items the handoff can queue. The document-size limit prevents one permitted item from being arbitrarily large. A small item count without a size boundary is not a useful payload-memory bound.

The source construction is explicit:

var channel = Channel.CreateBounded<Document>(
    new BoundedChannelOptions(capacity)
    {
        SingleReader = true,
        SingleWriter = true,
        FullMode = BoundedChannelFullMode.Wait
    });

The waiting behavior creates backpressure at this handoff. It does not automatically limit memory retained elsewhere in the application, such as the source array, embedding lookup, SQLite buffers or already materialized query results. The article therefore does not describe the complete process as bounded by two documents.

The channel's behavior is an explicit library contract, not a custom queue hidden inside the fixture. Microsoft describes bounded channels and their full-mode choices, including waiting for space. That supports the chosen handoff mechanism while leaving the application's byte limits and shutdown policy to the surrounding code. System.Threading.Channels.

A future multi-consumer version would introduce another question: can two revisions of the same source be processed concurrently without publishing them in the wrong order? The current single-consumer fixture avoids that scheduling problem and relies on version checks at storage. Increasing concurrency should therefore begin with a new correctness scenario, not merely a larger number in a configuration file. Faster producers cannot compensate for an index whose source revisions become inconsistent.

The fixture tracks the UTF-8 payload associated with the producer's waiting item, queued items and the consumer's current item. That count is useful for inspecting this pipeline boundary, but it is not a managed-heap measurement. It excludes object overhead and unrelated storage and can include an item waiting to enter the channel.

The distinction prevents a memory graph from becoming more impressive than the observation warrants. The pipeline records what it can count directly and leaves total retained and peak process memory as separate questions.

Commit a complete source revision atomically

05 / See the idea

An atomic source update has a visible failure path

Enlarge diagram

Each accepted source update runs inside a SQLite transaction. The importer removes the old chunks for that source, inserts the new chunks, updates the source record and commits. A deliberately injected exception before commit verifies that the earlier accepted chunks remain available.

This transaction boundary means a reader should not receive half of a source revision simply because the importer failed between deleting and replacing fragments. The fixture uses a local SQLite database and the .NET provider's transaction mechanism. It does not claim a distributed transaction spanning object storage, an external embedding service and a hosted search index. Microsoft.Data.Sqlite transactions.

The exact scope is important when adapting the example. If embedding generation occurs outside the transaction, the application needs to decide whether to stage the new representation before activation. If a remote search service accepts batches independently, the application needs a visibility policy for partially applied revisions.

The present local design keeps that problem small. A chunk can be stored without a vector, and the retrieval reader excludes it. The source update itself is atomic, while readiness for vector search is explicit.

A transaction also does not validate semantic correctness. It can atomically store the wrong paragraph boundaries. The parser and source interpretation need their own checks before the database commit becomes meaningful evidence of a good import.

The correct claim is therefore that the local update is all-or-nothing at the declared storage boundary, not that the entire document-processing problem has become transactional.

Cancellation leaves a state that can be explained

03 / Watch the mechanism

Cancel, resume, replay without extra chunks

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

The cancellation test stops the consumer after seven documents have been processed. The producer receives the same cancellation signal, so it cannot remain blocked indefinitely trying to write into an abandoned bounded channel.

Unexpected consumer failure needs the same shutdown discipline. The implementation cancels the shared token when the consumer throws, releasing a producer waiting for channel capacity. A separate test supplies an oversized first document followed by enough work to fill the channel; the intended size error must return within a bounded test wait while the seven previously committed chunks remain unchanged. This check prevents a valid rejection rule from turning into a pipeline deadlock.

The database then contains exactly seven chunks from the initial single-paragraph sources. The test reruns the complete twenty-document input. Already accepted versions return unchanged, and the remaining sources are committed. A second full replay leaves the count at twenty.

This establishes a useful recovery property: the importer can resume from a partially completed batch without pretending the whole batch was atomic and without duplicating the completed source versions.

The checkpoint is implicit in the source/version ledger, not a single global cursor. Replaying the input asks the store which source revisions are already accepted. That is appropriate for this small idempotent fixture. A larger pipeline may need explicit checkpoints, ordering or a manifest of required source versions to avoid repeatedly scanning unchanged work.

Cancellation here is a controlled request observed between source operations. It does not interrupt SQLite halfway through an arbitrary commit or simulate machine failure. The separate precommit-failure test establishes rollback at the transaction boundary; the cancellation test establishes producer/consumer shutdown and resumability.

Naming both mechanisms keeps the evidence precise. A test that proves cancellation between records should not be described as proof of every crash-recovery condition.

Keep completeness and retrieval quality in the same report

The completed index is checked against the same eighteen held-out query fixtures used by M08. Exact dot-product ranking uses the preserved real vectors, and candidate eligibility is applied using the tenant field carried through ingestion.

Every manually expected source ID remains in the exact top three for these queries after the resume and update tests restore the original snapshot. That result shows that this import path preserved the particular evidence represented by those fixtures.

It does not show that a new chunking policy would produce the same quality. The paragraph texts and their embeddings are unchanged from the original experiment. The laboratory deliberately avoids reusing a whole-document vector for a newly invented fragment.

The source ID is the relevance identity, while the chunk ID identifies the stored version and position. Comparing only chunk IDs after a version change could incorrectly treat a successfully refreshed source as unrelated evidence. Comparing only source IDs without checking version could hide stale content. The report needs both levels.

The companion vector-index article then examines a different question: how candidate preselection can lose relevant or exact-neighbor documents even when ingestion preserved them correctly. Keeping those stages separate makes a quality regression diagnosable.

No generated answers are assessed. A source appearing in the result window does not prove that a later model applies its numeric limit or negative condition correctly.

The executed correctness boundary

The shared executable passed fifty-six named assertions. They cover controlled cancellation, failure-driven producer shutdown, complete resume, idempotent replay, paragraph chunk count, source deletion, newer-version replacement, stale-version refusal, transaction rollback, missing-embedding exclusion and same-version content or tenant conflicts.

The query checks cover expected-source presence and eligible scope for each of the eighteen cases. Additional checks support the companion index experiment's geometry, deletion, generation and version behavior.

The run uses SDK 10.0.400, net10.0 and runtime 10.0.15. Microsoft.Data.Sqlite and its native bundle are pinned in the project and lockfile. The database files live in a newly created owned fixture directory and are removed when the run ends.

The normal command is:

dotnet run -c Release -- check NEW_PATH/vector-fixture-check ../correctness.json

The directory must not already exist. This prevents a reproducibility command from treating a user's existing database directory as disposable scratch space. The source package includes the code and immutable input manifest, not the model weights or generated build output.

The test count is not a performance or reliability percentage. Several assertions concern the same corpus and implementation. The useful evidence is the named behavior and its actual execution, retained with the source version.

Measure the work that this importer actually performs

The measurement mode creates eighty synthetic source identities from four copies of the twenty known paragraphs. Their text is unchanged, so each uses an existing verified vector. This is a scaled import-work fixture, not eighty independent semantic documents and not a new embedding benchmark.

The timed ingestion path includes the channel handoff, paragraph processing, hashing, local transaction work and vector serialization. It does not include reading PDF files, downloading objects, computing new embeddings or sending documents to a hosted search service.

The runner records elapsed ingestion time, process-wide managed allocation and the tracked payload high-water value from the handoff. The allocation counter covers the operation's managed work across its tasks. It does not represent retained memory, native SQLite allocation or peak working set.

The tracked payload value is a different observation. It describes the fixture's outstanding document text at a specific logical boundary. A small value there can coexist with larger process memory because the corpus, vector map and runtime are already present.

These distinctions matter more than a single headline throughput figure. If an application spends most of its import time extracting documents or computing embeddings, this result cannot estimate its end-to-end rate. The measurement answers whether this particular local storage pipeline completes its bounded work under the recorded configuration.

The integration owner runs all timed work in the exclusive measurement lane. The article's result section must use those actual records rather than extrapolate from correctness execution or a convenient stopwatch around one method.

The observed import result has a narrow memory boundary

04 / See the idea

Memory observations have different meanings

Enlarge diagram

The integration owner ran three fresh processes on the final implementation, including the consumer-failure shutdown correction. Each measured the same eighty-source fixture, completed all eighty records and removed its owned temporary database files afterward. The ingestion batch times were 149.2961, 151.7088 and 150.6377 milliseconds. The median was 150.6377 milliseconds for the declared batch. Earlier measurements were retained as superseded rather than mixed into this comparison.

The managed-allocation observations were 3,536,632, 3,528,752 and 3,529,712 bytes. These are process-wide allocation deltas during the import interval, not the size of the resulting database or memory retained after the operation. SQLite also has native work outside that managed counter.

The tracked outstanding text payload reached 447 bytes in each measured batch. That small number belongs to the channel-adjacent tracking rule, not the whole application. The corpus array, dictionary of vectors, serialized values, database connection objects and runtime storage are outside the definition of that payload count.

Observation	Executed result	Boundary
Source identities processed	80 of 80	Four copies of twenty known paragraphs
Median ingestion batch	150.6377 ms	Channel, hashing and local commits
Median managed allocation	3,529,712 bytes	Process-wide allocation delta
Tracked payload high-water	447 bytes	Waiting, queued and current text
Exact retrieval checks	18 of 18 evidence sets present	Original twenty-source snapshot

The eighty identities should not be described as eighty independently authored semantic documents. Their text deliberately repeats the twenty verified representations, allowing the experiment to exercise import work without inventing new embedding data.

The timings also do not establish a universal throughput. Three processes on a shared developer host provide a bounded local observation. The experiment uses a single consumer, small documents and a local SQLite file. A remote store or an embedding stage could dominate the total cost in another application.

The valuable result is that the import and its checks completed under explicit conditions. The memory observations remain separately named so a reader cannot accidentally promote a small channel-payload counter into a claim that the entire process used only a few hundred bytes.

Preserve the failed version instead of repairing history

The update tests reveal several different states that an operator might otherwise see only as an indexing failure. An old version is stale. A changed body under the same version is a conflict. A changed tenant under that same version is also a conflict. A new body without a known embedding is a committed source whose vector representation is not ready.

Those outcomes deserve different responses. Retrying a stale version cannot make it current. Repeating a conflicting revision without resolving the producer's contract can conceal a data-integrity problem. A missing embedding needs preparation work, not a fabricated vector or a misleading success label.

The fixture's report preserves those distinctions through named assertions. It does not automatically decide how a production operator should resolve each case. That decision depends on the source owner, the version policy and the cost of temporarily excluding the material.

A useful operational extension would keep the original envelope, refusal reason and accepted current version available for inspection. The point is to explain why the index looks the way it does, not to overwrite every inconvenient record until the import dashboard becomes green.

This also changes how a team should discuss quality. If a required source is absent because the new representation is incomplete, the retrieval score is not the first place to investigate. The source lifecycle explains the missing candidate before the metric is involved.

Keeping that explanation close to the index makes the next repair more precise. It is a practical benefit of treating ingestion as a versioned state transition rather than a stream of text that happens to end in a database.

An honest incomplete index is better than a falsely fresh one

The strongest practical objection is that excluding changed text until an embedding exists can reduce the useful result set. That is true. The alternative of attaching an old vector to new content does not remove the problem; it hides it.

A production system needs an explicit readiness policy. It might keep the old version active until the new representation is ready, make the new source temporarily unavailable, or provide a lexical fallback with a clear version contract. Each choice changes what readers can expect during an update.

The fixture chooses one visible behavior: the new chunk is stored without a vector and is not returned by the vector reader. This makes the missing stage observable. It does not claim that the choice is ideal for every user experience.

The same discipline applies to parser limitations. Plain paragraph splitting is acceptable for this corpus because the expected units are known. It would be irresponsible to attach the same confidence to a scanned contract, a table with merged cells or a footnote whose meaning depends on page layout without testing those structures.

The goal is to make each new capability earn its own evidence. Add a parser when its required structures are specified. Add embedding computation when its version and failure behavior are controlled. Add remote storage when its update and visibility contract has been verified.

The local reference is useful because it shows those boundaries rather than hiding them behind the word ingestion. It can move documents quickly enough to measure, preserve accepted versions and expose incomplete semantic preparation. That is a sounder foundation for scale than an import counter whose relationship to the resulting knowledge is unknown.

Source code and execution evidence

Download the complete lab, source record and raw execution results. This package supports the specific execution claims in this article. It is not a production certification.