Imported is not yet searchable.

The job finishes. Every input file was read. The index still contains a paragraph from an older revision, or an embedding that describes text no longer attached to it.

That is not merely a retrieval problem waiting downstream. It is an ingestion contract failure. Throughput tells us how quickly the pipeline moved work. It does not establish which source version the published index represents or whether the fragment preserves the condition that made the source meaningful.

Consider a policy with two sentences: suspended administrators may not delete projects; active administrators may delete projects. A chunker that cuts after “may not” separates the action from its condition. A reuse cache keyed only by document identity can then attach an earlier vector to the revised text. The import can be complete as a file-processing task and misleading as an evidence-producing task.

Try the revision model. Change the chunk size, introduce the revised policy and interrupt the commit. Compare atomic revision replacement with incremental publication. The model computes which chunks are visible and whether their fingerprints belong to the selected text.

It is a plain-text construction, not a PDF parser or an embedding service. Its value is showing the invariants that must survive before a real model or vector database enters the picture.

Identify the source before fragmenting it

A path is not always a durable source identity. A file can change in place. A document can move. Two imports can observe different bytes under the same name. If a later reader asks which source supported a result, “the file at this path” may no longer be a sufficient answer.

A useful revision record contains a stable source identity, a revision identity or content fingerprint, the parser/chunking policy and the expected set of chunks. The relationship should be inspectable independently of the queue that transported the work.

public sealed record SourceRevision(
    string SourceId,
    string RevisionId,
    string ContentHash,
    string ParserVersion);

public sealed record Chunk(
    string SourceId,
    string RevisionId,
    int Ordinal,
    string Text,
    string TextHash);

These records describe identities, not a complete storage design. A hash can detect changed bytes under the chosen encoding and algorithm; it does not establish that the source is authoritative or correct. Those are separate provenance and domain questions.

A chunk boundary is an editorial decision implemented in code

Fixed-size chunks are easy to generate. They can also divide a negation, unit, exception or permission condition from the statement it qualifies. More overlap may reduce that risk while increasing duplicate content and index size. It is not a guarantee that the relevant relationship remains intact.

For the policy example, inspect the actual text that a retriever could return. Does one chunk contain both “suspended” and “may not delete”? If the answer requires joining two chunks, does the retrieval contract make that join likely and traceable? A chunk count alone cannot answer either question.

The visual highlights a split condition when the chosen word budget divides the first sentence. This is a deterministic text-boundary check, not a semantic model. It asks a narrow useful question: did the selected rule keep this known condition together?

A production parser may use headings, paragraphs, tables or document structure. Each choice should be versioned because changing the parser changes the evidence units. The same source bytes under a different parser can legitimately produce a different index revision.

Reuse the representation only for the text it represents

An embedding cache can save expensive work, but its key must describe the representation. At minimum, distinguish the exact chunk text and the embedding method's relevant identity. Model snapshot, tokenization, pooling and normalization can all change the resulting vector even when dimensions remain the same.

Reusing a vector because “this is document A, chunk three” is unsafe if the content or chunk boundaries changed. An ordinal is a position inside a revision, not proof of content equivalence across revisions.

string cacheKey = string.Join(":",
    embeddingContractVersion,
    chunk.TextHash);

This is a sketch of the identity rule. A real implementation needs unambiguous encoding and a collision policy appropriate to the application. The point is that cache reuse is justified by representation equivalence, not by a convenient filename.

The model's deliberately broken cache mode reuses by ordinal. When the policy changes, a chunk can display a vector fingerprint that belongs to the prior text. The mismatch is computed and shown; no actual embedding quality is invented.

A bounded queue is not a bound on every retained byte

Channel.CreateBounded<T> can limit the number of items waiting in one handoff. It does not account automatically for the producer's current item, consumers' current items, parser buffers or collections retained elsewhere.

var channel = Channel.CreateBounded<Chunk>(new BoundedChannelOptions(8)
{
    FullMode = BoundedChannelFullMode.Wait,
    SingleWriter = true,
    SingleReader = true
});

This historical .NET channel API is present in the .NET 9.0.4 source. The queue's item count is one resource boundary. If chunks vary greatly in size, the same count can retain very different amounts of memory.

Bound the unit that matters and record what the bound excludes. A queue of eight giant documents is not equivalent to a queue of eight small normalized fragments. Backpressure should eventually reach the source of work rather than merely move an unbounded collection to another stage.

Publish a complete revision or keep the previous one

Suppose revision two contains four chunks. The importer writes two and crashes. If readers immediately see those rows alongside the previous revision, the index can describe a source that never existed: half old and half new.

One design stages the new revision, validates its complete chunk set, and then atomically changes the publication pointer. Another replaces all relevant rows inside a transaction that is appropriate for the chosen database and workload. The implementation details differ, but the reader-visible invariant should be clear.

The model contrasts these choices. With atomic publication, interrupting the staged import leaves the old complete revision visible. With incremental publication, the same interruption can expose a partial new set. Neither result is hidden behind a generic “job failed” message.

This does not mean an importer must hold one enormous transaction for an entire corpus. A source revision can be a smaller publication unit, and a corpus manifest can coordinate multiple units if the product needs a stronger snapshot. The unit of atomicity should follow the reader's consistency requirement, not the easiest loop to write.

Cancellation should leave an explainable state

A cancelled job can leave staged work that is safe to discard or resume. It should not leave a published revision whose completeness cannot be established. Record which revision was being built, which chunks were accepted and whether publication occurred.

Retries need stable identities. An exact duplicate chunk delivery can be recognized. A repeated identity with changed content is a conflict that needs a policy. Treating every repeated key as harmless can replace evidence under the same name while the index continues to claim it is unchanged.

The current laboratory companion contains actual transaction and restart evidence separately. This edition's model demonstrates the publication invariant without claiming those later executions existed at its historical context date.

Completeness and retrieval quality need different checks

An import can be mechanically complete and still generate poor evidence units. Conversely, a few useful search results do not prove that every expected source revision was imported correctly.

Check the revision manifest, chunk identities, text fingerprints and publication boundary. Separately, keep questions whose answers depend on a number, negation, scope or exception that the chunking policy could damage. Evaluate the retrieved text, not merely whether a vector was stored for every row.

A changed parser deserves both kinds of regression check. If it creates fewer chunks, that may be an efficiency gain or a loss of important structure. If it creates more chunks, that may improve local context or flood retrieval with duplicates. The right interpretation depends on the questions the index is meant to answer.

A defensible throughput claim starts after the contract is fixed

Only after source identity, representation reuse and publication semantics are stable does a throughput comparison answer a useful question. Record what the importer actually performs: parsing, hashing, embedding, persistence or reuse of already computed vectors.

An importer that reads preserved vectors measures a different boundary from one that runs a model for every chunk. A queue-capacity bound is not peak process memory. A small local corpus is not production scale. Keep those distinctions attached to any later measurements.

The output I want from ingestion is an index that can name the source revision behind each evidence unit and explain its incomplete states. Speed matters, but a fast pipeline that confidently publishes the wrong source is mostly an efficient way to distribute uncertainty.

GO FURTHER

Inspect the contemporary laboratory.

This is a newly reconstructed historical edition, not a claim of earlier authorship. Later executed experiments, their exact environments and original results are preserved in a separate notebook.

Open the current laboratory notebook ↗

Download the current reconstruction sources ↗