RAG Quality as a Distribution, Not a Single Score

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

RAG Quality as a Distribution, Not a Single Score

Dates and technology context

This is a retrospective editorial timeline. Topic context: 10 July 2026. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

The local ONNX retrieval fixture uses SDK10.0.400/runtime10.0.15 and was executed November2026. Agent Framework1.0 was already available by July2026, but the later runtime and observations were not.

A retrieval-augmented answer can look excellent in a demonstration and fail for a reader who asks the same question differently. It can cite a real document while relying on the wrong version. It can answer common questions well and fail precisely where the consequences are highest. A single average score hides these differences unusually well.

My proposed unit of evaluation is a distribution of outcomes within a meaningful question family. The family represents a user need. Paraphrases test how that need survives different wording. Repeated runs reveal variability. Separate retrieval and answer judgments tell us where a failure began. Hard constraints prevent a pleasing average from laundering an unacceptable result.

This approach costs more than running a small list of prompts once. It earns that cost by making a release decision more specific: what improved, for whom, under which conditions, and at what risk?

Define the task before selecting a metric

Consider a documentation assistant for a .NET platform. A reader might ask how to cancel a queued export, whether cancellation guarantees that no output was produced, or how to recover after losing the browser connection. These questions share vocabulary but require different evidence and different answers.

An evaluation set built only by generating questions from document headings may reward recognition of the documentation's own wording. Real readers bring incomplete context, obsolete product names, incorrect assumptions and competing goals. Their questions are not necessarily shaped like the source material.

Define families around decisions: configure a feature, diagnose a symptom, compare alternatives, recover from failure, or establish whether a claim applies to a particular version. Record the intended audience and consequences of being wrong. A beginner's setup question and an incident-response question should not silently receive identical weight.

For each family, identify acceptable outcomes. Sometimes the correct answer is a concise explanation with a citation. Sometimes it is a clarification question. Sometimes it is an explicit statement that the authorized corpus does not contain enough evidence. Treating all abstentions as failures encourages confident invention.

Separate retrieval from the answer

Retrieval asks whether the system found sufficient, relevant and authorized evidence. Answer evaluation asks whether the response used that evidence correctly and communicated its limits. Combining these judgments too early makes debugging unnecessarily mysterious.

Azure AI Search supports vector retrieval and combinations of text and vector queries. These are retrieval mechanisms; selecting one does not establish that a resulting answer is accurate. Hybrid retrieval combines ranked result sets, while the downstream application still decides what evidence to present to generation. Azure AI Search vector overview, hybrid search overview

For a question with known evidence, inspect whether that evidence appears in the retrieved candidates, whether it survives any reranking or filtering, and whether the answer actually uses it. A relevant document outside the final context can indicate a context-selection problem rather than an embedding problem.

For a question without supporting evidence, retrieval can return plausible neighbors. That is not sufficient support. An evaluator must distinguish thematic similarity from an answerable claim. Otherwise the system receives credit for finding documents that merely sound related.

Authorization belongs before answer generation. An answer that is accurate because it read another customer's material is a failure, regardless of fluency or citation quality. Security trimming through a filter is an application mechanism whose identity and access rules must be correct. Azure AI Search security trimming

Make variation intentional

01 / See the idea

Evaluate families, variants and repetitions

Enlarge diagram

For each question family, create several phrasings that preserve the intended need. Include concise wording, conversational wording, a plausible incorrect assumption and a version-qualified request where relevant. Have a human verify that the variants still belong together. A paraphrase that changes the question cannot reveal instability for the original question.

Then repeat each variant under a recorded configuration. Repetition is useful even when a model parameter appears to reduce randomness. Provider infrastructure, retrieval ties, corpus updates and application concurrency can create variation elsewhere. Do not promise reproducibility from a single setting.

Record the corpus snapshot, retrieval configuration, prompt version, application build and provider deployment identity. If a provider does not expose an immutable model build, record that limitation. A familiar deployment name is not proof that identical model weights served every run.

Keep raw outcomes with their family and variant identifiers. Without that hierarchy, repeated easy questions can dominate the average. Ten variants of a common question should not accidentally outweigh a single important recovery scenario simply because they were convenient to write.

Store observations, not only judgments

The following C# types are application-owned examples for a .NET 10 evaluation harness. They do not represent an Azure AI Search or Microsoft Agent Framework SDK API. They are illustrative and have not been executed.

public sealed record EvaluationCase(
    string FamilyId,
    string VariantId,
    string Question,
    string CorpusVersion,
    string[] RequiredEvidenceIds,
    bool MayRequireClarification);

public sealed record EvaluationObservation(
    string RunId,
    string FamilyId,
    string VariantId,
    string[] RetrievedEvidenceIds,
    string[] CitedEvidenceIds,
    string Answer,
    string Outcome,
    double ElapsedSeconds,
    bool UnauthorizedEvidenceObserved);

These records deliberately separate the scenario from what happened. The evaluator can revise a rubric without rewriting the original observation. In a real implementation, evidence references need document versions and locators, and raw text needs an appropriate privacy and retention policy.

Store failures too: timeouts, tool exceptions, malformed output and cancellation. If the harness silently removes unsuccessful runs, the resulting distribution describes only the system's survivors. That can make an unreliable release appear accurate.

Operational metadata should support explanation without contaminating model input. The answer generator should not receive expected evidence IDs simply because the evaluation harness possesses them. A test that leaks its answer key measures a different task.

Use hard checks for hard rules

Some requirements admit deterministic checks. A cited identifier must belong to the evidence bundle. The bundle must belong to the authorized scope. A response must have a permitted output shape. A required version qualifier must be present in a structured field when the contract demands it.

Other requirements do not reduce so easily. Citation membership does not establish that the cited passage supports the claim. A correctly formatted answer can still be wrong. A model-based judge can help identify semantic problems, but its judgment is another uncertain observation.

Use deterministic checks as gates and semantic assessment as a separate layer. Retain examples of disagreements between automated judgments and human review. Calibrate on difficult cases, including confident errors, justified abstentions and partially correct answers.

If using a model as a judge, isolate the rubric from untrusted answer text and source material. Do not assume the judge is immune to instructions embedded in what it evaluates. Record the judge configuration and preserve enough evidence to revisit a surprising score.

A hard failure should remain visible even when it is rare. An unauthorized disclosure does not become acceptable because hundreds of ordinary answers were helpful. The release gate should express that asymmetry directly instead of relying on a weighted average to imply it.

Compare distributions that answer decisions

02 / A concrete example

A higher total can hide a regression

Enlarge diagram

For each family, report the number of cases, repetitions, useful answers, justified clarifications, justified abstentions and harmful or unsupported answers. Show latency and operational failures alongside content quality. A response that arrives after the user gives up is not equivalent to one that arrives in time.

Avoid reporting only an overall percentage. Present the numerator and denominator, especially for small groups. A perfect result across a handful of cases is weak evidence about rare failures. The absence of an observed failure establishes only that it was not observed in that sample.

When comparing two configurations, run them against the same held-out cases and preserve pairing. Inspect cases that changed from acceptable to unacceptable, not just the net difference. A release can improve a common family while breaking a smaller but more consequential one.

Uncertainty estimates should respect the data's structure. Paraphrases of the same question and repeated runs on the same evidence are correlated. Treating every run as an independent user problem can exaggerate confidence. Resampling at the question-family level is one possible analysis strategy, but its suitability depends on how families were sampled and what population the claim concerns.

The key discipline is modesty about inference. A carefully measured evaluation set still describes the tasks it represents. It does not automatically establish quality for another language, customer domain or document collection.

Keep tuning separate from acceptance

03 / Watch the mechanism

Keep acceptance cases out of tuning

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

A team will naturally inspect failed examples and improve chunking, prompts or retrieval settings. Those examples have now become development material. Continuing to call them an untouched test set makes the evaluation less credible with every iteration.

Maintain a development set for diagnosis and a held-out set for release decisions. Refresh the latter through a controlled process as real needs change. Record when cases move between roles. A fresh random split of near-duplicate questions is not necessarily independent if the same underlying document passages appear everywhere.

Use a change log that connects an intervention to its hypothesis. Increasing the number of retrieved chunks might improve evidence recall while increasing distraction, latency and cost. Reranking might improve relevance while failing on an unfamiliar vocabulary. A change should earn its place through the dimensions it affects.

For an agent-based implementation, freeze tool permissions and available tools as part of the configuration. Microsoft Agent Framework reached its version 1.0 milestone before this article's historical context, but a framework release does not establish the quality of a particular application workflow. The harness remains responsible for observing behavior. Microsoft Agent Framework version 1.0 announcement

No specific provider model or framework method is required by the proposed evaluation contract. Keeping the observation format independent makes it easier to compare adapters without redefining success around whichever SDK is currently convenient.

A concrete release conversation

Imagine that a candidate improves ordinary configuration answers but increases unsupported answers for version-migration questions. This is a hypothetical decision scenario, not a measured result. A single average can make the candidate look better because configuration questions are more common in the test set.

The distribution suggests a more useful response. Inspect the migration family. Are obsolete documents outranking current ones? Are version filters missing? Does the model collapse incompatible instructions into a confident synthesis? Each explanation implies a different repair.

The release decision could be to keep the existing configuration for migration questions, add a clarification step when version information is missing, or postpone the candidate until retrieval handles the boundary correctly. Those are product decisions informed by evidence, not attempts to maximize one score.

Also examine the cost of uncertainty. Requiring clarification for every question can reduce unsupported answers while making the assistant frustrating. The goal is an appropriate response to ambiguity, not universal hesitation. Include user effort and task completion in the evaluation when those outcomes can be observed responsibly.

What this method cannot promise

No finite evaluation set proves that a retrieval system will never invent a claim or expose unauthorized evidence. Corpus drift, permission changes and new user needs can invalidate assumptions. Evaluation supports a bounded claim about observed behavior under a recorded configuration.

The method is also not free. Human review takes time, repeated runs consume resources, and maintaining representative cases requires domain expertise. For a small informational assistant, a modest set of carefully chosen families may be more useful than an elaborate scoring platform that nobody maintains.

Start with the distinctions that would change a release decision. Preserve evidence, separate hard failures from graded quality, and grow the set when a new class of failure appears. Complexity should follow a demonstrated need for discrimination.

A trustworthy RAG evaluation tells us where the system is dependable, where it hesitates usefully, and where it fails in ways we cannot accept. That is a richer claim than an average score. It is also a claim an engineering team can act on.

A later experiment measures retrieval, not generated answers

Execution note, prepared 12 November 2026. The earlier July 2026 discussion describes a broader evaluation design. The following is a later, completed local retrieval experiment, not evidence that the original design had already been run.

The shared laboratory with the retrieval-contract article executes a real sentence-embedding model locally in .NET through ONNX Runtime. It compares dense ranking with a simple lexical baseline. No hosted embedding endpoint, chat model or model-as-judge service participates.

That narrower boundary is intentional. Before asking whether an answer uses evidence correctly, we need to know which evidence the retriever supplied. The experiment can therefore report document relevance and rank, but cannot report answer truthfulness, correctness of reasoning or the quality of a natural-language explanation.

The model is the official sentence-transformers/all-MiniLM-L6-v2 snapshot at commit 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. Its ONNX graph and tokenizer artifacts are pinned and hash-verified. The model card describes its intended sentence-similarity use; that description is not a guarantee that it preserves every numeric distinction or negative condition in a particular corpus. Official model card.

The .NET run uses SDK 10.0.400, runtime 10.0.15 and Microsoft.ML.OnnxRuntime 1.22.1. The exact setup and code are part of the shared laboratory. These execution pins belong to the November appendix and do not silently revise the historical configuration discussed earlier.

Freeze the questions before seeing the rankings

The dataset contains 20 synthetic documents about limits, retention, retries, access and export scope. Its statements are invented for the experiment. They are not customer policies, product guarantees or descriptions of any real organization.

There are 18 evaluation queries in three families: numbers, negation and scope. Two additional development queries are excluded from the reported family metrics. The expected relevant document IDs were manually written before the first model inference and kept unchanged after seeing the results.

This makes the set held out from tuning in this experiment. It does not make it an independent sample of real users. The same project authored the fixtures and relevance judgments, and no claim is made that an independent annotation panel validated them. A later production evaluation would need to address that limitation.

The expected set can contain more than one document. A question comparing an organization's storage capacity with a per-file upload limit needs both relevant boundaries. A question about whether meeting attendance establishes completed training needs the training rule and the meeting context. Merely finding one plausible neighbor is insufficient for complete evidence recall.

The queries deliberately include paraphrases and related formulations. That helps expose sensitivity to wording, but it also means the observations are not independent examples of arbitrary user needs. Repeating a concept in several forms should not be mistaken for discovering several unrelated classes of success.

The dataset hash is recorded in both preprocessing and evaluation output. This connects the score to the actual questions and expected IDs. It does not prove the judgments are correct; it prevents an unexplained change in the set from being hidden behind the same result label.

Three metrics answer three different questions

For each query, recall at three is the fraction of its manually expected document set present in the top three. If two documents are required and both appear, recall is one. If only one appears, recall is one half. This is an evidence-coverage measure within a small result window, not an answer-quality score.

Reciprocal rank at three asks where the first relevant document appears. A relevant document first receives one, second receives one half, and third receives one third. If none appears in the window, the value is zero. It can reward a query that finds one relevant document early even when another required document is missing.

Hit at one asks whether the first result belongs to the expected set. This is deliberately simple. It matters when a downstream component gives disproportionate attention to the top result, but it does not establish that the full evidence requirement has been met.

The program computes these quantities directly from document IDs:

int relevant = ranked.Count(h => query.Expected.Contains(h.Id));
double recall = (double)relevant / query.Expected.Length;
int first = Array.FindIndex(ranked, h => query.Expected.Contains(h.Id));
double rr = first < 0 ? 0 : 1.0 / (first + 1);

Family summaries are arithmetic means over the six queries in that family. The report retains the individual rankings and scores so a reader can inspect which cases drove the aggregate. We do not infer confidence intervals from six related questions, nor label the result a benchmark of all semantic search.

The lexical baseline uses token-set Jaccard similarity. Dense retrieval uses dot products of normalized sentence vectors. Both rank the same tenant-eligible candidates and use the same stable identifier to break ties. The comparison is consequently interpretable, but the lexical baseline is not BM25, and the dense method is not an approximate nearest-neighbor index.

The same perfect recall can hide different first results

02 / A concrete example

The wrong scope can still be the top neighbor

Enlarge diagram

01 / See the idea

Perfect recall hid different first results

Enlarge diagram

The executed family results are:

Family	Method	Queries	Mean recall@3	Mean reciprocal rank@3	Relevant first result
Numbers	lexical	6	1.000	0.917	5 of 6
Numbers	dense	6	1.000	0.833	4 of 6
Negation	lexical	6	1.000	0.806	4 of 6
Negation	dense	6	1.000	1.000	6 of 6
Scope	lexical	6	1.000	1.000	6 of 6
Scope	dense	6	1.000	1.000	6 of 6

Every manually required document appeared within the top three in these cases. That result is useful, but the first-result column immediately shows why it should not be the only reported number.

For the question asking the maximum size of one uploaded file, both methods ranked the organization-storage document first. The per-file limit document appeared lower in the same window. The two documents share vocabulary, yet they answer different questions about different units of scope.

For the question about a single 25 MB document, dense retrieval again ranked the storage document ahead of the upload limit. The lexical baseline placed the expected document first. Conversely, the dense method ranked the relevant evidence first for all six negation-family queries, where the lexical baseline did so for four.

These observations do not prove that the embedding model understands negation or fails at arithmetic. No arithmetic answer was generated. They show where the selected representation and ranking placed manually relevant evidence for this corpus and these formulations.

The next useful investigation follows the failed distinction. If a product often asks per-file questions, the storage-versus-upload ambiguity deserves attention even when aggregate recall looks perfect. Changing the retrieval policy should then be evaluated on a separate acceptance set, rather than repeatedly tuning against these now-inspected failures and continuing to call them untouched evidence.

Tenant eligibility is a gate, not a relevance feature

Several Red-tenant documents closely resemble Blue-tenant documents but specify different permissions or values. This makes the dataset useful for observing what happens when relevance is allowed to operate without an eligibility boundary.

The real ranking path filters candidates to the query tenant and public documents before selecting neighbors. It returned zero out-of-scope documents across the evaluation results. A separate unfiltered dense diagnostic produced 17 forbidden slots across the 18 top-three lists.

The unit matters: 17 slots does not mean 17 distinct documents, 17 users or 17 queries. The same forbidden document can appear more than once. The count illustrates a concrete failure mode in the synthetic setup, not the frequency of information leakage in a deployed system.

The filter also should not be mistaken for a complete authorization implementation. The query's tenant is supplied by a trusted fixture, not derived from a real identity provider. A production service would need to establish that scope independently of model-generated arguments and untrusted user claims.

This is why eligibility should not be averaged together with relevance. A highly similar forbidden document is not partially successful evidence. In this experiment, such a result would fail a hard boundary even if it improved the ranking metric.

The distinction is especially important when presenting a result to a nontechnical decision maker. A dashboard with one quality score can hide whether an improvement came from finding better authorized evidence or from searching material the user was never permitted to access.

Separate implementation agreement from semantic quality

Before interpreting ranks, the laboratory checks that Python and .NET produce matching normalized vectors from the same ONNX graph and token IDs. The maximum observed component difference across 40 inputs was approximately 7.38 × 10^-9, within the declared 10^-5 tolerance. Repeated inference and an added masked-padding case also passed their numerical checks.

This evidence supports the implementation path: tensor inputs, model output shape, mask-aware aggregation and normalization agree closely between the two runtimes. It does not create an independent semantic judge. Both implementations use the same model and therefore can share the same representational limitations.

The tokenizer is a separate preprocessing stage. The .NET program consumes verified token IDs rather than accepting arbitrary raw text through a production API. This limitation is deliberate and documented, so a reader can reproduce the actual boundary without assuming that a complete end-to-end text service has been built.

The model and dataset hashes are checked before inference. The output retains per-query expected IDs, eligible-candidate counts, top-three IDs and similarity scores. These artifacts make a surprising ranking inspectable without rerunning a paid or remote service.

No answer model, semantic entailment model or independent human answer assessment was run. The experiment therefore cannot establish whether a generated explanation would preserve the negation, compare the correct limits or ask for clarification. Those remain separate acceptance questions.

Repeating inference does not create new user needs

Three fresh .NET processes repeated the same inference and evaluation path. They retained the same reported rankings and numerical checks. The local repeated-input check reported no component difference, and adding masked padding also preserved the result within the declared tolerance.

This is useful execution evidence, but the repetitions do not turn 18 authored questions into 54 independent user needs. The corpus, model and expected document sets are unchanged. Treating each process run as a new semantic sample would exaggerate what the experiment establishes.

The timed interval also answers a separate question. It covers encoding all 40 prepared items, pooling and reference comparison after session construction. It does not measure end-to-end user retrieval, answer generation or the human effort required to evaluate a result. The companion implementation article reports those timing boundaries and raw values.

This distinction keeps the evaluation honest in both directions. Stable repeated execution is not evidence of broad real-world coverage, while a limited corpus does not erase the value of having run and checked a real model locally. Each observation should support the claim it can actually bear.

Grow the evaluation because of a new failure, not a better headline

This small dataset has clear limits. It contains no broad language variation, no genuinely long documents, no changing corpus, no relevance judgments from real users and no dedicated unanswerable-query family. Exact brute-force ranking over a small eligible set also tells us little about the behavior of a large approximate index.

Those limits suggest specific next steps. Add independently reviewed cases from the intended audience. Include questions for which the authorized corpus cannot support an answer. Test document updates and scope changes. Introduce larger retrieval machinery only when its behavior becomes part of the product decision.

The observed numerical ambiguity is a good reason to add fresh questions about units and scope. It is not a reason to rewrite the expected answers until the preferred method wins. Once a failure informs an implementation change, that example has become development material, and a new acceptance set should test whether the change generalizes.

The completed experiment is valuable because it is small enough to inspect and honest about the work it performed. It demonstrates real local embeddings, an explicit baseline, tenant-gated candidates and family-level rank differences. It does not replace the larger evaluation design described earlier. It gives that design a concrete, reproducible starting point.

Historical versions and verification boundaries

These pins existed before the context date. Availability and static API checks do not establish a successful build or a production-safe patch level. Execution claims are limited to explicitly identified companion checks.

Component	Version	Available by	Source
.NET SDK	10.0.100	2025-11-11	Primary record
.NET runtime	10.0.0	2025-11-11	Primary record
C#	12.0	2024-12-03	Primary record
Microsoft.Agents.AI	1.0.0	2026-04-03	Primary record
OpenTelemetry	1.9.0	2024-06-14	Primary record

Download the complete laboratory and execution records.