Executable RAG Oracles for Numbers, Scope and Negation in C#

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

Executable RAG Oracles for Numbers, Scope and Negation in C#

Dates and technology context

This is a retrospective editorial timeline. Topic context: 9 December 2025. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

Saved ONNX rankings,89-check evaluator, mutation outcomes and timing are12Nov2026 evidence. Keep the20,000,000-byte example synthetic.

Prepared on 12 November 2026. This laboratory uses saved results from an earlier, actually executed local embedding experiment and adds a new, deterministic C# evaluation layer. It does not run a new language model or claim to measure generated-answer quality.

A retrieval system can find the paragraph containing the right number and still leave the application with the wrong answer. The number may describe a different population. Its unit may have been changed. The passage may contain a prohibition that the answer reverses. A citation can be valid while the statement attached to it is not.

Those distinctions are difficult to see when evaluation consists of asking another model whether the response looks good. A model-based judge can be useful, but some parts of the task deserve a more explicit contract. If a declared file limit is twenty million bytes, a candidate claiming a different quantity should not receive credit merely because its explanation is fluent.

This article implements a small oracle for structured propositions. An oracle here means a procedure that decides whether a candidate satisfies a known fixture contract. It is not an oracle of universal truth. Its authority comes from manually authored expected facts and explicit rules, whose limitations remain inspectable.

The resulting system separates two questions. Did retrieval supply the evidence required by the fixture? Does a proposed structured claim preserve the expected value, scope, version and polarity? The laboratory can answer those questions independently without pretending that it generated or assessed a complete natural-language answer.

Begin where the existing retrieval result stops

The local retrieval laboratory used a pinned public sentence-embedding model through ONNX Runtime in .NET. Its evaluation companion compared lexical and dense rankings over twenty synthetic documents and eighteen evaluation queries. Both methods placed all manually required evidence inside their top three on that small set, while their first-result behavior differed.

That is a useful result, but it does not establish what an answer would say. A downstream component could choose the wrong document, omit a condition or reverse a rule. The next evaluation boundary needs something more specific than another retrieval score.

AI05 therefore preserves the previous dataset, model manifest and result JSON as immutable inputs. Their hashes are checked before replay. The earlier model is not downloaded or invoked by this program. There are no artificial replacement vectors either: the program reads the actual recorded rankings and explicitly calls that operation replay.

The new work is a set of hand-authored structured propositions and a C# checker. Its candidate records are also supplied by the fixture. Some are deliberately correct; others contain a controlled defect. This lets the test establish whether the checker detects the intended error without relying on a model to happen to produce it.

That boundary is narrower than full RAG evaluation and more useful than disguising a fixture as a live answer experiment. It gives the application a tested vocabulary for errors that later generated answers may exhibit.

A numeric fact includes the thing being measured

01 / See the idea

A number needs its boundaries

Enlarge diagram

The fixture's first numeric statement concerns a per-file upload limit. Another concerns total storage per organization. Both contain storage-related language and numbers, but they do not describe the same quantity.

Representing either as only a numeric value loses the condition that makes comparison meaningful. The proposition also needs a scope, a unit, a source version, an evidence identifier and an eligible tenant. For permission statements, it needs an explicitly defined action and the allowed or prohibited polarity.

The application-owned type in Oracle.cs makes those dimensions visible:

public sealed record Proposition(
    string Id, string EvidenceId, string Version,
    string Tenant, string Scope, string Unit,
    decimal? Value, bool? Allowed);

This is a deliberately small schema. It does not model every temporal, geographic or contractual qualifier that a real policy could require. Its six facts cover an upload limit, organization storage, audit-log retention, ambiguous payment retries, suspended-user deletion and cross-team export permission.

The facts are independently authored from the ranking algorithm. Their values are not inferred from similarity scores or generated by the component being evaluated. That independence is limited: the same project designed the synthetic corpus and annotations, and no external annotation panel adjudicated them. The article does not turn a fixture into a representative benchmark by calling its expectations ground truth.

The important engineering property is that an implementation cannot improve its score by changing its own definition of correctness. The expected proposition remains a separate input to the check.

Exact arithmetic begins with an explicit unit convention

The laboratory defines MB as one million bytes and GB as one billion bytes for its synthetic contract. That convention is stated in code and documentation. It is not a claim that every source using those labels has been interpreted correctly by default.

The conversion method accepts only supported units:

public static decimal Bytes(decimal value, string unit) => unit switch
{
    "bytes" => value,
    "MB" => checked(value * 1_000_000m),
    "GB" => checked(value * 1_000_000_000m),
    _ => throw new ArgumentException("Unsupported unit")
};

The tests accept twenty MB and 0.02 GB as equivalent to twenty million bytes. They reject twenty times 1,024 squared bytes as a different value under the declared convention. The point is not to resolve every naming convention in the world. It is to prevent the checker from silently changing the convention while evaluating an answer.

The implementation uses decimal for these decimal quantities. Microsoft documents its decimal-scaled representation and the distinction from binary floating-point types. That property is useful here because the selected conversions should not introduce avoidable binary rounding into an exact fixture comparison. It does not make arbitrary measurements exact or remove the need to choose a rounding rule for other domains. C# numeric types.

An unsupported unit is a refusal, not an invitation to guess. If a future application needs additional units, it should extend the contract and add independently calculated cases. The same applies to a value outside the supported numeric range. The checker should preserve that limitation rather than manufacture a comparison from a failed conversion.

Evidence membership is necessary but insufficient

A candidate first has to refer to evidence actually present in the supplied bundle. The identifier must also match the evidence required by the expected proposition. A real document with the wrong identifier does not become supporting evidence because it discusses a similar subject.

The implementation then checks version, tenant and scope before comparing the numeric value or permission polarity. This order makes the reason for refusal explicit. A candidate with an old version should not be described merely as having a wrong number if the more fundamental problem is that it applies a different rule.

The core gate begins as follows:

if (!bundle.Contains(candidate.EvidenceId) ||
    candidate.EvidenceId != expected.EvidenceId)
    return new(false, "evidence");
if (candidate.Version != expected.Version)
    return new(false, "version");
if (candidate.Tenant != expected.Tenant)
    return new(false, "tenant");
if (candidate.Scope != expected.Scope)
    return new(false, "scope");

These checks demonstrate mechanical relationships inside a controlled fixture. The tenant string is not an authentication system. The version label is a synthetic fixture version, not an assertion about a vendor release. The evidence identifier is not a cryptographic proof that the paragraph supports the claim.

That last distinction should remain visible even in a production implementation. Membership says that the candidate cited an allowed item. Semantic support asks whether the item actually justifies the claim. The present oracle can decide support only for the narrow typed facts and rules that were manually specified.

The benefit of the gate is therefore not universal certainty. It is a smaller surface on which a plausible answer can hide a particular known mistake.

Compare quantities only when their scopes match

02 / See the idea

Exact arithmetic cannot repair the wrong scope

Enlarge diagram

The comparison helper refuses to compare propositions with different tenants, scopes or units, or with missing numeric values. For comparable propositions, it returns less, equal or greater. For the rest, it returns clarification-required.

The upload limit and total organization capacity are a useful example. Their values can both be expressed in bytes, but that does not make them interchangeable. Asking which is larger numerically may be trivial, while asking whether the organization capacity permits a larger individual upload is a different proposition.

The fixture's scope guard prevents that second conclusion from being smuggled in through a correct arithmetic operation. Arithmetic can be exact while the comparison is meaningless for the user's decision.

This also clarifies what a refusal should communicate. The checker is not saying that the numbers are unknown. It is saying that the requested relation has not been established under a common scope. A useful application could ask which limit the user means or present the two quantities separately with their definitions.

The implementation does not attempt that conversational repair. It exposes a typed state that a later interaction can handle. Keeping the state separate from the explanation makes it easier to test whether the application preserved the distinction before judging whether it communicated it well.

A more ambitious domain could require scope relationships rather than exact string equality. That would be additional domain logic, not a reason to discard scope from the model. The small example chooses a conservative rule whose behavior can be checked clearly.

Negation needs a proposition with a defined action

The permission facts in the fixture are not loose sentences labelled positive or negative. They name an action and a condition: retry after an ambiguous payment timeout, deletion by a suspended user, or export of another team's records.

This matters because a general statement such as the user is allowed can conceal the object of permission. Allowed to view an invoice does not imply allowed to delete a project. Permission within one team does not imply permission across an organization.

For these declared facts, the candidate's boolean must match the expected polarity and its unit must identify a permission proposition. A candidate that flips prohibited to allowed is rejected even if the evidence identifier, version and tenant are otherwise correct.

The test is intentionally simpler than understanding arbitrary natural-language negation. It does not parse a document and discover every exception. It evaluates a candidate already expressed in the contract. That transformation from natural language into the typed candidate remains a separate step requiring its own evaluation if a model is used to perform it.

The distinction prevents a common overclaim. A deterministic checker rejecting a flipped boolean is not evidence that the overall RAG system understands negation. It is evidence that one dangerous class of structured output can be detected once the proposition has been represented correctly.

For a consequential action, that can still be valuable. The application can refuse a known contradiction rather than treating fluent language as sufficient authorization.

Change one condition and require the expected result to change

03 / Watch the mechanism

A changed rule requires a changed candidate

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

Metamorphic testing examines how behavior should change under a controlled transformation. In this laboratory, the transformations are deliberately small: express the same quantity in another supported unit, change the applicable version and value, or invert a permission fact in the expected fixture.

Equivalent unit expressions should preserve acceptance. A changed version should invalidate an answer tied to the old rule. A changed rule with a correspondingly changed candidate should be accepted under the new fixture contract.

The version example changes the upload proposition from fixture-v1 with twenty million bytes to fixture-v2 with twenty-five million. The old candidate is rejected. A candidate carrying fixture-v2 and twenty-five MB is accepted. These are synthetic revisions, not an actual policy change in the underlying M08 corpus.

The corpus copies remain immutable. The changed proposition is a local test object, so the experiment does not rewrite previously accepted evidence to make a new scenario appear historical.

For the permission transformation, the expected rule is explicitly changed and the candidate changes with it. That demonstrates that the checker follows the supplied rule rather than hardcoding every permission as false. It does not establish that a model would discover the changed rule in an updated document.

These transformations are useful because they test a relation between cases. A checker that simply rejects everything could pass many negative tests. Positive transformed cases require it to preserve valid behavior as well.

Deliberate mutations test the checker, not the model

04 / See the idea

Six facts, six deliberate defects each

Enlarge diagram

For each of the six expected facts, the harness creates one valid candidate and six invalid variants. It changes the evidence identifier, version, tenant, scope, unit and either value or polarity. The expected outcome of each invalid variant is rejection.

All thirty-six variants were rejected in the executed correctness run. All six valid candidates were accepted. The report retains the named checks rather than collapsing them into a claim that the entire system is secure or semantically correct.

The mutations are intentionally one-at-a-time. If a candidate has the wrong tenant and the wrong value, rejection tells us little about whether the numeric rule works. A single changed condition gives the test a more useful oracle.

The harness also checks an empty evidence bundle. A numerically correct candidate still fails when its required source is unavailable. This prevents correctness of the number from laundering a missing evidence relationship.

The design is not a complete mutation-testing framework. It is a bounded set of manually chosen defects that correspond to the contract's dimensions. Additional cases should follow a new requirement or a discovered gap, not a desire to increase the displayed test count.

The executable's purpose is to make those dimensions inspectable. A reader can add a new supported unit, scope rule or failure case and see which acceptance condition must change.

Replay the actual rankings without inventing answers

The second part of the program reads the saved lexical and dense result lists from M08. There are thirty-six lists: eighteen queries for each method. For every list, the harness checks whether all expected evidence IDs appear within the top three and records whether the first result is relevant.

That replay confirms the earlier evidence-coverage result under the preserved dataset hash. It does not generate structured candidates from those lists. The candidate-oracle tests and retrieval replay are intentionally separate observations.

The distinction matters because a complete pipeline could fail between them. A model might receive all required evidence but emit a proposition with the wrong scope. It might emit a correct proposition with an unsupported citation. It might decline to answer even when evidence is sufficient. None of those behaviors occurred in this fixture because no answer generator was invoked.

The new oracle provides a way to evaluate some of those future structured outcomes. It does not supply the missing outcomes itself.

The replay also prevents an attractive but invalid conclusion from perfect recall. In the saved data, required documents appeared within the top three while some first-ranked documents had the wrong scope for the question. The oracle's evidence and scope dimensions explain why these are separate stages of correctness.

This is the delta from the earlier articles: M08 established an actual embedding and ranking path, M28 inspected its distribution, and this article implements a narrow set of answer-contract checks that can sit after retrieval. It does not repeat the embedding experiment or pretend to finish a generation evaluation that remains unperformed.

The executed correctness report

05 / See the idea

What the 89 assertions cover

Enlarge diagram

The project targets net10.0 with SDK 10.0.400 and runtime 10.0.15 pinned locally. It uses no third-party package. The ordinary correctness command is:

dotnet run -c Release -- check

The completed run passed 89 assertions. Their distribution is useful to report because not every assertion represents an independent semantic case:

Check group	Assertions	What it establishes
Preserved input hashes	3	Exact copied artifacts match the manifest
Valid structured facts	6	The checker accepts the declared fixture propositions
Deliberate mutations	36	The six controlled defect classes are rejected
Unit, scope, evidence and rule transformations	8	Selected equivalences and changes have the expected result
Recorded retrieval coverage	36	Expected evidence remains in each saved top-three list

The total is not an accuracy percentage. Thirty-six retrieval assertions inspect lists from one small synthetic corpus, and thirty-six mutation assertions are constructed around six facts. The report should remain attached to that structure.

The runner writes its complete result to correctness.json. It also supports a separately controlled timing command, but correctness does not depend on a speed threshold. A fast incorrect oracle would be a worse component, not a successful optimization.

The files needed to reproduce the checks are included with the source package. Model weights are unnecessary because the original embeddings are not recomputed here. The manifest identifies the saved real model result, which preserves provenance without making this program a new inference benchmark.

The small timing experiment measures the checker only

The integration owner executed three fresh processes in the exclusive measurement lane. Each process performed the correctness work and then timed ten thousand calls checking the same valid upload candidate. A short preparation loop preceded the timed interval. No retrieval or model execution occurred during those calls.

The batch times were 23.8376, 24.1475 and 24.3574 milliseconds. The median batch time was therefore 24.1475 milliseconds, or approximately 2.415 microseconds per call when divided by the fixed operation count. This is a median of process-average observations, not a percentile of individual latency.

Each batch reported 320,168 managed bytes allocated on the current thread. The checker returns a new verdict record, and the harness has fixed overhead; the result must not be described as an allocation-free validation path. The observation is the complete declared call in this fixture, not a detailed attribution of every allocation site.

The measurement is intentionally narrow. It uses one successful candidate, not a representative mixture of missing evidence, wrong versions and invalid units. It excludes deserializing an answer, fetching a source, tokenizing input or presenting a refusal to a user. It also does not compare this implementation with an alternative oracle design.

Those limits determine the useful conclusion. The code can perform its small deterministic check with the measured local cost. The result says nothing about end-to-end RAG latency, the economics of a model call or whether adding the check improves a user's completed task.

If a production design needs a different question answered, extend the experiment explicitly. A high-volume parser might need a mixed workload with allocation-site inspection. A source-backed validator might need remote lookup costs included. A user-facing workflow might need to measure how frequently refusals lead to clarification or unnecessary retries. None of those costs should be inferred from the present microsecond figure.

The timing output contains an additional assertion that all ten thousand calls were accepted, bringing its total check count to ninety. The ordinary correctness report remains at eighty-nine. Keeping those counts separate prevents a reader from treating repeated valid calls as ten thousand new semantic examples.

A useful refusal identifies the next decision

A checker returning false can become frustrating if the application discards the reason. The fixture returns a small reason code so a later workflow can distinguish evidence, version, tenant, scope, unit, value and polarity failures.

Those reasons should not all trigger the same response. A missing evidence identifier may require another retrieval step. A version mismatch may require asking which rule applies. A scope mismatch may require clarification rather than another attempt to generate the same comparison. A tenant failure should not be repaired by broadening access merely to obtain a convenient answer.

The reason code is not itself a complete user message. It is an internal contract that helps the next layer avoid guessing what went wrong. A human-readable explanation should preserve the distinction without exposing irrelevant implementation details or implying that the source says more than it does.

The next layer also needs a stopping rule. Repeatedly generating candidates until one happens to satisfy the checker can conceal uncertainty and accumulate cost. Acceptance by a narrow oracle does not prove that every important condition has been represented in its schema. The workflow must keep the oracle's scope visible even when it returns success.

Where this oracle must stop

The strongest objection is that a hand-authored oracle simply encodes what its author already believes. That is partly true, and it defines the responsibility of the expected-data author. The facts need review, their source interpretation must be defensible, and disagreements should remain visible.

The response is not to remove the oracle and trust an opaque score instead. It is to distinguish independently specified expectations from the system under test, preserve the source relationship and acknowledge the range of questions the oracle cannot decide.

This implementation cannot resolve arbitrary legal language, infer every temporal qualifier, verify the truth of an external document or determine whether a generated paragraph is persuasive and useful. It cannot establish production authorization from fixture tenant strings. It cannot prove that a future model will represent the proposition correctly.

It can reject a candidate whose declared evidence, version, tenant, scope, unit or value violates a known contract. That is a concrete and testable contribution.

The next integration would place a real answer-producing component before the checker, freeze a new acceptance set and preserve both accepted and rejected candidates. It would need to distinguish parser failures, unsupported propositions and genuinely ambiguous questions. A rejection should lead to a useful product behavior, such as clarification or escalation, rather than disappear into a hidden retry loop.

Until that integration is executed, the right claim is limited: the deterministic evaluation layer works for the declared fixture and preserves the distinctions that a broader answer evaluation will need. It makes the remaining uncertainty smaller and easier to name without pretending to eliminate it.

Source code and execution evidence

Download the complete lab, source record and raw execution results. This package supports the specific execution claims in this article. It is not a production certification.