> ## Content Index
> Fetch the complete content index at: https://devinside.cloud/llms.txt
> Use this file to discover other available public pages before exploring further.

# RAG That Keeps Its Evidence: Eligibility, Coverage and Licensed Claims
- URL: https://devinside.cloud/decision-apis-08-rag-evidence-sufficiency/
- Published: 2026-11-13T11:04:18.000Z
- Updated: 2026-11-13T22:06:15.000Z
- Description: Filter before Top K, check that the selected set covers every required fact, accept only sentences their citations license, and carry an evidence identity into the next stage.
- Author: Marcin Borecki
- Tags: AI Engineering, .NET & C#, Series: Decision APIs, #series-decision-apis, #dapi08

A support case contains two facts that should make an engineer slow down. A Gateway G2 stopped reporting telemetry after a reconnect, and the customer is under an active audit hold. The first fact points to a firmware investigation. The second changes what support may recommend while that investigation is open.

The knowledge base has an apparently perfect match: a document containing every query word and a direct rollback instruction. It applies to firmware 2.4.0; the customer runs 2.4.1\. Another document scores just as high and belongs to a different tenant. The audit-hold exception sits lower in the ranking. A RAG pipeline that ranks first and asks questions later will either answer from the wrong document or answer without the exception.

- Retrieval relevance, evidence sufficiency and answer support fail independently. Test each one with its own check.
- Apply eligibility (version, tenant, model) before Top K. Ranking the whole corpus and filtering afterwards can leave zero usable sources.
- Sufficiency is a property of the selected set: every required fact must be covered by an exact, curated passage. No relevance score can stand in for a missing fact.
- A citation is a pointer, not a licence. Accept a sentence only if the cited passage supports that proposition, and carry the evidence identity into the next stage.

## Start with the question the application actually supports

This is not an open chat question. In the support desk built through this series, the case arrives with a typed question identifier, `gateway-explain-and-next-step-v1`: explain the G2 telemetry issue on firmware 2.4.1 and give the permitted next step while an audit hold is active. The words sent to the retriever are a separate, shorter string: `Gateway G2 telemetry reconnect audit hold rollback`.

Keeping those apart matters. The question identifier selects an application contract, in the same way that a decision task selected a contract in [Part 3](https://devinside.cloud/decision-apis-03-designing-the-decision-contract/). The query only supplies words to a ranker. A different question identifier returns `UnsupportedQuestion` before any retrieval or drafting. Adding documents cannot make an unsupported question supported; it needs a new contract.

For this question, the answer needs two facts:

| Required fact    | Why the answer needs it                                    | Curated passage                                                                                                |
| ---------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| known-issue      | Explain the symptom without claiming a confirmed diagnosis | GW-241 v3, span s1: "G2 firmware 2.4.1 may stop sending telemetry after a reconnect."                          |
| audit-constraint | State the permitted next step under the hold               | HOLD-7 v2, span s1: "While an audit hold is active, collect a diagnostic export and do not roll back firmware" |

"May" in the first passage is doing work. It supports a known possibility, not a diagnosis of this device. The answer has to keep that modality, which is why the application will later accept only a sentence it knows to be licensed by that passage.

## Eligibility belongs before Top K, not after it

The teaching corpus has six documents. Each carries an ID, a version, a device model, a firmware version, a visibility (public or a tenant) and exact spans: offset, length, quotation and a curated fact label. The loader derives each span's offset and length from its quotation and rejects a span whose quotation does not occur in the body, so a span can never point at text the document does not contain.

The ranker is deliberately plain: it counts distinct query terms that also appear in the document, case-insensitively, and breaks ties by document ID. The scores are integers of lexical overlap, not probabilities and not a retrieval benchmark. Two of the documents were written to match the query well. A relevance function has no way to know that "2.4.0" is the wrong version for this customer or that a document belongs to someone else. Eligibility is an application rule, and the question is where it runs.

Explorable

### Rank then filter keeps nothing. Filter then rank keeps both required passages.

Global ranking, query overlap What the prompt receives 1 OLD-240 v9 score 7 firmware 2.4.0 2 PRIVATE v1 score 7 other tenant 3 GW-241 v3 score 4 known-issue 4 HOLD-7 v2 score 4 audit-constraint 5 G2-USB v1 score 2 no required fact 6 BILL-42 v1 score 0 billing Take 2, then filter (nothing left) both top results were ineligible known-issue missing audit-constraint missing AskForEvidence 

1. Take the global top two, then filter. Both winners are ineligible, so the prompt receives nothing and neither required fact is covered.
2. Same corpus, same query, same K. Filtering first removes OLD-240 and PRIVATE, and the top two eligible sources cover both required facts.
3. Correct order, smaller K. GW-241 is eligible and relevant, but on its own it cannot establish what support may do under the audit hold.
4. A larger K adds G2-USB, which shares two query terms and no required fact. Coverage is unchanged: more passages do not mean more evidence.

Scores and outcomes printed by the C# program in this article for the six-document teaching corpus.

With K equal to two, the order of two LINQ operators decides whether the case can be answered. "Apply metadata filters" is correct advice that hides this consequence. Vector databases make the same choice explicit. In Azure AI Search, `vectorFilterMode` can be `preFilter`, which applies the filter during graph traversal and guarantees K results if they exist, `postFilter`, which filters each shard's results, or the preview `strictPostFilter`, which filters the unfiltered global top K and can return zero results for selective filters and small K. That last mode is exactly the "rank, then filter" row above. `preFilter` is the default for indexes created after about October 2023; older indexes default to `postFilter`.

There is a second reason to filter early: what the model sees. In this example the diagnostic ranking contains the distractors, because the corpus is a trusted local fixture loaded into memory. The specialist request built later must carry only the selected sources, and the test to write is on the wire: serialize that request and assert that it contains GW-241 and excludes PRIVATE and OLD-240\. Filtering only what the user sees would be too late: the other tenant's text would already be in the prompt. In production, enforce the tenant filter inside the search query, before any text is materialized, as described in [RAG in .NET: retrieve the policy the caller can actually use](https://devinside.cloud/rag-in-net-start-with-the-retrieval-contract/).

## Coverage is a property of the set, not of a passage

Return to K equal to one in the correct order. GW-241 is eligible and on topic. It still says nothing about the audit hold, and no amount of confidence in that one passage can produce the missing constraint. Sufficiency is not "the best passage scored high". It is set coverage: every required fact must be represented by an accepted passage in the selected set.

That raises the question of what counts as an accepted passage. A fact label such as `audit-constraint` is metadata someone typed. Suppose HOLD-7 is revised to say "roll back firmware only after approval" and the editor leaves the old label in place. A coverage check that trusts the label would report the constraint as covered while the meaning has reversed. The rule in this application therefore checks the quotation itself against the curated proposition for that fact, both when it computes coverage and when it chooses a citation.

Code walkthrough

### Eligibility, coverage and the claim gate in about forty lines

```csharp
public static bool Eligible(Doc d, Ticket t) =>
    (d.Model == "*" || d.Model == t.Model) &&
    (d.Firmware == "*" || d.Firmware == t.Firmware) &&
    (d.Visibility == "public" || d.Visibility == t.Tenant);

public static List<Ranked> Rank(IEnumerable<Doc> corpus, string query, Ticket t)
{
    var q = Tokens(query);
    return corpus.Select(d => new Ranked(d, Tokens(d.Body).Count(q.Contains), Eligible(d, t)))
        .OrderByDescending(r => r.Score).ThenBy(r => r.Doc.Id, StringComparer.Ordinal).ToList();
}

// Filter first, then take K.
var selected = ranked.Where(x => x.Eligible).Take(k).Select(x => x.Doc).ToArray();

// A span counts only if its quote is exactly the curated proposition for its fact.
public static IEnumerable<(Doc Doc, Span Span)> QualifyingWitnesses(IEnumerable<Doc> selected) =>
    selected.SelectMany(d => d.Spans.Select(s => (Doc: d, Span: s)))
        .Where(w => Domain.Quotes.TryGetValue(w.Span.Fact, out var q) && w.Span.Quote == q)
        .OrderBy(w => w.Doc.Id, StringComparer.Ordinal).ThenBy(w => w.Span.Id, StringComparer.Ordinal);

public static string[] Missing(IEnumerable<Doc> selected, string[] required)
{
    var covered = QualifyingWitnesses(selected).Select(w => w.Span.Fact).ToHashSet(StringComparer.Ordinal);
    return required.Where(f => !covered.Contains(f)).ToArray();
}

public static Verdict Check(Claim c, IReadOnlyList<Doc> selected)
{
    var doc = selected.FirstOrDefault(d => d.Id == c.DocId && d.Version == c.DocVersion);
    if (doc is null) return new(false, "cited source is not in the selected evidence");
    var span = doc.Spans.FirstOrDefault(s => s.Id == c.SpanId);
    if (span is null || span.Fact != c.Fact) return new(false, "span does not carry this fact");
    if (span.Quote != Domain.Quotes[c.Fact]) return new(false, "span text is not the curated proposition");
    if (c.Text != Domain.Clauses[c.Fact]) return new(false, "the rule does not license this sentence");
    return new(true, "");
}
```

1. Eligibility is a plain predicate over metadata: device model, firmware version and visibility. It knows nothing about relevance, and relevance knows nothing about it.
2. The ranker counts distinct shared terms and breaks ties by ID. It also records eligibility, so the diagnostic view can show what was excluded and why.
3. The one line that decides the case: `Where` before `Take`. Swapping the two operators is the failure variant shown in the explorable.
4. A witness is a span whose quotation equals the curated proposition for its fact. A stale label on changed text is not a witness. Ordering by ID makes the chosen citation deterministic.
5. Coverage is set arithmetic: required facts minus facts with a qualifying witness. Any missing fact turns the outcome into `AskForEvidence`.
6. The claim gate checks the cited source and version, the span's fact, the span's text and finally the sentence itself against the licensed clause for that fact.

Running the program against the teaching corpus prints the ordering result, the K = 1 gap and two variations of HOLD-7:

```text
K=2 take-then-filter: []  filter-then-take: [GW-241, HOLD-7]  missing=
K=1 take-then-filter: []  filter-then-take: [GW-241]  missing=audit-constraint
changed quote, old tag: missing=audit-constraint
version bump only: missing=[] snapshot ev-3803b9ab28ca85b85
question gateway-free-question: UnsupportedQuestion
```

The "changed quote, old tag" line is the important one. HOLD-7 v3 still carries the `audit-constraint` label, the span is structurally valid, and coverage still refuses it, because the text no longer states the proposition the contract depends on. A version bump with the same quotation is accepted, but it changes the evidence identity (more on that below).

This strictness has a cost. A harmless rewording of HOLD-7 also needs a person to re-curate it. That is acceptable for a small support rulebook with controlled statements and a review step; it is not a general method for open-domain question answering. The outcome name `SufficientForTemplate` says so: the selected set covers the requirements of this template under this rule version, not every reasonable reading of the customer's message.

## A citation is not a licence

Now the generator produces a candidate answer with three clauses, each carrying a citation. Two reproduce the permitted propositions. The third says "Roll back firmware now." and cites GW-241, a real, eligible, selected source. What should each kind of check conclude?

Worked example

### All three clauses have real citations. Only two are licensed.

Candidate clauses Four checks per clause "Firmware 2.4.1 on G2 gateways may stop sending telemetry after a reconnect." cites GW-241 v3 s1 "Because an audit hold is active, please send us a diagnostic export; we will not roll back..." cites HOLD-7 v2 s1 "Roll back firmware now." cites GW-241 v3 s1 1 source and version are selected 2 span carries the claimed fact 3 span text is the curated quote 4 sentence is the licensed clause citation present: 3 of 3 pass Licensed: 2 clauses. Withheld: the rollback instruction. 

1. The candidate has three clauses and three citations, all pointing at sources in the selected evidence. The gate runs four checks on each clause.
2. Clause 1 cites GW-241 s1\. The source is selected, the span carries `known-issue`, its text is the curated quote and the sentence is the licensed clause.
3. Clause 2 cites HOLD-7 s1 and passes the same four checks for `audit-constraint`. It keeps the "do not roll back" constraint intact.
4. Clause 3 cites the same real span as clause 1, so checks 1 to 3 pass. Check 4 fails: no rule licenses "Roll back firmware now." from that passage.
5. A check for citation presence, or even for eligible citations, accepts all three clauses. Only the proposition check withholds the instruction that contradicts the hold.

Verdicts printed by the program in this article: the rollback clause is rejected with "the rule does not license this sentence".

The rejected sentence has syntactic provenance and no valid route through the contract. That is the defect worth catching, because it is the one users act on. The program also rejects a clause that cites HOLD-9, which is not in the selected evidence, with "cited source is not in the selected evidence". Both are different from a missing fact, and both are reported as such.

Be clear about what this gate is. It selects licensed clauses; it does not judge entailment. A sophisticated paraphrase that a human would accept is rejected, and the gate cannot tell you whether the rulebook itself is right. Research on citation-bearing answers makes the same separation: the ALCE benchmark evaluates fluency, correctness and citation quality as separate dimensions, because a cited answer is not automatically a correct or fully supported one. A free-form generator needs proposition-level support checks with measured false accepts and false rejects; numbers, scope and negation are good places to start, as in [executable RAG oracles in C#](https://devinside.cloud/executable-rag-oracles-for-numbers-scope-and-negation-in-c/).

## Worked example: the naive pipeline and the one that keeps its evidence

#### Rank, prompt, render

```csharp
var top = await search.QueryAsync(query, top: 2, ct);       // global ranking
var allowed = top.Where(d => Eligible(d, ticket)).ToList(); // filter afterwards
var answer = await llm.AnswerAsync(question, top, ct);      // prompt sees both
return answer.Citations.Any() ? answer : Refuse();          // "has a citation"
```

The prompt contains another tenant's document, the audit exception never reaches it, and a rollback instruction with a real citation passes the only check.

#### Contract, filter, cover, license, carry

```csharp
if (ticket.QuestionId != Domain.QuestionId) return Outcome.UnsupportedQuestion;
var selected = Rank(corpus, query, ticket).Where(x => x.Eligible).Take(k)
                                          .Select(x => x.Doc).ToArray();
var missing = Missing(selected, required);
if (missing.Length > 0) return Outcome.AskForEvidence(missing);
var draft = await drafter.DraftAsync(ticket, selected, ct);  // selected only
var verdicts = draft.Claims.Select(c => Check(c, selected)).ToList();
return new EvidencePacket(ticket, selected, required, verdicts,
                          SnapshotId(ticket, selected));
```

Three different outcomes for three different repairs, and a packet the next stage can verify without retrieving again.

The drafter here can be a model or a template renderer. With a finite set of licensed clauses, a renderer that joins them is the stronger baseline: it cannot invent the rollback sentence in the first place. A model earns its place when the answer space is larger than the rulebook, and then it needs a support check you have evaluated.

## Carry the evidence into the next decision

The result of this stage is an `EvidencePacket`, not a string: the typed ticket, the selected source objects with versions and spans, the required and missing facts, the claim verdicts, and an identity for the selected evidence. That identity is a SHA-256 digest over length-prefixed fields: the question and case facts, then each selected document's ID, version, visibility, text and span positions, in a fixed order.

Two identities are worth keeping apart. A corpus fingerprint identifies everything the retriever could have seen; it changes when any document changes, used or not, and it is useful for reproduction. The selected-evidence identity covers only what the answer relied on. Bumping HOLD-7 from version 2 to 4 with the same quotation leaves coverage intact and changes the identity from `ev-d9602773866f...` to `ev-3803b9ab28ca...`, even though the draft can stay word for word the same. That is deliberate: a reviewer who approved the answer approved it on specific evidence.

[Part 9](https://devinside.cloud/decision-apis-09-bounded-agent-coordination/) consumes this packet directly: specialists must cite its spans, and an insufficient packet stops coordination before any specialist call. [Part 10](https://devinside.cloud/decision-apis-10-human-review-resume-idempotency/) includes the evidence identity in the operation the human approves, so a later change of sources cannot inherit an earlier approval.

Changed data is handled by ranking again, not by freezing a hash. A newly added passage with the exact permitted quotation can contribute coverage. When several passages support the same fact, citation selection takes the first qualifying witness ordered by document ID and then span ID with ordinal comparison, so the choice does not depend on enumeration order. A passage that only carries the right label, with different text, cannot displace the exact witness.

## Pitfalls

When a fact is missing, raising K looks like the obvious fix. Sometimes it works; often it adds passages like G2-USB that match query words and carry no required fact, and it pushes more ineligible text toward the prompt if the filter runs late. Fix the order first, then ask whether the required passage exists at all.

- **Filtering the display, not the prompt.** Assert on the serialized model input that excluded sources are absent.
- **Trusting fact labels.** Labels drift when text is edited. Check the quotation, or re-curate on every content change.
- **Averaging support scores.** A packet with one unsupported instruction is not "90% supported". Report each rejected clause.
- **Treating "unsupported question" as "no documents".** One needs a new contract, the other new content. Keep the outcomes distinct.
- **Reusing an approval across evidence changes.** Bind approval to the selected-evidence identity, not to the draft text alone.

## Which check answers which question

| Question                                      | Mechanism                                                                          | Fails as                               |
| --------------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------- |
| Is this question type supported?              | Typed question identifier and contract                                             | UnsupportedQuestion                    |
| May this source be used for this case?        | Eligibility predicate, enforced before Top K (in search: preFilter)                | Source excluded, never sent to a model |
| Is the selected set enough?                   | Coverage of required facts by exact curated passages                               | AskForEvidence with the missing facts  |
| Does the cited passage support this sentence? | Fact-to-clause licence (finite domain) or an evaluated support check (open domain) | Clause withheld, reason recorded       |
| Did the evidence change since review?         | Selected-evidence identity in the approval                                         | Stale approval refused                 |

## Checklist

- Every request carries a typed question identifier separate from the retrieval query.
- Eligibility (tenant, version, product) is applied before Top K, and the model input is tested for excluded sources.
- Each question type declares its required facts, and coverage is computed over the selected set.
- Coverage and citations rely on exact passages, not on labels alone.
- Answer clauses are accepted only when the cited passage licenses that proposition.
- Unsupported question, missing evidence and unlicensed clause are three different outcomes.
- The evidence packet carries a selected-evidence identity that downstream stages and approvals bind to.

**Series:** Decision APIs in .NET, part 8 of 10\. Previous: [Part 7, local models and Polish routing](https://devinside.cloud/decision-apis-07-local-models-polish-evaluation/). Next: [Part 9, when an agent coordinator has a real choice](https://devinside.cloud/decision-apis-09-bounded-agent-coordination/).

## Further reading

- [Vector query filters in Azure AI Search](https://learn.microsoft.com/en-us/azure/search/vector-search-filters?ref=devinside.cloud) (Microsoft Learn): `preFilter`, `postFilter` and `strictPostFilter`, and when each returns fewer than K results.
- [Security filters for trimming results in Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search?ref=devinside.cloud) (Microsoft Learn): enforcing who may see which document inside the query.
- [Gao, Yen, Yu and Chen, Enabling Large Language Models to Generate Text with Citations](https://arxiv.org/abs/2305.14627?ref=devinside.cloud) (ALCE, 2023): fluency, correctness and citation quality as separate dimensions.
- [RFC 6234, US Secure Hash Algorithms](https://www.rfc-editor.org/rfc/rfc6234?ref=devinside.cloud): the SHA-256 definition behind the evidence identity.