What Does a Correct RAG Answer Actually Cost?

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

What Does a Correct RAG Answer Actually Cost?

Dates and technology context

This is a retrospective editorial timeline. Topic context: 24 October 2025. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

The cost packet was assembled in November 2026 from later retrieval observations and explicitly synthetic cost assumptions. It does not retroactively establish an October 2025 provider bill or accepted-answer study.

Two retrieval-augmented generation configurations answer the same question. One uses a cheaper model and returns fewer tokens. The other retrieves more evidence, spends longer checking it and occasionally asks for clarification. If the comparison stops at the model invoice, the first configuration looks like the obvious winner. If its answer sends a person into twenty minutes of correction, the comparison has stopped at the wrong boundary.

“Cost per answer” is easy to calculate because software produces an answer-shaped object. “Cost per correct, usable answer” requires a judgement about what the object enables someone to do. That judgement is inconvenient, but it is also the reason to build the system.

This laboratory design treats an accepted task as the unit of analysis. It proposes a small .NET 9 reference application, a controlled document collection and a declared evaluation set. It does not report an executed benchmark or claim that one model is more economical than another. Its purpose is to make that claim testable without hiding failed attempts outside the denominator.

The proposed fixed baseline uses SDK 9.0.101, runtime 9.0.0, Semantic Kernel 1.45.0, Azure.Search.Documents 11.6.0 and OpenTelemetry 1.11.2. Its control model is Azure OpenAI GPT-4.1, version 2025-04-14, whose Azure launch predates this context. The Semantic Kernel release and NuGet publication records establish availability. The package graph, deployment region and model calls have not been tested. All workflow code below is original pseudocode, not a vendor API sample.

Define the task before the retrieval settings

Consider a hypothetical internal assistant answering questions about equipment warranty terms. The corpus contains several product families, document versions and jurisdiction-specific schedules. A question such as “Can this part be replaced?” is underspecified unless the assistant knows which product, purchase date and warranty schedule apply.

A convincing paragraph is not an adequate acceptance condition. For an answerable case, the proposed rubric requires the right conclusion, the relevant qualifying conditions and a citation to the applicable document version. For an unanswerable case, it requires a useful clarification or abstention. For a question outside the user's access boundary, it requires the system to preserve that boundary.

These outcomes should not all be called “correct answer” without qualification. Report them separately, then define which count as successful task completion for the experiment. A refusal can be correct behaviour while still leaving a business task unresolved. That unresolved work belongs in the workflow cost, even if it does not belong in the error count.

Build the expected answers independently of the system under test. A small set written and reviewed by people who understand the documents is preferable to a large set generated by the same model that will later be judged. Keep disputed cases visible. If two reviewers cannot agree on the source's meaning, the case may reveal a documentation problem rather than a model failure.

Give the corpus a version

01 / See the idea

Freeze evidence and processing separately

Enlarge diagram

The experiment needs more than a directory called documents. Record the exact source files, their hashes, their effective dates and the rules used to extract text. Store the chunking configuration and the relationship from each indexed chunk back to a document and passage.

Otherwise a comparison can change several things at once. A new model may be evaluated against a fresher index. A new chunk size may accidentally exclude a difficult appendix. A parser update may fix the very table that caused earlier failures. The resulting improvement could be real, but the proposed explanation would remain untested.

Use a corpus manifest as the stable reference. Changes to source documents create a new corpus version. Changes to the extraction or indexing pipeline create a new processing version. Keeping those identities separate allows a later reviewer to ask whether the system improved because the model changed or because the evidence became easier to retrieve.

Azure AI Search had generally available vector search from November 2023. That establishes availability for the October 2025 contextual baseline. It does not imply that every current retrieval feature or API existed then. The reference implementation should pin the historical SDK and use only the capabilities it actually verifies.

Follow the whole request through the accounting boundary

For each task, retain a ledger of attempts rather than only the final response. The ledger should identify retrieval calls, model calls, retries, validation steps, latency and the eventual acceptance decision. Shared indexing and storage costs should be reported separately with the allocation rule used to distribute them.

A retry is not a new user success. It is another attempt to satisfy the original task. A system that quietly discards failed attempts before exporting its cost report can make an expensive workflow appear efficient. Keep the task identifier stable across attempts and give each attempt its own identifier.

Human work needs a careful boundary too. Record active review time separately from waiting time. Include correction and escalation effort if the experiment's objective is workflow efficiency. Do not claim that a minute of saved review becomes a minute of realised business value automatically. That requires an additional assumption about whether the freed capacity can actually be used.

The following pseudocode describes the accounting relationship, not a runnable Semantic Kernel API example:

task = create_task(question, corpus_version, access_context)
for attempt in permitted_attempts(task):
    evidence = retrieve(question, corpus_version, access_context)
    draft = generate(question, evidence, model_configuration)
    record_attempt(task, evidence, draft, usage, elapsed_time)
    if stopping_rule_satisfied(draft):
        break
decision = independent_review(task, final_output)
record_task_outcome(task, decision, review_effort, correction_effort)

Semantic Kernel's stable C# 1.x foundation was available before this baseline. Its presence does not remove the need for the ledger. A framework can orchestrate calls while the application still loses track of which attempts belong to the same user request.

Make the denominator difficult to manipulate

02 / A concrete example

Equal unit cost, different unfinished work

Enlarge diagram

The core measure is straightforward once the boundary is explicit: divide the declared costs of all evaluated tasks by the number of accepted task outcomes. Report the acceptance rate beside it. A low cost per accepted task with a very low acceptance rate may describe a useful specialist tool, but it does not describe a general replacement for the workflow.

Use a toy example to see why both numbers matter. Suppose one configuration spends 30 accounting units on 100 tasks and 60 are accepted. Another spends 45 units and 90 are accepted. Each costs half a unit per accepted task under this narrow accounting rule. That equality says nothing about the remaining work, the severity of failures or the distribution of latency. These numbers are invented solely to illustrate the arithmetic.

Now include escalation. The first configuration leaves forty tasks for another route; the second leaves ten. If that route is costly, the apparent tie disappears. If the first configuration correctly refuses a distinct class of dangerous requests, the extra refusals may be desirable. The experiment needs the reason behind each outcome before a cost ranking can support a decision.

Avoid dividing by a success count that is estimated from a tiny or convenient subset. If reviewers inspect only the easiest outputs, the reported denominator is not the denominator of the workload. Either review all tasks in the bounded fixture or declare a sampling method with its uncertainty and limitations.

Compare architectures, not just model names

Choose a small number of configurations with a clear reason for each. A baseline might retrieve a fixed number of passages and generate once. Another might rerank evidence. A third might ask for clarification when the required product identity is missing. Keep the model constant while testing retrieval changes, then keep retrieval constant while testing a model change.

Only after those comparisons should the experiment examine combinations. Otherwise it becomes impossible to distinguish the contribution of better retrieval from that of a more capable generator. An impressive final result with no causal account is difficult to maintain when one component changes.

The strongest alternative to a more elaborate RAG pipeline may be a narrower interaction. A structured product selector could eliminate ambiguity more reliably than asking a model to infer the intended warranty from conversation. A curated table could answer a common question without generation. Include such an alternative if it addresses the same task.

This can feel unfair to the AI system because the simpler interface changes the user experience. In fact, the comparison reveals where the value resides. If a small interaction change solves the problem, the team has learned something more useful than which model won a leaderboard. The objective is a successful workflow, not preserving the complexity of the first proposal.

Cost has a distribution, and so does failure

An average cost can hide a small group of tasks that consume repeated retrieval and model calls. An average latency can hide those same tasks missing the user's deadline. Group the results by case type, document complexity and failure category, while avoiding categories so narrow that every task becomes its own explanation.

Inspect the long tail manually. Did the system retry an impossible request? Did it retrieve multiple near-identical versions of a document? Did a generated answer repeatedly fail the same validation rule? Each pattern suggests a different intervention. A general instruction to “use fewer tokens” does not address them equally.

Set stopping rules before execution. A maximum number of attempts, an overall deadline and a defined escalation route make the experiment bounded. They also create observable outcomes: completed, abstained, timed out, failed validation or escalated. Treating all unfinished tasks as missing data would erase precisely the behaviour that the experiment is meant to expose.

The evaluation should also distinguish wrong conclusions from weak evidence. A correct conclusion supported by the wrong document version is fragile. It may pass today's question accidentally and fail tomorrow's variation. Preserve that distinction in the rubric so cost reductions do not reward a system for making plausible guesses.

Try to make the preferred design lose

03 / Watch the mechanism

Make failure stay in the ledger

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

A useful challenge set contains realistic sources of ambiguity: a superseded document, a scanned table, conflicting wording, a question with missing context and a document that the current user must not access. Add cases in which the answer genuinely is absent. The point is to test the design's boundaries, not to construct an impossible puzzle collection.

Freeze a held-out set before tuning. If every failure becomes a prompt adjustment and the same cases are then used to report improvement, the evaluation gradually becomes a demonstration. Keep the development set and the final comparison set separate, and record any reason for changing either.

Falsification should be explicit. The claim that a cheaper model reduces total task cost fails if extra retries and review consume the savings. The claim that more retrieval improves quality fails if irrelevant passages reduce acceptance on the held-out set. The claim that an automated validator protects the workflow fails if it accepts a wrong but well-formatted answer.

NIST's generative AI profile provides a broader catalogue of risks to consider when deciding which failures matter. It does not supply the expected answers for this corpus. That remains the responsibility of the experiment's domain owner.

Use the result to make a bounded choice

The final report should contain the task set, corpus and processing manifests, all attempts, acceptance decisions, review effort and the allocation rule for shared costs. State the actual execution date. Leave empirical results unfilled until the fixture has run and the outputs have been independently judged.

The decision might be to use one configuration for routine questions and another for cases with conflicting evidence. It might be to retain human approval for a narrow action, improve the corpus before changing the model, or stop using generation for a problem that a structured lookup solves better. Each is a legitimate result if the evidence supports it.

The important improvement is conceptual. Once the unit becomes an accepted task, engineering effort follows the work that determines value: resolving ambiguity, obtaining the right evidence, controlling retries and making uncertainty visible. Token efficiency still matters. It simply takes its place inside a larger account of what the system must accomplish before anyone can call the answer useful.

A later packet connects observed retrieval with explicit assumptions

Contemporary laboratory note, prepared 12 November 2026. The original October 2025 sections describe a proposed complete RAG experiment. The following packet uses later executed retrieval and ledger evidence; it does not claim that the earlier model configuration was run.

The new packet begins with the saved results of the local retrieval laboratory and its family-level evaluation. Those experiments performed real local embedding inference and compared dense ranking with a lexical baseline. They did not generate answers.

That distinction controls the accounting. We have an observed ranking outcome for each of 18 queries. We do not have a measured answer-acceptance decision, a provider invoice or a record of human correction effort. The packet therefore keeps actual monetary cost, accepted generated answers and human minutes as null.

The exercise is a sensitivity calculation: apply explicitly invented accounting units to observed retrieval outcomes and examine which assumptions change the comparison. It is not an estimate of the real price of a correct RAG answer. The title's larger question remains open until the complete workflow supplies the missing evidence.

The shared Python packet is small enough to inspect. It reads frozen JSON artifacts, verifies their hashes, calculates exact fractions and checks its expected denominators. It performs no network access, model inference or timed benchmark. The original experiments are not rerun by reading their results.

This is a useful intermediate step because it prevents unknown parts of the cost model from being smuggled in as observations. A planning assumption can still be valuable when its status is explicit.

A retrieval hit is not an accepted answer

01 / See the idea

Observed counts, synthetic rates

Enlarge diagram

The observed first result was relevant for 15 of 18 lexical queries and 16 of 18 dense queries. Both methods included all manually required evidence within their top three for this small synthetic corpus. These are different observations, and neither equals the number of correct generated answers.

For the packet, the qualifying denominator is explicitly the number of queries with a relevant first-ranked document. That is a deliberately narrow proxy chosen to make the calculation concrete. The output names it top1_qualifying, not accepted answers.

Why not use the top-three result instead? We could, if the planning question concerned an interaction that inspects all three. The denominator would then be 18 for both methods. But that would describe a different workflow assumption. Choosing whichever denominator makes a preferred method look cheaper would make the comparison meaningless.

The complete RAG task would require further judgments. Did the answer use the relevant evidence? Did it preserve the difference between an upload limit and total storage? Did it communicate a missing condition rather than invent a conclusion? The retrieval result cannot answer those questions simply because the expected document was available somewhere in the window.

The same caution applies to failures. A nonrelevant first result does not prove that a downstream answer would be wrong. It identifies a condition under which the downstream component would need to find and use another result correctly. The cost of doing so remains an assumption until observed.

Keeping these distinctions in field names is part of the engineering. It makes later misuse of the output harder, even when the numbers are copied into a presentation.

Four scenarios show how the missing cost can reverse the comparison

02 / See the idea

An assumed follow-up cost changes the ordering

Enlarge diagram

The packet assigns one synthetic unit to each lexical retrieval and two synthetic units to each dense retrieval. These numbers are invented, not derived from the inference timings or a service tariff. It then assigns a hypothetical follow-up cost to each query whose first result was not relevant.

With four units per follow-up, the totals are:

Method	Queries charged	First-result qualifiers	Follow-ups assumed	Synthetic total	Units per qualifier
Lexical	18	15	3	30	2.00
Dense	18	16	2	44	2.75

All 18 initial queries contribute to the numerator. The three or two nonqualifying queries do not disappear merely because they are absent from the denominator. The follow-up is a hypothetical cost attached to those cases; it does not turn them into additional accepted outcomes.

Now change only the assumed follow-up cost to 20 units. The lexical total becomes 78 and the dense total becomes 76. Dividing by the same qualifying counts gives 5.20 and 4.75 respectively. The ordering reverses because the method with fewer first-result misses incurs fewer assumed follow-ups.

This reversal does not establish that dense retrieval is economically preferable. It shows that the conclusion depends on an unmeasured part of the workflow. The next useful observation concerns what those follow-ups actually require, whether they occur, and whether the top-result proxy represents the intended user task.

The exact arithmetic is simple. The discipline is to keep the status of each input visible. Observed counts and synthetic rates can coexist in a model without pretending that their combination is a measured invoice.

Preserve failed work and unknown usage in the ledger

The agent cost-ledger laboratory supplies a separate, executed example of accounting integrity. Its saved trace contains a task decision arriving before all attempt information, an unknown usage quantity, a later revision and a duplicate event.

The packet checks that an incomplete history does not produce a precise ratio. It also checks that the final synthetic ledger contains nine known units and one accepted task, including work on a task ultimately rejected. Replaying a duplicate observation does not change the saved snapshot.

These are properties of that ledger fixture. They do not validate the invented rates in the retrieval scenario, and the two laboratories have not been integrated into one live agent. Their connection here is analytical: one supplies observed retrieval outcomes, while the other demonstrates why an accounting record must preserve attempt identity and incomplete information.

A real combined system would need a stable task identifier across retrieval, generation, review and correction. It would also need a rule for allocating shared work such as indexing. The packet does not fill those gaps with an arbitrary per-query fraction and then call it measured cost.

Unknown is especially important. A missing usage report should not be treated as zero simply because zero makes the aggregation convenient. A precise ratio built from incomplete costs is more misleading than a partial report that says which observation is missing.

Reproduce the calculation without reproducing a benchmark

The source package contains a manifest, frozen input artifacts and check.py. From the packet directory, running the script verifies the input hashes, performs the arithmetic and writes result.json.

The ratio helper uses exact rational arithmetic:

def ratio(total, accepted):
    return None if accepted == 0 else str(Fraction(total, accepted))

The zero-denominator result is unknown rather than an invented zero cost. The supplied checks verify the four synthetic totals and ratios, the 18 unique query identifiers for each method, and the observed 15/16 qualifying counts. They also check selected ledger and authority records used by the companion diligence articles.

The completed packet run passed 26 named validation groups. This is an executed data-validation and aggregation script, not a new performance experiment or a new review of generated answers. Hash verification establishes that the packet read the preserved bytes; it does not independently certify the semantic interpretation of those bytes.

The source package makes that interpretation inspectable. A reader can change a planning assumption in a separate copy and see which conclusion depends on it. They should preserve the original result and label the modified scenario rather than overwrite the evidence history.

The observed retrieval counts remain separate from the synthetic rate card in the output. This separation matters when another reader wants to challenge an assumption: they can replace a planning rate without rewriting the source observation. They should also preserve the original scenario as a named comparison rather than silently changing what an earlier published conclusion meant. The packet is designed to make that distinction inspectable, not to make one chosen cost model authoritative.

Decide what must be measured next

The strongest objection is that synthetic rates are too arbitrary to support a commercial decision. That is correct if they are presented as prices. Their legitimate role is narrower: expose which missing variable could change the ordering and therefore deserves attention.

If follow-up effort dominates the comparison, observe a representative follow-up process. If indexing cost matters, record the corpus refresh pattern and choose an explicit allocation rule. If answer quality matters, evaluate generated answers under an independent rubric rather than substituting retrieval recall.

The next experiment should also allow the simpler system to win. A structured selector might remove the ambiguity between a per-file limit and organization capacity without a more expensive model. A human might inspect three results quickly enough that the first-result proxy is inappropriate. Either finding would improve the decision by changing the model to match the work.

The useful deliverable is therefore not a claim that one configuration is cheaper. It is a transparent statement of what has been observed, what has been assumed and which missing observation would make the cost comparison meaningful. That is how a ledger becomes an instrument for better decisions instead of a machine for producing convincing fractions.

Historical versions and verification boundaries

These pins existed before the context date. Availability and static API checks do not establish a successful build or a production-safe patch level. Execution claims are limited to explicitly identified companion checks.

Component	Version	Available by	Source
OpenTelemetry	1.11.2	2025-03-04	Primary record
OpenTelemetry.Extensions.Hosting	1.11.2	2025-03-04	Primary record
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.11.2	2025-03-04	Primary record
Microsoft.SemanticKernel	1.45.0	2025-04-03	Primary record
Azure.Search.Documents	11.6.0	2024-07-17	Primary record
.NET SDK	9.0.101	2024-12-03	Primary record
.NET runtime / ASP.NET Core shared framework	9.0.0	2024-12-03	Primary record
Azure OpenAI GPT-4.1	gpt-4.1 / 2025-04-14	2025-04-14	Primary record

Download the complete laboratory and execution records.