Local Models and Polish Routing: Evaluate the Errors, Not the Average

A support desk routes English tickets well with a small local model, so the team tries the same setup on Polish. The first Polish result looks confident: the router sends a billing request to technical support, and the winning score is ahead of the runner-up by 0.18. The English version of the same ticket went to billing correctly, by a margin of only 0.02.

That inversion is the subject of this part of the series. A local model changes where inference runs and who operates it. It does not turn a similarity score into a probability, and it does not make a model trained for one language reliable in another. You find out what it does by evaluating paired cases and reading every error, not by reading one accuracy number.

Local is a location, not a capability

Earlier parts of this series built a decision contract around native decision answers: a choice with a probability for every option and a confidence value, a noul with the probability of yes, and a score on an ordered rubric. Part 5 wrote a C# client for the System One wire format. Can that client simply point at a local endpoint?

Only if the local service implements the same contract. Two do. Ollama announced support for decision models in version 0.35, served at POST /v1/systemone on the usual localhost:11434 address, with models such as nimble and the experimental tev1 and tev1:0.8b. Basal, an inference engine for "small, fast, calibrated typed-decision models for Polish and English", documents the same POST /v1/systemone shape with state and named, typed questions. A request to either looks like the hosted one:

{
  "model": "tev1:0.8b",
  "state": { "message": "Dawny problem z telemetrią jest naprawiony. Teraz proszę usunąć podwójną opłatę z faktury." },
  "questions": {
    "team": {
      "type": "choice",
      "instructions": "Which team should investigate this ticket first?",
      "criteria": {
        "technical": "Device faults, connectivity, firmware",
        "billing": "Invoices, charges, credits",
        "returns": "An explicit request to send the device back",
        "other": "Anything else, such as office hours or contact details"
      }
    }
  }
}

Implementing the protocol is not the same as being good at your task. The tev1 model page states that the model has not been fully tested on languages other than English. A Polish routing task on that model is an experiment, and the response still has to pass the validation from Part 3 before its probabilities mean anything. An ordinary chat endpoint such as Ollama's /api/chat is a different contract altogether: it returns generated text, and no amount of prompt engineering turns that text into a calibrated distribution.

Many teams start one step further away from a decision API: with a small embedding model they already run for search. That is a legitimate baseline, provided nobody mistakes its output for a decision probability. The rest of this article evaluates such a baseline, because its failure modes are exactly the ones a native model must also be tested for.

Freeze the model, the preprocessing and the decision rule

The baseline here is the kind of router many teams build first. Take sentence-transformers/all-MiniLM-L6-v2, pinned to a commit, exported to ONNX and run in process. Its model card lists English as its language, describes 384-dimensional sentence embeddings for retrieval, clustering and similarity, and states that input longer than 256 word pieces is truncated. It is not advertised as a multilingual classifier. That is the point of testing it on Polish: the result tells you whether "it works in English" carries over.

The decision rule is deliberately simple. Encode the ticket and four English prototype descriptions (technical troubleshooting, billing correction, an explicit return, office and contact information) with the same tokenizer, the same masked mean pooling and the same L2 normalization. Choose the prototype with the highest cosine similarity. For unit vectors, cosine similarity is just the dot product:

// Both vectors are L2-normalized by the pooling step, so |a| = |b| = 1.
double similarity = ticket.Vector.Zip(prototype.Vector).Sum(v => v.First * v.Second);
double margin = ranked[0].Score - ranked[1].Score;   // winner minus runner-up

On two random normalized 384-dimensional vectors, that dot product and the full cosine formula agree to 12 decimal places, so the shortcut is safe as long as normalization really happens. Pin everything that can change the vectors: the model commit, the ONNX file hash, the tokenizer vocabulary and configuration, the pooling rule and the prototype texts. A graph with identical weights but different token preparation is a different system, which moving ONNX inference behind HTTP shows in detail.

Note what the rule does not produce. No softmax is applied, and even if it were, a softmax over four cosines would only rescale distances. The cosine is a similarity in embedding space and the margin is the gap between the best two. Neither is called confidence in the code, and neither should be called confidence in a dashboard.

Design pairs around meaning, not word replacement

A bilingual evaluation set is not twelve English tickets run through a translator. Each pair should hold the intended meaning fixed while the language changes, and each case should exercise a cue that a router can get wrong. The teaching set in this article has twelve pairs, labelled under one rubric: which team should investigate first. Among them:

• a direct fault ("The gateway stopped sending telemetry after the last update");
• a negated return ("I don't want to return the device. I just need it to connect again");
• a historical fault followed by a current billing task (P03, the case in the opening);
• an explicit return after failed troubleshooting, and a rejection of further troubleshooting;
• a Polish return request typed without diacritics ("Chce odeslac urzadzenie i dostac zwrot pieniedzy");
• a resolved invoice followed by a question about weekend phone hours.

The twelfth pair is deliberately ambiguous: "It still does not work. Please help." / "To nadal nie działa. Proszę o pomoc." The referent is missing. It could be the device, the payment form or the return process. Forcing a gold label would hide that missing context, so the pair has no reference label. The router still returns a nearest prototype, because that is what nearest-prototype routing does; the evaluation records the prediction and excludes it from correctness.

The wrong answer can win by the widest margin

Start with the pair from the opening. The English ticket says the telemetry fault is history and the charge is the task. The Polish ticket says the same thing. What do the four similarity scores look like for each?

Nothing in that picture is a bug. A plausible reading is that an English-trained vocabulary still recognizes pieces of "telemetrią" and "problem", which pull the sentence toward the technical prototype, while "opłata" and "faktura" share nothing with "charge" and "invoice". Whatever the cause in a given model, the margin faithfully reports a distance in embedding space. It does not report correctness.

Read the result with the denominator intact

One pair is an anecdote. The harness below runs the whole set and keeps three measurements apart that dashboards like to merge: correctness per language, agreement between the two languages, and agreement on a wrong answer.

record RoutedText(string Pair, string Lang, string? Reference, string Predicted, double Margin)
{
    public bool Correct => Reference == Predicted;
}

RoutedText Route(string pair, string lang, string? reference, params double[] cosines)
{
    var ranked = cosines.Select((s, i) => (Label: Labels[i], Score: s))
                        .OrderByDescending(x => x.Score).ToArray();
    return new(pair, lang, reference, ranked[0].Label, ranked[0].Score - ranked[1].Score);
}

static void Report(IReadOnlyList<RoutedText> rows)
{
    foreach (string lang in new[] { "en", "pl" })
    {
        var l = rows.Where(r => r.Lang == lang && r.Reference is not null).ToList();
        Console.WriteLine($"{lang} correct {l.Count(r => r.Correct)} / {l.Count}");
    }

    var pairs = rows.GroupBy(r => r.Pair)
        .Select(g => (En: g.Single(x => x.Lang == "en"), Pl: g.Single(x => x.Lang == "pl")))
        .ToList();
    Console.WriteLine($"pairs agreeing {pairs.Count(p => p.En.Predicted == p.Pl.Predicted)} / {pairs.Count}");

    var labeled = pairs.Where(p => p.En.Reference is not null).ToList();
    Console.WriteLine($"labeled pairs agreeing on a wrong label " +
        $"{labeled.Count(p => p.En.Predicted == p.Pl.Predicted && !p.En.Correct)} / {labeled.Count}");
    Console.WriteLine($"unlabeled pairs outside the denominator {pairs.Count - labeled.Count}");
}

// Sweep a margin threshold: what is automated, what is wrong, what correct work is lost.
foreach (double t in new[] { 0.0, 0.02, 0.05, 0.10, 0.15 })
foreach (string lang in new[] { "en", "pl" })
{
    var labeled = rows.Where(r => r.Lang == lang && r.Reference is not null).ToList();
    var auto = labeled.Where(r => r.Margin >= t).ToList();
    Console.WriteLine($"margin>={t:F2} {lang}: automated {auto.Count}/{labeled.Count}, " +
        $"wrong among automated {auto.Count(r => !r.Correct)}, " +
        $"correct sent to review {labeled.Count(r => r.Margin < t && r.Correct)}");
}

Run on the twelve illustrative pairs on .NET 10, the report prints:

en correct 10 / 11
pl correct 6 / 11
pairs agreeing 7 / 12
labeled pairs agreeing on a wrong label 1 / 11
unlabeled pairs outside the denominator 1

Read it slowly. Pooled over both languages, that is 16 of 22, which sounds like a router that is right most of the time. Split by language, it is right 10 times in 11 in English and 6 times in 11 in Polish, which is a different product. Agreement of 7 in 12 could suggest the languages roughly match, but one of those seven agreements (P10, the resolved invoice followed by a phone-hours question) is both languages choosing billing when the reference is other. That is the failure that agreement alone would reward: a model that always answers "technical" agrees with itself on every pair.

Then list the errors, because the aggregate cannot tell you what to fix:

A pattern appears: most errors involve a sentence with two topics, one historical or rejected and one current. Nearest-prototype similarity has no notion of "current task"; it averages the whole sentence. That is a hypothesis to test, not a conclusion. The run does not reveal the model's internals, and the rows only show where to look next.

A margin threshold is a trade, and it trades differently per language

The natural reaction is to add a safety rule: automate only when the margin is large, and send the rest to people. How much does that buy in each language, and what does it cost?

The threshold that looked like a safety mechanism behaves like a volume control. It removes the low-margin errors, which were the easy ones to catch, and leaves the confident error in place. In English it removes correct work quickly because the English margins are compressed by mixed-topic sentences. Neither effect shows up in a single accuracy figure, and both would change on a different sample.

Two rules follow. First, if you do use a margin rule, choose it per language on one set of cases and evaluate it on another; a threshold picked by looking at these twelve pairs and then reported on the same twelve is a description of the data, not a test. Second, report the lost automation next to the avoided errors. The review queue has a cost and a capacity, which is the subject of Part 4 on thresholds and error cost. A threshold on a calibrated probability can be derived from those costs. A threshold on a cosine margin can only be tuned.

Inspect preprocessing before blaming the language

When a Polish case fails, it is tempting to write "the model does not understand Polish" and stop. Several cheaper explanations should be ruled out first, and the harness should store what is needed to rule them out:

• Truncation. The encoder drops everything after 256 word pieces. A long ticket whose current request comes last can lose it. Store the token count and a truncation flag for every text; none of the short teaching tickets comes close, but production tickets with signatures and quoted replies do.
• Segmentation. A tokenizer built on an English vocabulary usually splits Polish words into more and smaller pieces. Store token IDs and pieces, so that "more pieces" can be checked against "wrong label" instead of assumed.
• Orthography. P07 fails, and it has no diacritics. That does not show that missing diacritics caused the failure, because the set has no Polish version of the same sentence with diacritics. Add that within-language contrast, freeze it, then rerun.
• Prototypes. All four prototypes are English. That is a fixed design choice, which isolates the effect of the ticket language. A follow-up should add Polish prototypes as a separate arm, not as a silent change to the existing one.

Each item changes one variable while the pairs stay fixed. That is what makes the next run comparable to this one.

Worked example: testing a native local model on the same pairs

The next experiment replaces the similarity router with a native decision model served locally. The mistake to avoid is changing the task, the data and the model at the same time.

// New model, new prompt wording, new test tickets, and the first
// probability above 0.8 is treated as "safe to automate".
var answer = await client.DecideAsync(ticket.Text, ct);
if (answer.Probabilities[answer.Choice] > 0.8)
    await router.SendAsync(ticket, answer.Choice, ct);

// Pin: runtime version, model digest, task version, criteria text.
var raw = await local.PostSystemOneAsync(Requests.Team(text), ct);
var answer = DecisionContract.Validate(raw);        // Part 3 rules
if (answer is not Decision.Valid valid)
    return Row.Invalid(pair, lang, raw);             // kept, not dropped
return Row.Native(pair, lang, reference,
    valid.Choice, valid.Probabilities, valid.Confidence);

Before reading any number, record the runtime version (0.35 or later for Ollama's /v1/systemone), the exact model digest, the request and the raw response. Passing contract validation establishes compatibility. Running the bilingual pairs gives you the first semantic evidence. Neither establishes calibration: a probability of 0.9 means "right nine times in ten" only after you have checked it on enough labeled cases in each language, with the Brier score and reliability analysis from Part 4. Basal's documentation describes per-layout calibration files and marks answers without one as unvalidated, which is the right instinct: calibration is a measured property of a model on a distribution, not a label on the API.

Pitfalls

• Forcing a label on an ambiguous case. It inflates or deflates accuracy depending on which label you pick. Keep it unlabeled and route it to a clarification step in the product.
• Counting agreement as quality. Agreement is necessary for a bilingual product but says nothing about correctness. Always print wrong agreement next to it.
• Reporting accuracy only on automated cases. After a threshold, the automated slice looks excellent because the hard cases left. Report the full denominator and the review volume.
• Calling a local run a benchmark. Initialization, cache state and thread settings dominate small runs. Record timings as diagnostics of one run, not as a latency claim.
• Silent model updates. Local serving puts updates in your hands. Pin the digest and treat a pull of a new tag as a new candidate that reruns the pairs.

Which local option fits which job

Location matters for privacy, cost and operations, and those are real reasons to run locally. It is not a reason to expect better decisions. Choose the option by the contract you need and the behavior you have measured in each language.

Checklist

• The local endpoint implements the contract your client expects (/v1/systemone for System One clients), checked with a real request.
• Model commit or digest, tokenizer, pooling, prototypes and runtime version are pinned and recorded with every run.
• Evaluation pairs preserve meaning, each exercises a named cue, and ambiguous pairs carry no reference label.
• The report prints correctness per language, pair agreement, wrong agreement and the full error list.
• No similarity score or margin is labeled confidence anywhere in code, logs or dashboards.
• Any threshold is chosen per language on one set of cases, evaluated on another, and reported with the review volume it creates.
• Token counts and truncation flags are stored per text, so preprocessing can be ruled in or out.

Series: Decision APIs in .NET, part 7 of 10. Previous: Part 6, Azure: two contracts, one workflow. Next: Part 8, RAG that keeps its evidence.