What a Decision Probability Actually Computes: Logits to Labels

Series: Decision APIs in .NET, part 2 of 10. Previous: Part 1, one judgment inside a working support desk. Next: Part 3, designing the decision contract.

Three services route the same support ticket and return the same object:

{"department": "technical", "probability": 0.88}

The first generated those characters token by token under a JSON schema. The second ran a classification head with one output per department. The third scored each complete department name as a string and compared the results. The client deserializes three identical records. Yet the 0.88 means something different in each case, fails in different ways and costs a different amount of compute to produce.

Before a router treats that number as "the probability this ticket belongs to technical support", it needs to know which calculation produced it and over which alternatives. This article derives the mechanisms with small, fully specified numbers. Every figure below was computed by the C# program in the article; the token probabilities are a teaching construction, not a measurement of any commercial model.

A probability is a share of a denominator someone chose

At each output position a language model assigns a score, called a logit, to every token in its vocabulary. A token may be a word, a fragment of a word, punctuation or another encoded unit. Logits are not probabilities: they can be negative or much larger than one. Softmax turns a finite vector of scores into non-negative values that sum to one:

p(i) = exp(z(i)) / sum over j of exp(z(j))

Take two candidates with logits 2 and 0. Their exponentials are about 7.389 and 1, so the first gets 7.389 / 8.389 = 0.880797 and the second 0.119203. Nothing in the arithmetic knows whether the candidates are departments, Yes and No tokens or chess moves. The meaning comes from the task around it.

Now add a third candidate with logit 2. The first candidate's logit has not moved, yet its share falls to 0.468311. What happens when a taxonomy grows over time? Use the slider to add candidates one at a time.

This is why a candidate set needs a version. A technical=0.88 from a two-label task and a technical=0.47 from a three-label task are not evidence of a model regression; they answer different conditional questions. If the new label describes a real category that used to be forced into the old ones, the new task can be more useful despite lower-looking numbers. Part 3 makes that version a field of the contract.

Compute softmax without overflow, and know what you discarded

Exponentiating a large logit directly overflows: Math.Exp(1002) returns Infinity in .NET, and the division that follows produces NaN. Subtracting the maximum score first does not change the result, because the common factor exp(-max) cancels between numerator and denominator:

public static double[] Softmax(double[] logits)
{
    if (logits.Length == 0 || logits.Any(x => !double.IsFinite(x)))
        throw new ArgumentException("A nonempty finite logit vector is required.");
    double max = logits.Max();
    double[] w = logits.Select(x => Math.Exp(x - max)).ToArray();
    double sum = w.Sum();
    return w.Select(x => x / sum).ToArray();
}

With logits 1002 and 1000 the method computes exp(0) and exp(-2) and returns 0.880797 and 0.119203, the same as for 2 and 0. At least one weight is exactly 1, so the denominator cannot underflow to zero for a non-empty finite vector. Rejecting NaN, infinity and an empty vector is an input rule, not a calibration technique. Very unlikely alternatives can still underflow to zero; if later code needs their log probabilities, keep values in log space rather than taking the logarithm of a rounded zero.

The bigger question is which logits went in. Keeping only two vocabulary entries and renormalizing them discards the mass on every other token. Google's ShieldGemma notebook does exactly this, deliberately: it takes the logits of the Yes and No tokens at the last prompt position, stacks them and applies a two-way softmax. A result of 0.99 between Yes and No can coexist with a model that mostly wanted to begin an explanation. That is a sound scoring method for a model trained and evaluated that way. It is not a general description of every endpoint that returns a probability.

A label is a path of tokens, and the path has an end

Real labels often span several tokens. Suppose, in a deliberately invented tokenizer, tech repair needs two tokens and billing needs one. Each also needs an end marker, so the scorer can tell billing from billing details. If a scorer looks only at the first position, which label wins, and is that the label with the highest probability?

The product of conditional probabilities is the probability of the exact complete answer. Long products underflow, so scorers add log probabilities instead and apply the stable softmax to the totals:

log P(label | input) = sum over t of log P(token t | input, earlier label tokens)

tech repair + end:  log 0.55 + log 0.20 + log 0.90 = -2.312635
billing + end:      log 0.30 + log 0.95             = -1.255266
softmax of the two totals                          -> 0.2578125, 0.7421875

A scorer supplies each candidate's earlier tokens while it evaluates the next one, which is usually called teacher forcing. That is different from sampling a free answer and hoping its tokens reveal the likelihood of every alternative. It also means scoring costs one forward pass per candidate path unless the implementation shares the common prefix.

The end marker matters because labels share prefixes. Scoring the prefix billing without the terminator treats billing details, billing address and every other continuation as if they were the intended label. A real scoring protocol must fix the answer template, the leading whitespace and the stop symbol, and changing any of them can change the result.

A cleaner design gives labels short, neutral identifiers and puts the descriptions in the prompt. Even then, tokenization, leading spaces and the distribution of prompts still matter. Inspect the actual tokenizer before assuming two labels are one token each.

Constrained decoding changes what can be emitted, not what is true

A constrained decoder prevents invalid continuations under a grammar or JSON schema. Conceptually it masks disallowed tokens, then samples or selects from what remains. At the start of an enum value it permits only tokens that begin technical or billing; after the first branch, only continuations consistent with that branch survive. What does the renormalized distribution say about the model's belief?

That may be exactly the decoding procedure you want. It is not independent evidence of correctness. OpenAI's Structured Outputs guide says the same from the provider side: the schema constrains completed, non-refusal output, but the application must still handle refusals, incomplete responses and API errors, and validate any business rules the schema cannot express.

Do not try to rebuild a full candidate distribution from returned log probabilities either. The Chat Completions top_logprobs parameter returns at most 20 alternatives per generated position, and the reference notes that fewer may come back. Those are alternatives for the tokens that were generated, not scores for every multi-token label you care about. Reading the first position of a generated enum gives you the first-token view from the previous section, with its reversal risk.

A classification head has a different output space

A conventional classifier maps an input representation h to one score per class with a learned matrix and bias:

z = W h + b
p = softmax(z)

Its output dimension is the class set, so no label is ever spelled through a vocabulary, and training can reward the correct class directly. Computing h may still mean running a transformer over the whole ticket, so a short output does not imply a cheap request. The cost of the head itself is usually small next to the cost of reading the input.

The trade-off is flexibility. Adding a genuinely new class, such as the firmware team in the slider above, normally requires new training data and a new head. A model that reads label descriptions from the prompt can absorb a changed candidate set without retraining, at the price of the tokenization and normalization issues above. A stable, well-labelled routing task is an excellent place to try a small classifier first; a taxonomy that changes monthly with few examples per class may justify the more flexible approach.

Specialized decision models add a further variable. TypeSafe describes its Jev model as a new architecture with a parallel sampler, trained with what it calls Reinforcement Learning for Calibrated Decisions, and states that higher confidence means higher accuracy. Those are provider claims; the announcement publishes no calibration measurements, and a C# client cannot inspect the internals. Whether the probabilities are calibrated on your tickets is something you measure, as Part 4 does.

What a native decision API actually hands you

Both native interfaces return a full distribution for a choice and a separate confidence. They are not the same number. System One documents confidence as a value computed from how the probabilities are spread, high for a single peak and low for a flat shape, and its own documentation example shows a choice with probability 0.40 and confidence 0.20. OpenAI's Decisions guide says choice and score answers return a distribution and a separate confidence field, without publishing a formula. A client should keep both, and must not treat confidence as the probability of the selected choice.

For ordered rubrics both APIs return a score equal to the probability-weighted average of the level indices, alongside the per-level distribution. The average can fall between levels. Part 3 shows why the distribution has to travel with it.

// Week 1: two labels.  Week 6: three labels.
if (todayTechnical < lastMonthTechnical - 0.2)
    alerts.Raise("routing model regressed");
// 0.88 -> 0.47 fires, although nothing got worse

public sealed record Assessment(
    string TaskVersion,          // which candidate set
    string Procedure,            // "native-choice", "label-scoring", ...
    ImmutableDictionary<string, double> Probabilities,
    double? Confidence);
// compare only assessments with equal TaskVersion and Procedure

Keep four comparisons separate

The same application can compare several mechanisms, but each comparison asks a different question and holds different things constant:

A rule can produce a correct label with no probability. A generated enum may have the same limitation. If the downstream policy requires a distribution, those methods will send cases to review even when their labels are right. That is a mismatch of information, not proof of worse classification. The reverse shortcut misleads too: wrapping a rule's answer in a one-hot distribution makes it look maximally certain while saying nothing about its observed error rate.

A fair performance comparison counts the work each method needs. Scoring every candidate path can cost far more than emitting one enum, although batching and shared-prefix reuse change that. A classification head can be tiny relative to computing the input representation. Do not infer latency from the size of the JSON, and never time a recorded response against live inference.

Which mechanism gives you which number

Checklist

• Every stored probability records its candidate set version and the procedure that produced it.
• Probabilities are compared only across identical candidate sets; a taxonomy change starts a new baseline.
• Multi-token labels are scored as complete paths with a fixed template and end marker, or replaced by short neutral identifiers.
• Length-normalized scores are named as such and never displayed as probabilities.
• Schema-valid output is still evaluated for correctness on labelled cases.
• confidence and the selected choice's probability are stored as separate fields.
• Softmax is computed with the maximum subtracted, and non-finite inputs are rejected.

When the router returns a number, the useful next question is now precise: probability of which event, conditional on which candidates, computed how? Part 3 turns those answers into a contract a program can validate, and Part 4 asks whether the numbers are good enough to choose an action. For the evaluation side of the same idea, see treating quality as a distribution, not a single score.

Series: Previous: Part 1. Next: Part 3, designing the decision contract.