Designing a Decision Contract That Preserves Meaning in C#

Series: Decision APIs in .NET, part 3 of 10. Previous: Part 2, what a decision probability actually computes. Next: Part 4, thresholds from the cost of errors.

Two reviewers label the same support message. One picks returns, the other technical, and both can defend the choice:

The Aurora M4 stopped sending telemetry after firmware 2.8. Should I return it?

The first reviewer labelled the outcome the customer mentioned. The second labelled the team that should investigate first. Measuring their agreement now would report a defect in the task definition as if it were a defect in the reviewers, and any model evaluated against either set of labels would inherit the confusion.

A decision contract starts with that disagreement. It defines a question people can answer consistently, then the values a program may accept as an answer to it. The JSON shape is one part. The meaning of each label, the relationships between fields and the states that signal failure matter at least as much. This article builds the complete C# contract for the series' routing task, support-routing-v1; the native clients in Part 5 normalize both vendors' responses into it.

Write the disagreement into the rubric

The department question is "Which team should investigate first?" It classifies the requested investigation, not every topic in the message and not the eventual resolution. Four choices need descriptions that separate them, and the exclusions do most of the work:

The last exclusion closes an easy escape. "Telemetry fails; I do not know the firmware" is still a technical investigation. Missing evidence is a different dimension, so the contract asks about it separately, as the predicate enough_evidence. Otherwise other drifts between "outside the taxonomy" and "inside it but poorly described", and its frequency stops meaning anything.

Mixed tickets stay hard. If telemetry and a duplicated invoice both need independent work, a single first-team label depends on a business ordering rule nobody wrote down. Part 1's prototype handled this with a fifth mixed choice. In support-routing-v1 such tickets are flagged for adjudication before they become reference labels; a later version can add sub-cases if the workflow can use them. Either way the task owner decides the representation, not the annotators one ticket at a time.

In the evaluation set, store an annotation reason and an ambiguity flag next to each label. An unresolved ambiguity is not a confidently known gold answer, and counting it as one penalizes a model for disagreeing with a coin flip.

Three answer kinds preserve three different meanings

A choice selects one candidate from a set. A predicate estimates whether one proposition holds. A score places the content on an ordered rubric. If all three land in an unqualified double Value, some downstream method will eventually compare an urgency mean with a probability threshold. Separate records make that a compile error:

public abstract record Answer(string Name);

public sealed record ChoiceAnswer(string Name, string Choice,
    ImmutableDictionary<string, double> Probabilities, double Confidence) : Answer(Name);

public sealed record PredicateAnswer(string Name, double Probability) : Answer(Name);

public sealed record ScoreAnswer(string Name, double Score,
    ImmutableDictionary<string, double> Probabilities, double Confidence) : Answer(Name);

public sealed record RefusalAnswer(string Name) : Answer(Name);

The collections are immutable, so copying a record cannot leave a shared dictionary that another consumer mutates after validation. The constructors describe what is representable; validation decides whether the values answer the task. The questions mirror the answers: ChoiceQuestion carries its candidates, ScoreQuestion its ordered levels, and a DecisionTask pairs a version string with its questions.

The names map onto both native protocols. OpenAI Decisions calls the three kinds predicate, choice and score, returns probability for a predicate, and can return a per-question refusal. System One calls its yes/no question type noul and returns the value in a field of the same name; its published answer union has no refusal variant. The provider-neutral C# type can represent a refusal, but that does not give every wire protocol the capability. Part 5 turns a System One body with type: "refusal" into an invalid response.

For enough_evidence, 0.94 refers to the proposition "device, firmware and symptom are supplied". It is not the probability that the department choice is correct, and no code should combine the two without a model of how they relate.

A score is a mean, and the mean hides the tail

The urgency question uses three ordered levels:

0  No service interruption
1  Intermittent loss with a workaround
2  Continuous loss without a workaround

Both native APIs return score as the probability-weighted average of the level indices, together with the per-level distribution and a separate confidence. With level probabilities 0.2, 0.5 and 0.3, the score is 0 x 0.2 + 1 x 0.5 + 2 x 0.3 = 1.10. That is not a 110% probability, and dividing it by two does not give the probability that a specialist is needed. How much does the mean alone tell you about the worst level?

Keep the distribution next to the score, everywhere. A routing policy that cares about continuous outages must read P(level 2), not the mean, and must never round the mean into a level. The rubric is also not a ratio scale of damage: level 2 does not promise twice the loss of level 1. When an action depends on cost, ask a question about that action. Part 4's urgent-specialist-binary-v1 is a separate binary question for exactly that reason, and it must receive predictions for its own proposition, not an urgency mean or its confidence.

Here is an authored System One score answer for the course fixture, in the documented shape:

{
  "type": "score",
  "score": 1.10,
  "confidence": 0.70,
  "probabilities": {"0": 0.20, "1": 0.50, "2": 0.30}
}

The adapter copies score into ScoreAnswer.Score, keeps the string level indices as keys and stores confidence separately. It then validates the typed answer against the original ScoreQuestion. A response cannot define its own rubric by supplying familiar field names. OpenAI returns the same information as an array of entries carrying the level index, its label and its probability; Part 5's adapter also checks that each returned label matches the requested one before normalizing.

Confidence needs the same restraint. System One documents choice confidence as computed from how the distribution is spread; it is not the probability of the selected choice. OpenAI documents a separate field without publishing a formula. The contract checks that confidence is finite and within [0, 1] and preserves it, but does not impose a cross-provider formula that neither vendor promised.

Validate relationships, not just shapes

Every rule in the validator comes from a contradiction a schema would accept. If the request asked three questions, two answers are incomplete, and an extra unrequested answer is not a substitute. Names provide identity, so order may differ. Each answer must be the kind its question asked for: this task defines urgency as a score, so a predicate named urgency fails even when its value happens to lie in [0, 1].

For a choice, every requested candidate must appear exactly once in the distribution, every probability must be finite and in [0, 1], and the sum must be one within a declared tolerance. The selected choice must be a candidate and must have maximal probability; a tie allows either maximal candidate, because this task defines no tie-break. For a score, the distribution must cover exactly the requested level indices, and the score must be finite, inside the index range and equal to the weighted mean. Range checking alone would accept score = 0.50 beside the distribution above.

Which check stops which defect? Slide through eight responses, one valid and seven each broken in exactly one way.

The validator is short enough to read in full. The choice check mirrors the score check: distribution first, then membership and maximality, then the confidence range.

public static Validation Validate(
    DecisionTask task, ImmutableArray<Answer> answers)
{
    var asked = task.Questions.Select(q => q.Name).ToHashSet();
    var given = answers.Select(a => a.Name).ToList();
    if (given.Count != given.Distinct().Count() || !asked.SetEquals(given))
        return new("$.answers", "answer names must match the questions once");

    foreach (var answer in answers)
    {
        var question = task.Questions.Single(q => q.Name == answer.Name);
        string at = "$.answers." + answer.Name;
        var r = (question, answer) switch
        {
            (_, RefusalAnswer) => Validation.Ok,
            (PredicateQuestion, PredicateAnswer p) =>
                Unit(p.Probability, at + ".probability"),
            (ChoiceQuestion q, ChoiceAnswer c) => CheckChoice(q, c, at),
            (ScoreQuestion q, ScoreAnswer s) => CheckScore(q, s, at),
            _ => new(at, $"{answer.GetType().Name} does not answer "
                         + question.GetType().Name),
        };
        if (r.Path is not null) return r;
    }
    return Validation.Ok;
}

static Validation CheckScore(ScoreQuestion q, ScoreAnswer s, string at)
{
    var keys = Enumerable.Range(0, q.Levels.Length)
        .Select(i => i.ToString()).ToImmutableArray();
    var d = Distribution(keys, s.Probabilities, at + ".probabilities");
    if (d.Path is not null) return d;
    if (!double.IsFinite(s.Score) || s.Score < 0 || s.Score > keys.Length - 1)
        return new(at + ".score", "outside the level index range");
    double mean = keys.Select((k, i) => i * s.Probabilities[k]).Sum();
    if (Math.Abs(s.Score - mean) > MeanTolerance)
        return new(at + ".score", $"{s.Score} disagrees with mean {mean:F2}");
    return Unit(s.Confidence, at + ".confidence");
}

The shared distribution check is the place where version mismatches and truncated responses surface:

static Validation Distribution(ImmutableArray<string> expected,
    ImmutableDictionary<string, double> p, string at)
{
    if (!p.Keys.ToHashSet().SetEquals(expected))
        return new(at, "keys must be exactly the requested candidates");
    foreach (var (k, v) in p)
        if (Unit(v, at + "." + k) is { Path: not null } bad) return bad;
    double sum = p.Values.Sum();
    return Math.Abs(sum - 1) > SumTolerance
        ? new(at, $"sum {sum:F3} is not 1") : Validation.Ok;
}

The tolerances are explicit: the sum may differ from one by at most 0.002, and the reported mean from the computed mean by at most 0.02. They suit two-decimal fixtures and are the contract's own choice, not vendor accuracy guarantees. A production client should set them from the protocol's actual precision and keep unrounded values when it has them.

double sum = probs.Values.Sum();          // 0.70
probs = probs.ToImmutableDictionary(
    kv => kv.Key, kv => kv.Value / sum);   // now sums to 1
// a dropped candidate, a truncated body and a
// wrong response shape all look identical now

var v = Validator.Validate(task, answers);
if (v.Path is not null)
    return Assessment.Invalid(raw, v.Path, v.Error);
// $.answers.department.probabilities  sum 0.700 is not 1
// the raw body is kept for investigation

Reject duplicate properties at the parser

This body is syntactically valid JSON, and ambiguous to an application:

{"choice": "technical", "choice": "returns"}

RFC 8259 says object names should be unique and warns that receivers behave unpredictably when they are not. In .NET the default is last-wins: both JsonDocument.GetProperty("choice") and JsonSerializer.Deserialize into a record return returns. Another consumer of the same bytes, a log pipeline or a different language's parser, might read the first value. The application and its audit trail would then disagree about what the model said.

.NET 10 adds AllowDuplicateProperties to both JsonDocumentOptions and JsonSerializerOptions. Set it to false and parsing throws JsonException ("Duplicate property 'choice' encountered during deserialization"), including for duplicates nested inside array elements. That replaces the hand-written recursive walker older code needed:

var strict = new JsonDocumentOptions { AllowDuplicateProperties = false };
using JsonDocument doc = JsonDocument.Parse(body, strict);   // throws on repeats

This is a different defect from duplicate answers. OpenAI's array could contain two separate objects both named department, with no repeated property inside either. The parser accepts that; the validator's name check rejects it. Both checks are needed, at different layers.

Refusal is a state, not an answer

OpenAI Decisions can refuse one question while answering the others. The normalized result keeps the refusal and the remaining answers; the client in Part 5 marks the call Refused, so the application can avoid automatic action without losing the useful parts. The validator accepts a RefusalAnswer for any question for the same reason: it is a legitimate outcome, not a malformed one.

A malformed response is different. A missing confidence field where the native profile requires one is invalid. An authentication failure means no answer was received at all. None of these becomes an extra department, and none becomes a default.

Methods that only produce a label, such as a rule or a generated enum, should not be forced into this native profile. Represent them separately, for example as a LabelOnlyAssessment. Converting a bare label into a one-hot distribution makes a statistical claim nobody measured. A capability-aware application can compare raw labels across methods and require a real distribution only for policies that use one.

Evolve the task, not just the enum

Suppose a firmware specialist team becomes available. Adding firmware to the candidate list changes three things at once: what a reviewer considers the correct label, which alternatives compete in the model's distribution, and which destinations exist downstream. That is a new task, support-routing-v2.

Keep v1 reference labels and outputs attached to the v1 definition, and re-annotate affected cases under v2 instead of reading old technical labels as mistakes. A model that gave technical 0.93 under v1 may split its mass between technical and firmware under v2, as the slider in Part 2 showed. That can be entirely consistent with the new task. The validator does not judge whether the new taxonomy is good; it prevents the silent mixture of two of them, which is what the v1-under-v2 response above demonstrated.

The same applies to rewording the rubric, reordering score levels or changing what counts as enough evidence. A version identifies the semantic definition, not a release of the assembly. Store the definition with the evaluation data so a later reader can recover the question each prediction answered.

The whole contract on eighteen responses

The program behind this article runs the validator over four valid and fourteen invalid responses and compares each result with an expected path. On .NET 10 it printed:

ok   canonical                    -> accepted
ok   reordered answers            -> accepted
ok   partial refusal              -> accepted
ok   tied maximum                 -> accepted
ok   missing answer               -> $.answers  answer names must match the questions once
ok   extra answer                 -> $.answers  answer names must match the questions once
ok   duplicate answer name        -> $.answers  answer names must match the questions once
ok   wrong kind for urgency       -> $.answers.urgency  PredicateAnswer does not answer ScoreQuestion
ok   predicate out of range       -> $.answers.enough_evidence.probability  1.2 is not a finite value in [0, 1]
ok   NaN probability              -> $.answers.department.probabilities.technical  NaN is not a finite value in [0, 1]
ok   sum 0.70                     -> $.answers.department.probabilities  sum 0.700 is not 1
ok   missing candidate            -> $.answers.department.probabilities  keys must be exactly the requested candidates
ok   choice not maximal           -> $.answers.department.choice  selected choice is not maximal
ok   unknown choice               -> $.answers.department.choice  'firmware' is not a candidate
ok   score disagrees with mean    -> $.answers.urgency.score  0.5 disagrees with mean 1.10
ok   score outside index range    -> $.answers.urgency.score  outside the level index range
ok   confidence out of range      -> $.answers.urgency.confidence  1.7 is not a finite value in [0, 1]
ok   v1 distribution under v2     -> $.answers.department.probabilities  keys must be exactly the requested candidates
18 passed, 0 failed

These checks establish how the declared contract behaves. They do not establish that technical is the right label for any real ticket, or that a returned 0.94 is calibrated. The rubric, the validator and the evaluation set answer three different questions, and keeping all three explicit is what turns an API response into something you can reason about rather than merely deserialize.

Pitfalls

• Rounding the score into a level. A split distribution rounds to a level the model gave zero probability.
• Reading confidence as the chosen probability. Both vendors return them as separate quantities; store both.
• One double Value for every answer kind. The type system is the cheapest validator you have.
• Renormalizing invalid distributions. It hides truncation, dropped candidates and version mixtures behind a clean sum.
• Last-wins JSON parsing. The default in .NET; set AllowDuplicateProperties = false on every untrusted decision response.
• Editing the enum in place. A new candidate is a new task version with its own labels and baselines.

Which layer catches which defect

Checklist

• Each label has an inclusion and an exclusion written down, with counterexamples.
• Ambiguous evaluation cases carry a flag and an annotation reason, and are not scored as gold.
• Choice, predicate, score and refusal are distinct C# types.
• Score answers keep their distribution; no code rounds or rescales the mean.
• Decision responses are parsed with AllowDuplicateProperties = false.
• The validator checks names, kinds, keys, ranges, sums and cross-field relations, and reports a JSON path.
• Invalid responses are rejected with the raw body kept, never repaired.
• Every stored prediction references a task version that identifies its semantic definition.

The contract tells you whether a response is a valid answer to the question. It cannot tell you what to do with a valid 0.93. That is Part 4, which derives thresholds from the cost of each mistake. The same habit of putting a typed, validated value between language and action appears in typed intent between language and SQL.

Series: Previous: Part 2. Next: Part 4, thresholds from the cost of errors.