> ## Content Index > Fetch the complete content index at: https://devinside.cloud/llms.txt > Use this file to discover other available public pages before exploring further. # Does Agent Oversight Actually Stop the Effect? Mutation Testing the Control Boundary - URL: https://devinside.cloud/does-agent-oversight-actually-stop-the-effect-mutation-testing-the-control-boundary/ - Published: 2026-02-26T10:45:00.000Z - Updated: 2026-11-12T17:08:28.000Z - Description: Six deliberately broken executor variants show whether independent effect assertions detect missing actor, scope, replay, expiry, digest and version controls. - Author: Marcin Borecki - Tags: Engineering, Performance & Runtime, #ai08, #devinside-review START WITH THE MECHANISM Interactive companion and laboratory update: 12 November 2026\. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates. THE FULL INVESTIGATION ## Does Agent Oversight Actually Stop the Effect? Mutation Testing the Control Boundary Editorial context: 26 February 2026 · Prepared and updated: 12 November 2026 Dates and technology context This is a retrospective editorial timeline. Topic context: 26 February 2026\. First publicly available: 12 November 2026\. Current text, visuals and laboratory revisions were prepared on 12 November 2026. MEAI11.0.0, six mutants,56 scenario evaluations and actual effect counts/timing remain an executed12Nov2026 update. A test passes because an unauthorized request returns an error. That sounds reassuring until we ask a second question: did the forbidden action happen before the error was returned? An agent's explanation, an approval screen and a policy log can all describe the intended control without proving that the control prevents the effect. The useful observation lies on the other side of the boundary. If authorization is removed, can the test see the change in what the system actually does? This article uses deliberate faults to answer that question. A small C# executor checks actor identity, resource scope, proposal binding, expiry, resource version and replay protection before calling a local effect recorder. Six selectable mutants disable those controls. A set of independently specified scenarios then observes whether effects occurred. The experiment is executed code, not a checklist of tests that someone might write later. The baseline passes all eight scenarios. Each of the six selected mutants is detected by at least one scenario. The complete matrix is retained, including the response text and expected and actual effect counts. The shared suite also exercises actual `Microsoft.Extensions.AI` function invocation with a scripted local provider. There are no customer systems, payments, external actions or live model calls. The recorder stores synthetic proposals in memory. These results establish sensitivity to six deliberately chosen faults in one reference implementation. They do not certify security against all attacks or establish that a model behaves safely. ## Start with the effect that must not happen 01 / See the idea ### The effect recorder is outside the return message ![The effect recorder is outside the return message. The test owns the recorder and states the expected effect count independently. A reassuring response string cannot erase an effect that already crossed the boundary.](https://devinside.cloud/content/images/2026/11/devinside/AI08-v1.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/AI08-v1.svg) The test owns the recorder and states the expected effect count independently. A reassuring response string cannot erase an effect that already crossed the boundary. Interpretation and evidence - Observe the consequence. - Do not derive expected values from the guard. - Keep the recorder deliberately simple. synthetic controller explanation; hypothetical cases explicitly labeled The fixture's effect is a synthetic `commit-import` proposal reaching an `EffectRecorder`. A valid proposal identifies an operation, resource, resource version, action and expiry. An approval binds a digest of that proposal to an actor. The test does not treat the executor's return string as the authority for whether the effect occurred. It creates the recorder independently and reads its contents after execution. The executor can return `ActorDenied`, `Expired` or `Applied`, but the expected effect count comes from the test scenario, not from the returned label. For an authorized request, the expected count is one. For a wrong actor, wrong resource, expired proposal, changed proposal or stale resource version, it is zero. For an exact replay, the total remains one after two calls. For reuse of a consumed approval identity on another operation, the total also remains one. Those are small expectations that can be explained without reading the guard implementation. That independence matters. A test that computes its expected result by calling the same authorization helper can reproduce the helper's mistake and still pass. The recorder is not a full external system. It does not persist across process restarts, implement network retries or independently authenticate the executor. It is sufficient for this particular question: did the tested control allow the forbidden call to cross its effect boundary? A later integration test would need an appropriate isolated receiver and its own observations. A human approval UI can help a reviewer understand a proposed action. It cannot replace this effect invariant. If the server ignores the binding or accepts stale authority, a carefully designed screen becomes an accurate display of a decision the backend failed to enforce. ## Make the reference executor small enough to challenge The important part of the executor is a short sequence of deterministic checks. In the baseline, each must pass before the recorder is called: ```csharp if (actor.Id != "alice" || approval.Actor != actor.Id) return "ActorDenied"; if (p.Resource != "input-A") return "ScopeDenied"; if (p.Action != "commit-import") return "ActionDenied"; if (p.Digest() != approval.Digest) return "DigestDenied"; if (now >= p.Expires) return "Expired"; if (p.ResourceVersion != ResourceVersion) return "VersionDenied"; ``` Replay checks follow, then the effect is recorded and the operation and approval identities are retained. The actual source surrounds execution with an in-process lock. That coordinates calls to this executor instance; it is not a distributed transaction or proof against arbitrary concurrent mutation of external state. The actor object represents trusted context supplied by the host. The approval object is also a fixture object, not a signed artifact from a production approval service. The test deliberately supplies valid and invalid combinations to verify the control algorithm. It does not claim that constructing a C# record authenticates a person. The digest uses a declared representation of the proposal fields. Length prefixes distinguish string components. It provides a comparison of content, not a signature or proof that a trusted reviewer authorized it. A real application would need a trusted approval store or another authenticated binding mechanism, together with a canonical cross-language format if several implementations produce the digest. This narrowness is an advantage for mutation testing. When a scenario fails, the reader can identify which invariant changed without searching through a large platform. The example should remain small enough that its limitations are as visible as its successful checks. ## Seed a fault that changes behavior The mutants are explicit code paths selected by an enum. They are not comments describing hypothetical problems. For example, the actor guard in the running source is conditional on the selected mutant: ```csharp if (mutant != Mutant.SkipActor && (actor.Id != "alice" || approval.Actor != actor.Id)) return "ActorDenied"; ``` Selecting `SkipActor` removes that protection for the experiment. The wrong-actor scenario supplies otherwise valid scope, action, digest, expiry and resource version, so the removed actor check is not masked by an unrelated rejection. The effect recorder then contains an operation when the independent expectation was zero. This is the central discipline in constructing a useful mutant: make the faulty behavior reachable. If the request also had the wrong scope, the scope check might still block it and the actor mutant would survive for an uninteresting reason. The test would not have established that actor validation mattered. The other mutants widen scope, remove replay protection, ignore expiry, skip proposal-digest validation or skip the resource-version check. They are chosen because they correspond to consequential control failures, not because a mutation tool happened to generate many syntactic variations. This is hand-seeded behavioral mutation, not an automated Stryker run or a complete enumeration of source mutations. The terminology is consistent with the basic distinction between a killed mutant, whose altered behavior makes a test fail, and a survivor, whose tests still pass. Compile errors and runtime infrastructure errors are different outcomes and should not be presented as demonstrated control sensitivity. [Stryker's mutant states and metrics](https://stryker-mutator.io/docs/mutation-testing-elements/mutant-states-and-metrics/?ref=devinside.cloud). Our harness does not count a compilation failure as a kill. Every selected variant is compiled into the executable and run against the same scenario set. Its matrix records concrete effect-count disagreement. If a variant threw an unexpected infrastructure exception, the run would fail instead of quietly converting that into a successful security result. ## Keep the scenario correct except for the challenged condition 02 / Watch the mechanism ### Change the proposal after the approval ![Change the proposal after the approval. The executed digest scenario extends the proposal expiry after its approval was constructed. The current time remains valid for both values, so only the binding guard can detect that the approved content changed.](https://devinside.cloud/content/images/2026/11/devinside/AI08-v2.svg) Change the proposal after the approvalThe executed digest scenario extends the proposal expiry after its approval was constructed. The current time remains valid for both values, so only the binding guard can detect that the approved content changed.AI08 / MOTIONChange the proposal after the approvalApproval bindsoriginal expiryProposal expirychangesBaseline rejectsdigest mismatchSkipDigest mutant: effect count 1Local synthetic fixture. No live-model quality claim. Play animation[Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/AI08-v2.svg) Static view. Play to see how the mechanism unfolds. The executed digest scenario extends the proposal expiry after its approval was constructed. The current time remains valid for both values, so only the binding guard can detect that the approved content changed. Interpretation and evidence - Keep other guards satisfied. - Change one consequential condition. - Check for a forbidden effect. synthetic controller explanation; hypothetical cases explicitly labeled The wrong-scope scenario changes the resource to `input-B` but creates an approval matching that proposal. This is intentional. Approval binding and resource authorization are distinct controls. A matching approval object in the fixture does not give the actor access to every resource. Removing only the scope guard must become observable. The expiry scenario uses `now == Expires`. The baseline's boundary is inclusive: authority is no longer valid at that instant. An off-by-one interpretation should not be hidden by testing only an obviously old timestamp. The changed-proposal scenario creates approval for one expiry, then extends the proposal's expiry without changing its approval digest. At the test's current time both expiry values are still in the future, so the expiry guard does not mask the content-binding failure. The point is not that this particular extension immediately changes the effect; it is that unapproved content crossed the binding boundary. The stale-resource scenario changes the executor's resource version after the proposal and approval have been constructed. The proposal remains internally consistent, but it no longer refers to the current resource state. Removing the version check allows an effect under a stale assumption. These cases are ordered interleavings: approve, change a specific condition, execute. They do not depend on a random thread race. The fixture does not claim to test simultaneous resource changes during an external call. That harder time-of-check/time-of-use boundary requires a storage or receiver contract beyond this in-memory executor. The scenarios teach a transferable lesson: controls can conceal one another. A meaningful negative test keeps the other prerequisites valid so the challenged control is the reason the effect is prevented. ## Replay protection has two different questions An exact retry should not create a second effect. A different operation should not reuse an approval identity that has already authorized an effect. These expectations sound similar but require different observations. The first scenario calls the executor twice with the same operation and approval. The baseline returns the existing operation state on the repeat and leaves the recorder with one effect. The second scenario first executes a valid operation. It then creates another proposal and a matching digest while retaining the same approval identity. This is a deliberately malformed approval-issuance situation in the fixture: the executor must still notice that the approval identity was consumed. The recorder must remain at one effect after the second request. The `ReplayApproval` mutant disables the replay-protection section, which contains both the operation receipt and consumed-approval checks. It is therefore one selected behavioral mutant spanning two related guards, not a claim that exactly one source token was changed. Both replay scenarios detect it and observe two effects instead of one. That distinction should remain visible in the report. If an automated tool later mutates each guard independently, it may produce two separate mutants and a different denominator. Comparing the resulting percentage directly with this hand-seeded set would be meaningless. The replay state is held in dictionaries and a set inside one executor. Restart durability, retention windows and multi-instance coordination are excluded. The test establishes the current instance's behavior. A real operation ledger would have to preserve the same invariant at a durable shared boundary before the result could be generalized to a service deployment. ## Use the real Microsoft function path without a live model The shared laboratory references `Microsoft.Extensions.AI` 11.0.0, restored for a net10.0 application running on .NET 10.0.15\. The public package registration reports publication on 10 November 2026\. The exact package graph is retained in the lockfile rather than inferred from a live documentation banner. [Microsoft.Extensions.AI 11.0.0 package](https://www.nuget.org/packages/Microsoft.Extensions.AI/11.0.0?ref=devinside.cloud). `ScriptedChat` implements `IChatClient`. It returns a predetermined `FunctionCallContent`, observes the subsequent function result and returns a final message. `FunctionInvokingChatClient` performs the actual tool invocation. This tests the framework's plumbing without pretending that a model generated, understood or selected the request. The executor is captured behind a local function with a named parameter. Trusted actor context is supplied by the host-side closure, not by a model argument. The corresponding integration test uses the wrong actor and verifies zero effects after the SDK completes the round trip. ```csharp string ExecuteStep(string stepId) => guard.Execute(proposal, approval, trustedActor, now); var options = new ChatOptions { Tools = [ AIFunctionFactory.Create( (Func)ExecuteStep, "execute_step") ] }; ``` This shortened registration illustrates the same boundary as the runnable helper. The generated function schema maps model-supplied arguments to the declared method parameters. The framework does not make those arguments trustworthy merely by deserializing them. [AIFunctionFactory.Create documentation](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.ai.aifunctionfactory.create?ref=devinside.cloud). The integration is deliberately configured with serial function execution and at most four provider iterations. Those limits prevent the demonstration from becoming an open-ended conversation. They do not replace actor or scope checks inside the executor. During development, the real function test caught a mismatch between the scripted argument name and the generated method parameter. That was repaired by aligning the explicit function signature. A fake test that called the delegate directly would not have exposed the mismatch, which is why both deterministic unit boundaries and the actual SDK path are useful. ## The executed matrix shows which test detects which fault 03 / See the idea ### Every seeded fault was detected ![Every seeded fault was detected. The matrix is generated from the actual mutation report. A cross marks an effect-count violation with the mutant active; a dot means that scenario still satisfied its invariant.](https://devinside.cloud/content/images/2026/11/devinside/AI08-v3.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/AI08-v3.svg) The matrix is generated from the actual mutation report. A cross marks an effect-count violation with the mutant active; a dot means that scenario still satisfied its invariant. Interpretation and evidence - Six chosen mutants were killed. - The baseline passes all eight scenarios. - No claim of exhaustive security coverage. executed local report; see evidence and benchmark-results.json The baseline passes all eight scenarios. Each of the six selected mutants is killed by an effect-count disagreement. The complete matrix has seven variants, including the baseline, and eight scenarios: 56 executed scenario evaluations. | Variant | Detecting scenario | Expected effects | Observed effects | | -------------- | ------------------ | ---------------- | ---------------- | | SkipActor | wrong-actor | 0 | 1 | | WidenScope | wrong-scope | 0 | 1 | | ReplayApproval | replay | 1 | 2 | | ReplayApproval | approval-reuse | 1 | 2 | | IgnoreExpiry | expired | 0 | 1 | | SkipDigest | changed-proposal | 0 | 1 | | SkipVersion | changed-resource | 0 | 1 | These are actual local test outcomes stored in `evidence/mutation-results.json`. The JSON also retains scenarios that did not fail for each variant. That matters because the experiment should show targeted sensitivity rather than an executor that simply fails every request. The authorized scenario passes for the baseline and all six mutants. Each selected mutant weakens a control, so it should not prevent a valid effect. The absence of accidental breakage in that path makes the negative-result interpretation clearer, although it is not a substitute for a larger positive test set. There are no survivors in this selected set. That is a fact about six intentional faults, not a 100-percent security guarantee. We selected the mutants and scenarios together around known invariants. The result does not establish coverage of omitted guards, malformed payload classes, denial-of-service behavior, persistence failures or framework vulnerabilities. The shared suite reports 32 assertions because some assertions summarize the scenario matrix and others test replanning and integration. The report does not pretend that the number 32 is the number of mutation scenarios or that two articles represent two independent experimental runs. Source hashes and provenance connect both articles to the same accepted code. ## A green error-message test can miss the real regression 04 / A concrete example ### A denial label alone is a weak oracle ![A denial label alone is a weak oracle. This is a hypothetical additional counterexample, not an executed mutant in the current matrix. A component could return a denial after creating an effect, so a response-only assertion would miss the consequence.](https://devinside.cloud/content/images/2026/11/devinside/AI08-v4.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/AI08-v4.svg) This is a hypothetical additional counterexample, not an executed mutant in the current matrix. A component could return a denial after creating an effect, so a response-only assertion would miss the consequence. Interpretation and evidence - Label illustrative cases honestly. - Test the destination as well as the response. - A mock call count is not always the invariant. synthetic controller explanation; hypothetical cases explicitly labeled Imagine replacing the effect-count oracle with a test that only checks whether the response string belongs to a list of expected labels. A buggy executor might write the effect and then return a denial. That test could pass while the actual invariant failed. Our current mutants do not implement every possible response-label bug. The example is a proposed counterexample explaining why the recorder sits outside the response contract. It should not be described as an additional executed mutant unless that variant is added and run. Similarly, asserting that an authorization method was called is weaker than asserting that an unauthorized effect did not occur. The method can return a result that the caller ignores. A mock invocation count may verify a coding pattern while missing the behavior the user actually needs. An independent recorder is not magical. If the executor can mutate the test's expected values or if the recorder shares the same faulty predicate, independence collapses. In this fixture, expected counts are literal scenario requirements and the recorder appends every received proposal without making authorization decisions. That intentionally simple observer cannot silently apply the same guard and hide the executor's mistake. The test should also avoid overclaiming from absence. Zero recorded effects establishes that this local receiver was not called successfully by the tested path. It does not prove that no other side effect occurred elsewhere in a larger application. When the real system writes to several destinations, each consequential boundary needs an observation or a clearly stated limitation. ## What to do with a survivor A surviving mutant is a question, not automatically proof that the production code is broken. The altered behavior may be unreachable, equivalent under the specified contract, masked by another guard or genuinely undetected by the tests. Start by confirming that the intended mutation was active and that the relevant path ran. Then construct an input for which the baseline and mutant should differ while other requirements remain satisfied. If no such input exists under the current contract, document the equivalence or redundancy rather than manufacturing a test solely to improve a score. If the behavior is consequential and reachable, improve the oracle or the scenario. For example, a replay-protection mutant may survive when a test issues only one call. An expiry mutant may survive when every test uses a comfortably valid timestamp. A digest mutant may survive when the changed field is independently rejected by scope validation. The corrective action should remain connected to the invariant. Counting more lines, adding more mocks or generating thousands of random requests does not guarantee that the missing distinction becomes observable. A single carefully constructed counterexample can be more useful than a large volume of nondiscriminating tests. This is also a practical stopping rule. Once the selected important fault classes are demonstrably detected and the positive behavior remains valid, deliver that evidence with its limits. Do not keep expanding a bounded experiment merely because additional mutations can always be imagined. ## Timing the control does not measure safety 05 / See the idea ### Control-path timing is not a security score ![Control-path timing is not a security score. The authorized and denied cases execute different local work, and the shared report also includes the replanning controller. Mutant detection comes from invariant failures, not timing.](https://devinside.cloud/content/images/2026/11/devinside/AI08-v5.svg) [Enlarge diagram](https://devinside.cloud/content/images/2026/11/devinside/AI08-v5.svg) The authorized and denied cases execute different local work, and the shared report also includes the replanning controller. Mutant detection comes from invariant failures, not timing. Interpretation and evidence - A faster mutant can still be wrong. - Denial may exit before later checks. - No live-model latency is measured. executed local report; see evidence and benchmark-results.json The shared benchmark measures a fresh authorized executor episode and a fresh denied executor episode, plus the companion replanning controller. Each executor episode constructs a recorder, executor, proposal, digest and approval, invokes the control and checks the effect count. The timed loop does not invoke a live model, SDK function middleware, a network service or a durable store. It includes object allocation and digest work in the local episode. It therefore describes one cost of applying the reference control, not the latency of an AI product. The denied case exits at the actor guard and does less work than the authorized case. A lower time for denial would not mean weaker or stronger security. It would show a different executed path. Likewise, the mutation experiment does not select a faster mutant as a candidate optimization: its purpose is to demonstrate that removing the guard violates an invariant. The benchmark uses seven samples of one thousand episodes after two warm-ups and records managed allocation on the current thread. It is a custom, bounded stopwatch experiment. It does not measure retained heap, concurrent contention or the cost of cryptographic verification in a real approval system. The integrator ran the shared measurement on 12 November 2026 at 14:01 UTC, on .NET 10.0.15, with exit code zero. The table retains every configured workload. These are seven within-process samples per workload, expressed per fresh in-memory episode; they are not percentiles of user request latency. | Episode | Median µs | Observed µs range | Allocated bytes/episode | | ------------------- | --------- | ----------------- | ----------------------- | | replan-controller | 2.181 | 1.703–2.324 | 1424–1424.42 | | authorized-executor | 3.751 | 3.643–3.915 | 1808–1808 | | denied-executor | 1.164 | 1.151–1.218 | 768–768 | The authorized executor episode had a median of 3.751 microseconds and allocated 1,808 managed bytes. The denied episode had a median of 1.164 microseconds and allocated 768 bytes. Denial stops at the actor guard in this workload; successful execution continues through content checks and records an effect and replay state. Subtracting these numbers would not isolate the cost of authorization. The paths differ in several operations, and both include fixture-object construction. The measurement also excludes SDK middleware and external storage. It is useful as a baseline for this complete local episode, not as a universal price for making an agent safe. The replanning row is included because the raw report is shared with the companion article. It does not enter the mutant-kill result. Mutation sensitivity came from observed effect-count disagreements, not from performance: a faster mutant that creates a forbidden effect is still a failed control. No timing was collected for every mutant, and none is needed to establish that the selected tests detect their behavior. The report keeps these evidence types separate so a fast successful run cannot be mistaken for a security result, or a killed mutant for a claim about system throughput. ## Do not let the lock imply a larger atomicity guarantee The executor locks its own check-and-record sequence. Within the intended single-instance fixture, that keeps its operation receipt and approval-use checks together with the recorder call. It does not lock a remote resource, a separate approval service or another process with its own executor instance. The resource-version test deliberately changes the value before execution begins. It proves that the baseline notices that ordered change and that the SkipVersion mutant does not. It does not prove behavior when another thread changes resource state between the local check and an external commit. Solving that problem may require a conditional write or a transaction at the destination that owns the resource. The mutation matrix should therefore be read as a map of observed local boundaries. Replacing the recorder with a real service would introduce acknowledgment loss, duplicated delivery, uncertain outcomes and durability requirements. Those conditions cannot be covered simply by adding more names to this table without implementing the corresponding receiver behavior. This restraint also protects the meaning of the positive test. One successful authorized local call shows that the baseline is not merely denying everything. It does not establish that a full product remains usable under every legitimate workflow. Functional acceptance and mutation sensitivity should reinforce each other while retaining their different scopes. ## Turn oversight into a falsifiable claim The phrase human oversight can describe several different things: a reviewer seeing a proposal, a stored approval, a server checking its binding, or an operator resolving a later uncertain result. Only some of those mechanisms directly prevent the initial effect. The useful engineering statement is specific: this action cannot reach this receiver unless these conditions hold. Mutation testing asks whether the tests notice when one of those conditions stops being enforced. That is a stronger question than whether a happy-path demonstration contains an approval step. The laboratory can be reproduced with its local SDK pin and package lockfile: ```powershell dotnet restore --locked-mode dotnet run -c Release -- check ../evidence dotnet bin/Release/net10.0/ControlsLab.dll measure ../benchmark-results.json ``` The first two commands establish the dependency and correctness evidence. The measurement command belongs in the exclusive benchmark lane and must not be interpreted as another mutation result. The AI02 and AI08 lab directories contain identical source copies with recorded hashes; their results describe one shared implementation. A production system will need stronger persistence, authentication and external-effect contracts than this in-memory example. It may also need automated mutation tooling, fuzzing and integration tests. Those are additional layers, not reasons to skip a small test that can already prove an important guard is being observed. The core lesson is modest and actionable. Do not ask only whether a control is present. Remove it in a safe fixture and ask whether the expected forbidden effect becomes visible. If the test still says everything is fine, the next repair may belong to the test's understanding of the requirement rather than to the control's implementation. ## Source code and execution evidence [Download the complete lab, source record and raw execution results](https://devinside.cloud/content/files/2026/11/AI08-lab.zip). This package supports the specific execution claims in this article. It is not a production certification.