Measuring the AI Review Queue in a Reference Repository

START WITH THE MECHANISM

Interactive companion and laboratory update: 12 November 2026. The editorial date places the topic on the timeline; this current explanation and its measured evidence retain their actual preparation and execution dates.

THE FULL INVESTIGATION

Measuring the AI Review Queue in a Reference Repository

Dates and technology context

This is a retrospective editorial timeline. Topic context: 24 April 2026. First publicly available: 12 November 2026. Current text, visuals and laboratory revisions were prepared on 12 November 2026.

The collector records actual local command events from November2026. It is not the proposed April2026 team review study and did not measure human reviewers.

A team starts producing pull requests more quickly. The activity feed is busy, the implementation time looks better and the number of proposed changes rises. Yet the people reviewing those changes feel increasingly behind. Work reaches the review queue sooner and then waits there longer. Whether the team has become more effective depends on where the measurement stops.

That hypothetical pattern is easy to turn into a slogan about AI creating more work. It is equally easy to dismiss as temporary resistance to a useful tool. Neither explanation is sufficient. The review queue is a system with arrival rates, service capacity, variable work and feedback loops. Its behaviour deserves an experiment rather than a moral judgement about how code was produced.

The original design proposes a reference-repository study for a .NET 10 application, intended to measure accepted changes, reviewer effort and rework alongside implementation time. No participants have been recruited and no team-study results are reported here. The later implementation section adds a real command-event collector with a narrower boundary; it does not establish the proposed productivity claim.

The proposed repository baseline is SDK 10.0.100 and runtime 10.0.0, available in the Microsoft release metadata before the contextual date. No AI coding model or proprietary review API is fixed by this protocol: the actual intervention must be named and versioned in the run manifest before execution. The record shape below remains original pseudocode for the broader study. The separately described Python collector does not establish a restored .NET study repository or completed participant measurements.

Define completion beyond the first pull request

The first decision is what counts as done. A submitted patch, a passing build, an approved change and a change that satisfies the intended behaviour are different milestones. The experiment should record them separately rather than choose whichever milestone makes the preferred workflow look strongest.

For a reference repository, define a small service with tests and a documented behaviour contract. Use synthetic data and no real external effects. Select tasks that can be independently accepted: add a validation rule, correct a cancellation path, improve a query boundary or extend a response without breaking a client contract.

Do not define acceptance solely through tests written by the person or agent making the change. Those tests are useful implementation evidence, but they can repeat the same misunderstanding as the code. Keep an independent acceptance set or a reviewer-authored check derived from the task statement.

The historical contextual baseline is April 2026, after the .NET 10 release. A future execution must record its actual runtime, tool versions and date. A reference repository makes the workload inspectable; it does not make the result representative of every team or codebase.

Treat review as several kinds of work

01 / See the idea

Waiting and effort are different clocks

Enlarge diagram

Review time is not one continuous activity. A reviewer may first establish what the change intends, then inspect its implementation, run checks, ask questions and revisit the patch after a response. Waiting between those activities is important to delivery time but should not be counted as active effort.

Record active review sessions and elapsed time separately. A pull request opened late on Friday can wait a long time without demanding much reviewer attention. A patch reviewed immediately can consume substantial effort even if its wall-clock lead time is short. Confusing those cases makes process changes difficult to evaluate.

Classify the questions reviewers ask. Some concern an unclear requirement. Others concern a defect, an unsupported assumption, missing evidence or a mismatch with the repository's conventions. These categories reveal whether the implementation workflow changed the burden of understanding, rather than merely moving a timestamp.

Do not assume that a long review is a bad review. A difficult change may deserve careful attention. The relevant comparison is whether the review effort is proportionate to the task and whether it helps produce an accepted outcome. Speed becomes useful only when the acceptance boundary remains meaningful.

Build tasks that are comparable without pretending they are identical

Two software tasks are rarely interchangeable. One may touch a familiar module while another crosses a poorly documented boundary. Randomly assigning a small number of tasks to two conditions does not guarantee that the conditions receive equally difficult work.

Use task families and record their expected challenges before execution. For example, a family might contain two comparable validation changes in separate endpoints, or two recovery changes with similar dependencies. Have someone who is not implementing them review the task descriptions for hidden differences.

A paired design can reduce some variation, but it introduces learning effects if the same person solves the same problem twice. Use separate, related tasks rather than identical repeats, and vary the order of conditions. Record familiarity with the repository. A developer's first encounter with a project should not be silently compared with their fifth as though the only difference were an AI assistant.

The 2024 DORA report provides a reason to examine team outcomes alongside individual experience: its findings describe benefits and tradeoffs around AI adoption rather than a simple universal productivity gain. DORA Research 2024. This proposed experiment would not reproduce that research. It would test a much narrower question in one declared repository.

Keep the intervention explicit

“AI-assisted” can mean autocomplete, a chat consultation, generated tests or an agent making a large patch. If those uses are mixed without recording them, the experiment cannot explain what changed.

Specify the permitted workflow for each condition. In the assisted condition, record which tool was available, what context it received and whether it could modify files or run checks. Do not collect unrelated private conversations. Preserve only the task-relevant artifacts required to understand the implementation and its acceptance.

Keep the human acceptance responsibility stable across conditions. If assisted changes receive more scrutiny merely because they are labelled as assisted, the study may measure reviewer expectations rather than patch quality. Where practical, conceal the assigned condition from reviewers. Do not promise perfect blinding: style, comments and patch structure may still reveal how work was produced.

The intervention should also specify when an implementer is allowed to stop. “The code compiles” is a different stopping rule from “the behaviour contract is checked and the evidence is attached.” A tool can accelerate the first while contributing less to the second. The experiment should make that distinction visible instead of treating an early handoff as a completed improvement.

Record the queue rather than reconstruct it from memory

02 / A concrete example

Faster arrival can grow the queue

Enlarge diagram

Capture events as the work moves. A minimal record can distinguish task ready, implementation started, first patch ready, review started, changes requested, revised patch ready and final acceptance. Attach a stable task identity so a rewritten pull request does not become a new independent success.

The following is data-model pseudocode for the study, not an executed tracking system:

TaskRecord:
    task_id
    task_family
    assigned_condition
    repository_revision
    implementation_sessions[]
    review_sessions[]
    patch_revisions[]
    acceptance_checks[]
    outcome

ReviewSession:
    started_at
    ended_at
    question_category
    linked_patch_revision
    evidence_requested

Measure accepted throughput as the count of tasks satisfying the independent acceptance contract within the observation period. Also report tasks still waiting, tasks abandoned and tasks requiring substantial redesign. Excluding unfinished work would make a workflow that starts many difficult changes look better than one that finishes a smaller, coherent set.

Avoid reducing the study to a count of review comments. One important comment can uncover a serious defect; ten comments may concern wording. Record the consequence of a finding and the rework it requires. The goal is to understand the work, not to reward or punish visible activity.

Introduce a realistic limit on reviewer attention

An experiment with an always-available reviewer can establish patch-level review effort. It cannot show how a queue behaves when several implementers share limited attention. Both studies are useful, but they answer different questions.

For the queue experiment, define a fixed review schedule or a bounded amount of reviewer time. Keep it the same across conditions. Allow the queue to form, then record its age distribution and the number of tasks waiting at each observation point. Averages alone can hide a few changes that remain blocked for most of the study.

The arithmetic does not require a grand theory. If a hypothetical workflow proposes eight reviewable changes per day while the available review process can accept six comparable changes, unfinished work grows unless something else changes. The numbers are illustrative, and real tasks are not equal-sized units. Their purpose is to show why faster arrival cannot by itself establish faster completion.

The next design decision might be smaller patches, better evidence at handoff, more reviewer capacity or less work in progress. Automatically increasing implementation capacity would not resolve a review bottleneck. Equally, restricting implementation may waste capacity if the true problem is unclear acceptance criteria. Measure enough of the flow to distinguish those explanations.

Look for the mechanism behind a difference

Suppose an assisted condition produces faster first patches but longer active reviews. That is a finding to explain, not an immediate verdict. The patches may be larger, the evidence may be weaker, or reviewers may be less familiar with the chosen approach. The condition may also have attempted more demanding solutions than the task required.

Inspect representative revisions. Did a request for a small behaviour change become a broad refactor? Did generated tests verify the implementation's own assumptions? Did the patch introduce an unfamiliar dependency? Or did the assistant produce clearer code while the review process failed to take advantage of the attached evidence?

A useful experiment allows the assisted condition to win on more than one route. It might reduce implementation time without changing review effort. It might make review easier by producing a smaller, better-explained patch. It might be particularly helpful for one task family and unhelpful for another. Report those distinctions instead of forcing a single productivity multiplier.

The same discipline applies to a negative result. A configuration that performs poorly with an inadequate repository brief does not establish that every AI-assisted workflow is ineffective. It establishes a limitation of the tested workflow. The next intervention should address the proposed mechanism, rather than repeat the same comparison until a preferred result appears.

State how the study could mislead

03 / Watch the mechanism

Follow one task through rework

Enlarge diagram

Static view. Play to see how the mechanism unfolds.

Participants may change their behaviour because they know they are being observed. Reviewers may become faster as they learn the task families. A synthetic repository may have cleaner boundaries than a long-lived production system. A short study may overrepresent setup costs and underrepresent learning.

Record those limits in the design, then choose which can be reduced. Randomized order can help with time effects. A practice task can reduce initial tool unfamiliarity. Independent acceptance checks can expose defects that a friendly reviewer might overlook. None of those measures turns a small study into a universal statement about an entire profession.

The strongest objection is that measurement itself may interfere with normal development. That is why the recording burden should be small and tied to observable events. Do not ask participants to narrate every thought or convert the study into employee surveillance. Collect the evidence necessary for the process question, with clear agreement about its use.

The falsification conditions should be equally concrete. The claim of improved delivery fails if accepted throughput does not improve under the declared reviewer constraint. The claim of unchanged quality fails if the independent checks reveal more consequential defects. The claim that evidence-rich handoffs reduce review effort fails if reviewers still need to reconstruct the same missing reasoning.

Optimise for a system that can finish

The study's report should include task definitions, assignment rules, repository versions, permitted workflows, event records, acceptance outcomes and exclusions. A clear result should be attached to its actual execution date. Until that work is performed, there is no honest percentage to place in the headline.

The useful outcome is a decision about the next workflow. Perhaps the team should ask the assistant for a smaller patch and an explicit acceptance note. Perhaps review capacity needs to move closer to implementation. Perhaps a certain task family is worth automating while another benefits from more direct human design work.

Faster coding is valuable when the rest of the delivery system can convert it into accepted change. Measuring the review queue makes that conversion visible. It lets a team improve the point where work actually becomes trustworthy, rather than celebrating the point where the work first becomes available for somebody else to understand.

A later collector records what actually ran

Implementation note, prepared 12 November 2026. The original April 2026 sections describe a proposed study of team workflow. The following executable records local command events; it does not complete that human study or establish an AI productivity effect.

The new collector has a deliberately smaller job. It launches bounded validation commands, records their start and end, retains the revision of the script being executed and applies a stated machine acceptance predicate. This closes the gap between proposing an event record and demonstrating that a small record can be produced from actual execution.

It does not label command time as developer time. It does not infer review effort from a process duration or treat a successful validation command as an accepted feature. Those distinctions are essential because the proposed study above concerns a social and technical workflow, while this first implementation observes only a narrow automated boundary.

The shared packet lives with the diagnostic article. Its inputs are frozen first-party artifacts: recorded retrieval outcomes, versioned diagnostic evidence and selected operation traces. The collector starts child processes that validate or interpret those inputs. No model inference, database workload or remote service is rerun.

This is useful infrastructure for a later study, provided its limits remain explicit. A trustworthy event collector should first be able to say what it actually saw before anyone asks it to support a larger explanation of productivity.

Keep task identity separate from command completion

01 / See the idea

Four events, one explicit task identity

Enlarge diagram

Each observed command has a stable task identifier within the collection run. The events are ready, started, ended and acceptance. Every event also carries the script revision and an actor label identifying the fixture command runner.

The label is intentionally not developer or reviewer. There was no person implementing a change or performing a code review during these observations. A future human study would need different event sources and an agreed method of collecting them.

The ended event records the process exit code, elapsed command duration and a hash of standard output. The acceptance event records whether the exit code and structured output satisfy the declared predicate. A command that exits does not automatically pass that predicate.

The collector includes a deliberately invalid fixture. That command should be rejected, and the record should preserve the rejection rather than drop the task from the dataset. Its failure is expected by the collection scenario, but it remains a rejected child command. Confusing those two levels would produce misleading counts.

This is the same distinction a real workflow needs. A harness can successfully test that an invalid change is refused; that does not mean the invalid change was accepted. Reports should identify whether they count test scenarios, task outcomes or implementation attempts.

The revision hash makes the event's meaning more stable. If the acceptance predicate changes, a later report can distinguish runs rather than assuming that every event named accepted represents the same standard.

Synthetic queue cases test arithmetic, not people

The packet also contains two synthetic event cases for the reporting algorithm. One has ready, start and end positions of zero, three and seven; the other has positions of one, seven and nine. These are invented units, not clock readings from people or processes.

The expected total wait is nine units, and the expected total execution is six units. One case is accepted. The assertions check that the algorithm keeps waiting separate from execution and counts the intended outcome.

These fixtures are useful precisely because the answer is known before the code runs. They can catch an arithmetic mistake without requiring a real team to reproduce a particular queue shape. Their existence does not turn the values into an empirical finding.

The actual command collector writes a different record with an explicit actual_command_observations mode. Its UTC timestamps and elapsed durations come from execution. The synthetic cases remain in the correctness report and are never merged into the observed run as though they were additional participants.

That separation should survive export. A spreadsheet containing both kinds of row can easily create a convincing but meaningless average if provenance is lost. The schema therefore needs to say not only what a number is, but how it was obtained.

For a later team study, the same rule would distinguish observation, imputation, simulation and a proposed scenario. All can support reasoning, but they cannot be pooled casually and called measured productivity.

A successful validator is not a successful workflow

The child commands exercise two kinds of work. One checks a diagnostic packet against preserved evidence. Another replays recorded retrieval outputs through a small substitution-contract fixture. Both can be useful components of engineering work, but neither represents the full process of understanding a request, implementing a change and reviewing its consequences.

The collector's acceptance rule is correspondingly narrow: a zero exit code and structured status indicating that the fixture passed. It does not inspect whether a user received a valuable outcome. The final report leaves human review time and team productivity as unknown.

This limitation is not a reason to discard the implementation. It tells us where the next evidence source belongs. A real study would need task definitions, implementation revisions, independently assessed acceptance and a consented account of relevant human effort. The automated collector could contribute execution events without pretending to supply the missing human observations.

The distinction also prevents an easy form of optimism. Automated checks can become faster while people spend longer resolving unclear requirements. More checks can run while the same difficult changes remain unaccepted. A command-level improvement is one input to the workflow question, not its answer.

When presenting the collector, the useful statement is therefore concrete: these commands ran, these artifacts were examined, these predicates passed or failed, and these durations belong to those process boundaries. Anything broader needs further evidence.

Revision identity matters when acceptance changes

Suppose a future version of the validator adds a required source-version check. An output accepted by the earlier script might be rejected by the later one. That does not automatically indicate that the underlying task became worse; the standard changed.

The event record should preserve that distinction through revision identity. Comparing acceptance rates without it could make a stricter check look like a productivity regression or a weaker check look like a delivery improvement.

The same issue arises with human review. A team may learn that a class of changes needs an additional failure scenario. Later reviews take longer because they assess a more complete contract. A simple before-and-after comparison can misinterpret that learning if the acceptance definition is treated as constant.

This is why the original study proposal records task families and the intervention explicitly. The collector demonstrates only a small part of that discipline, but it makes the principle executable: an event is attached to the version of the process that gave the event meaning.

A later analysis should therefore ask whether the comparison preserves the task, the quality threshold and the relevant operating conditions. If any of them changed, the change should be part of the explanation rather than hidden inside a single throughput number.

The observed run contains commands, not reviewers

02 / See the idea

Three observed command durations

Enlarge diagram

The integration owner executed the collector once in the exclusive measurement lane. It recorded three child commands and twelve events: ready, started, ended and acceptance for each command. Two commands satisfied the stated predicate; the deliberately invalid fixture was rejected with exit code three.

Recorded command	Observed duration	Exit code	Machine acceptance
Diagnostic packet check	58.691 ms	0	Accepted
Recorded substitution check	59.716 ms	0	Accepted
Deliberately invalid fixture	56.781 ms	3	Rejected

These durations include starting the Python child process, reading its input files, performing the selected validation and producing its output. They are single observations, not medians or latency percentiles. Their similarity does not establish that the underlying activities take equivalent human effort, and the rejected command being shorter says nothing about the efficiency of rejecting real work.

The collector uses an elapsed-time clock for duration and records UTC separately for event identity. Wall-clock timestamps should not be treated as a substitute for the measured interval. No person reviewed a code change during this run, and no assisted-versus-unassisted comparison was performed.

The useful result is that a real execution produced the intended event sequence, revision identity and outcome distinction. The larger team-study question remains open exactly where it was open before: people, representative tasks and independently assessed work still need their own evidence.

What would be needed for the original team-level claim

To study whether AI assistance changes accepted team throughput, the experiment still needs people doing representative work, an explicit intervention, comparable tasks and an acceptance procedure that does not merely repeat the implementer's assumptions. None of those can be manufactured from local process timestamps.

The collector can support that future work by preserving execution events and revisions. It cannot infer when a developer was thinking, whether a reviewer understood the change, or how a delayed clarification affected the task. Those observations require an appropriate collection method and agreement about their use.

The study would also need to preserve incomplete and abandoned work. A system that records only commands that ran successfully would omit tasks blocked before execution. The current deliberate rejection is a small reminder of that reporting problem, not a substitute for observing it in a team.

The useful next step is therefore to define the missing boundary, not to rename the available data. Command execution remains command execution. A synthetic queue remains a fixture. A proposed study remains unperformed until its participants, tasks and outcomes exist.

This restraint makes the technical contribution more credible. We now have a small executable collector and a reportable event contract. We still do not have a causal result about AI-assisted development, and the article should make that distinction as easy to see as the code itself.

Historical versions and verification boundaries

These pins existed before the context date. Availability and static API checks do not establish a successful build or a production-safe patch level. Execution claims are limited to explicitly identified companion checks.

Component	Version	Available by	Source
.NET SDK	10.0.100	2025-11-11	Primary record
.NET runtime / ASP.NET Core shared framework	10.0.0	2025-11-11	Primary record

Download the complete laboratory and execution records.