Topics

Measurement

Tools and honest measurement: repeated runs with stated limits.

Findings

4 findings, each linking to its write-up and limits

  • BT-1The local model scored no measurably lower on GSM8K problems with new numbers: 168 of 184 originals right against 161 of 184 twins+3.8 ptsInterval includes zero
  • BT-2The hosted model's drop on the number twins was no different from the local model's: 175 of 184 against 172 of 184+2.2 ptsInterval includes zero
  • BT-3When a model got a number twin wrong, its answer was never the original problem's answer0 of 30
  • BT-439 of the local model's 656 replies were lost to a harness problem, not wrong answers; regraded, it matched the hosted model on these itemsGSM8K 177 vs 175 of 184; HumanEval 78 vs 75 of 80

Experiments

22 experiments

Difference in percentage points, 95% interval

Controlled trial · OKL

Is a code guide worth more to a small local model?

When a small local model must change a storefront to match a backend it can't see, does a generated guide to the backend help it, and does the guide help more than the backend's source code?

OKF test G's four cross-repo changes, rerun with Codex on a small open-weight model on my own machine. With the backend's generated guide, it got all 20 changes right, against 4 of 20 with no guide. With the backend's source code it got 19 of 20, slightly more cheaply. As for the hosted model, the guide stands in for the code but doesn't beat it.

Difference in percentage points, 95% interval

Controlled trial · SPS

Does splitting a feature across coding agents beat one agent?

When a planner splits a feature into parts for coding agents working at the same time, does the team get more of it right than one agent, and does giving each coder only its part's requirements lose details at the joins?

A planner split each of six features into three parts, and three Codex coders built them at the same time. Whether each coder saw only its own part's requirements or the whole task, the team passed no more of the hidden acceptance tests than one agent did (61.6% and 52.9% against 62.4%), at 2.3 to 2.5 times the cost and with no saving in time. The split added a new failure: parts that didn't fit, so the application wouldn't start.

Difference in percentage points, 95% interval

Controlled trial · SWT

Does a team of coding agents beat one agent?

Does a team of coding agents (planner, coder, reviewer) get more of a feature right than one agent, and does any gain survive when one agent gets the same spend?

I gave Codex six feature tasks, either alone or as a team: a planner, a coder and a reviewer. One agent passed 56.8% of the hidden acceptance tests; the team passed 25.7%, at higher cost. The reason was the hand-off: only the planner saw the task, and its summary lost the details. Giving every agent the task file brought the team level with one agent (54.1%), still at higher cost.

Difference in percentage points, 95% interval

Controlled trial · PE

Does an AI's visible thinking drive its answer? I planted errors in it to find out

If a wrong number is planted in the middle of a reasoning model's thinking, does its answer follow the error, does it catch it, or does the answer come from somewhere else?

I edited a number in the middle of a local reasoning model's thinking on 144 maths problems. A third of the time it carried the error to a wrong answer without a flicker of doubt. It caught about half. And in 1 run in 10, its thinking concluded one answer while its reply quietly gave another.

Difference in percentage points, 95% interval

Controlled trial · BT

Do benchmark scores show real ability, or remembered answers?

If a benchmark item is rewritten with new numbers, names or function name but the same logic, does a model score lower on the rewrite, as it would if it had memorised the original?

I rewrote GSM8K and HumanEval items with new numbers, new names or a renamed function, and asked a model running on my desk and a hosted one to solve both versions. Neither scored measurably lower on the rewrites, and no wrong answer on a rewrite was the original's answer.

Difference in percentage points, 95% interval

Controlled trial · DJ

Can an AI reviewer tell a broken change from a working one?

Shown a task and the change a coding agent made for it, can a model tell whether the change passes the hidden tests, and does the agent's own success report sway it?

Shown a task and a coding agent's change, gpt-5.6-luna called 57 of 96 right, no better than tossing a coin. The agent's own success report did not sway it, a doubtful one made it reject more of everything, and its confidence meant nothing.

Hidden tests passed of 16, one bar per run: plain Codex 4 and 4; plan plus approver 6, 13, 16; EGM plus approver 16, 5, 16

Pilot · EGM

Spec first with the Elephant-Goldfish Model, and an AI approver (pilot EGM)

Does writing a spec with Google's Elephant-Goldfish Model before any code is written make a coding agent build a hard feature more correctly, and how much of any gain comes from the person who approves the spec?

On a hard feature, a Claude approver reviewing the design lifted Codex from 4 of 16 hidden tests to as many as 16, whether the design came from Google's Elephant-Goldfish spec process or from a plain plan. EGM added nothing over the plain plan, at about five times the cost. A pilot: three runs per arm.

Difference in percentage points, 95% interval

Controlled trial · TSK

Telling the agent to test, with and without a skill

When the agent is told to write a focused API test of its change, does giving it the repository's verification skill help it write one that really checks the change, does the skill's design matter, and does a gate that checks for such a test work better than telling it?

One paragraph asking for a focused API test got the agent to write a real test of its change in 56 of 60 runs. Adding the repository's verification skill added nothing. A hook that sent the agent back once did as well. None of it made the changes measurably more correct.

Difference in percentage points, 95% interval

Controlled trial · ESK

Telling the agent to use its skills

With the same repository guidance in place, does telling the agent to use its skills, in the prompt or in AGENTS.md, raise the share of changes that pass independent acceptance tests?

Told to use the repository's skills, the agent read the verification skill every time, but mostly ran a smoke test instead of testing its change. Reading an instruction is not the same as carrying it out.

Difference in percentage points, 95% interval

Controlled trial · RSK

Repository skills

Does guidance written for one repository (an AGENTS.md and skills on how to build, test, verify and change it) raise the share of changes that pass independent acceptance tests, when the agent is left to discover it?

Guidance written for this repository, left for the agent to find, did not measurably make its changes more often correct; the agent rarely opened the part on checking its work.

Difference in percentage points, 95% interval

Controlled trial · VVS

Verification, skills or effort?

On tasks where the obvious change compiles but is wrong when the app runs, which gives more correct changes per dollar: a short instruction to verify, the Superpowers skill pack, or more reasoning effort?

Letting the agent think a little longer made its changes more often correct; an instruction to verify and a skill pack did not.

Difference in percentage points, 95% interval

Controlled trial · SKT

Skill packs across 16 tasks

On a varied set of real change tasks, do Superpowers or Matt Pocock's skills raise the share of changes that pass independent acceptance tests, and at what cost in tokens?

Two popular skill packs for coding agents did not reliably make the agent's changes more correct; one cost about three times as much.

Ratio, 95% interval (1× is no change)

Controlled trial · J

Agent-optimised repository

Does a repository set up for agents do better than the same repository left as it is?

Experiment J: a fully set-up repository cost more tokens with no reliable gain in correctness.

Bar chart of tokens per question for multi-level callers-of-callers questions, with and without the code graph

Evaluation · AGWT

AgentGraphWorkTree evaluation

Are the graph's facts correct, and when does the graph reduce agent tokens?

Code-graph accuracy and agent token cost: multi-hop questions cheapen; simple lookups do not overall.

1.30× (run 1); 1.07× (rerun)

Controlled trial · A

Click guide trials

On a smaller library the agent can read, does a generated guide improve answers or change cost?

On Click, a generated guide did not improve correctness and cost more tokens.

1 / 24

Controlled trial · D1

Cross-repo questions, unread backend

When a storefront change depends on a backend the agent cannot read, can it answer questions that need backend facts?

With the backend unread, agents answered 1 of 24 storefront-to-backend questions correctly.

+40.3% tokens; +28.7% cost

Controlled trial · DOCLAYER

Documentation-layer study

Given that you have some documentation for agents, which layout costs least, and do agents find the right parts unprompted?

Unpublished comparison of documentation layouts: heavy always-loaded context cost more; lean skills were discovered unprompted.

3 / 20 → 19 / 20

Controlled trial · F

Cross-repo change tasks, small

When the agent must change the storefront using facts only the backend states, how much does a backend guide help?

Hidden-test passes on small cross-repo changes: 3 of 20 with nothing, 19 of 20 with a backend guide.

3 / 20 → 18 / 20

Controlled trial · G

Cross-repo change tasks, harder

Does a backend guide still help on harder cross-repo changes that need several facts or parent-class fields?

Harder cross-repo changes: 3 of 20 with nothing, 18 of 20 with a backend guide (level with source).

Bar chart of cross-repository change tasks by kind of knowledge supplied, Codex, 16 runs per arm

Controlled trial · H1

What kind of facts? Change tasks

For harder cross-repo changes, what kind of facts help — contract, short note, guide, or source?

On harder cross-repo changes, contract, note, guide and source all beat nothing and did not differ reliably from each other.

guide/source 21/24; contract 8/24; note 0/24

Controlled trial · H2

Internal call-structure questions

For questions about internal call structure across a boundary, which knowledge kind works?

For internal call-structure questions, only the guide and the source worked (21 of 24 each).

14 / 16

Controlled trials · OKF

OKFExperiment and the worked example

When a dependency is unread or stale, and when verification can be rewritten, what decides the outcome?

Controlled trials on facts across a repository boundary, stale knowledge, and a reproducible worked example with agent trials.

Other topics: Context, Verification, Safety and cost, Skills, Review, Briefs, Parallel, Harness