Synthesis
What the experiments show
33 experiments · 89 findings · 9 topics
A connected body of controlled trials, not a list of one-off tests. Start with three results, then read by the problem you are trying to solve. Every number is read from the experiment it comes from, with its limits.
Three results to read first
If you read three, read these. Each shows the result, its interval and its limit.
Does a team of agents beat one agent?
A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money
−31.2 pts
Difference in percentage points, 95% interval · Interval excludes zero
Limit one model at one effort level, six features in one Java codebase, team instructions written by me
Does a team of coding agents beat one agent? (SWT-1) →Is code that works also secure?
With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole
18%
Proportion, 95% interval
Limit one model, one small service, eight deliberately chosen tasks; the baseline breaches are concentrated in one of them
The endpoint worked. Was it secure? (SEC-1) →Can false evidence mislead a security agent?
One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence
+67.5 pts
Difference in percentage points, 95% interval · Interval excludes zero
Limit one local model, 40 synthetic scenarios, a minimal agent that was asked to report suspected manipulation but had no other hardening
Can manipulated evidence mislead an AI security agent? (EM-1) →
How to read these results
- Controlled trial
- The same tasks run under different set-ups, with the runs counted. A run is one agent attempt at one task.
- pts
- Percentage points: the gap between two pass rates. +10 pts means, for example, 60% against 50%.
- 95% interval
- The range the true difference plausibly lies in. If it includes no change (0 pts, or 1× for a ratio), the trial cannot tell the set-ups apart.
- No measurable difference
- The test could not tell the set-ups apart with this many cases. It does not mean they are equal.
- Pre-registered or exploratory
- Pre-registered hypotheses were published before the runs. Exploratory analyses were added after seeing results, so treat them as leads.
- Limit
- What the result does not cover, such as one model or a small task set. It sits next to every finding.
New to all this? A short route in →
By engineering problem
The same experiments, grouped by the question an engineer is trying to answer. Each line is a finding record: its number, interval and limits are on the write-up it links to. The short reading under each question is my wording.
When does extra context help an agent?
My reading In these trials, supplying facts the agent could not otherwise see made the difference. Extra set-up for facts it could already read mostly added cost.
- D1-1With the backend unread, agents answered 1 of 24 storefront-to-backend questions correctly, at about 785k tokens per question on average1 / 24
- OKF-2When the backend facts a storefront change depended on were supplied as a guide, hidden-test passes on cross-repository changes went from 3 of 20 with nothing to 19 of 20 on four small tasks and 18 of 20 on four harder ones.3 → 18–19
- OKL-2The guide did no better than the backend's own source code: 20 of 20 against 19 of 20, and source runs were slightly faster and cheaper+5 ptsNo measurable difference
- J-1On a 1,173-file Java backend, the copy set up for agents (generated guide, code graph, three skills, configured AGENTS.md) used 35–48% more tokens per run than the plain copy, and its pass rate was not reliably different.+35–48%Interval excludes 1×
- A-1On Click, the default guide did not make agents cheaper: against no guide it cost 1.30× the tokens in the first run and 1.07× when rerun without the confirm-in-source sentence. Accuracy was already at ceiling with or without a guide.1.30× (run 1); 1.07× (rerun)
What happens when the facts are out of date or wrong?
My reading Agents repeated out-of-date and planted facts with confidence. One sentence telling the agent to confirm in source helped with drift.
- OKF-1After four renames in a backend the agent could not read, agents given the out-of-date guide gave the old fact in 14 of 16 drift answers; none of the wrong answers hedged.14 / 16
- B3-1On Click after three planted refactors, an agent under an efficiency prompt followed the out-of-date guide in 8 of 12 drift answers. Adding one sentence to the pointer, to read the source to confirm before relying on it, cut that to 0 of 12 at the same cost (77k against 79k tokens).8/12 → 0/12
- E-1When some entries in a backend guide were deliberately wrong and the source could not be checked, labelling facts [high] or [medium] and telling the agent to say when it relied on a guess raised disclosure from 0 of 16 answers to 6 of 16. Wrong answers fell only from 11 to 8 of 16, which is not reliable.0/16 → 6/16
- RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5+90 ptsInterval excludes zero
- RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known13/15
Do teams of agents beat one agent?
My reading Not in these trials. A team did worse until every member had the full task, and then only matched one agent.
- SWT-1A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money−31.2 ptsInterval excludes zero
- SWT-2Giving every team member the task file raised the team from 25.7% to 54.1% of hidden tests passed, level with one agent, at about 1.6 times its cost+28.5 ptsInterval excludes zero
- SPS-1Splitting a feature across three coders working at the same time, each given only its part's requirements, passed 61.6% of hidden acceptance tests; one agent alone passed 62.4%, at less than half the cost−0.8 ptsInterval includes zero
- SPS-2In 4 of 72 split runs the application would not start at all, against none of 36 one-agent runs4 of 72
Can you trust an agent, or an AI reviewer, on whether the work is right?
My reading Agents usually said the change worked when it had failed, and agent reviewers approved failing work. Check with something the agent does not control.
- SKT-3In 28 of the 30 failed runs, the agent said its change worked; this held in every arm28/30
- SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed69 of 71
- SWT-3Every time an agent reviewer approved a change, the change failed hidden acceptance tests: 24 of 24 approvals24 of 24
- SPS-3Agent reviewers approved code that failed hidden acceptance tests 14 times out of 1514 of 15
- DJ-1Shown the task and the diff, the AI judge called 57 of 96 changes right: it caught 21 of 48 broken changes and passed 36 of 48 working ones, no measurable difference from guessing0.59Interval excludes zero
- DJ-4The judge's confidence did not track whether it was right: it averaged 87 out of 100 when right and 86 when wrong87 vs 86
Do skills and instructions change what the agent does?
My reading Reading a skill is not the same as acting on it, and telling the agent to use skills mostly did not raise the pass rate. A specific instruction did change what the agent did. In VVS, more reasoning effort did more than the skills.
- ESK-1Reading is not doing: told to follow the verification skill, the agent read it in every run, but a test of its change was detected in only 15 of 60 runs (prompt) and 10 of 60 (AGENTS.md); most runs ran the skill's smoke-test example instead15/60 and 10/60
- ESK-2Telling the agent to use its skills did not measurably raise the pass rate: 42 of 60 runs passed when told in the prompt and 43 when told in AGENTS.md, against 37 with no guidance and 39 with the guidance left to discover+8 to +10 pts, not significantNo measurable difference
- SKT-1Neither skill pack reliably raised the pass rate: Superpowers passed 24 of 32 runs, the same as with no skills, and Pocock passed 27 of 32, a gap small enough to be chance (Holm p = 0.38)+9.4 ptsInterval includes zero
- RSK-1Repository guidance left for the agent to discover did not measurably raise the pass rate: 40 of 60 runs passed, against 38 of 60 without it (p = 0.40)40 vs 38 of 60 · not measurableNo measurable difference
- TSK-1A one-paragraph instruction to write a focused API test got the agent to write one that fails without its change and passes with it in 56 of 60 runs; given only the task, it wrote none56/60 vs 0/60
- VVS-1Medium reasoning effort, with no skills and no instruction, passed 49 of 60 runs against 36 of 60 at low effort: better on 9 tasks and worse on 1 (Holm p = 0.041)+21.7 ptsInterval excludes zero
Is an agent secure when it works?
My reading Working and secure were different things. In EM, a false fact was a stronger attack than a blunt injected instruction, and a check against an authoritative record helped.
- SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole18%
- SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request38 of 40
- EM-1One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence+67.5 ptsInterval excludes zero
- EM-2An injected instruction in a ticket comment changed 0 of 120 decisions and was flagged as manipulation in all 120 runs+0.8 ptsNo measurable difference
- EM-4With a tool that checks authoritative records, 359 of 360 manipulated runs reached the right decision+42.5 ptsInterval excludes zero
By topic
Each topic page lists its findings first, then the experiments behind them.
- Measurement4 findings · 22 experiments
- Context37 findings · 19 experiments
- Verification48 findings · 16 experiments
- Safety and cost30 findings · 8 experiments
- Skills29 findings · 8 experiments
- Review18 findings · 5 experiments
- Briefs11 findings · 3 experiments
- Parallel10 findings · 3 experiments
- Harness7 findings · 2 experiments
The experiments
The write-ups behind the findings, newest first.
- Grilling or a good prompt? Does the skill beat a two-sentence instruction?
Does Matt Pocock's grilling skill recover more of the rules only the developer knows than a short prompt that makes the agent list its open decisions and wait, or than an interview capped at three questions? Does grilling's branch-by-branch walk win where one answer opens new questions, and what does each cost the developer?
- Grill me first? Does letting the agent interview you get you what you wanted?
When a coding agent interviews the developer before building (Matt Pocock's grilling skill), does it build more of what the developer actually wanted than working from the ticket, and is that from the skill, from being allowed to ask, or from the developer's answers? Do grill-with-docs's glossary and decision records help a later ticket?
- Can a repository hijack its coding agent?
If a repository or ticket contains an instruction to leak secrets and change git configuration, does a coding agent working an ordinary ticket carry it out, and does one paragraph of warning in the prompt stop it?
- The endpoint worked. Was it secure?
When a coding agent adds endpoints to a Spring Boot API from ordinary tickets, how often are they working but insecure, and does a security skill, a security gate of public tests, or both make secure endpoints more likely?
- Three AI agents agreed. Did that make them right?
When security evidence has been tampered with, does a team of AI agents catch the error, repeat it, or make it worse, and does a stronger model reviewing the team's work fix it?
- Is a code guide worth more to a small local model?
When a small local model must change a storefront to match a backend it can't see, does a generated guide to the backend help it, and does the guide help more than the backend's source code?
- Does splitting a feature across coding agents beat one agent?
When a planner splits a feature into parts for coding agents working at the same time, does the team get more of it right than one agent, and does giving each coder only its part's requirements lose details at the joins?
- Does a team of coding agents beat one agent?
Does a team of coding agents (planner, coder, reviewer) get more of a feature right than one agent, and does any gain survive when one agent gets the same spend?
Five rules to start with
The shortest practical version of the work: five rules, each tied to a finding. Each links to the write-up with its limits.
- 1Give the agent the facts it cannot see3 → 18–19
When the code a change depends on lives in another repository, a guide to it is the difference between mostly failing and mostly passing.
Limit 5 runs per task; identifiers renamed by me; one model
OKFExperiment and the worked example (OKF-2) → - 2A stale guide is worse than none14 / 16
Out-of-date facts get repeated with confidence. One sentence telling the agent to confirm in source cut the repeated errors from 8 of 12 to 0 of 12 (B3-1).
Limit the drift was planted; 4 runs per question; one model
OKFExperiment and the worked example (OKF-1) → - 3More set-up is not a better result+35–48%
A repository fitted out the recommended way cost more per run without a reliable gain in correctness.
Limit one repository; one model; mechanical edits and structure questions; 8 runs per task per arm; four ingredients tested together
Agent-optimised repository (J-1) → - 4Do not take “it works” on trust28/30
Most failed runs ended with the agent saying the change worked. Check with something the agent does not control.
Limit the agent could not see the hidden tests; one model at low reasoning effort
Skill packs across 16 tasks (experiment SKT) (SKT-3) → - 5Run a friction loop, and fix causes in code before prose15.3 → 7.7 turns
Ask agents to record what blocked them, review the notes weekly, and fix the repeated cause as high up as you can. In one trial, documenting the test order still left a note in 1 of 3 runs; a script that ran the migrations first left none, and the runs used about half the tokens (647k to 314k).
Limit One FastAPI task, Codex, three runs per arm; the scripted arm was run after the others and no interval excluded “no difference”.
Chapter 6, the harness and the friction loop →
Follow new experiments
Each experiment is published here with its method, data and limits. The channel explains the work in video; the site holds the evidence.