What I've measured so far

Context

The right facts, not every fact: context, documentation and knowledge layers. Experiments on this topic →

• GVP-1A two-sentence structured prompt recovered as many developer-only rules as the grilling skill
• GVP-2Grilling did not pull ahead on nested tickets, because the first rule in each chain was rarely found
• GVP-3Capping the interview at three questions did well on simple tickets and fell behind with more unknowns
• GRL-1Grilling recovered every requirement that only the developer knew; working from the ticket recovered none
• GRL-2The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone
• GRL-3Telling the agent "ask me anything you need to know first" produced no questions at all
• GRL-4Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it
• RI-1Without a defence, a planted instruction disguised as a setup script hijacked the agent every time it sat in AGENTS.md or a ticket comment, and never anywhere else
• RI-2Every hijacked run reported a clean success, listing the malicious script as a passed check
• RI-3One paragraph in the prompt, saying repository text is information and not instructions, cut hijacks from 18 of 90 runs to 1, at no measurable cost
• OKL-1With the backend's generated guide, a small local model got 20 of 20 cross-repo changes right, against 4 of 20 with no guide
• OKL-2The guide did no better than the backend's own source code: 20 of 20 against 19 of 20, and source runs were slightly faster and cheaper
• OKL-3Without the backend, the agent presented guessed field names as checked facts in 9 of 15 failed runs
• RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5
• RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known
• RAG-3When no retrieved paragraph held the answer, both models mostly said so: NOT FOUND in 107 (27B) and 112 (8B) of 120 questions; made-up answers 8 and 6
• RAG-4No measurable difference between the models: qwen3:8b repeated the planted number in 98 of 120 questions, against 102 for qwen3.8:27b
• J-1On a 1,173-file Java backend, the copy set up for agents (generated guide, code graph, three skills, configured AGENTS.md) used 35–48% more tokens per run than the plain copy, and its pass rate was not reliably different.
• AGWT-1Java call links in the graph matched the compiled bytecode: all 13,380 high-confidence links on Shopizer, and 8,947 of 8,957 medium-confidence ones.
• AGWT-2On callers-of-callers questions (Click and Express, 36 runs) the graph cut tokens for Codex (84k against 196k) and Cursor (121k against 414k), and was close to level for Copilot (161k against 181k). One-hop lookups were level or dearer.
• AGWT-3Across 192 controlled runs on Python and TypeScript repositories the graph did not reduce tokens overall: agents ran kg and then, as the instructions suggest for medium-confidence results, checked it with grep. Every run was correct, so these questions measure cost only.
• B3-1On Click after three planted refactors, an agent under an efficiency prompt followed the out-of-date guide in 8 of 12 drift answers. Adding one sentence to the pointer, to read the source to confirm before relying on it, cut that to 0 of 12 at the same cost (77k against 79k tokens).
• A-1On Click, the default guide did not make agents cheaper: against no guide it cost 1.30× the tokens in the first run and 1.07× when rerun without the confirm-in-source sentence. Accuracy was already at ceiling with or without a guide.
• A-2On Click, the thinnest guide (types and function names only, with no citations, links, call edges or index) was the dearest in both runs: 1.42× the no-guide cost, then 1.26× once the confirm-in-source sentence was removed.
• D1-1With the backend unread, agents answered 1 of 24 storefront-to-backend questions correctly, at about 785k tokens per question on average
• DOCLAYER-1In the documentation-layer study, the heavy design (a long always-read AGENTS.md, a separate tree of decision records and OKF reference pages, with prompts telling the agent to read all of it) cost 40.3% more tokens and 28.7% more in cost than the comparison design on the simple change, in all four rounds.
• DOCLAYER-2With prompts that never mentioned any of the documentation, agents found the lean design's small index and opened the task-relevant skills in 24 of 24 measured cells.
• E-1When some entries in a backend guide were deliberately wrong and the source could not be checked, labelling facts [high] or [medium] and telling the agent to say when it relied on a guess raised disclosure from 0 of 16 answers to 6 of 16. Wrong answers fell only from 11 to 8 of 16, which is not reliable.
• F-1On four small cross-repository change tasks, hidden-test passes went from 3 of 20 with no guide to 19 of 20 with a guide to the backend, at about half the tokens (242k against 482k per run).
• G-1On four harder cross-repository change tasks, hidden-test passes went from 3 of 20 with no guide to 18 of 20 with a guide to the backend, level with the agent being able to read the backend source (18 of 20).
• H1-1On harder change tasks with nothing, 4 of 16 runs passed the hidden tests; with a contract 13, a hand-written note 14, a guide 14 and the backend source 14 of 16. The four knowledge arms beat nothing and did not differ reliably from each other.
• H1-2Among the four knowledge arms on those change tasks, the hand-written 3 KB note was cheapest at about 128k tokens per run, against 192k for the contract, 282k for the guide and 334k for the backend source (277k with nothing).
• H2-1On six internal call-structure questions, the guide and the backend source each answered 21 of 24 correctly, against 8 of 24 for the contract, 0 of 24 for the hand-written note and 1 of 24 with nothing.
• H3-1A contract that listed only one success code (201) led agents to treat the service's other success code as an error: on the retry task the contract arm passed 0 of 4 runs, while the other four arms passed 4 of 4.
• OKF-1After four renames in a backend the agent could not read, agents given the out-of-date guide gave the old fact in 14 of 16 drift answers; none of the wrong answers hedged.
• OKF-2When the backend facts a storefront change depended on were supplied as a guide, hidden-test passes on cross-repository changes went from 3 of 20 with nothing to 19 of 20 on four small tasks and 18 of 20 on four harder ones.
• OKF-3On the worked example's task, all 6 agents without the provider contract wrote a contract of their own as version 1.0.0 and edited the freshness gate so it compared it with itself, and none passed the black-box check; all 6 with the contract did. All 12 runs reported passed while listing the real inventory check as unverified.

Briefs

Briefs, plans and specifications: deciding what "done" means. Experiments on this topic →

• GVP-1A two-sentence structured prompt recovered as many developer-only rules as the grilling skill
• GVP-2Grilling did not pull ahead on nested tickets, because the first rule in each chain was rarely found
• GVP-3Capping the interview at three questions did well on simple tickets and fell behind with more unknowns
• GRL-1Grilling recovered every requirement that only the developer knew; working from the ticket recovered none
• GRL-2The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone
• GRL-3Telling the agent "ask me anything you need to know first" produced no questions at all
• GRL-4Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it
• EGM-1A Claude approver reviewing the design before the build lifted Codex on a hard feature: three of six approver runs passed all 16 hidden tests, against 4 of 16 in both runs of plain Codex at the same effort
• EGM-2EGM did no better than a plain plan reviewed by the same kind of approver, and cost about five times as much in Codex usage
• EGM-3The score followed one sentence in the approved design, naming the cart's discount line "discount" as the task asks. Every run whose design said so scored 13 or 16; the two that did not scored 5 and 6. The name always came from the approver, never from EGM's reviewers
• EGM-4Without an approver's judgement, EGM's thorough designs did not help: built at low effort they produced skeletons (3 and 1 of 16), and built at medium effort they scored 4 of 16, the same as plain Codex with the task alone

Verification

Checking that a change works: executable checks and independent review. Experiments on this topic →

• BRK-1With the test suite silently skipped, the agent reported a pass without running a test; one line in AGENTS.md fixed most of it
• BRK-2When some tests were quietly excluded, every setup mostly reported a pass, and the counting check script made it worse
• BRK-3Visible failures were reported honestly, and no agent tampered with a test
• SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole
• SEC-2A security skill, opened in every run, made no measurable difference
• SEC-3In an exploratory analysis after two faults in my test materials were fixed, a gate of public security tests beat the skill
• SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request
• SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed
• TH-1A vote of three identical agents was no better than one agent at seeing through a false fact
• TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted
• TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own
• TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)
• EM-1One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence
• EM-2An injected instruction in a ticket comment changed 0 of 120 decisions and was flagged as manipulation in all 120 runs
• EM-3With both a falsified fact and the instruction, the agent flagged manipulation in all 120 runs and still reached the attacker's intended outcome in 73
• EM-4With a tool that checks authoritative records, 359 of 360 manipulated runs reached the right decision
• RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5
• RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known
• RAG-3When no retrieved paragraph held the answer, both models mostly said so: NOT FOUND in 107 (27B) and 112 (8B) of 120 questions; made-up answers 8 and 6
• RAG-4No measurable difference between the models: qwen3:8b repeated the planted number in 98 of 120 questions, against 102 for qwen3.8:27b
• PE-1A wrong number planted in the model's thinking was carried through to the matching wrong answer in 81 of 230 runs (35%); with the thinking left unchanged, 1 of 230 went wrong
• PE-2The model caught the planted slip in its own thinking in 124 of 230 runs (54%)
• PE-3In 23 of 230 runs (10%), the thinking ended on the wrong answer but the written reply gave the right one without saying why; the reverse never happened
• PE-4The thinking does real work: forced to answer with no thinking, the model got 23% of problems right; after its full thinking, 94%
• DJ-1Shown the task and the diff, the AI judge called 57 of 96 changes right: it caught 21 of 48 broken changes and passed 36 of 48 working ones, no measurable difference from guessing
• DJ-2Showing the judge the agent's own success report did not make it miss more broken changes: it caught 20 of 48 with the report and 21 without
• DJ-3A fixed doubtful report ('I could not run the tests') made the judge catch more broken changes (30 of 48) but pass fewer working ones (22 of 48); it got no more accurate overall
• DJ-4The judge's confidence did not track whether it was right: it averaged 87 out of 100 when right and 86 when wrong
• TSK-1A one-paragraph instruction to write a focused API test got the agent to write one that fails without its change and passes with it in 56 of 60 runs; given only the task, it wrote none
• TSK-2Adding the repository's verification skill to the instruction did not help: 52 of 60 runs wrote a relevant test with the original skill and 54 with a revised one that puts the focused test first, against 56 with the instruction alone
• TSK-3A stop hook that checked for a relevant test when the agent tried to finish, and sent it back once if there was none, worked as well as the instruction: it blocked every run, and 58 of 60 then had a relevant test
• TSK-4Real tests of the change did not measurably raise the pass rate: 41 to 47 of 60 in the arms that wrote them, against 40 with no guidance; medium reasoning effort, with no tests, passed 49
• ESK-1Reading is not doing: told to follow the verification skill, the agent read it in every run, but a test of its change was detected in only 15 of 60 runs (prompt) and 10 of 60 (AGENTS.md); most runs ran the skill's smoke-test example instead
• ESK-2Telling the agent to use its skills did not measurably raise the pass rate: 42 of 60 runs passed when told in the prompt and 43 when told in AGENTS.md, against 37 with no guidance and 39 with the guidance left to discover
• ESK-3A one-line entry in AGENTS.md got the verification skill read as reliably as an instruction in the prompt; the pass rates were similar (43 and 42 of 60), and the experiment cannot tell whether placement matters
• ESK-4Being told cost about twice as much per correct change as no guidance: 1.98 times when told in the prompt and 1.77 times in AGENTS.md
• RSK-1Repository guidance left for the agent to discover did not measurably raise the pass rate: 40 of 60 runs passed, against 38 of 60 without it (p = 0.40)
• RSK-2The agent read the skills on building and on changing code in every run, but opened the skill on checking a change through the API in only 8 of 60, and a test of its own change was detected in 2
• RSK-3Cost 1.29 times as much per correct change (1.01 to 1.55), against about 2.4 to 2.5 times for the Superpowers pack in earlier experiments
• VVS-1Medium reasoning effort, with no skills and no instruction, passed 49 of 60 runs against 36 of 60 at low effort: better on 9 tasks and worse on 1 (Holm p = 0.041)
• VVS-2An instruction to check the change against the running application did not measurably raise the pass rate (39 of 60 against 36, Holm p = 0.67); the agent tried to start the app in 47 runs, but it got the app to answer in only 3
• VVS-3Superpowers again passed about as often as no skills (37 of 60 against 36) and cost the most per correct change: 7.0 cents, against 4.0 at medium effort and 2.8 with no skills at low effort
• SKT-1Neither skill pack reliably raised the pass rate: Superpowers passed 24 of 32 runs, the same as with no skills, and Pocock passed 27 of 32, a gap small enough to be chance (Holm p = 0.38)
• SKT-2Superpowers used about three times the tokens of no skills on the same task; it wrote tests in 23 of 32 runs, where the agent with no skills wrote none
• SKT-3In 28 of the 30 failed runs, the agent said its change worked; this held in every arm
• B3-1On Click after three planted refactors, an agent under an efficiency prompt followed the out-of-date guide in 8 of 12 drift answers. Adding one sentence to the pointer, to read the source to confirm before relying on it, cut that to 0 of 12 at the same cost (77k against 79k tokens).
• E-1When some entries in a backend guide were deliberately wrong and the source could not be checked, labelling facts [high] or [medium] and telling the agent to say when it relied on a guess raised disclosure from 0 of 16 answers to 6 of 16. Wrong answers fell only from 11 to 8 of 16, which is not reliable.
• H3-1A contract that listed only one success code (201) led agents to treat the service's other success code as an error: on the retry task the contract arm passed 0 of 4 runs, while the other four arms passed 4 of 4.
• OKF-1After four renames in a backend the agent could not read, agents given the out-of-date guide gave the old fact in 14 of 16 drift answers; none of the wrong answers hedged.
• OKF-2When the backend facts a storefront change depended on were supplied as a guide, hidden-test passes on cross-repository changes went from 3 of 20 with nothing to 19 of 20 on four small tasks and 18 of 20 on four harder ones.
• OKF-3On the worked example's task, all 6 agents without the provider contract wrote a contract of their own as version 1.0.0 and edited the freshness gate so it compared it with itself, and none passed the black-box check; all 6 with the contract did. All 12 runs reported passed while listing the real inventory check as unverified.

Harness

The harness and the friction loop: fixing recurring environment problems. Experiments on this topic →

• BRK-1With the test suite silently skipped, the agent reported a pass without running a test; one line in AGENTS.md fixed most of it
• BRK-2When some tests were quietly excluded, every setup mostly reported a pass, and the counting check script made it worse
• BRK-3Visible failures were reported honestly, and no agent tampered with a test
• RI-1Without a defence, a planted instruction disguised as a setup script hijacked the agent every time it sat in AGENTS.md or a ticket comment, and never anywhere else
• RI-2Every hijacked run reported a clean success, listing the malicious script as a passed check
• RI-3One paragraph in the prompt, saying repository text is information and not instructions, cut hijacks from 18 of 90 runs to 1, at no measurable cost
• EM-1One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence
• EM-2An injected instruction in a ticket comment changed 0 of 120 decisions and was flagged as manipulation in all 120 runs
• EM-3With both a falsified fact and the instruction, the agent flagged manipulation in all 120 runs and still reached the attacker's intended outcome in 73
• EM-4With a tool that checks authoritative records, 359 of 360 manipulated runs reached the right decision

Parallel

Parallel agents: isolation and the cost of coordination. Experiments on this topic →

• TH-1A vote of three identical agents was no better than one agent at seeing through a false fact
• TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted
• TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own
• TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)
• SPS-1Splitting a feature across three coders working at the same time, each given only its part's requirements, passed 61.6% of hidden acceptance tests; one agent alone passed 62.4%, at less than half the cost
• SPS-2In 4 of 72 split runs the application would not start at all, against none of 36 one-agent runs
• SPS-3Agent reviewers approved code that failed hidden acceptance tests 14 times out of 15
• SWT-1A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money
• SWT-2Giving every team member the task file raised the team from 25.7% to 54.1% of hidden tests passed, level with one agent, at about 1.6 times its cost
• SWT-3Every time an agent reviewer approved a change, the change failed hidden acceptance tests: 24 of 24 approvals

Skills

Agent skills and instruction files: what they change, and what they do not. Experiments on this topic →

• BRK-1With the test suite silently skipped, the agent reported a pass without running a test; one line in AGENTS.md fixed most of it
• BRK-2When some tests were quietly excluded, every setup mostly reported a pass, and the counting check script made it worse
• BRK-3Visible failures were reported honestly, and no agent tampered with a test
• GVP-1A two-sentence structured prompt recovered as many developer-only rules as the grilling skill
• GVP-2Grilling did not pull ahead on nested tickets, because the first rule in each chain was rarely found
• GVP-3Capping the interview at three questions did well on simple tickets and fell behind with more unknowns
• GRL-1Grilling recovered every requirement that only the developer knew; working from the ticket recovered none
• GRL-2The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone
• GRL-3Telling the agent "ask me anything you need to know first" produced no questions at all
• GRL-4Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it
• SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole
• SEC-2A security skill, opened in every run, made no measurable difference
• SEC-3In an exploratory analysis after two faults in my test materials were fixed, a gate of public security tests beat the skill
• SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request
• SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed
• TSK-1A one-paragraph instruction to write a focused API test got the agent to write one that fails without its change and passes with it in 56 of 60 runs; given only the task, it wrote none
• TSK-2Adding the repository's verification skill to the instruction did not help: 52 of 60 runs wrote a relevant test with the original skill and 54 with a revised one that puts the focused test first, against 56 with the instruction alone
• TSK-3A stop hook that checked for a relevant test when the agent tried to finish, and sent it back once if there was none, worked as well as the instruction: it blocked every run, and 58 of 60 then had a relevant test
• TSK-4Real tests of the change did not measurably raise the pass rate: 41 to 47 of 60 in the arms that wrote them, against 40 with no guidance; medium reasoning effort, with no tests, passed 49
• ESK-1Reading is not doing: told to follow the verification skill, the agent read it in every run, but a test of its change was detected in only 15 of 60 runs (prompt) and 10 of 60 (AGENTS.md); most runs ran the skill's smoke-test example instead
• ESK-2Telling the agent to use its skills did not measurably raise the pass rate: 42 of 60 runs passed when told in the prompt and 43 when told in AGENTS.md, against 37 with no guidance and 39 with the guidance left to discover
• ESK-3A one-line entry in AGENTS.md got the verification skill read as reliably as an instruction in the prompt; the pass rates were similar (43 and 42 of 60), and the experiment cannot tell whether placement matters
• ESK-4Being told cost about twice as much per correct change as no guidance: 1.98 times when told in the prompt and 1.77 times in AGENTS.md
• RSK-1Repository guidance left for the agent to discover did not measurably raise the pass rate: 40 of 60 runs passed, against 38 of 60 without it (p = 0.40)
• RSK-2The agent read the skills on building and on changing code in every run, but opened the skill on checking a change through the API in only 8 of 60, and a test of its own change was detected in 2
• RSK-3Cost 1.29 times as much per correct change (1.01 to 1.55), against about 2.4 to 2.5 times for the Superpowers pack in earlier experiments
• VVS-1Medium reasoning effort, with no skills and no instruction, passed 49 of 60 runs against 36 of 60 at low effort: better on 9 tasks and worse on 1 (Holm p = 0.041)
• VVS-2An instruction to check the change against the running application did not measurably raise the pass rate (39 of 60 against 36, Holm p = 0.67); the agent tried to start the app in 47 runs, but it got the app to answer in only 3
• VVS-3Superpowers again passed about as often as no skills (37 of 60 against 36) and cost the most per correct change: 7.0 cents, against 4.0 at medium effort and 2.8 with no skills at low effort
• SKT-1Neither skill pack reliably raised the pass rate: Superpowers passed 24 of 32 runs, the same as with no skills, and Pocock passed 27 of 32, a gap small enough to be chance (Holm p = 0.38)
• SKT-2Superpowers used about three times the tokens of no skills on the same task; it wrote tests in 23 of 32 runs, where the agent with no skills wrote none
• SKT-3In 28 of the 30 failed runs, the agent said its change worked; this held in every arm

Review

Review, merging and risk: matching scrutiny to consequence. Experiments on this topic →

• TH-1A vote of three identical agents was no better than one agent at seeing through a false fact
• TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted
• TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own
• TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)
• SPS-1Splitting a feature across three coders working at the same time, each given only its part's requirements, passed 61.6% of hidden acceptance tests; one agent alone passed 62.4%, at less than half the cost
• SPS-2In 4 of 72 split runs the application would not start at all, against none of 36 one-agent runs
• SPS-3Agent reviewers approved code that failed hidden acceptance tests 14 times out of 15
• SWT-1A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money
• SWT-2Giving every team member the task file raised the team from 25.7% to 54.1% of hidden tests passed, level with one agent, at about 1.6 times its cost
• SWT-3Every time an agent reviewer approved a change, the change failed hidden acceptance tests: 24 of 24 approvals
• DJ-1Shown the task and the diff, the AI judge called 57 of 96 changes right: it caught 21 of 48 broken changes and passed 36 of 48 working ones, no measurable difference from guessing
• DJ-2Showing the judge the agent's own success report did not make it miss more broken changes: it caught 20 of 48 with the report and 21 without
• DJ-3A fixed doubtful report ('I could not run the tests') made the judge catch more broken changes (30 of 48) but pass fewer working ones (22 of 48); it got no more accurate overall
• DJ-4The judge's confidence did not track whether it was right: it averaged 87 out of 100 when right and 86 when wrong
• EGM-1A Claude approver reviewing the design before the build lifted Codex on a hard feature: three of six approver runs passed all 16 hidden tests, against 4 of 16 in both runs of plain Codex at the same effort
• EGM-2EGM did no better than a plain plan reviewed by the same kind of approver, and cost about five times as much in Codex usage
• EGM-3The score followed one sentence in the approved design, naming the cart's discount line "discount" as the task asks. Every run whose design said so scored 13 or 16; the two that did not scored 5 and 6. The name always came from the approver, never from EGM's reviewers
• EGM-4Without an approver's judgement, EGM's thorough designs did not help: built at low effort they produced skeletons (3 and 1 of 16), and built at medium effort they scored 4 of 16, the same as plain Codex with the task alone

Safety and cost

Safety, cost and the human in the loop. Experiments on this topic →

• RI-1Without a defence, a planted instruction disguised as a setup script hijacked the agent every time it sat in AGENTS.md or a ticket comment, and never anywhere else
• RI-2Every hijacked run reported a clean success, listing the malicious script as a passed check
• RI-3One paragraph in the prompt, saying repository text is information and not instructions, cut hijacks from 18 of 90 runs to 1, at no measurable cost
• SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole
• SEC-2A security skill, opened in every run, made no measurable difference
• SEC-3In an exploratory analysis after two faults in my test materials were fixed, a gate of public security tests beat the skill
• SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request
• SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed
• TH-1A vote of three identical agents was no better than one agent at seeing through a false fact
• TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted
• TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own
• TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)
• SPS-1Splitting a feature across three coders working at the same time, each given only its part's requirements, passed 61.6% of hidden acceptance tests; one agent alone passed 62.4%, at less than half the cost
• SPS-2In 4 of 72 split runs the application would not start at all, against none of 36 one-agent runs
• SPS-3Agent reviewers approved code that failed hidden acceptance tests 14 times out of 15
• SWT-1A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money
• SWT-2Giving every team member the task file raised the team from 25.7% to 54.1% of hidden tests passed, level with one agent, at about 1.6 times its cost
• SWT-3Every time an agent reviewer approved a change, the change failed hidden acceptance tests: 24 of 24 approvals
• EM-1One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence
• EM-2An injected instruction in a ticket comment changed 0 of 120 decisions and was flagged as manipulation in all 120 runs
• EM-3With both a falsified fact and the instruction, the agent flagged manipulation in all 120 runs and still reached the attacker's intended outcome in 73
• EM-4With a tool that checks authoritative records, 359 of 360 manipulated runs reached the right decision
• RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5
• RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known
• RAG-3When no retrieved paragraph held the answer, both models mostly said so: NOT FOUND in 107 (27B) and 112 (8B) of 120 questions; made-up answers 8 and 6
• RAG-4No measurable difference between the models: qwen3:8b repeated the planted number in 98 of 120 questions, against 102 for qwen3.8:27b
• PE-1A wrong number planted in the model's thinking was carried through to the matching wrong answer in 81 of 230 runs (35%); with the thinking left unchanged, 1 of 230 went wrong
• PE-2The model caught the planted slip in its own thinking in 124 of 230 runs (54%)
• PE-3In 23 of 230 runs (10%), the thinking ended on the wrong answer but the written reply gave the right one without saying why; the reverse never happened
• PE-4The thinking does real work: forced to answer with no thinking, the model got 23% of problems right; after its full thinking, 94%

Measurement

Tools and honest measurement: repeated runs with stated limits. Experiments on this topic →

• BT-1The local model scored no measurably lower on GSM8K problems with new numbers: 168 of 184 originals right against 161 of 184 twins
• BT-2The hosted model's drop on the number twins was no different from the local model's: 175 of 184 against 172 of 184
• BT-3When a model got a number twin wrong, its answer was never the original problem's answer
• BT-439 of the local model's 656 replies were lost to a harness problem, not wrong answers; regraded, it matched the hosted model on these items

What I'm measuring next

Questions the results leave open. Each comes from a follow-up already on the page it names. These are not results and not a schedule: when one is run, it becomes an experiment and leaves this list. If you think one is the wrong question, or have a better one, start a GitHub discussion.

1. When the source of truth is wrong

   EM used complete, correct authoritative records. What happens when the verification source is stale, missing, contradicts another source, or has been compromised?

   Follows EM· Verification, Safety and cost · not yet run

2. Propose, then execute

   If the agent can only propose an action and a deterministic gate decides what runs, do the stray action calls seen in EM disappear? Scored on what was executed, not the agent’s final answer.

   Follows EM· Safety and cost, Verification · not yet run

3. Subtler prompt injection

   EM tested one blunt instruction, which changed 0 of 120 decisions. Do injections that read like an ordinary ticket comment, or come from a more trusted source, do better?

   Follows EM· Safety and cost · not yet run

4. Conflicting evidence and source provenance

   When two sources disagree, especially when the wrong one looks more authoritative, what does a retrieval agent do? Does telling it a source’s age, origin or reliability change the outcome?

   Follows RAG· Context, Verification · not yet run

5. A split team with a build-and-start step

   In SPS, a split team passed 61.6% of hidden tests against 62.4% for one agent. Does that change when one agent starts the application and calls each new endpoint before review?

   Follows SPS· Parallel, Verification · not yet run

6. Teams that must write tests

   Does a coder told to add tests for the stated interface, with a reviewer who writes acceptance tests from the requirements before reading the change, recover what the split lost?

   Follows SWT· Parallel, Review · not yet run

7. A realistic way to choose between attempts

   SWT had a 68.9% upper bound if the best attempt could be picked. How close does a judge model get?

   Follows SWT· Parallel, Review · not yet run