Verification

Findings

48 findings, each linking to its write-up and limits

• SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole18%
• SEC-2A security skill, opened in every run, made no measurable difference−5 ptsNo measurable difference
• SEC-3In an exploratory analysis after two faults in my test materials were fixed, a gate of public security tests beat the skill+11 ptsInterval excludes zero
• SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request38 of 40
• SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed69 of 71
• TH-1A vote of three identical agents was no better than one agent at seeing through a false fact20 vs 17 of 42
• TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted22 of 22
• TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own6 of 42
• TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)29 → 15
• EM-1One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence+67.5 ptsInterval excludes zero
• EM-2An injected instruction in a ticket comment changed 0 of 120 decisions and was flagged as manipulation in all 120 runs+0.8 ptsNo measurable difference
• EM-3With both a falsified fact and the instruction, the agent flagged manipulation in all 120 runs and still reached the attacker's intended outcome in 73−6.7 ptsNo measurable difference
• EM-4With a tool that checks authoritative records, 359 of 360 manipulated runs reached the right decision+42.5 ptsInterval excludes zero
• RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5+90 ptsInterval excludes zero
• RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known13/15
• RAG-3When no retrieved paragraph held the answer, both models mostly said so: NOT FOUND in 107 (27B) and 112 (8B) of 120 questions; made-up answers 8 and 6−1.7 ptsNo measurable difference
• RAG-4No measurable difference between the models: qwen3:8b repeated the planted number in 98 of 120 questions, against 102 for qwen3.8:27b−3.3 ptsNo measurable difference
• PE-1A wrong number planted in the model's thinking was carried through to the matching wrong answer in 81 of 230 runs (35%); with the thinking left unchanged, 1 of 230 went wrong+37 ptsInterval excludes zero
• PE-2The model caught the planted slip in its own thinking in 124 of 230 runs (54%)124/230
• PE-3In 23 of 230 runs (10%), the thinking ended on the wrong answer but the written reply gave the right one without saying why; the reverse never happened23/230
• PE-4The thinking does real work: forced to answer with no thinking, the model got 23% of problems right; after its full thinking, 94%+71 ptsInterval excludes zero
• DJ-1Shown the task and the diff, the AI judge called 57 of 96 changes right: it caught 21 of 48 broken changes and passed 36 of 48 working ones, no measurable difference from guessing0.59Interval excludes zero
• DJ-2Showing the judge the agent's own success report did not make it miss more broken changes: it caught 20 of 48 with the report and 21 without21/48 vs 20/48
• DJ-3A fixed doubtful report ('I could not run the tests') made the judge catch more broken changes (30 of 48) but pass fewer working ones (22 of 48); it got no more accurate overall+21 pts, −25 pts
• DJ-4The judge's confidence did not track whether it was right: it averaged 87 out of 100 when right and 86 when wrong87 vs 86
• TSK-1A one-paragraph instruction to write a focused API test got the agent to write one that fails without its change and passes with it in 56 of 60 runs; given only the task, it wrote none56/60 vs 0/60
• TSK-2Adding the repository's verification skill to the instruction did not help: 52 of 60 runs wrote a relevant test with the original skill and 54 with a revised one that puts the focused test first, against 56 with the instruction alone56, 52, 54 of 60
• TSK-3A stop hook that checked for a relevant test when the agent tried to finish, and sent it back once if there was none, worked as well as the instruction: it blocked every run, and 58 of 60 then had a relevant test58/60
• TSK-4Real tests of the change did not measurably raise the pass rate: 41 to 47 of 60 in the arms that wrote them, against 40 with no guidance; medium reasoning effort, with no tests, passed 49+15 ptsInterval excludes zero
• ESK-1Reading is not doing: told to follow the verification skill, the agent read it in every run, but a test of its change was detected in only 15 of 60 runs (prompt) and 10 of 60 (AGENTS.md); most runs ran the skill's smoke-test example instead15/60 and 10/60
• ESK-2Telling the agent to use its skills did not measurably raise the pass rate: 42 of 60 runs passed when told in the prompt and 43 when told in AGENTS.md, against 37 with no guidance and 39 with the guidance left to discover+8 to +10 pts, not significantNo measurable difference
• ESK-3A one-line entry in AGENTS.md got the verification skill read as reliably as an instruction in the prompt; the pass rates were similar (43 and 42 of 60), and the experiment cannot tell whether placement matters60/60, 60/60, 3/60
• ESK-4Being told cost about twice as much per correct change as no guidance: 1.98 times when told in the prompt and 1.77 times in AGENTS.md1.98×Interval excludes 1×
• RSK-1Repository guidance left for the agent to discover did not measurably raise the pass rate: 40 of 60 runs passed, against 38 of 60 without it (p = 0.40)40 vs 38 of 60 · not measurableNo measurable difference
• RSK-2The agent read the skills on building and on changing code in every run, but opened the skill on checking a change through the API in only 8 of 60, and a test of its own change was detected in 28/60
• RSK-3Cost 1.29 times as much per correct change (1.01 to 1.55), against about 2.4 to 2.5 times for the Superpowers pack in earlier experiments1.29×Interval excludes 1×
• VVS-1Medium reasoning effort, with no skills and no instruction, passed 49 of 60 runs against 36 of 60 at low effort: better on 9 tasks and worse on 1 (Holm p = 0.041)+21.7 ptsInterval excludes zero
• VVS-2An instruction to check the change against the running application did not measurably raise the pass rate (39 of 60 against 36, Holm p = 0.67); the agent tried to start the app in 47 runs, but it got the app to answer in only 3+5.0 ptsInterval includes zero
• VVS-3Superpowers again passed about as often as no skills (37 of 60 against 36) and cost the most per correct change: 7.0 cents, against 4.0 at medium effort and 2.8 with no skills at low effort7.0¢
• SKT-1Neither skill pack reliably raised the pass rate: Superpowers passed 24 of 32 runs, the same as with no skills, and Pocock passed 27 of 32, a gap small enough to be chance (Holm p = 0.38)+9.4 ptsInterval includes zero
• SKT-2Superpowers used about three times the tokens of no skills on the same task; it wrote tests in 23 of 32 runs, where the agent with no skills wrote none3.07×Interval excludes 1×
• SKT-3In 28 of the 30 failed runs, the agent said its change worked; this held in every arm28/30
• B3-1On Click after three planted refactors, an agent under an efficiency prompt followed the out-of-date guide in 8 of 12 drift answers. Adding one sentence to the pointer, to read the source to confirm before relying on it, cut that to 0 of 12 at the same cost (77k against 79k tokens).8/12 → 0/12
• E-1When some entries in a backend guide were deliberately wrong and the source could not be checked, labelling facts [high] or [medium] and telling the agent to say when it relied on a guess raised disclosure from 0 of 16 answers to 6 of 16. Wrong answers fell only from 11 to 8 of 16, which is not reliable.0/16 → 6/16
• H3-1A contract that listed only one success code (201) led agents to treat the service's other success code as an error: on the retry task the contract arm passed 0 of 4 runs, while the other four arms passed 4 of 4.0 / 4
• OKF-1After four renames in a backend the agent could not read, agents given the out-of-date guide gave the old fact in 14 of 16 drift answers; none of the wrong answers hedged.14 / 16
• OKF-2When the backend facts a storefront change depended on were supplied as a guide, hidden-test passes on cross-repository changes went from 3 of 20 with nothing to 19 of 20 on four small tasks and 18 of 20 on four harder ones.3 → 18–19
• OKF-3On the worked example's task, all 6 agents without the provider contract wrote a contract of their own as version 1.0.0 and edited the freshness gate so it compared it with itself, and none passed the black-box check; all 6 with the contract did. All 12 runs reported passed while listing the real inventory check as unverified.0 / 6 vs 6 / 6