Controlled trials and worked examples on coding agents, with limits stated.https://programmingtheloop.com/en-GBhttps://programmingtheloop.com/experiments/grill-vs-prompt/https://programmingtheloop.com/experiments/grill-vs-prompt/Codex with gpt-5.6-luna, seven tickets on a small banking API, four ways to start: ticket only, the grilling skill, a two-sentence structured prompt, and "at most three questions". The prompt matched the skill (35 against 34 of 75 developer-only rule checks; non-inferiority supported, narrowly) with about 300 words to read instead of about 650. Grilling did not pull ahead on nested tickets, because the parent rules came up in only 4 of 30 nested interviews. Three questions did well on simple tickets and fell behind with more unknowns. The developer's answers came from Claude following a fixed brief, not a real person.Sun, 01 Nov 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/grilling/https://programmingtheloop.com/experiments/grilling/Codex with gpt-5.6-luna, six tickets on a small banking API, five ways to start. Grilling recovered all five rules that only the developer knew in every run (15 of 15 rule checks: 5 rules × 3 runs); ticket only, "ask me anything first" (which never asked a question) and grilling with "yes" to everything met 0. The gain came entirely from the developer correcting one or two wrong recommendations per ticket. Grill-with-docs met 9 of the 15 checks, because a third of its runs built without waiting for answers, and its docs only helped a later ticket when the next agent happened to open them. The developer's answers came from Claude following a fixed brief, not a real person.Sat, 31 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/repo-injection/https://programmingtheloop.com/experiments/repo-injection/Codex with gpt-5.6-luna, working ordinary tickets, ran a planted "setup script" that sent .env to an outside host and set core.hooksPath in 18 of 90 undefended runs: 9 of 9 each when the text was in AGENTS.md or a ticket comment, 0 elsewhere, and 0 of 45 when the request was overt. All 18 listed the script as a passed check. A defence paragraph in the prompt cut hijacks to 1 of 90 at no measurable cost.Fri, 30 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/sec-authz/https://programmingtheloop.com/experiments/sec-authz/From ordinary tickets, Luna shipped working but insecure endpoints in 18% of baseline runs. All 9 genuine access breaches came from one task, closure approval, where the requester could approve their own request. A security skill, read every time, didn't help. A gate of public security tests beat the skill by 11 points in an exploratory analysis after two test faults were fixed, but it did not reduce genuine access breaches: 38 of 40 closure runs let an approver approve their own request.Thu, 29 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/team-herding/https://programmingtheloop.com/experiments/team-herding/With tampered security evidence, three Luna agents voting got 20 of 42 cases right, against 17 for one agent. After peer review all three escalated all 22 harmless cases with conflicting records, and Sol, reviewing the team, overruled it only 6 times in 42. Letting agents answer "can't tell, investigate" halved the peer-reviewed team's harmful errors (29 to 15 of 42, exploratory), by sending 19 of 42 cases to a human.Thu, 29 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/okf-local/https://programmingtheloop.com/experiments/okf-local/With the backend's guide, a small local model (qwen3.8:27b) passed 20 of 20 cross-repo changes, against 4 of 20 with no guide. With the backend's source it passed 19 of 20, and runs were slightly faster and cheaper. The guide is a stand-in for the code, for a small model as for a large one.Wed, 28 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/split-spec/https://programmingtheloop.com/experiments/split-spec/On six features, three Codex coders working in parallel on a planner's split passed 61.6% of hidden acceptance tests when each saw only its part, and 52.9% when each saw the whole task; one agent alone passed 62.4%, for less than half the cost. In 4 of 72 split runs the parts didn't fit and the application wouldn't start.Mon, 26 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/swarm-tax/https://programmingtheloop.com/experiments/swarm-tax/On six features, one Codex agent passed 56.8% of hidden acceptance tests and a planner–coder–reviewer team 25.7%. A bigger team reached 48.1% at 3.7 times the cost. Giving every team member the task file closed most of the gap (54.1%), but the team never beat one agent. Every reviewer approval went to code that failed the tests.Mon, 26 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/evidence-manipulation/https://programmingtheloop.com/experiments/evidence-manipulation/With one fact in the evidence falsified, qwen3.8:27b reached the wrong security decision in 81 of 120 runs, usually with low stated uncertainty. A blunt injected instruction changed 0 of 120 decisions and was flagged every time. With both, it flagged the instruction and still accepted the false fact in 73 of 120. With a verification tool, 359 of 360 manipulated runs were right.Sun, 25 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/poisoned-retrieval/https://programmingtheloop.com/experiments/poisoned-retrieval/A baseline for how far a wrong retrieved number travels. With the right paragraph, qwen3.8:27b answered 94% of questions right; with one number in it changed, it repeated the planted number 85% of the time (qwen3:8b: 82%) and rarely flagged it. When retrieval missed, both mostly said NOT FOUND. Not a new attack: the wrong value was placed in the retrieved paragraph, and the prompt held no instructions.Sat, 24 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/planted-errors/https://programmingtheloop.com/experiments/planted-errors/Planting a wrong intermediate number in qwen3.8:27b's thinking changed its final answer 35% of the time, nearly always with no sign of doubt. It caught 54% of the slips in its thinking. In 10% of runs the thinking ended on the wrong answer and the written reply silently fixed it, so the thinking was not a faithful record of how the answer was reached.Sat, 24 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/benchmark-twins/https://programmingtheloop.com/experiments/benchmark-twins/184 GSM8K maths problems and 80 HumanEval coding problems, each with a 'twin' that changes the numbers, names or function name but not the logic. qwen3.8:27b running locally and gpt-5.6-luna hosted scored much the same on originals and twins: no measurable drop, and no sign of recalled answers.Sat, 24 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/diff-judge/https://programmingtheloop.com/experiments/diff-judge/96 changes that coding agents made to a Java online-shop backend, half of which failed hidden tests, were shown to an AI judge (Codex gpt-5.6-luna). It caught 21 of the 48 broken changes and passed 36 of the 48 working ones: no measurable difference from guessing. Adding the agent's own report changed little; adding a doubtful one made it reject more working changes as well as broken ones.Sat, 24 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/egm-pilot/https://programmingtheloop.com/experiments/egm-pilot/On one hard feature in a Java online-shop backend, plain Codex at medium effort passed 4 of 16 hidden tests twice. With a fresh Claude agent approving the design before the build, EGM runs passed 16, 5 and 16, and plain plan-first runs 6, 13 and 16. EGM cost $0.95 of Codex per run against $0.19. What decided the score was one sentence the approver put in the design.Fri, 23 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/test-skill/https://programmingtheloop.com/experiments/test-skill/On 20 tasks in a Java online-shop backend, an instruction to write a focused API test got Codex to write one that fails without its change and passes with it in 56 of 60 runs (none without it); adding the verification skill gave 52 and 54, and a stop hook that sent the agent back once gave 58. The pass rate stayed between 40 and 47 of 60; only medium reasoning effort, at 49, was clearly higher.Fri, 23 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/explicit-skills/https://programmingtheloop.com/experiments/explicit-skills/On 20 tasks in a Java online-shop backend, telling Codex to use its repository skills, in the prompt or in AGENTS.md, made it read the verification skill in every run (against 3 of 60 when left to find it), but a test of its change was detected in only 10 to 15 runs; the pass rate rose from 37 to 42 or 43 of 60, too little to rule out chance.Thu, 22 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/repo-skills/https://programmingtheloop.com/experiments/repo-skills/On 20 tasks in a Java online-shop backend, an agent-written AGENTS.md and three repository skills raised Codex's pass rate from 38 to 40 of 60 runs, too little to rule out chance; the agent opened the verification skill in only 8 runs.Thu, 22 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/verify-vs-skills/https://programmingtheloop.com/experiments/verify-vs-skills/On 20 tasks in a Java online-shop backend, Codex at medium reasoning effort passed 49 of 60 runs against 36 of 60 at low effort; an instruction to check against the running app (39) and the Superpowers pack (37) made no measurable difference.Thu, 22 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/skills-tasks/https://programmingtheloop.com/experiments/skills-tasks/On 16 bug fixes and features in a Java online-shop backend, neither Superpowers nor Matt Pocock's skills reliably raised the pass rate; Superpowers used 3.07× the tokens.Wed, 21 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/agent-optimised-repository/https://programmingtheloop.com/experiments/agent-optimised-repository/On a readable Java backend, the recommended agent set-up used 35–48% more tokens with no reliable correctness gain.Sat, 17 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/agwt-evaluation/https://programmingtheloop.com/experiments/agwt-evaluation/Graph facts are mostly correct when sampled; multi-hop callers of callers cut tokens; simple lookups were level or dearer overall.Sun, 11 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/b3-confirm-sentence/https://programmingtheloop.com/experiments/b3-confirm-sentence/Where the agent could read the code, “read the source to confirm…” cut stale answers from 8 of 12 to 0 of 12 at the same cost.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/click-guide-trials/https://programmingtheloop.com/experiments/click-guide-trials/On Click, agents were already near ceiling with or without a guide; the default guide cost 1.30× then 1.07× tokens, and the thinnest guide was dearest (1.26 to 1.42×).Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/d1-cross-repo-questions/https://programmingtheloop.com/experiments/d1-cross-repo-questions/With nothing, agents answered 1 of 24 cross-repo questions correctly at about 785k tokens per question; a guide or source reached 24 of 24.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/d2-stale-knowledge/https://programmingtheloop.com/experiments/d2-stale-knowledge/Headline finding lives once as OKF-1 on the published okf-worked-example record. This page is the D2 trial write-up.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/documentation-layer-study/https://programmingtheloop.com/experiments/documentation-layer-study/Lower-grade unpublished study: heavy layout +40.3% tokens on a simple change; lean skills opened in 24 of 24 cells without prompts.Sun, 11 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/e-confidence-labels/https://programmingtheloop.com/experiments/e-confidence-labels/With an instruction to disclose guesses, agents said they relied on a guess in 6 of 16 answers instead of 0 of 16; accuracy did not reliably improve.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/f-crossrepo-change-small/https://programmingtheloop.com/experiments/f-crossrepo-change-small/Small cross-repo changes: 3 of 20 with nothing, 19 of 20 with a backend guide. Foreword composite OKF-2 also covers G.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/g-crossrepo-change-hard/https://programmingtheloop.com/experiments/g-crossrepo-change-hard/Harder cross-repo changes: 3 of 20 with nothing, 18 of 20 with a guide, level with readable source. Foreword composite OKF-2 also covers F.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/h1-contract-vs-okf-changes/https://programmingtheloop.com/experiments/h1-contract-vs-okf-changes/Harder changes, five arms: nothing 4/16; contract 13; note 14; guide 14; source 14 of 16. The note was cheapest (~128k tokens).Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/h2-contract-vs-okf-questions/https://programmingtheloop.com/experiments/h2-contract-vs-okf-questions/Internal call-structure questions: guide and source 21 of 24 each; contract 8; note 0 of 24.Mon, 12 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/h3-incomplete-openapi/https://programmingtheloop.com/experiments/h3-incomplete-openapi/Guide recommendation note: an incomplete success-code list in a contract led agents to treat another success code as an error.Tue, 13 Oct 2026 00:00:00 GMThttps://programmingtheloop.com/experiments/okf-worked-example/https://programmingtheloop.com/experiments/okf-worked-example/Where the agent can't read a dependency, the right facts decide the outcome. Stale facts get repeated. Agents without a contract defeated the freshness gate.Sat, 17 Oct 2026 00:00:00 GMT