REPOSITORYOKFExperiment ↗
~1,650
controlled runs in the research repository
A project-level total across experiments on generated knowledge, contracts and stale documentation. Not a single study’s sample size.
Independent research on coding agents
Coding agents are fast.
Are they right?
Practical experiments exploring what makes coding agents effective, where they fail, and which engineering practices justify their cost.
Evidence-led guidance from real repositories, measured trials and reproducible code.
FIELD NOTES / VOL. 01First edition · October 2026
THE EXECUTION LOOPSYSTEM / 01
THE RIGHT FACTS, NOT EVERY FACT.FIELD GUIDE TO THE AGENT–REPOSITORY LOOP
Small trials. Stated limits.
What the measurements say.
EXPERIMENT JPREPARED REPOSITORY
+35–48%
more tokens, no reliable correctness gain
One large Java repository; the guide, graph, skills and instructions were tested together. Differences were within noise.
Chapter 3, with limitsCROSS-REPOSITORYHIDDEN-TEST PASSES
3 → 18–19/20
when the missing dependency facts were supplied
Storefront changes against an unseen backend. Five runs per task; one model; identifiers renamed by the author.
Read the case studyThese are observations from particular repositories and tasks. Each finding is linked to its method and limitations; none is a universal success rate.
A result worth reading slowly · Experiment J
More repository knowledge did not make this agent more correct.
On a 1,173-file Java monolith, the prepared copy bundled a generated guide, code graph, three operating skills and configured AGENTS.md. It cost more tokens. The measured correctness differences were not reliable.
Measured finding / Chapter 3
Read the full analysis Limits One repository and model; mechanical edits and structure questions; 8 runs per task per arm; four ingredients tested together.
CORRECT ANSWERS
RUNS / TASKExperiment J · Pass counts
Plain repositoryPrepared repository
Mechanical changes32 trials
Structure questions24 trials
0%25%50%75%100%
The book, made navigable
Learn the engineering around the agent.
Eleven chapters, reorganised for the web. Move from task design through context and verification to the practices that keep work safe and measurable.
Browse all lessons01Agent foundationsGive work a clear brief, a bounded change and an observable finish.Start with the loop02Context & knowledgeSupply the facts an agent cannot discover, and keep them authoritative.Chapter 3 · The evidence03Verification & harnessTurn a plausible answer into a result a repository can actually check.Chapter 5 · Proof04Scaling & measurementReview parallel work and measure the cost of the engineering around it.Chapter 7 · Parallel work
Research log
Published evidence. Open questions.
PUBLISHEDJRepository optimisationToken cost and correctness in a readable Java monolith.FIRST EDITION PLANNEDKKnowing When You Don’t KnowCan signposting help an agent find the authoritative dependency?NO RESULTS
Experiment K remains a proposal: two tasks, five repetitions per arm, and 30 planned runs. No outcomes are available.
A permanent edition · October 2026
Programming the Loop
What I Measured with Coding Agents
The original first edition, preserved as published. Read it as a free PDF or follow the chapters online with source links and navigable evidence.
About the research
Evidence first.
Certainty earned.
These are practical trials, not universal laws. Findings describe what happened in specific repositories under specific conditions. Recommendations are engineering judgement, clearly separated from those observations.