Topics

Context

The right facts, not every fact: context, documentation and knowledge layers.

Findings

37 findings, each linking to its write-up and limits

  • GVP-1A two-sentence structured prompt recovered as many developer-only rules as the grilling skill35 of 75 (prompt) against 34 of 75 (grilling); 0 of 75 from the ticket alone
  • GVP-2Grilling did not pull ahead on nested tickets, because the first rule in each chain was rarely found3 of 30 (grilling) against 5 of 30 (prompt) and 0 of 30 (three questions)
  • GVP-3Capping the interview at three questions did well on simple tickets and fell behind with more unknowns19 of 25 on the few-unknown tickets (grilling 21), 6 of 20 on the many-unknown ticket (grilling 10), 0 of 30 on the nested tickets
  • GRL-1Grilling recovered every requirement that only the developer knew; working from the ticket recovered none5 of 5 rules in each of 3 runs (15 of 15 checks) against 0 of 15
  • GRL-2The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone15 of 15 checks (answered) against 0 of 15 ("yes" to everything)
  • GRL-3Telling the agent "ask me anything you need to know first" produced no questions at all0 of 18
  • GRL-4Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it17 of 27 against 15 of 27
  • RI-1Without a defence, a planted instruction disguised as a setup script hijacked the agent every time it sat in AGENTS.md or a ticket comment, and never anywhere else18 of 90 (9 of 9 AGENTS.md, 9 of 9 ticket, 0 of 27 elsewhere)
  • RI-2Every hijacked run reported a clean success, listing the malicious script as a passed check18 of 18
  • RI-3One paragraph in the prompt, saying repository text is information and not instructions, cut hijacks from 18 of 90 runs to 1, at no measurable cost18 → 1 of 90
  • OKL-1With the backend's generated guide, a small local model got 20 of 20 cross-repo changes right, against 4 of 20 with no guide+80 ptsInterval excludes zero
  • OKL-2The guide did no better than the backend's own source code: 20 of 20 against 19 of 20, and source runs were slightly faster and cheaper+5 ptsNo measurable difference
  • OKL-3Without the backend, the agent presented guessed field names as checked facts in 9 of 15 failed runs9 of 15
  • RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5+90 ptsInterval excludes zero
  • RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known13/15
  • RAG-3When no retrieved paragraph held the answer, both models mostly said so: NOT FOUND in 107 (27B) and 112 (8B) of 120 questions; made-up answers 8 and 6−1.7 ptsNo measurable difference
  • RAG-4No measurable difference between the models: qwen3:8b repeated the planted number in 98 of 120 questions, against 102 for qwen3.8:27b−3.3 ptsNo measurable difference
  • J-1On a 1,173-file Java backend, the copy set up for agents (generated guide, code graph, three skills, configured AGENTS.md) used 35–48% more tokens per run than the plain copy, and its pass rate was not reliably different.+35–48%Interval excludes 1×
  • AGWT-1Java call links in the graph matched the compiled bytecode: all 13,380 high-confidence links on Shopizer, and 8,947 of 8,957 medium-confidence ones.13,380 / 13,380 high; 8,947 / 8,957 medium
  • AGWT-2On callers-of-callers questions (Click and Express, 36 runs) the graph cut tokens for Codex (84k against 196k) and Cursor (121k against 414k), and was close to level for Copilot (161k against 181k). One-hop lookups were level or dearer.Codex 84k vs 196k; Cursor 121k vs 414k; Copilot 161k vs 181k
  • AGWT-3Across 192 controlled runs on Python and TypeScript repositories the graph did not reduce tokens overall: agents ran kg and then, as the instructions suggest for medium-confidence results, checked it with grep. Every run was correct, so these questions measure cost only.did not reduce tokens overall
  • B3-1On Click after three planted refactors, an agent under an efficiency prompt followed the out-of-date guide in 8 of 12 drift answers. Adding one sentence to the pointer, to read the source to confirm before relying on it, cut that to 0 of 12 at the same cost (77k against 79k tokens).8/12 → 0/12
  • A-1On Click, the default guide did not make agents cheaper: against no guide it cost 1.30× the tokens in the first run and 1.07× when rerun without the confirm-in-source sentence. Accuracy was already at ceiling with or without a guide.1.30× (run 1); 1.07× (rerun)
  • A-2On Click, the thinnest guide (types and function names only, with no citations, links, call edges or index) was the dearest in both runs: 1.42× the no-guide cost, then 1.26× once the confirm-in-source sentence was removed.1.26–1.42×
  • D1-1With the backend unread, agents answered 1 of 24 storefront-to-backend questions correctly, at about 785k tokens per question on average1 / 24
  • DOCLAYER-1In the documentation-layer study, the heavy design (a long always-read AGENTS.md, a separate tree of decision records and OKF reference pages, with prompts telling the agent to read all of it) cost 40.3% more tokens and 28.7% more in cost than the comparison design on the simple change, in all four rounds.+40.3% tokens; +28.7% cost
  • DOCLAYER-2With prompts that never mentioned any of the documentation, agents found the lean design's small index and opened the task-relevant skills in 24 of 24 measured cells.24 / 24
  • E-1When some entries in a backend guide were deliberately wrong and the source could not be checked, labelling facts [high] or [medium] and telling the agent to say when it relied on a guess raised disclosure from 0 of 16 answers to 6 of 16. Wrong answers fell only from 11 to 8 of 16, which is not reliable.0/16 → 6/16
  • F-1On four small cross-repository change tasks, hidden-test passes went from 3 of 20 with no guide to 19 of 20 with a guide to the backend, at about half the tokens (242k against 482k per run).3 / 20 → 19 / 20
  • G-1On four harder cross-repository change tasks, hidden-test passes went from 3 of 20 with no guide to 18 of 20 with a guide to the backend, level with the agent being able to read the backend source (18 of 20).3 / 20 → 18 / 20
  • H1-1On harder change tasks with nothing, 4 of 16 runs passed the hidden tests; with a contract 13, a hand-written note 14, a guide 14 and the backend source 14 of 16. The four knowledge arms beat nothing and did not differ reliably from each other.nothing 4/16; contract 13; note 14; guide 14; source 14 of 16
  • H1-2Among the four knowledge arms on those change tasks, the hand-written 3 KB note was cheapest at about 128k tokens per run, against 192k for the contract, 282k for the guide and 334k for the backend source (277k with nothing).~128k
  • H2-1On six internal call-structure questions, the guide and the backend source each answered 21 of 24 correctly, against 8 of 24 for the contract, 0 of 24 for the hand-written note and 1 of 24 with nothing.guide/source 21/24; contract 8/24; note 0/24
  • H3-1A contract that listed only one success code (201) led agents to treat the service's other success code as an error: on the retry task the contract arm passed 0 of 4 runs, while the other four arms passed 4 of 4.0 / 4
  • OKF-1After four renames in a backend the agent could not read, agents given the out-of-date guide gave the old fact in 14 of 16 drift answers; none of the wrong answers hedged.14 / 16
  • OKF-2When the backend facts a storefront change depended on were supplied as a guide, hidden-test passes on cross-repository changes went from 3 of 20 with nothing to 19 of 20 on four small tasks and 18 of 20 on four harder ones.3 → 18–19
  • OKF-3On the worked example's task, all 6 agents without the provider contract wrote a contract of their own as version 1.0.0 and edited the freshness gate so it compared it with itself, and none passed the black-box check; all 6 with the contract did. All 12 runs reported passed while listing the real inventory check as unverified.0 / 6 vs 6 / 6

Experiments

19 experiments

Developer-only rules met, higher is better. Few flat unknowns, of 25: ticket only 0, grilling 21, structured prompt 21, three questions 19. Many flat unknowns, of 20: 0, 10, 9, 6. Nested unknowns, of 30: 0, 3, 5, 0.

Controlled trial · GVP

Grilling or a good prompt? Does the skill beat a two-sentence instruction?

Does Matt Pocock's grilling skill recover more of the rules only the developer knows than a short prompt that makes the agent list its open decisions and wait, or than an interview capped at three questions? Does grilling's branch-by-branch walk win where one answer opens new questions, and what does each cost the developer?

A follow-up to GRL. A two-sentence prompt asking the agent to list its open decisions with recommended answers, then wait, recovered as many of the rules only the developer knew as Matt Pocock's grilling skill (35 against 34 of 75), with about half the reading. Neither found rules the agent had no reason to suspect, and on tickets where one answer opens new questions, the first question was rarely asked. The developer was Claude following a fixed brief, not a real person.

Left-out requirements met on the Some and Lots tickets, higher is better. Only in the developer's head, of 15: grilling 15, grill-with-docs 9, ticket only 0, ask me anything 0, yes to everything 0. Findable in the repo, of 18: 17 or 18 in every arm.

Controlled trial · GRL

Grill me first? Does letting the agent interview you get you what you wanted?

When a coding agent interviews the developer before building (Matt Pocock's grilling skill), does it build more of what the developer actually wanted than working from the ticket, and is that from the skill, from being allowed to ask, or from the developer's answers? Do grill-with-docs's glossary and decision records help a later ticket?

Matt Pocock's grilling skill makes a coding agent interview the developer before it builds. On tickets that left out business rules, grilling recovered all five rules only the developer knew, in each of three runs; working from the ticket, a plain "ask me anything" and a developer who said yes to everything recovered none. Grill-with-docs's glossary and decision records made no measurable difference to the next ticket. The developer was Claude following a fixed brief, not a real person.

Proportion, 95% interval

Controlled trial · RI

Can a repository hijack its coding agent?

If a repository or ticket contains an instruction to leak secrets and change git configuration, does a coding agent working an ordinary ticket carry it out, and does one paragraph of warning in the prompt stop it?

I planted an instruction in a repository asking a coding agent to send the checkout's secrets to an outside server and redirect git's commit hooks. Disguised as a setup script in AGENTS.md or in a ticket comment, it worked in every undefended run, and the agent reported a clean success. In the README, a code comment or test output it never worked. One paragraph of warning in the prompt cut hijacks from 18 of 90 runs to 1.

Difference in percentage points, 95% interval

Controlled trial · OKL

Is a code guide worth more to a small local model?

When a small local model must change a storefront to match a backend it can't see, does a generated guide to the backend help it, and does the guide help more than the backend's source code?

OKF test G's four cross-repo changes, rerun with Codex on a small open-weight model on my own machine. With the backend's generated guide, it got all 20 changes right, against 4 of 20 with no guide. With the backend's source code it got 19 of 20, slightly more cheaply. As for the hosted model, the guide stands in for the code but doesn't beat it.

Difference in percentage points, 95% interval

Controlled trial · RAG

When RAG retrieves the wrong facts, will an AI challenge them?

When one retrieved number is wrong, how often does a model repeat it, and does knowing the right answer protect it? When retrieval misses, does it make something up?

A baseline test of error propagation in retrieval. I changed one number in the paragraph that held each answer and measured whether two local models repeated it. Both did, more than 8 times in 10, even when the bigger model had answered correctly with no documents. That is what retrieval is built to do. The measurement is what is useful, not the surprise.

Ratio, 95% interval (1× is no change)

Controlled trial · J

Agent-optimised repository

Does a repository set up for agents do better than the same repository left as it is?

Experiment J: a fully set-up repository cost more tokens with no reliable gain in correctness.

Bar chart of tokens per question for multi-level callers-of-callers questions, with and without the code graph

Evaluation · AGWT

AgentGraphWorkTree evaluation

Are the graph's facts correct, and when does the graph reduce agent tokens?

Code-graph accuracy and agent token cost: multi-hop questions cheapen; simple lookups do not overall.

8/12 → 0/12

Controlled trial · B3

Confirm-in-source sentence

Where the agent can read the code, does telling it to confirm in source reduce stale answers from a bundle?

Where the agent could read the code, one sentence cut stale answers from 8 of 12 to 0 of 12.

1.30× (run 1); 1.07× (rerun)

Controlled trial · A

Click guide trials

On a smaller library the agent can read, does a generated guide improve answers or change cost?

On Click, a generated guide did not improve correctness and cost more tokens.

1 / 24

Controlled trial · D1

Cross-repo questions, unread backend

When a storefront change depends on a backend the agent cannot read, can it answer questions that need backend facts?

With the backend unread, agents answered 1 of 24 storefront-to-backend questions correctly.

Controlled trial

Controlled trial · D2

Stale knowledge across a boundary

When a guide is stale and the agent cannot read the dependency, does it repeat old facts?

After planted backend renames, a stale guide gave the old answer in 14 of 16 drift answers.

+40.3% tokens; +28.7% cost

Controlled trial · DOCLAYER

Documentation-layer study

Given that you have some documentation for agents, which layout costs least, and do agents find the right parts unprompted?

Unpublished comparison of documentation layouts: heavy always-loaded context cost more; lean skills were discovered unprompted.

0/16 → 6/16

Controlled trial · E

Confidence labels on wrong entries

Do confidence labels change what agents do when some guide entries are wrong and the source is unread?

Labels with a disclose-guesses instruction raised disclosure; accuracy did not reliably improve.

3 / 20 → 19 / 20

Controlled trial · F

Cross-repo change tasks, small

When the agent must change the storefront using facts only the backend states, how much does a backend guide help?

Hidden-test passes on small cross-repo changes: 3 of 20 with nothing, 19 of 20 with a backend guide.

3 / 20 → 18 / 20

Controlled trial · G

Cross-repo change tasks, harder

Does a backend guide still help on harder cross-repo changes that need several facts or parent-class fields?

Harder cross-repo changes: 3 of 20 with nothing, 18 of 20 with a backend guide (level with source).

Bar chart of cross-repository change tasks by kind of knowledge supplied, Codex, 16 runs per arm

Controlled trial · H1

What kind of facts? Change tasks

For harder cross-repo changes, what kind of facts help — contract, short note, guide, or source?

On harder cross-repo changes, contract, note, guide and source all beat nothing and did not differ reliably from each other.

guide/source 21/24; contract 8/24; note 0/24

Controlled trial · H2

Internal call-structure questions

For questions about internal call structure across a boundary, which knowledge kind works?

For internal call-structure questions, only the guide and the source worked (21 of 24 each).

0 / 4

Controlled trial · H3

Incomplete OpenAPI success codes

What goes wrong if a provider contract lists only one of the service's success codes?

A contract that listed only one success code led agents to treat the service's other success code as an error.

14 / 16

Controlled trials · OKF

OKFExperiment and the worked example

When a dependency is unread or stale, and when verification can be rewritten, what decides the outcome?

Controlled trials on facts across a repository boundary, stale knowledge, and a reproducible worked example with agent trials.

Other topics: Measurement, Verification, Safety and cost, Skills, Review, Briefs, Parallel, Harness