Documentation-layer study

The question

Given that you have some documentation for agents, which layout costs least, and do agents find the right parts unprompted?

Experiment

DOCLAYER

Key findings

• DOCLAYER-1

  In the documentation-layer study, the heavy design (a long always-read AGENTS.md, a separate tree of decision records and OKF reference pages, with prompts telling the agent to read all of it) cost 40.3% more tokens and 28.7% more in cost than the comparison design on the simple change, in all four rounds.

  • Six documentation layouts on copies of one FastAPI and React application. Four independent rounds per design and task. Cost is taken from the provider transcript.
  • The lean design, which the study recommends: a 19-line AGENTS.md index, task-scoped skills opened only when needed, and OKF reference pages with the reasons folded in.
  • Over eight rounds, the lean design was cheaper than the baseline in 7 of 8 rounds on both change tasks (two-sided binomial p = 0.07): suggestive, not confirmed.
  • Recreated on five other open-source repositories, one run per cell: comprehension was cheaper in four of five, and code-change cost moved both ways, from +86.4% to −64.3% tokens in individual cells.
  • The heavy design was also run with prescriptive prompts, so prompt and layout are confounded.

  Limitsauthor's unpublished document; numbers can't be independently checked · heavy design also run with prescriptive prompts (prompt and layout confounded) · lower-grade evidence than published OKF / J trials

  Source ↗

  +40.3% tokens; +28.7% cost

  Measuretoken delta vs comparison arm on simple change

  Samplefour independent rounds per design and task

• DOCLAYER-2

  With prompts that never mentioned any of the documentation, agents found the lean design's small index and opened the task-relevant skills in 24 of 24 measured cells.

  • The lean design is a 19-line AGENTS.md that is only an index, plus one task-scoped skill per task family (model-field-change, write-migration, add-api-endpoint, regenerate-frontend-client, add-ui-action, verify-work).
  • On the five other repositories, three of five generated documentation layers contained a task-relevant factual error, and one layer's omissions caused two first-attempt verification failures that the plain baseline avoided.

  Limitsunpublished document; numbers can't be independently checked · lower-grade evidence

  Source ↗

  24 / 24

  Measurecells where lean skills were opened unprompted

Question

Given that you have some documentation for agents, which layout costs least, and do agents find the right parts unprompted?

Prediction

None published. The study document itself is unpublished.

Set-up

Six documentation layouts on copies of one FastAPI and React application; four independent rounds per design and task; cost from the provider transcript. External check on five other open-source repositories (one run per cell). Guide Chapter 3 case study. Trial set-up published as AgentTrialRig examples/fastapi-doc-layers; figures from the unpublished study document.

Results

Findings DOCLAYER-1 and DOCLAYER-2 are above. Guide also: lean cheaper than baseline in 7 of 8 rounds on both change tasks (two-sided binomial p = 0.07) — suggestive, not confirmed; external repos: comprehension cheaper in four of five, code-change cost both ways, three of five generated layers contained a task-relevant factual error.

The trial set-up

Does an agent-facing documentation layer change what a coding agent spends on real change tasks? This trial is ported from the original Codex task matrix: one FastAPI, SQLModel and PostgreSQL application (the upstream full-stack-fastapi-template) cloned with byte-identical code and different documentation layers.

Arm	Repository	Layer
baseline	repo-baseline	none
skill	repo-agent-skill	AGENTS.md index plus six task-scoped skills
recommended	repo-agent-recommended	the same skills, with the load-bearing facts folded into the knowledge docs

Two tasks: simple_change (add an order_by parameter to the item listing) and complex (an archive and soft-delete feature across backend, migration and frontend). The prompts are the original baseline prompts; the self-reported metrics block is dropped because trialrig measures tokens and tool calls itself.

What a pilot showed

Three runs per arm of simple_change with Codex, baseline against the recommended layer: 2 of 3 verified in each arm, and a relative cost of 1.12 (interval 0.78 to 1.47) for the recommended layer, which is no clear difference. After a single run each it had looked like baseline failing and recommended passing. Three runs per arm is far too few to rank the layers; the report estimates about 39 runs per arm to detect a 20% difference at that spread.

The complex task

Three runs per arm of complex (the archive feature across backend, migration and frontend) with Codex gpt-5.6-luna, baseline against the recommended layer: all six verified, and the mean cost was the same, about 5.1 million tokens per run in both arms (relative cost 1.00, interval 0.83 to 1.09). The runs were long (3.4 to 6.6 million tokens each, mostly cached input). At that spread about 9 runs per arm would be needed to detect a 20% difference. On this task and agent the documentation layer made no measurable difference to cost or correctness.

Source: AgentTrialRig examples/fastapi-doc-layers (pinned to commit e54514a).

Limits

Guide’s own framing: lower-grade evidence; you can’t check its numbers against a public results file; heavy arm confounded with prompts; external cells are single runs; where this study and Experiment J pull in different directions, the book follows J.

Reproduce

Trial set-up: AgentTrialRig examples/fastapi-doc-layers. The study document is unpublished, so the figures cannot be re-derived from a public table.

Evidence and reproducibility

Every link is pinned to the commit the figures on this page were read from.

• Full trial report ↗
• Source repository ↗

Related experiments

• Is a code guide worth more to a small local model?When a small local model must change a storefront to match a backend it can't see, does a generated guide to the backend help it, and does the guide help more than the backend's source code?
• Agent-optimised repositoryDoes a repository set up for agents do better than the same repository left as it is?
• AgentGraphWorkTree evaluationAre the graph's facts correct, and when does the graph reduce agent tokens?

Question, correction, or a result that differs from mine? Join the discussion ↗

← All experiments