Topics

Briefs

Briefs, plans and specifications: deciding what "done" means.

Findings

11 findings, each linking to its write-up and limits

  • GVP-1A two-sentence structured prompt recovered as many developer-only rules as the grilling skill35 of 75 (prompt) against 34 of 75 (grilling); 0 of 75 from the ticket alone
  • GVP-2Grilling did not pull ahead on nested tickets, because the first rule in each chain was rarely found3 of 30 (grilling) against 5 of 30 (prompt) and 0 of 30 (three questions)
  • GVP-3Capping the interview at three questions did well on simple tickets and fell behind with more unknowns19 of 25 on the few-unknown tickets (grilling 21), 6 of 20 on the many-unknown ticket (grilling 10), 0 of 30 on the nested tickets
  • GRL-1Grilling recovered every requirement that only the developer knew; working from the ticket recovered none5 of 5 rules in each of 3 runs (15 of 15 checks) against 0 of 15
  • GRL-2The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone15 of 15 checks (answered) against 0 of 15 ("yes" to everything)
  • GRL-3Telling the agent "ask me anything you need to know first" produced no questions at all0 of 18
  • GRL-4Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it17 of 27 against 15 of 27
  • EGM-1A Claude approver reviewing the design before the build lifted Codex on a hard feature: three of six approver runs passed all 16 hidden tests, against 4 of 16 in both runs of plain Codex at the same effortapprover runs 16, 5, 16, 6, 13, 16 vs plain 4, 4
  • EGM-2EGM did no better than a plain plan reviewed by the same kind of approver, and cost about five times as much in Codex usage12.3 vs 11.7 of 16; $0.95 vs $0.19
  • EGM-3The score followed one sentence in the approved design, naming the cart's discount line "discount" as the task asks. Every run whose design said so scored 13 or 16; the two that did not scored 5 and 6. The name always came from the approver, never from EGM's reviewers13 to 16 vs 5 and 6
  • EGM-4Without an approver's judgement, EGM's thorough designs did not help: built at low effort they produced skeletons (3 and 1 of 16), and built at medium effort they scored 4 of 16, the same as plain Codex with the task alone3, 1 (low); 4, 4 (medium) vs 4, 2 (low); 4, 4 (medium)

Experiments

3 experiments

Developer-only rules met, higher is better. Few flat unknowns, of 25: ticket only 0, grilling 21, structured prompt 21, three questions 19. Many flat unknowns, of 20: 0, 10, 9, 6. Nested unknowns, of 30: 0, 3, 5, 0.

Controlled trial · GVP

Grilling or a good prompt? Does the skill beat a two-sentence instruction?

Does Matt Pocock's grilling skill recover more of the rules only the developer knows than a short prompt that makes the agent list its open decisions and wait, or than an interview capped at three questions? Does grilling's branch-by-branch walk win where one answer opens new questions, and what does each cost the developer?

A follow-up to GRL. A two-sentence prompt asking the agent to list its open decisions with recommended answers, then wait, recovered as many of the rules only the developer knew as Matt Pocock's grilling skill (35 against 34 of 75), with about half the reading. Neither found rules the agent had no reason to suspect, and on tickets where one answer opens new questions, the first question was rarely asked. The developer was Claude following a fixed brief, not a real person.

Left-out requirements met on the Some and Lots tickets, higher is better. Only in the developer's head, of 15: grilling 15, grill-with-docs 9, ticket only 0, ask me anything 0, yes to everything 0. Findable in the repo, of 18: 17 or 18 in every arm.

Controlled trial · GRL

Grill me first? Does letting the agent interview you get you what you wanted?

When a coding agent interviews the developer before building (Matt Pocock's grilling skill), does it build more of what the developer actually wanted than working from the ticket, and is that from the skill, from being allowed to ask, or from the developer's answers? Do grill-with-docs's glossary and decision records help a later ticket?

Matt Pocock's grilling skill makes a coding agent interview the developer before it builds. On tickets that left out business rules, grilling recovered all five rules only the developer knew, in each of three runs; working from the ticket, a plain "ask me anything" and a developer who said yes to everything recovered none. Grill-with-docs's glossary and decision records made no measurable difference to the next ticket. The developer was Claude following a fixed brief, not a real person.

Hidden tests passed of 16, one bar per run: plain Codex 4 and 4; plan plus approver 6, 13, 16; EGM plus approver 16, 5, 16

Pilot · EGM

Spec first with the Elephant-Goldfish Model, and an AI approver (pilot EGM)

Does writing a spec with Google's Elephant-Goldfish Model before any code is written make a coding agent build a hard feature more correctly, and how much of any gain comes from the person who approves the spec?

On a hard feature, a Claude approver reviewing the design lifted Codex from 4 of 16 hidden tests to as many as 16, whether the design came from Google's Elephant-Goldfish spec process or from a plain plan. EGM added nothing over the plain plan, at about five times the cost. A pilot: three runs per arm.

Other topics: Measurement, Context, Verification, Safety and cost, Skills, Review, Parallel, Harness