Grill me first? Does letting the agent interview you get you what you wanted?

The question

When a coding agent interviews the developer before building (Matt Pocock's grilling skill), does it build more of what the developer actually wanted than working from the ticket, and is that from the skill, from being allowed to ask, or from the developer's answers? Do grill-with-docs's glossary and decision records help a later ticket?

Experiment

GRL

Runs

114

Agents

Codex CLI 0.157 (gpt-5.6-luna (hosted))

Key findings

• GRL-1

  Grilling recovered every requirement that only the developer knew; working from the ticket recovered none

  • on the Lots tickets, grilling met 1.67 more left-out requirements per run than ticket only (pre-registered H1, Holm p = 0.017)
  • requirements findable in the repo were met in 17 or 18 of 18 runs in every arm; the whole difference is in the ones only the developer knew
  • on clear tickets every arm passed; grilling added only cost (about 500 words to read, 2–3 times the wall time)

  Limitsthe developer was Claude following a fixed brief, not a real person; one agent, one small repository, 3 runs per cell · only five distinct rules; no interval is given, because counting repeated runs of the same rule as independent would make it look more precise than it is · compared with a one-line "ask me anything", not with a structured prompt that requires blocking questions; this shows the value of compulsory structured clarification, not yet the extra value of this particular skill

  Source ↗

  5 of 5 rules in each of 3 runs (15 of 15 checks) against 0 of 15

  Measurerequirements only the developer knew, met on the Some and Lots tickets, counted only when the ticket's own tests passed (higher is better)

  Sample5 distinct rules, each checked in 3 runs per arm; the 15 checks are not 15 independent rules

• GRL-2

  The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone

  • grilling against yes-to-everything: +1.25 per run, Holm p = 0.0004 (pre-registered H3a)
  • yes-to-everything against ticket only: identical observed results, 0 against 0 (H3b, p = 1.0); that is not proof the two are equivalent in general
  • yes-to-everything asked as many questions (8–9 per run); the developer corrected 1–2 wrong recommendations per run on the Some and Lots tickets

  Limitsa real developer may sometimes catch a wrong recommendation even when skimming

  Source ↗

  15 of 15 checks (answered) against 0 of 15 ("yes" to everything)

  Measurerequirements only the developer knew, met (higher is better)

  Sample5 distinct rules, each checked in 3 runs per arm

• GRL-3

  Telling the agent "ask me anything you need to know first" produced no questions at all

  • every run said it needed no clarification and built straight away; it scored the same as ticket only

  Limitsone wording of the invitation; one model

  Source ↗

  0 of 18

  Measureruns in which the agent asked the developer anything

  Sample18

• GRL-4

  Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it

  • pre-registered H5 not supported: +0.17 per run, Holm p = 0.81
  • the carried-over rule was met in 2 of 12 docs runs, both on the ticket whose ADR file name matched the follow-up, and 0 of 12 without docs
  • grill-with-docs built without waiting for the developer's answers in 6 of 18 runs; one such run wrote an ADR stating the opposite of the developer's rule

  Limitsfollow-ups started from the plain app, without the first ticket's code · 12 docs sets and 24 runs, with docs that were unevenly written and unevenly read: this doesn't show that docs are ineffective

  Source ↗

  17 of 27 against 15 of 27

  Measureleft-out requirements met on the follow-up ticket, docs against no docs (higher is better)

  Sample24

Question

Every ticket leaves things out. The person who wrote it knows the business rule nobody wrote down, the decision from last month’s meeting, the edge case that bit them before. An agent working from the ticket can only guess.

Matt Pocock’s grilling skill turns that around: before it writes any code, the agent interviews you. Its sibling, grill-with-docs, does the same and also writes what was settled into the repository as a glossary and short decision records (ADRs). It’s an attractive idea, and it costs the developer time. I wanted to know what the interview actually gets you, where that comes from, and when it’s worth it.

The short answer: grilling recovered every rule that only the developer knew, but only because the developer corrected the agent’s wrong guesses. A developer who just says “yes” gets nothing from it, and a plain “ask me anything” got no questions at all. The developer here was Claude following a fixed brief, not a real person.

How grilling works

The skill tells the agent to work through the decisions a ticket implies one branch at a time, ask everything it can ask now as numbered questions, give its own recommended answer to each (so the developer can just say “yes”), look facts up in the code itself rather than asking, and not start building until both sides agree. In practice that meant two or three rounds and eight to eleven questions per ticket in all, and most of the replies were one word.

Prediction

Before any run I published the design and five hypotheses (README.md):

• H1. On the tickets that leave out the most, grilling meets more left-out requirements than the ticket alone.
• H2. Grilling against a plain “ask me anything first”: no direction predicted.
• H3. On requirements only the developer knows, grilling beats grilling with “yes” to everything, and “yes” to everything is no better than the ticket alone.
• H4. On clear tickets, grilling is no better and costs more.
• H5. Grill-with-docs’s docs help a later ticket done with nobody to ask.

Set-up

The agent: Codex CLI 0.157 with hosted gpt-5.6-luna at medium effort, in a bubblewrap sandbox. Each run was one Codex session, with every developer reply sent back into it.

The work: six new tickets on SEC-1’s Spring Boot accounts API. Two leave out nothing, two leave out two requirements and two leave out four. Each left-out requirement is either findable in the repository (a README convention, a pattern in the code) or exists only in the developer’s head. An example of the second kind: when staff add notes to an account, support users may only add internal notes, because anything a customer sees counts as a message from the bank and has to come from an admin.

The developer was Claude following a fixed brief, not a real person. Each ticket had a private brief, written before any run, and fixed rules for replying: answer only the question asked; “yes” to a recommendation that matches the brief; correct one that doesn’t; point to the code for questions about the code; “your call” for things the brief doesn’t care about. Afterwards a separate judge model read every reply against its brief and the agent’s message, looking for anything the replies gave away that the agent hadn’t asked about. It found nothing in 54 interviews.

Scoring: hidden tests run after each session. The main score is left-out requirements met, counted only in runs that pass the ticket’s own tests.

Stage 2: each ticket with something left out has a follow-up ticket, done in a fresh session with nobody to ask. Each follow-up quietly depends on one rule the first interview settled. Twelve runs got the glossary and ADRs that one grill-with-docs run wrote; twelve got nothing.

Size: 6 tickets × 5 arms × 3 runs = 90, plus 24 follow-up runs. The agent’s bill for all 114 was $2.52 at list prices.

Results

Findings GRL-1 to GRL-4 are at the top of the page.

What each arm built

Left-out requirements met, counted only when the ticket’s own tests passed (higher is better):

Arm	Clear tickets passed	Some: met / possible	Lots: met / possible	Only in the developer’s head: 5 rules × 3 runs
Ticket only	6 of 6	5 / 12	14 / 24	0 / 15
“Ask me anything first”	6 of 6	6 / 12	15 / 24	0 / 15
Grilling	6 of 6	12 / 12	24 / 24	15 / 15
Grill-with-docs	6 of 6	9 / 12	20 / 24	9 / 15
Grilling, “yes” to everything	6 of 6	5 / 12	13 / 24	0 / 15

• H1, confirmed: on the Lots tickets grilling met 1.67 more of the 4 left-out requirements per run than the ticket alone (Holm p = 0.017).
• H2: “ask me anything” never asked anything, so it scored like the ticket alone.
• H3a, confirmed: grilling beat “yes” to everything by 1.25 per run on the rules only the developer knew (Holm p = 0.0004).
• H3b, identical observed results; equivalence not established: “yes” to everything and the ticket alone both met 0 of 15 (p = 1.0). No difference showed up, but a test like this can’t prove the two are the same in general.

The “of 15” counts are five distinct rules, each checked in three runs per arm. Grilling recovered all five in every run, and the ticket alone recovered none of them in any run. With only five rules, that is strong evidence on these tickets but a small base for a general claim.

• H4, as predicted: every arm passed the clear tickets; grilling only added cost there.

What it cost

Per run, means (lower is cheaper):

	Ticket only	Grilling	Grill-with-docs
Rounds the developer answered	0	2.3–2.7	1.0–1.2
Questions asked	0	8–11	4–5
Words the developer read	0	486–704	247–272
Words the developer typed	0	17–85	8–40
Agent wall time	about 1 minute	2.5–6 minutes	2–5 minutes
Agent cost, list price	$0.013–0.019	$0.025–0.030	$0.025–0.036

The ranges run from the Clear to the Lots tickets. Most of what Claude typed was “Yes.” On the Some and Lots tickets it corrected 1–2 recommendations per grilling run, and those corrections were the whole gain.

The next ticket: did the docs help?

• H5, not supported: with the docs, 17 of 27 left-out requirements were met; without, 15 of 27 (Holm p = 0.81, no measurable difference). With 24 runs and docs that were unevenly written and unevenly read, this doesn’t show that docs are ineffective.
• Written knowledge has to pass three gates: captured, correct, and found. 6 of the 12 docs sets stated the carried-over rule, 4 of the 12 follow-up runs opened any doc, and 2 of 12 met the rule. That chain matters more than the grilling skill itself: it’s the same problem as stale knowledge across a boundary (D2), where a wrong guide was believed, and the OKF trials, where a guide helped when the agent was given it.
• The docs helped exactly where they held the rule and the next agent opened them. That happened only on the freeze ticket, whose ADR was called 0001-account-freeze-audit.md. On the case-closing ticket all three ADRs recorded the rule, and none of the follow-up agents opened them.
• Docs can be wrong with authority. A grill-with-docs run that built before the developer could answer wrote an ADR saying case assignees are stored “without resolving” them, the opposite of the developer’s rule.

Checks

• All 114 runs were graded, none rerun or dropped. Every session ran gpt-5.6-luna at medium effort, and no agent touched the hidden tests or briefs.
• The developer audit found no leaks in 54 interviews; in a dry run it had caught both leaks I planted on purpose.
• Nine interview turns hung inside Codex’s subagent wait call after posting their questions; they were ended and answered as usual (amendment log).
• Two runs failed their ticket tests (ticket only and “yes” to everything, both on the case-closing ticket): the new route ended up under the existing /api/cases prefix.

Limits

• The developer was Claude following a fixed brief, not a real person. It knew the complete answer and followed a fixed correction policy. Real developers forget, give confident wrong answers, miss constraints, disagree with each other, and sometimes say yes without reading. So this measures how well the agent extracts ground truth the developer has, not whether an interview produces correct requirements in a real team. The audit shows the replies didn’t give away more than was asked.
• No structured-prompt arm. The cheap alternative tested was a one-line “ask me anything”, which never asked. A short instruction that requires the agent to list the missing decisions and wait for answers wasn’t tested, so this shows the value of compulsory structured clarification, not yet the extra value of this particular skill.
• One agent, one small repository, six tickets, three runs per cell, five developer-only rules. The large effects (every rule in every run against none) are clear on these tickets; the smaller ones, such as grill-with-docs against grilling, are description.
• I wrote the left-out requirements. A fresh reader checked them for fairness before the run, but which ones I chose shapes the result.
• The follow-up tickets started from the plain app, not from the first ticket’s code, to isolate the docs. In real life the code would carry some of the rules.
• The arms weren’t fully blind. The agents usually named their skill, so the developer could often tell which arm it was answering.

What this means: impact and risks

This part is my interpretation, not a measurement.

Grilling works for a narrow reason. It creates a moment, before any code exists, where the developer corrects one or two wrong assumptions. On tickets that depend on unwritten business rules, that moment was worth everything; on clear tickets it was worth nothing.

The recommended answers are a trap for busy people. Because the agent’s recommendations are usually right, it’s tempting to say “yes” down the list. Doing that gave exactly the same result as never being asked, while feeling thorough.

“Ask me if you need anything” isn’t a substitute. Given permission, this agent concluded every time that it didn’t need anything.

Written decisions only help if they’re captured, right, and found. Grill-with-docs sometimes wrote down its own guesses as decisions, and the next agent mostly didn’t look. Each gate failed on its own here; fixing the skill doesn’t fix discovery.

Configuration recommendations

For developers and teams using an agent on tickets. Each says how strong its evidence is.

Recommendation	Why	Strength
Use a grilling-style skill on tickets that depend on unwritten rules (who may do what, what the customer sees, what gets recorded)	All 5 such rules recovered in every run, against none from the ticket alone	Strong on these tickets: pre-registered, Holm p = 0.017 and 0.0004; five rules, and no structured-prompt comparison
Read each recommendation; the job is to spot the wrong one or two	“Yes” to everything recovered none of the 5 rules in any run	Strong: pre-registered, p = 0.0004
Skip it on tickets that spell everything out	Every arm passed the clear tickets; grilling added reading and 2–3× the wall time	Descriptive
Don’t rely on “ask me anything first”	0 of 18 runs asked a question	Descriptive, one wording
With grill-with-docs, say “ask, then wait for my answers before building”	It built without waiting in 6 of 18 runs	Descriptive; the fix is untested
Point AGENTS.md at GLOSSARY.md and docs/adr/, and name each ADR after the rule it records	The docs helped only when the next agent opened them, and it opened only an ADR whose name matched its ticket	Descriptive; untested
Review ADRs an agent writes, like code	One recorded the opposite of the developer’s rule	Descriptive

What would be worth testing next?

These are proposals, not results.

• Grilling against a short structured prompt. Four arms: ticket only; the grilling skill; a prompt that requires blocking questions and answers before coding; and a targeted interview limited to the three most consequential unknowns. Measure hidden requirements met, wrong assumptions accepted, developer reading and total cost. That says whether the skill is worth installing or a plain instruction gets most of the benefit for less interaction. Now done: Grilling or a good prompt? (GVP).
• Real developers answering the same briefs, to see how often people say yes without reading or answer wrongly.
• Grill-with-docs told to wait for answers, to see whether it then matches grilling.
• The follow-up ticket with an AGENTS.md line pointing at the docs.
• A larger repository, where more of the unwritten rules leave traces in the code and asking matters less.

Reproduce

Everything is in experiments/2026-10-grilling:

• the tickets, developer briefs, hidden tests, reference implementations, skills and driver (tasks/, hidden/, reference/, arms/, scripts/);
• per run: every developer reply with its rule labels, the agent’s messages and event stream, the grade, the audit and the diff (results/main, results/pilot);
• the analysis, checks and diagrams.

The Codex homes, which hold the login, are not published. The commands are in REPORT.md, section 10.

Evidence and reproducibility

Every link is pinned to the commit the figures on this page were read from.

• Full trial report ↗
• Prediction, written before the runs ↗
• Source repository ↗

Related experiments

• Grilling or a good prompt? Does the skill beat a two-sentence instruction?Does Matt Pocock's grilling skill recover more of the rules only the developer knows than a short prompt that makes the agent list its open decisions and wait, or than an interview capped at three questions? Does grilling's branch-by-branch walk win where one answer opens new questions, and what does each cost the developer?
• Can a repository hijack its coding agent?If a repository or ticket contains an instruction to leak secrets and change git configuration, does a coding agent working an ordinary ticket carry it out, and does one paragraph of warning in the prompt stop it?
• The endpoint worked. Was it secure?When a coding agent adds endpoints to a Spring Boot API from ordinary tickets, how often are they working but insecure, and does a security skill, a security gate of public tests, or both make secure endpoints more likely?

Question, correction, or a result that differs from mine? Join the discussion ↗

← All experiments