Skills

Findings

29 findings, each linking to its write-up and limits

• GVP-1A two-sentence structured prompt recovered as many developer-only rules as the grilling skill35 of 75 (prompt) against 34 of 75 (grilling); 0 of 75 from the ticket alone
• GVP-2Grilling did not pull ahead on nested tickets, because the first rule in each chain was rarely found3 of 30 (grilling) against 5 of 30 (prompt) and 0 of 30 (three questions)
• GVP-3Capping the interview at three questions did well on simple tickets and fell behind with more unknowns19 of 25 on the few-unknown tickets (grilling 21), 6 of 20 on the many-unknown ticket (grilling 10), 0 of 30 on the nested tickets
• GRL-1Grilling recovered every requirement that only the developer knew; working from the ticket recovered none5 of 5 rules in each of 3 runs (15 of 15 checks) against 0 of 15
• GRL-2The gain came from the developer's corrections: grilling with "yes" to everything did no better than the ticket alone15 of 15 checks (answered) against 0 of 15 ("yes" to everything)
• GRL-3Telling the agent "ask me anything you need to know first" produced no questions at all0 of 18
• GRL-4Grill-with-docs's glossary and decision records made no measurable difference to a later ticket; they helped only when the rule was captured correctly and the next agent found it17 of 27 against 15 of 27
• SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole18%
• SEC-2A security skill, opened in every run, made no measurable difference−5 ptsNo measurable difference
• SEC-3In an exploratory analysis after two faults in my test materials were fixed, a gate of public security tests beat the skill+11 ptsInterval excludes zero
• SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request38 of 40
• SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed69 of 71
• TSK-1A one-paragraph instruction to write a focused API test got the agent to write one that fails without its change and passes with it in 56 of 60 runs; given only the task, it wrote none56/60 vs 0/60
• TSK-2Adding the repository's verification skill to the instruction did not help: 52 of 60 runs wrote a relevant test with the original skill and 54 with a revised one that puts the focused test first, against 56 with the instruction alone56, 52, 54 of 60
• TSK-3A stop hook that checked for a relevant test when the agent tried to finish, and sent it back once if there was none, worked as well as the instruction: it blocked every run, and 58 of 60 then had a relevant test58/60
• TSK-4Real tests of the change did not measurably raise the pass rate: 41 to 47 of 60 in the arms that wrote them, against 40 with no guidance; medium reasoning effort, with no tests, passed 49+15 ptsInterval excludes zero
• ESK-1Reading is not doing: told to follow the verification skill, the agent read it in every run, but a test of its change was detected in only 15 of 60 runs (prompt) and 10 of 60 (AGENTS.md); most runs ran the skill's smoke-test example instead15/60 and 10/60
• ESK-2Telling the agent to use its skills did not measurably raise the pass rate: 42 of 60 runs passed when told in the prompt and 43 when told in AGENTS.md, against 37 with no guidance and 39 with the guidance left to discover+8 to +10 pts, not significantNo measurable difference
• ESK-3A one-line entry in AGENTS.md got the verification skill read as reliably as an instruction in the prompt; the pass rates were similar (43 and 42 of 60), and the experiment cannot tell whether placement matters60/60, 60/60, 3/60
• ESK-4Being told cost about twice as much per correct change as no guidance: 1.98 times when told in the prompt and 1.77 times in AGENTS.md1.98×Interval excludes 1×
• RSK-1Repository guidance left for the agent to discover did not measurably raise the pass rate: 40 of 60 runs passed, against 38 of 60 without it (p = 0.40)40 vs 38 of 60 · not measurableNo measurable difference
• RSK-2The agent read the skills on building and on changing code in every run, but opened the skill on checking a change through the API in only 8 of 60, and a test of its own change was detected in 28/60
• RSK-3Cost 1.29 times as much per correct change (1.01 to 1.55), against about 2.4 to 2.5 times for the Superpowers pack in earlier experiments1.29×Interval excludes 1×
• VVS-1Medium reasoning effort, with no skills and no instruction, passed 49 of 60 runs against 36 of 60 at low effort: better on 9 tasks and worse on 1 (Holm p = 0.041)+21.7 ptsInterval excludes zero
• VVS-2An instruction to check the change against the running application did not measurably raise the pass rate (39 of 60 against 36, Holm p = 0.67); the agent tried to start the app in 47 runs, but it got the app to answer in only 3+5.0 ptsInterval includes zero
• VVS-3Superpowers again passed about as often as no skills (37 of 60 against 36) and cost the most per correct change: 7.0 cents, against 4.0 at medium effort and 2.8 with no skills at low effort7.0¢
• SKT-1Neither skill pack reliably raised the pass rate: Superpowers passed 24 of 32 runs, the same as with no skills, and Pocock passed 27 of 32, a gap small enough to be chance (Holm p = 0.38)+9.4 ptsInterval includes zero
• SKT-2Superpowers used about three times the tokens of no skills on the same task; it wrote tests in 23 of 32 runs, where the agent with no skills wrote none3.07×Interval excludes 1×
• SKT-3In 28 of the 30 failed runs, the agent said its change worked; this held in every arm28/30