Telling the agent to test, with and without a skill

Question

In my explicit skills experiment, I told the agent to use the repository’s skills. It then read the verification skill, verify-api, in every run. But it mostly did not do what the skill asks. The skill says to write a focused test of the change; instead, most runs ran its first example, a smoke test that only checks the shop starts.

That left two questions:

1. Does the skill add anything beyond telling the agent to write the test? In that experiment, the skill and the instruction to use it came together.
2. Was the shortcut caused by the skill’s design? It gave the smoke-test command first, and the advice to write a focused test second.

I also added a third: does checking the agent’s work beat telling it? Codex can run a hook, a command it calls at fixed points. A stop hook runs when the agent tries to finish, and can send it back with a reason.

Prediction

Before any run, I wrote down and published the design, the hypotheses and how the results would be compared (README.md).

The hypotheses, about how many runs wrote a relevant test (defined below):

• H1. The instruction plus the skill beats the instruction alone.
• H2. The revised skill beats the original.
• H3. The gate beats the instruction.

Set-up

The agent. OpenAI’s Codex command-line agent, model gpt-5.6-luna, at low reasoning effort (how long the model thinks before acting), except one arm. Each run started from a clean configuration.

The code. shopizer, an open-source online-shop backend written in Java, served over a REST API. Every run started from the same version.

Six ways of running the agent (called arms):

The instruction:

Before you say you are done, add a focused test that checks your change through the API, calling the endpoint the way a client would, and run it. It should fail without your change and pass with it.

The task:

The two skill arms add one sentence: “Read and follow .agents/skills/verify-api/SKILL.md, which explains how to do this in this repository.”

The revised skill only reorders and rewords the original, which an agent wrote in the repository skills experiment. It now asks first for a focused test that calls the changed endpoint, and should fail without the change and pass with it. The smoke test comes second, as a pattern to copy, with the note that it only checks the shop starts, “so running it does not verify a change”.

The gate. When the agent tries to finish, the hook runs the same relevance check as the grader (below), on the agent’s copy of the code. If there is no relevant test, it sends the agent back once, saying what failed but not how to fix it. For example: “Before you finish: no test of your own was found in your change. Add a test that checks your change through the API, and run it. It should fail without your change and pass with it.” The second time, it always lets the agent finish.

The work. The same 20 tasks as the last three experiments, each run three times in each arm: 20 × 6 × 3 = 360 runs, all completed.

What counts as a relevant test

In the last experiment, I counted tests by looking for certain names in the code the agent added. This time the grader checks each one by running it:

1. It runs the tests the agent added or changed, with its change. All must pass.
2. It takes the original code, adds the agent’s test files, and runs them again. At least one must fail.

A test that passes both times does not check the change. I checked this method on the last experiment’s 27 test-writing runs before starting. All 27 were relevant, so the earlier counts had been slightly low, not inflated.

How a run is graded

The agent never sees how it will be judged. When it finishes, a grader adds hidden tests (65 across the 20 tasks) and runs the project’s whole test suite. The hidden tests start the shop and use its API as a real client would. A run passes only if every hidden test passes and none of the project’s existing tests breaks.

Results

Findings TSK-1 to TSK-4 are at the top of the page.

Tokens are the units of text the model reads and writes, and what you pay for; M is millions. Costs are estimates at list prices; the 360 runs cost $15.12 together. Run time is the median.

Telling was enough

• The instruction worked. Told to write a focused test, the agent wrote and ran one in every run. In 56 of 60, the test failed on the original code and passed with the change. Given only the task, it never wrote one.
• The skill added nothing. With the original skill, 52 of 60 runs wrote a relevant test; with the revised one, 54. Both are within chance of the instruction alone. The skill also made each run about a quarter more expensive.
• The shortcut disappeared. No run, in any arm, ran only the smoke test. In the last experiment, told to follow the same original skill, most runs did. What changed was the direct instruction, not the skill’s order.
• The misses were tests left failing (4 to 7 runs per told arm), or tests that also passed on the original code (0 to 2).

The gate

• Every run was sent back. Given only the task, the agent never wrote a test before trying to finish.
• After one block, 58 of 60 had a relevant test. The other two were let through, as designed: one with a failing test, one with a test that did not check the change.
• The gate agreed with the grader in all 60 runs.
• It knew. The agent’s notes referred to the hook: “The repository hook requires a regression test of the API itself, so I’m adding endpoint-level assertions”. None touched the hook’s files.
• It took longer. Runs took a median of 220 seconds, because each check runs the agent’s tests twice. Cost per correct change was about the same as the instruction’s.

Tests and correct changes

The arms that wrote real tests passed 41 to 47 of 60, against 40 with no guidance.

• Telling, with or without the skill, made no difference to the pass rate (41 to 43).
• The gate passed 47, 7 more than no guidance and 6 more than the instruction. That is not significant here, but it is the most of any guided arm. After being sent back, 20 gated runs changed the code itself, not only tests, and 14 of those passed. Writing the test after the fix sometimes exposed a gap in it. This was found after the results, and does not show those runs would otherwise have failed.
• Medium reasoning effort passed 49, with no tests at all. It is the only arm that is clearly better than no guidance.
• A relevant test is not a correct change. 11 gated runs had one and still failed the hidden tests. On the two tasks where the obvious change breaks another feature, the guided arms wrote relevant tests in most runs, and passed 0 or 1 of 6 each. A test of the change does not catch a break elsewhere.
• Failed runs still mostly said they had worked: 55 of 68 in the arms that wrote tests, 30 of 31 in the two that did not.

How the arms were compared

The unit is the task, not the run, because some tasks are simply harder than others. For each task, I took the number of runs with a relevant test in one arm (0 to 3) and subtracted the number in the other. To ask whether the result could easily be chance, the test flips the sign of each task’s difference in every possible way, and counts how often the total comes out at least as large as the real one. Because there are three hypotheses, the results are adjusted upwards (Holm’s method).

None is supported. On pass rate, compared the same way but without adjustment: medium effort against no guidance, +15 points (+5 to +27, chance 0.03); the gate against no guidance, +12 points (0 to +25, chance 0.15); the gate against the instruction, +10 points (−2 to +23, chance 0.22).

Checks on the runs

Before writing up, I checked these counts against the record of every command the agent ran:

• Every run was fully graded. In all 360, every hidden test ran.
• Every run has a relevance result, and no check failed to run.
• Only the two skill arms read the skill, in all of their runs.
• No run could see the hidden tests.

Limits

• One codebase; one agent and model; low reasoning effort except one arm. The gate was not tried at medium effort.
• One verification skill, revised by me. I know the tasks, but the revision only reorders the original.
• The instruction already said what the skill says. A skill carrying knowledge the agent lacks might add more.
• The medium-effort and gate arms ran in their own batches, alongside the others but not interleaved with them.
• The relevance check shows a test tells the change from the original code, not that it tests what the task asked for.
• Only 2 tasks were neighbour breaks.
• I wrote the tasks and hidden tests; all are published for checking.

Across the last five experiments

If you want your agent to write a test of its change, say so plainly, or check for one when it tries to finish. Both worked here, and the skill added nothing to either. But a test of the change did not make the change measurably more correct. In five experiments, the only thing that did was more reasoning effort.

Reproduce

Everything is in experiments/2026-10-test-skill:

• the instruction and both skills exactly as used (arms/);
• the relevance check and the gate (scripts/relevance.sh, scripts/gate.sh);
• the tasks and hidden tests (tasks/, hidden/);
• the raw runs (results/stage1, results/stage1-medium, results/stage1-gate);
• the analysis and chart scripts (scripts/).

The commands are in REPORT.md, section Reproducing.

Question, correction, or a result that differs from mine? Join the discussion ↗

← All experiments