Telling the agent to use its skills

Question

In my repository skills experiment, an agent wrote guidance for one codebase: an AGENTS.md file and three skills (short instruction files the agent can open when it judges them relevant). The guidance sat in the repository for the agent to find. It did not measurably help.

The agent read two of the skills every time. It opened the third, verify-api, on how to check a change through the shop’s API, in only 8 of 60 runs. So that experiment measured guidance the agent chose to read, not guidance it followed.

This experiment tells the agent to use the skills. The guidance is the same, byte for byte. The difference is an instruction to use it, given in one of two places:

• in the prompt, which a user writes for each task;
• in AGENTS.md, the file Codex reads for project instructions, which a team writes once.

Prediction

Before any run, I wrote down and published the design, the hypotheses and how the results would be compared (README.md).

The hypotheses: told in the prompt, a larger share of runs pass than with no guidance (H1), and than with the guidance left to discover (H2).

The AGENTS.md arm was added while the first three were running, before I had looked at any result, with two hypotheses of its own: it beats no guidance (H3), and it beats the guidance left to discover (H4). It is recorded in the README’s amendment log.

Set-up

The agent. OpenAI’s Codex command-line agent, model gpt-5.6-luna, at low reasoning effort (how long the model thinks before acting). Each run started from a clean configuration.

The code. shopizer, an open-source online-shop backend written in Java, served over a REST API. Every run started from the same version.

Four ways of running the agent (called arms):

The instruction at the start of the prompt:

Use this repository's skills in .agents/skills for this task:
1. Before you change any code, read and follow .agents/skills/safe-change/SKILL.md.
2. To build and run tests, follow .agents/skills/build-test/SKILL.md.
3. Before you say you are done, read and follow .agents/skills/verify-api/SKILL.md, and check your change through the API as it says.

The task:

The entry added to AGENTS.md uses the same words as step 3:

## Before you finish

Before you say you are done, read and follow .agents/skills/verify-api/SKILL.md, and check your change through the API as it says.

It names only the verification skill, because the agent already read the other two in every run when left to discover them.

The work. 20 tasks, each run three times in each arm: 20 × 4 × 3 = 240 runs, all completed.

The guidance

Written by Codex at high effort in the repository skills experiment, without sight of the tasks or tests, and used unedited:

• AGENTS.md: 8 lines on how the code is organised, the key commands, and which skill to use when.
• build-test: how to build, run every test, or run one test class without a network connection.
• safe-change: how a request passes through the code, which errors become which HTTP status codes, and to search for every other user of shared code before changing it.
• verify-api: the shop does not start on its own here, so check a change with a test that starts it and sends real requests. It gives, first, the command for an existing smoke test, which only checks that the shop starts and answers a health check. Then it says: “For an API change, add or update a focused … integration test”, calling the endpoint as a client would, and checking both the status and the body.

The tasks

The same 20 tasks as the verification experiment, unchanged, each written the way a shop owner or support team would report it.

• Runtime traps (18). The obvious change compiles and looks right, but is wrong when the shop runs.
• Neighbour breaks (2). The obvious change breaks another feature that shares the code: REVEDIT (editing reviews, where creating reviews must keep working) and BOXVIS (hiding content boxes from shoppers, where the back office must still see them).

How a run is graded

The agent never sees how it will be judged. When it finishes, a grader adds hidden tests (65 across the 20 tasks) and runs the project’s whole test suite. The hidden tests start the shop and use its API as a real client would. A run passes only if every hidden test passes and none of the project’s existing tests breaks.

Results

Findings ESK-1 to ESK-4 are at the top of the page. In short:

Tokens are the units of text the model reads and writes, and what you pay for; M is millions. Costs are estimates at list prices; the 240 runs cost $6.70 together. Run time is the median.

Reading is not doing

This is the main result. Telling the agent to use the skills changed what it read. It changed much less what it did.

• It read the skill. Told, in either place, the agent opened verify-api in all 60 runs. Left to discover it, 3.
• It took the shortcut. Most runs then ran the skill’s smoke test, which checks only that the shop starts. It ran in 49 runs told in the prompt and 47 told in AGENTS.md. In 23 and 37 runs it was the only test the agent ran. The step the skill asks for next, writing a test of the change, was detected in 15 and 10 runs.
• It did not ignore the skill. It took the action in it that looked like verification and was easiest, rather than the one that checks the change.
• It said so. In some runs, its notes admitted the gap, for example: “the specific customer address PATCH/GET flow was not covered by an existing integration test”. It still reported success.
• A test went with passing. Runs with a test of their change passed 13 of 15 (prompt) and 9 of 10 (AGENTS.md), against 29 of 45 and 34 of 50 that did not. This does not show that the test caused the pass: the agent may write one more readily when it already understands the task.
• What it claimed. Every failed run, in all four arms (79 runs), still said its change worked.
• Neighbour breaks. Being told to use safe-change, which says to check every other user of shared code, did not prevent them: 4 to 5 runs broke a neighbouring feature in each arm, of the 6 neighbour-break runs.

How a test was counted: the change the agent left adds a test that calls the API (using TestRestTemplate, ServicesTestSupport or MockMvc). This is an automatic check of the code, not a reading of each test by hand. It shows a test was added, not that it exercised the change.

How the arms were compared

The unit is the task, not the run, because some tasks are simply harder than others. For each task, I took the number of passing runs in one arm (0 to 3) and subtracted the number in the other. To ask whether the result could easily be chance, the test flips the sign of each task’s difference in every possible way, and counts how often the total comes out at least as large as the real one. Because there are two tests in each pair, the counts are adjusted upwards (Holm’s method).

None is close to the usual 0.05 threshold. Being told points the right way in all four, but a gain of 8 to 10 points is smaller than this experiment can show.

Told in the prompt and told in AGENTS.md gave similar results: 42 and 43 runs, the same on 17 tasks. That does not show placement makes no difference. The experiment is too small to show it, and the two instructions differed: the prompt named all three skills, AGENTS.md only the verification one. Left to discover, the guidance gained 2 runs over none, as in the repository skills experiment (40 against 38 there).

Checks on the runs

Before writing up, I checked these counts against the record of every command the agent ran:

• The guidance was where it should be. It was present in every run of the three arms with guidance, and in none without.
• The skills were really read. In the two arms that were told, the skill’s text shows in the logged output of 118 of 120 runs; in the other two, the log kept only part of a long output.
• The AGENTS.md entry was in effect. That arm had no instruction in its prompt, yet it opened verify-api in every run, against 3 of 60 for the same guidance without the entry.
• Every run was fully graded, except one whose change did not compile, which counts as a fail.
• No run could see the hidden tests.

Limits

• One codebase; one agent and model; low reasoning effort only.
• One set of guidance, written by an agent. The verification skill puts the smoke-test command first and the advice to write a focused test second. A skill that gave only the second might be followed more fully. This experiment cannot say.
• The instruction adds emphasis on checking, not only a pointer to the skills. Any gain could come from either.
• The AGENTS.md arm ran later than the other three, in a second batch. Its instruction named only the verification skill, so the prompt and AGENTS.md arms are not a pure test of placement.
• Tests of the change were detected automatically from the code the agent left, not checked by hand.
• Only 2 tasks were neighbour breaks.
• The tasks were chosen in the verification experiment using runs without guidance, which slightly favours that arm.
• I wrote the tasks and hidden tests; all are published for checking.

Across the last four experiments

Telling the agent gave the highest pass counts of the written instructions so far (42 and 43 of 60, against 37), but the difference could be chance. Each correct change cost 1.8 to 2.0 times as much. In an earlier experiment on the same tasks, medium reasoning effort did better (49 against 36 of 60, a real difference) at 1.4 times the cost.

If you want your agent to read a skill, one line in AGENTS.md was enough here. Getting it followed is a different matter: the agent read the instruction to write a test, and mostly ran the example instead.

Reproduce

Everything is in experiments/2026-10-explicit-skills:

• the guidance exactly as used (arms/guidance/, and arms/guidance-agentsmd/ with the extra entry);
• the prompt instruction (arms/explicit-prefix.md);
• the tasks and hidden tests (tasks/, hidden/);
• the raw runs (results/stage1, results/stage1-agentsmd);
• the analysis and chart scripts (scripts/).

The commands are in REPORT.md, section Reproducing.

Question, correction, or a result that differs from mine? Join the discussion ↗

← All experiments