Cross-repo change tasks, small

The question

When the agent must change the storefront using facts only the backend states, how much does a backend guide help?

Experiment

F

Runs

60

Agents

Codex (gpt-5.6-luna)

Key finding

• F-1

  On four small cross-repository change tasks, hidden-test passes went from 3 of 20 with no guide to 19 of 20 with a guide to the backend, at about half the tokens (242k against 482k per run).

  • Four tasks, each to create a small storefront module using a fact only the backend states: the discount-code field on an add-to-cart request, the backend's paging names (size, offset), the reviews path (product/{id}/reviews), and the purchase flag in the public config response (purchasingEnabled). Hidden Node tests, 5 runs per task per condition.
  • Passes by task (f1, f2, f3, f4): no guide 3, 0, 0, 0; guide 5, 5, 4, 5; guide with confidence labels 5, 3, 2, 5.
  • The 3 passes without a guide were all on the first task, where the field name (voucher) is a word an agent can guess. No guide against guide: 3 against 19 of 20, p < 0.001.
  • Labels did not help: 15 of 20 against 19 of 20 for the plain guide (p about 0.18, not a reliable difference), at 235k tokens per run.

  Limits5 runs per task · identifiers renamed by me · one model · task wording ambiguity noted for one prefix case (RESULTS.md F)

  Source ↗

  3 / 20 → 19 / 20

  Measurehidden-test passes of 20

  Sample5 runs per task × 4 tasks

Question

When the agent must change the storefront using facts only the backend states, how much does a backend guide help on small tasks?

Prediction

None written down for F in the Guide.

Set-up

Four small ES-module tasks; hidden Node tests; arms include no OKF, OKF, OKF with confidence labels. Codex. RESULTS.md section F; results/F-crossrepo-change-tasks.md.

Results

Finding F-1 is above. Foreword composite OKF-2 also covers G and is not redefined here.

F. Cross-repo change tasks (60 runs)

Question tested: when the agent must change the storefront using facts that only the backend states, how much does the backend’s guide help, and do confidence labels change that?

Setup. Four tasks, each “create this small ES module in the storefront”, each checked by a hidden Node test that was shown to fail on the unchanged storefront and pass on a reference solution:

1. buildAddToCartRequest: add the backend’s field for a discount code (voucher, renamed from the real name).
2. buildSearchRequest: use the backend’s names for page size and offset (size, offset).
3. buildReviewsRequest: use the path the backend declares (product/{id}/reviews; the storefront’s own code uses products/...).
4. canPurchaseOnline: read the backend’s purchase flag from its public config response (purchasingEnabled).

5 runs per task per condition.

Condition	Passed (of 20)	f1	f2	f3	f4	Tokens per run (95% CI)
1. No OKF	3	3	0	0	0	482k (301k to 681k)
2. OKF	19	5	5	4	5	242k (203k to 286k)
3. OKF with confidence labels	15	5	3	2	5	235k (205k to 270k)

• No OKF against OKF: 3 against 19 of 20 (p<0.001), at about half the tokens. The 3 passes without a guide were all on task 1, where the backend’s field name (voucher) is a common word an agent can guess.
• OKF against OKF with labels: 19 against 15 of 20 (p≈0.18), not a reliable difference at this size. The labelled condition’s five failures were two wrong paging field names (start/startIndex for offset) and three on the reviews path, one of which added a /api/v1 prefix that the task did not clearly rule out. The plain guide failed once, on the same task. The task wording should have been clearer about the prefix.
• One task (4) was ambiguous in the first draft: the backend has an internal setting and a public response field with similar meanings. The wording was fixed before any real run.

Source: docs/RESULTS.md (pinned to commit 5a09027).

Trial report (raw output)

Baseline arm: no_okf. Relative cost is each run’s tokens divided by the baseline’s mean for the same question; 1.00 means equal, lower is cheaper.

codex

arm	runs	correct	tokens per run (95% CI)	relative cost (95% CI)	verdict	turns	used tool	seconds
no_okf	20	- (verified 3/20)	482k (301k to 681k)	1.00	baseline	5.0	-	48
okf	20	- (verified 19/20)	242k (203k to 286k)	0.80 (0.51 to 1.15)	no clear difference	5.2	20/20	27
okf_confidence	20	- (verified 15/20)	235k (205k to 270k)	0.74 (0.51 to 1.02)	no clear difference	5.2	20/20	30

Run-to-run spread (coefficient of variation of relative cost) is 0.87. To detect a 20% difference with 80% power you would need about 296 runs per arm (larger differences need fewer); this trial has 20.

Tokens by question (mean, thousands)

question	codex/no_okf	codex/okf	codex/okf_confidence
f1_cart_request	652k	165k	193k
f2_search_request	357k	221k	242k
f3_reviews_request	130k	247k	210k
f4_store_config	790k	335k	294k

Token totals are not comparable between agents (see adapters.py). Treat a verdict as evidence only when the interval excludes 1.00, and as provisional below 8 runs per arm.

Full report: results/F-crossrepo-change-tasks.md (pinned to commit 5a09027).

Limits

5 runs per task; renamed identifiers; one model. Guide/RESULTS note task-wording issues on a prefix and an ambiguous config field (fixed before counted runs).

Reproduce

OKFExperiment. See repository README.

Longer write-up: Test F in the OKFExperiment details.

Evidence and reproducibility

Every link is pinned to the commit the figures on this page were read from.

• Full trial report ↗
• Source repository ↗

Related experiments

• Is a code guide worth more to a small local model?When a small local model must change a storefront to match a backend it can't see, does a generated guide to the backend help it, and does the guide help more than the backend's source code?
• Agent-optimised repositoryDoes a repository set up for agents do better than the same repository left as it is?
• AgentGraphWorkTree evaluationAre the graph's facts correct, and when does the graph reduce agent tokens?

Question, correction, or a result that differs from mine? Join the discussion ↗

← All experiments