Topics

Parallel

Parallel agents: isolation and the cost of coordination.

Findings

10 findings, each linking to its write-up and limits

  • TH-1A vote of three identical agents was no better than one agent at seeing through a false fact20 vs 17 of 42
  • TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted22 of 22
  • TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own6 of 42
  • TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)29 → 15
  • SPS-1Splitting a feature across three coders working at the same time, each given only its part's requirements, passed 61.6% of hidden acceptance tests; one agent alone passed 62.4%, at less than half the cost−0.8 ptsInterval includes zero
  • SPS-2In 4 of 72 split runs the application would not start at all, against none of 36 one-agent runs4 of 72
  • SPS-3Agent reviewers approved code that failed hidden acceptance tests 14 times out of 1514 of 15
  • SWT-1A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money−31.2 ptsInterval excludes zero
  • SWT-2Giving every team member the task file raised the team from 25.7% to 54.1% of hidden tests passed, level with one agent, at about 1.6 times its cost+28.5 ptsInterval excludes zero
  • SWT-3Every time an agent reviewer approved a change, the change failed hidden acceptance tests: 24 of 24 approvals24 of 24

Experiments

3 experiments

Harmful errors, lower is better, of 20 genuine threats and 22 harmless cases: one Luna agent 5 and 16; three voting 7 and 15; three after peer review 7 and 22; one agent with the same token budget 4 and 16; Sol reviewing the team 2 and 17; Sol alone 2 and 18. Of 42 falsified cases, right, sent for investigation and harmful: three voting 20, 0, 22, and with a can't-tell answer allowed 13, 11, 18; peer review 13, 0, 29, and with can't-tell allowed 8, 19, 15.

Controlled trial · TH

Three AI agents agreed. Did that make them right?

When security evidence has been tampered with, does a team of AI agents catch the error, repeat it, or make it worse, and does a stronger model reviewing the team's work fix it?

AI agents triaged security alerts whose evidence contained one false fact, contradicted by another record. A vote of three agents did no better than one. After reading each other's conclusions, all three escalated every harmless alert with conflicting records. A stronger model reviewing the team rarely overruled it. Allowing a "can't tell, investigate" answer halved the peer-reviewed team's harmful errors by sending cases to a human.

Difference in percentage points, 95% interval

Controlled trial · SPS

Does splitting a feature across coding agents beat one agent?

When a planner splits a feature into parts for coding agents working at the same time, does the team get more of it right than one agent, and does giving each coder only its part's requirements lose details at the joins?

A planner split each of six features into three parts, and three Codex coders built them at the same time. Whether each coder saw only its own part's requirements or the whole task, the team passed no more of the hidden acceptance tests than one agent did (61.6% and 52.9% against 62.4%), at 2.3 to 2.5 times the cost and with no saving in time. The split added a new failure: parts that didn't fit, so the application wouldn't start.

Difference in percentage points, 95% interval

Controlled trial · SWT

Does a team of coding agents beat one agent?

Does a team of coding agents (planner, coder, reviewer) get more of a feature right than one agent, and does any gain survive when one agent gets the same spend?

I gave Codex six feature tasks, either alone or as a team: a planner, a coder and a reviewer. One agent passed 56.8% of the hidden acceptance tests; the team passed 25.7%, at higher cost. The reason was the hand-off: only the planner saw the task, and its summary lost the details. Giving every agent the task file brought the team level with one agent (54.1%), still at higher cost.

Other topics: Measurement, Context, Verification, Safety and cost, Skills, Review, Briefs, Harness