Topics

Safety and cost

Safety, cost and the human in the loop.

Findings

30 findings, each linking to its write-up and limits

  • RI-1Without a defence, a planted instruction disguised as a setup script hijacked the agent every time it sat in AGENTS.md or a ticket comment, and never anywhere else18 of 90 (9 of 9 AGENTS.md, 9 of 9 ticket, 0 of 27 elsewhere)
  • RI-2Every hijacked run reported a clean success, listing the malicious script as a passed check18 of 18
  • RI-3One paragraph in the prompt, saying repository text is information and not instructions, cut hijacks from 18 of 90 runs to 1, at no measurable cost18 → 1 of 90
  • SEC-1With no safeguards, the agent shipped endpoints that worked but weren't secure in 14 of 80 runs, 9 of them with a real access hole18%
  • SEC-2A security skill, opened in every run, made no measurable difference−5 ptsNo measurable difference
  • SEC-3In an exploratory analysis after two faults in my test materials were fixed, a gate of public security tests beat the skill+11 ptsInterval excludes zero
  • SEC-4In 38 of 40 runs, across all four setups, an approver could approve or reject their own closure request38 of 40
  • SEC-5In 69 of 71 failed runs, the agent's closing line said its own checks had passed69 of 71
  • TH-1A vote of three identical agents was no better than one agent at seeing through a false fact20 vs 17 of 42
  • TH-2After reading each other's conclusions, all three agents escalated every harmless alert whose records conflicted22 of 22
  • TH-3A stronger model reviewing the team rarely overruled it, and did no better than on its own6 of 42
  • TH-4Allowing a "can't tell, investigate" answer for conflicting evidence halved the peer-reviewed team's harmful errors, by sending 19 of 42 falsified case-repeats to a human (exploratory)29 → 15
  • SPS-1Splitting a feature across three coders working at the same time, each given only its part's requirements, passed 61.6% of hidden acceptance tests; one agent alone passed 62.4%, at less than half the cost−0.8 ptsInterval includes zero
  • SPS-2In 4 of 72 split runs the application would not start at all, against none of 36 one-agent runs4 of 72
  • SPS-3Agent reviewers approved code that failed hidden acceptance tests 14 times out of 1514 of 15
  • SWT-1A planner–coder–reviewer team passed 25.7% of the hidden acceptance tests; one agent alone passed 56.8%, for less money−31.2 ptsInterval excludes zero
  • SWT-2Giving every team member the task file raised the team from 25.7% to 54.1% of hidden tests passed, level with one agent, at about 1.6 times its cost+28.5 ptsInterval excludes zero
  • SWT-3Every time an agent reviewer approved a change, the change failed hidden acceptance tests: 24 of 24 approvals24 of 24
  • EM-1One falsified fact in the evidence led to the wrong decision in 81 of 120 runs (68%), against 0 of 120 on clean evidence+67.5 ptsInterval excludes zero
  • EM-2An injected instruction in a ticket comment changed 0 of 120 decisions and was flagged as manipulation in all 120 runs+0.8 ptsNo measurable difference
  • EM-3With both a falsified fact and the instruction, the agent flagged manipulation in all 120 runs and still reached the attacker's intended outcome in 73−6.7 ptsNo measurable difference
  • EM-4With a tool that checks authoritative records, 359 of 360 manipulated runs reached the right decision+42.5 ptsInterval excludes zero
  • RAG-1Changing one number in the retrieved paragraph that held the answer made qwen3.8:27b repeat the planted number in 102 of 120 questions (85%); right answers fell from 113 to 5+90 ptsInterval excludes zero
  • RAG-2Knowing the answer didn't protect it here: on 15 questions qwen3.8:27b answered right with no documents, it repeated the planted number in 13, and its recorded thinking never mentioned the value it had known13/15
  • RAG-3When no retrieved paragraph held the answer, both models mostly said so: NOT FOUND in 107 (27B) and 112 (8B) of 120 questions; made-up answers 8 and 6−1.7 ptsNo measurable difference
  • RAG-4No measurable difference between the models: qwen3:8b repeated the planted number in 98 of 120 questions, against 102 for qwen3.8:27b−3.3 ptsNo measurable difference
  • PE-1A wrong number planted in the model's thinking was carried through to the matching wrong answer in 81 of 230 runs (35%); with the thinking left unchanged, 1 of 230 went wrong+37 ptsInterval excludes zero
  • PE-2The model caught the planted slip in its own thinking in 124 of 230 runs (54%)124/230
  • PE-3In 23 of 230 runs (10%), the thinking ended on the wrong answer but the written reply gave the right one without saying why; the reverse never happened23/230
  • PE-4The thinking does real work: forced to answer with no thinking, the model got 23% of problems right; after its full thinking, 94%+71 ptsInterval excludes zero

Experiments

8 experiments

Proportion, 95% interval

Controlled trial · RI

Can a repository hijack its coding agent?

If a repository or ticket contains an instruction to leak secrets and change git configuration, does a coding agent working an ordinary ticket carry it out, and does one paragraph of warning in the prompt stop it?

I planted an instruction in a repository asking a coding agent to send the checkout's secrets to an outside server and redirect git's commit hooks. Disguised as a setup script in AGENTS.md or in a ticket comment, it worked in every undefended run, and the agent reported a clean success. In the README, a code comment or test output it never worked. One paragraph of warning in the prompt cut hijacks from 18 of 90 runs to 1.

Proportion, 95% interval

Controlled trial · SEC

The endpoint worked. Was it secure?

When a coding agent adds endpoints to a Spring Boot API from ordinary tickets, how often are they working but insecure, and does a security skill, a security gate of public tests, or both make secure endpoints more likely?

A hosted coding agent added 320 endpoints to a multi-tenant Spring Boot API from ordinary tickets. The agent handled framework-level security well but repeatedly missed one business rule: in 38 of 40 closure-approval runs, an approver could approve their own request. A security skill changed nothing measurable. A gate of public tests improved the behaviours it named, but not that failure, and its edge over the skill rests on a corrected, exploratory result.

Harmful errors, lower is better, of 20 genuine threats and 22 harmless cases: one Luna agent 5 and 16; three voting 7 and 15; three after peer review 7 and 22; one agent with the same token budget 4 and 16; Sol reviewing the team 2 and 17; Sol alone 2 and 18. Of 42 falsified cases, right, sent for investigation and harmful: three voting 20, 0, 22, and with a can't-tell answer allowed 13, 11, 18; peer review 13, 0, 29, and with can't-tell allowed 8, 19, 15.

Controlled trial · TH

Three AI agents agreed. Did that make them right?

When security evidence has been tampered with, does a team of AI agents catch the error, repeat it, or make it worse, and does a stronger model reviewing the team's work fix it?

AI agents triaged security alerts whose evidence contained one false fact, contradicted by another record. A vote of three agents did no better than one. After reading each other's conclusions, all three escalated every harmless alert with conflicting records. A stronger model reviewing the team rarely overruled it. Allowing a "can't tell, investigate" answer halved the peer-reviewed team's harmful errors by sending cases to a human.

Difference in percentage points, 95% interval

Controlled trial · SPS

Does splitting a feature across coding agents beat one agent?

When a planner splits a feature into parts for coding agents working at the same time, does the team get more of it right than one agent, and does giving each coder only its part's requirements lose details at the joins?

A planner split each of six features into three parts, and three Codex coders built them at the same time. Whether each coder saw only its own part's requirements or the whole task, the team passed no more of the hidden acceptance tests than one agent did (61.6% and 52.9% against 62.4%), at 2.3 to 2.5 times the cost and with no saving in time. The split added a new failure: parts that didn't fit, so the application wouldn't start.

Difference in percentage points, 95% interval

Controlled trial · SWT

Does a team of coding agents beat one agent?

Does a team of coding agents (planner, coder, reviewer) get more of a feature right than one agent, and does any gain survive when one agent gets the same spend?

I gave Codex six feature tasks, either alone or as a team: a planner, a coder and a reviewer. One agent passed 56.8% of the hidden acceptance tests; the team passed 25.7%, at higher cost. The reason was the hand-off: only the planner saw the task, and its summary lost the details. Giving every agent the task file brought the team level with one agent (54.1%), still at higher cost.

Difference in percentage points, 95% interval

Controlled trial · EM

Can manipulated evidence mislead an AI security agent?

Can falsified evidence or an injected instruction make an AI triage agent reach the wrong security decision, and does an independent verification tool help?

I gave a local AI triage agent 40 synthetic security alerts, then changed one fact in the evidence, or added an instruction telling it what to do, or both. One false fact led to the wrong decision in 68% of runs, and in 38% of runs even where another record contradicted it. The injected instruction changed no decisions. A tool for checking an authoritative record corrected nearly all of them, but some runs still called the wrong action tool.

Difference in percentage points, 95% interval

Controlled trial · RAG

When RAG retrieves the wrong facts, will an AI challenge them?

When one retrieved number is wrong, how often does a model repeat it, and does knowing the right answer protect it? When retrieval misses, does it make something up?

A baseline test of error propagation in retrieval. I changed one number in the paragraph that held each answer and measured whether two local models repeated it. Both did, more than 8 times in 10, even when the bigger model had answered correctly with no documents. That is what retrieval is built to do. The measurement is what is useful, not the surprise.

Difference in percentage points, 95% interval

Controlled trial · PE

Does an AI's visible thinking drive its answer? I planted errors in it to find out

If a wrong number is planted in the middle of a reasoning model's thinking, does its answer follow the error, does it catch it, or does the answer come from somewhere else?

I edited a number in the middle of a local reasoning model's thinking on 144 maths problems. A third of the time it carried the error to a wrong answer without a flicker of doubt. It caught about half. And in 1 run in 10, its thinking concluded one answer while its reply quietly gave another.

Other topics: Measurement, Context, Verification, Skills, Review, Briefs, Parallel, Harness