Six experiments. Each had its kill condition written down, with numbers, before the run. Four came back against us. The one that didn't is at the top — it is also the only thing here we'd call a product claim.
Two frontier models played the frozen three-opponent pool twice: once alone, once with the SNHP engine handing them a recommendation each turn. Same scenarios, same opponents, same inference settings — the only difference is the advice.
Unaided, Claude Haiku 4.5 ended 43% of 360 negotiations no better than walking away. Advised: 19%. Sonnet 5 went 29% → 21%. Own-utility rose for both (+0.058 Haiku, +0.037 Sonnet). Every one of those cells is p=0.0001, and all of it repeats on a held-out set never used before, where Haiku goes 45% → 16%. It closes more, too: against an opponent that never concedes, deal rate goes 28% → 56%.
What we got wrong, since that is the point of registering. We predicted advice would barely move Sonnet — it moved it clearly. We predicted both registered downside statistics would improve; only one did. The worst-decile outcome (CVaR@10) is unchanged in all four cells. So the honest claim is narrower than the one we wrote down: SNHP cuts how often you fall to the floor. It does not raise the floor.
Registered in full — statistics, kill condition, and both predictions — before a single advised match ran. PREREG-pool.md Amendment 3 · full results
python -m arena.gauntlet.llm_tier --amendment 3 # the paid run (checkpointed) python -m arena.gauntlet.advice_analysis # the verdict, no LLM
We expected a buying agent under pressure — fake urgency, hidden fees, creeping anchors — to eventually accept above list price.
Unguarded frontier models never broke: zero above-list acceptances across 144 adversarial sessions, in both regimes, against six attacks each verified to bind. The guard's own zero-leak number is mechanical — a property of the function, not a capability — so it proves nothing alone.
python -m research.spendguard.run --full python -m research.spendguard.analyze python -m research.spendguard.run --full --regime blind python -m research.spendguard.analyze --regime blind
We predicted you'd need package deals to close machine trades. Price-only haggling got to 94.2% of our deal rate — the registered kill bar was 80%. We retracted the claim and kept the spec, because packages still capture more surplus (0.928 vs 0.889 price-only, 0.880 plain checkout).
Under tight deadlines price-only recovers only 0.696 of NX's deal formation — that re-run is exploratory, not pre-registered, and says so when you run it. nx/SPEC.md · conformance suite
python -m nx.experiment python -m nx.experiment --regime constrained # exploratory, self-labelled python -m pytest nx/test_nx.py -q # conformance
Design #1 measured the pair. Against a counterparty that finds good trades from its own side, even a plain anchor-and-split bargainer rides along. Registered bar p<0.01; the public set gave p=0.0128 and the held-out set had the naive baseline ahead (p=0.76). Retired.
Design #2 was picked after seeing the data that suggested it, which makes it worthless without a held-out check. We ran one and it failed: the naive baseline was ahead, 0.6662 vs 0.6152, p=0.43. Thrown away.
Design #3 changed the opponent, not the metric — three fixed scripted opponents — and cleared a held-out set that had never been used: gap +0.1086, p=0.0001, n=360, and against all three opponents individually. See the certificate →
python -m arena.gauntlet.separation # designs #1 and #2 python -m arena.gauntlet.pool_experiment # design #3 python -m arena.gauntlet.certify --run engine
We put two unaided frontier models — Claude Sonnet 5 and Haiku 4.5 — through the same frozen pool the engine is certified against, scored on the same own-utility metric. We wrote the prediction down first: that neither would beat split-the-difference. It was wrong. Both separate upward on both scenario sets — Sonnet +0.095 / +0.072, Haiku +0.059 / +0.060, every p=0.0001 — a margin near the engine's own certified edge (+0.107 / +0.109).
So a raw model, just handed the scoresheet, does most of what the engine
does against these scripted opponents. That is a real, pre-registered miss
on our part, and we report it straight. It does not touch the
certified claim — the engine still separates (section 1) — and these rows
are reported, never signed. One caveat we log rather than bury: Sonnet was
measured with the API's default thinking on and Haiku without, so the two
are not under identical inference config. The older runs on the retired
capture metric are kept as the honest record. Detail:
arena/gauntlet/certs/CERTIFIED-TABLE.md, section 3.
Each one is a research artifact, not a product. They run in a browser — full index →