Sonnet 5.5 came close to Opus 5.5 at 41% of the cost

The After bot answered all 64 cases with each model, graded by the same Opus judge, 3 runs per case. Sonnet 5.5 passed 165 of 192 runs and Opus 5.5 174, at $9.57 against $23.49 per 1,000 replies. Haiku 5.5 cost $0.82 and passed 129. The judge itself is steady: grading the same replies twice changed few verdicts.

Cost and quality

Passing runs, and what 1,000 replies cost

Cost is the answering model plus the Haiku router at list prices, per 1,000 replies with a warm cache. The eval's own judge is not included.

Haiku 5.5
$0.82
129 of 192 passing runs
Needs a second domain
63 of 99
One domain
66 of 93
Expected facts present
90%
Two runs disagree
27%
Answer + router per 1,000
$0.50 + $0.32
Sonnet 5.5
$9.57
165 of 192 passing runs
Needs a second domain
78 of 99
One domain
87 of 93
Expected facts present
96%
Two runs disagree
10%
Answer + router per 1,000
$9.25 + $0.32
Opus 5.5
$23.49
174 of 192 passing runs
Needs a second domain
85 of 99
One domain
89 of 93
Expected facts present
98%
Two runs disagree
10%
Answer + router per 1,000
$23.17 + $0.32

Is the gap bigger than run-to-run noise?

A run-permutation test by case, on how often each model passed each case:

Haiku 5.5 to Sonnet 5.5: 31 cases pass more often on Sonnet 5.5, 6 more often on Haiku 5.5, p < 0.001; counting cases instead of runs, p < 0.001.
Sonnet 5.5 to Opus 5.5: 8 cases pass more often on Opus 5.5, 6 more often on Sonnet 5.5, p = 0.099; counting cases instead of runs, p = 0.790.
Haiku 5.5 to Opus 5.5: 30 cases pass more often on Opus 5.5, 3 more often on Haiku 5.5, p < 0.001; counting cases instead of runs, p < 0.001.

So Sonnet 5.5 and Opus 5.5 could not be told apart with 3 runs per case. That is not proof they are equally good: a gap of a few cases would not show at this size. The run test holds these 64 cases fixed; counting cases as a sample of questions gives a much weaker signal still.

Reply length

The rule nothing grades

The response rules ask for replies under about 250 words, but neither the code checks nor the judge look at length, and the judge rewards complete answers. If replies over a cutoff counted as failures, which model comes out ahead would depend on where the line is drawn, so length is a caveat on the ranking, not a way to settle it.

Answer model	Median words	Over 250 words	Passing runs
Haiku 5.5	238	84 of 192	129
Sonnet 5.5	270.5	128 of 192	165
Opus 5.5	269	141 of 192	174

Passing runs if replies over the cutoff counted as failures
Cutoff	Haiku 5.5	Sonnet 5.5	Opus 5.5
200 words	50	25	16
250 words (the rule)	79	58	50
300 words	102	137	161
350 words	121	164	173

Words are counted by splitting on spaces, so markdown markers count too.

The judge

Grading the same replies again

Every pass or fail above comes from one Opus judge. To check it, 192 stored replies from an earlier single run of each model were graded again by Opus and by two cheaper models, with no bot run again.

The same Opus judge changed 6 of 192 verdicts when it graded again.

So the judge is a small part of the run-to-run noise: two runs of the same model disagree on 10% to 27% of cases, against 3% of verdicts for the judge alone. Most cases that flip between runs flip because the answer changed.

Judge	Facts graded the same as the first Opus pass	Verdicts changed	Cost to grade 64 replies
Opus 5.5, graded again	99%	6 of 192	$1.16
Haiku 5.5	97%	17 of 192	$0.04
Sonnet 5.5	96%	17 of 192	$0.50

Agreeing with Opus is not the same as being right. So a person also graded a blind sample of the replies, fact by fact, without seeing the judge's verdict or the answering model.

On 13 sampled replies (48 facts), the eval's judge agreed with the person on 92% of facts (86% to 98%); in all 4 disagreements it failed a fact the person passed.

The person passed every reply the judge passed, and would also have passed 4 of the 7 it failed. In those, the reply left out one detail of a fact, such as how often free reports are available or that the effect depends on the scoring model, that the person let pass and the judge did not. So the pass rates on this site are likely strict.

The replies were drawn at random within each answer model and the judge's own pass or fail, which over-samples failures; weighted back to each group's real share, agreement is 94%. On the unweighted sample, Cohen's kappa is 0.63. Whole verdicts agreed on 9 of 13 replies. Among the extra replies picked because the judge looked inconsistent, it once passed a fact the person marked missing. Every labeled reply came from the After bot, so whether the judge is as strict with Before's replies was not measured.

Haiku 5.5 agreed with the person on 2 more facts, by letting some of the same details pass: too few replies to choose a judge by.

Judge against a person's labels	Facts	Agree	Stricter
Opus 5.5 (the eval's judge)	48	92% (86% to 98%)	4
Haiku 5.5	48	96% (90% to 100.0%)	2
Opus 5.5, graded again	48	92% (86% to 98%)	4
Sonnet 5.5	48	92% (85% to 98%)	4

Answer-model runs from 2026-10-23 20:00 UTC, 3 runs per case, with the skills in place then, every model at low effort, so the comparison holds for that setting only. Cost assumes a warm cache: the eval reuses six members, so their profiles are read from the cache; a reply to a member seen for the first time costs roughly a fifth more with Sonnet or Opus. The judge regrades are of earlier single runs (after-haiku, after-opus, after-sonnet). List prices; the router's share is recorded per run.