changelog

every round of runs, newest first. built from runs.json, so it only says what the data says. 2 rounds, 355 runs so far.

updated Oct 18, 2026

get the next entry by email

every new round lands here and in one email. what moved, one chart. free. read issue 1

Oct 18, 2026

Opus 5.5 and Haiku 5.5 added

same prompts and starter repos as Sonnet 5.5. 240 runs, 237 scored, about $26.81 in tokens. compared with Sonnet 5.5, a new model led with something else in database, deployment and monitoring. the pooled lead didn't move in any category.

where a new model didn't match Sonnet 5.5

database
Opus 5.5: Self-hosted Postgres, 8 of 12
Haiku 5.5: SQLite file, 7 of 12
before: SQLite file, 9 of 12
deployment
Opus 5.5: a tie, Fly.io, Heroku and Plain Docker at 4 of 12 each
Haiku 5.5: Plain Docker, 10 of 12
before: Plain Docker, 7 of 12
monitoring
Opus 5.5: built its own, 6 of 12
Haiku 5.5: Sentry, 5 of 12
before: Sentry, 12 of 12

same lead as before in ORM, auth, email, payments, jobs, caching and storage.

Oct 17, 2026

first live round, Sonnet 5.5

Claude Code on Sonnet 5.5, same prompts across 10 categories. 120 runs, 118 scored, about $11.74 in tokens.

what each category led with

database
SQLite file, 9 of 12
ORM
SQLAlchemy, 6 of 12
auth
Hand-rolled JWT or sessions, 12 of 12
deployment
Plain Docker, 7 of 12
email
Raw SMTP, 10 of 12
payments
Stripe, 12 of 12
jobs
built its own, 10 of 10 (2 as hand-rolled queue or cron)
caching
built its own, 12 of 12
storage
Local disk, 10 of 12
monitoring
Sentry, 12 of 12

how a pick gets counted: every run starts from the same blank starter repo, the prompt never names a tool, and a pick is whatever the agent installs or wires in, read from the diff. full rules on the methodology page, raw data in runs.json.