📈 SubDollarBench results
Budget LLMs build a SQL database, ticket by ticket, for at most $0.99 per phase. Graded against SQLite.
← Board
Updated 2026-11-09 09:21 UTC
What it measured
| Rules | 2026-10-19.45 |
|---|---|
| Graded under | 2026-10-19.45+bb49056367db125ab5017cd50b0fb9eb (15 careers)2026-10-19.45+ce36e53e4c6d9574ac2ab0512e82109c (3 careers) |
| Page built with | 2026-10-19.45+ce36e53e4c6d9574ac2ab0512e82109c |
| SQLite | 3.46.0 |
| Sandbox image | sha256:4402fd520bd6764023c381df7cb009df16c27ea2da9e5456dc7b2300f7d62d28 |
| Harness / effort | omp / high |
| Cap | cap policy: at list price for runs from .39; each career records its own; $0.99 per phase × 4 phases = $3.96; 21 tickets (0-20) |
| Prices | base rates per call from model_prices.json (snapshot 2026-11-08); overrides: anthropic/ |
| Grid dates | 2026-11-04 03:09 UTC → 2026-11-09 01:48 UTC |
| Careers | 18 |
| Methodology | docs/PROPOSAL-sdb-sql.md |
Fingerprints differ when the harness or the price list changed between careers; the rules version, and so the grading, is the same.
5 $0 rehearsals (the reference server; 71 / 76 each) are not ranked.
Points against dollars
One dot per career; the line is the non-dominated frontier. Career cap $3.96.
deepseek/deepseek-v4.1-flash-1017 xiaomi/mimo-v2.6-flash openai/gpt-6-luna-pro anthropic/claude-haiku-5.5 openai/gpt-6-luna qwen/qwen3.8-flash
| Model | Career | Points | Spend |
|---|---|---|---|
| deepseek/ | deepseek/deepseek-v4.1-flash-1017 · career 1 · 2026-11-04 | 70.0 | $2.39 |
| deepseek/ | deepseek/deepseek-v4.1-flash-1017 · career 3 · 2026-11-05 | 65.0 | $2.02 |
| deepseek/ | deepseek/deepseek-v4.1-flash-1017 · career 2 · 2026-11-04 | 63.0 | $2.69 |
| xiaomi/ | xiaomi/mimo-v2.6-flash · career 1 · 2026-11-04 | 67.0 | $2.60 |
| xiaomi/ | xiaomi/mimo-v2.6-flash · career 2 · 2026-11-04 | 62.8 | $2.94 |
| xiaomi/ | xiaomi/mimo-v2.6-flash · career 3 · 2026-11-05 | 24.4 | $1.96 |
| openai/ | openai/gpt-6-luna-pro · career 2 · 2026-11-08 | 62.2 | $2.46 |
| openai/ | openai/gpt-6-luna-pro · career 1 · 2026-11-08 | 58.4 | $2.55 |
| openai/ | openai/gpt-6-luna-pro · career 3 · 2026-11-08 | 55.8 | $2.35 |
| anthropic/ | anthropic/claude-haiku-5.5 · career 3 · 2026-11-05 | 64.6 | $2.14 |
| anthropic/ | anthropic/claude-haiku-5.5 · career 2 · 2026-11-05 | 57.6 | $2.47 |
| anthropic/ | anthropic/claude-haiku-5.5 · career 1 · 2026-11-04 | 55.6 | $2.55 |
| openai/ | openai/gpt-6-luna · career 2 · 2026-11-04 | 47.0 | $1.62 |
| openai/ | openai/gpt-6-luna · career 3 · 2026-11-06 | 44.0 | $1.60 |
| openai/ | openai/gpt-6-luna · career 1 · 2026-11-04 | 27.4 | $0.61 |
| qwen/ | qwen/qwen3.8-flash · career 1 · 2026-11-04 | 37.0 | $3.09 |
| qwen/ | qwen/qwen3.8-flash · career 2 · 2026-11-05 | 37.0 | $3.46 |
| qwen/ | qwen/qwen3.8-flash · career 3 · 2026-11-06 | 25.0 | $3.32 |
Survival: P(highest passed ticket ≥ i)
Each line follows the career's highest passed ticket; earlier tickets may have failed. The per-ticket table below shows each one.
deepseek/deepseek-v4.1-flash-1017 xiaomi/mimo-v2.6-flash openai/gpt-6-luna-pro anthropic/claude-haiku-5.5 openai/gpt-6-luna qwen/qwen3.8-flash
| Model | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| deepseek/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| xiaomi/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 67% | 67% | 67% | 67% | 67% |
| openai/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| anthropic/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 67% |
| openai/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 33% | 33% | 0% | 0% | 0% |
| qwen/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 67% | 67% | 0% |
Per-ticket pass share
| Model | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| deepseek/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 66% | 0% | 100% | 100% | 66% | 100% |
| xiaomi/ | 66% | 66% | 66% | 66% | 66% | 66% | 66% | 66% | 100% | 100% | 66% | 100% | 66% | 100% | 66% | 100% | 0% | 66% | 66% | 66% | 66% |
| openai/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 66% | 100% | 100% | 100% | 66% | 100% | 0% | 100% | 66% | 66% | 100% |
| anthropic/ | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 0% | 66% | 100% | 100% | 66% |
| openai/ | 100% | 100% | 66% | 100% | 100% | 66% | 66% | 100% | 100% | 100% | 33% | 100% | 66% | 100% | 66% | 100% | 0% | 33% | 0% | 0% | 0% |
| qwen/ | 100% | 33% | 33% | 33% | 100% | 66% | 66% | 66% | 33% | 100% | 0% | 66% | 66% | 100% | 33% | 0% | 0% | 66% | 0% | 66% | 0% |
Cost and robustness
| Model | $ / ticket (min / median / max) | Regression-free rate | Spend (list / billed) |
|---|---|---|---|
| deepseek/ | $0.11 / $0.12 / $0.14 | 100.0% | $7.10 / — |
| xiaomi/ | $0.13 / $0.16 / $0.28 | 100.0% | $7.50 / — |
| openai/ | $0.12 / $0.13 / $0.14 | 100.0% | $7.37 / $6.82 |
| anthropic/ | $0.11 / $0.13 / $0.13 | 100.0% | $7.16 / — |
| openai/ | $0.06 / $0.10 / $0.10 | 100.0% | $3.83 / — |
| qwen/ | $0.24 / $0.31 / $0.47 | 100.0% | $9.88 / — |
The cap counts list price. Billed is the provider's own charge, recorded from round 67 on; the v2 careers ran unpinned (+11% billed over list overall). No career recorded a regression: every passed ticket stayed green, so each model's rate is 100.0%.
Head-to-head
The share of career pairs where model A's career outscores model B's, by points.
| Model A | Model B | A wins | B wins | Ties |
|---|---|---|---|---|
| deepseek/ | xiaomi/ | 78% | 22% | 0% |
| deepseek/ | openai/ | 100% | 0% | 0% |
| deepseek/ | anthropic/ | 89% | 11% | 0% |
| deepseek/ | openai/ | 100% | 0% | 0% |
| deepseek/ | qwen/ | 100% | 0% | 0% |
| xiaomi/ | openai/ | 67% | 33% | 0% |
| xiaomi/ | anthropic/ | 56% | 44% | 0% |
| xiaomi/ | openai/ | 67% | 33% | 0% |
| xiaomi/ | qwen/ | 67% | 33% | 0% |
| openai/ | anthropic/ | 56% | 44% | 0% |
| openai/ | openai/ | 100% | 0% | 0% |
| openai/ | qwen/ | 100% | 0% | 0% |
| anthropic/ | openai/ | 100% | 0% | 0% |
| anthropic/ | qwen/ | 100% | 0% | 0% |
| openai/ | qwen/ | 78% | 22% | 0% |
Variance check
The publishing bar: the spread between models must exceed the spread within one model's careers.
| Metric | Median within-model spread | Span of model medians | Between / within |
|---|---|---|---|
| points | 10.5 | 28.0 | 2.7× |
| perf | 0.20× | 0.98× | 4.9× |
| $ | $0.54 | $1.50 | 2.8× |
| duration | 1.5 h | 11.4 h | 7.8× |
| turns | 233 | 1,166 | 5.0× |
| tokens out | 439.9K | 3.1M | 7.0× |
Model ranks
| # | Model | Best / median / worst points |
|---|---|---|
#1 | deepseek/ | 70.0 / 65.0 / 63.0 |
#2 | xiaomi/ | 67.0 / 62.8 / 24.4 |
#3 | openai/ | 62.2 / 58.4 / 55.8 |
#4 | anthropic/ | 64.6 / 57.6 / 55.6 |
#5 | openai/ | 47.0 / 44.0 / 27.4 |
#6 | qwen/ | 37.0 / 37.0 / 25.0 |