Method
How every run on this site is graded, priced and timed.
Grading#
Hidden tests are tests the model never sees. A run passes its gate when every test passes, its own new ones included, and no existing test file was changed, renamed or deleted. New test files are fine.
On real app work, scores count every run and costs count the runs that passed the gate. On deep suite v4, a task is resolved when its hidden tests pass and the tests near it keep passing. On the explain task, an automatic check grades the answer.
Typical means the middle run, or the midpoint of the two middle runs.
Extras#
Extras are the plugins, skills, connectors and apps each CLI loads by default. The 30 September runs turned them off. Launch-day runs before that day ran with them on.
Effort#
Effort sets how much a model thinks before and between its steps. Claude Code picks it for each Claude model but Claude Haiku 5.5. I set it for the rest. The effort table lists each model's effort and CLI release.
Show the effort table
| Model | CLI | Effort | Set by |
|---|---|---|---|
| Claude Sonnet 5 | Claude Code 2.1.278, 2.1.283 and 2.1.284 | high | Claude Code's defaultdefault |
| Claude Opus 5.5 | Claude Code 2.1.280, 2.1.283 and 2.1.284 | medium | Claude Code's defaultdefault |
| Claude Sonnet 5.5 | Claude Code 2.1.284 | medium | Claude Code's defaultdefault |
| Claude Fable 5.1 | Claude Code 2.1.283 and 2.1.284 | high | Claude Code's defaultdefault |
| GPT-5.6 Luna | Codex CLI 0.153.3 | low | set by meset by me |
| GPT-6 Luna | Codex CLI 0.156.0 | low | set by meset by me |
| GPT-5.6 Luna, GPT-6 Luna, GPT-5.6 Sol, GPT-6 Sol and GPT-6 Astra | Codex CLI 0.157.1 | medium | set by meset by me |
| GPT-6.1 Sol, GPT-6 Sol, GPT-5.6 Sol, GPT-5.6 Luna, GPT-6 Luna and GPT-6 Astra | Codex CLI 0.159.0 | medium | set by meset by me |
| Claude Haiku 5.5 | Claude Code 2.1.284 | medium, and xhigh on the Haiku vs Luna page | set by meset by me |
| Claude Fable 5.2 | Claude Code 2.1.321 | medium | Claude Code's defaultdefault |
| GPT-6 Luna | Codex CLI 0.159.0 | xhigh, on the Haiku vs Luna page | set by meset by me |
| GPT-5.6 Luna and GPT-6 Luna | Codex CLI 0.159.0 | low, on the ASD-STE100 page | set by meset by me |
| Gemini 4 Argon | Cursor CLI 2026.10.01 | medium, on the Gemini 4 Argon page | set by meset by me |
| Muse Wonder | Cursor CLI 2026.10.01 | medium, on the Muse Wonder page | set by meset by me |
| Grok 4.7 | Cursor CLI 2026.10.01 | medium, on the Grok 4.7 page | set by meset by me |
| GPT-6.1 Terra | Codex CLI 0.159.0 | medium, on the GPT-6.1 Terra page | set by meset by me |
GPT-5.6 Luna, GPT-6 Luna, GPT-5.6 Sol, GPT-6 Sol and GPT-6 Astra appear more than once, because each row is one release and effort a model ran at.
Claude Code picks the effort for each Claude model but Claude Haiku 5.5. I logged the effort it sent on 28 September 2026, on Claude Code 2.1.284, and assume earlier releases sent the same. For Claude Haiku 5.5 and the GPT models I set the effort on every counted run, and Codex CLI's replies confirmed it. Cursor CLI names each effort as its own model, and I ran the medium one. Claude Fable 5.2, released on 9 November, ran on Claude Code 2.1.321. Every other Claude run on this site after 28 September used Claude Code 2.1.284, except three checks of newer releases, on 7 October, 22 October and 4 November: Claude Haiku 5.5's speed test on 2.1.293, Claude Opus 5.5's release check on 2.1.308 and a release check of Claude Opus 5.5 and Claude Sonnet 5.5 on 2.1.318.
List prices#
A run's cost is its tokens at the model's list prices, with Claude's cache writes at the one-hour rate Claude Code uses. These are estimates, not bills. Runs share a cache, so a run's cost depends partly on the runs before it. Each run is priced at the rates of its day.
A cost is a range when a log leaves out what sets the price. Codex CLI logs 0 cache writes, so a GPT run's uncached prompt tokens run from OpenAI's input rate to its cache-write rate, 1.25 times input. Anthropic charges more for a Claude Haiku 5.5 prompt over 100k tokens. xAI charges twice as much for every token once a prompt reaches 200k, and Cursor CLI reports only each run's totals, so a Grok 4.7 run that read that much costs a range.
The table scrolls sideways on a small screen.
| Model | Input | Cache read | Cache write | Output | Checked |
|---|---|---|---|---|---|
| GPT-6.1 Sol | $2.00 | $0.10 | not counted | $10.00 | 29 September 2026 |
| GPT-6 Sol | $2.00 | $0.20 | not counted | $10.00 | 29 September 2026 |
| GPT-6 Astra | $10.00 | $1.00 | not counted | $50.00 | 29 September 2026 |
| GPT-6 Luna | $0.10 | $0.01 | not counted | $0.50 | 29 September 2026 |
| GPT-5.6 Sol | $4.00 | $0.40 | not counted | $20.00 | 29 September 2026 |
| GPT-5.6 Luna | $0.20 | $0.02 | not counted | $1.20 | 29 September 2026 |
| Claude Opus 5.5 | $4.00 | $0.20 | $8.00 | $20.00 | 29 September 2026 |
| Claude Sonnet 5.5 | $2.00 | $0.20 | $4.00 | $10.00 | 29 September 2026 |
| Claude Sonnet 5 | $2.00 | $0.20 | $4.00 | $10.00 | 29 September 2026 |
| Claude Fable 5.1 | $10.00 | $0.25 | $20.00 | $50.00 | 29 September 2026 |
| Claude Haiku 5.5 | $0.10 | $0.01 | $0.20 | $0.50 | 7 October 2026 |
| Claude Haiku 5.5, a prompt over 100k tokens | $0.50 | $0.05 | $1.00 | $2.50 | 7 October 2026 |
| Muse Wonder | $4.00 | $0.20 | not counted | $20.00 | 16 October 2026 |
| Gemini 4 Argon | $2.00 | $0.10 | not counted | $10.00 | 16 October 2026 |
| Grok 4.7 | $2.00 | $0.50 | not counted | $6.00 | 24 October 2026 |
| Grok 4.7, a prompt of 200k tokens or more | $4.00 | $1.00 | not counted | $12.00 | 24 October 2026 |
| GPT-6.1 Terra | $1.00 | $0.05 | not counted | $5.00 | 26 October 2026 |
| Claude Fable 5.2 | $8.00 | $0.20 | $16.00 | $40.00 | 29 September 2026 |
US dollars per million tokens, standard tier. OpenAI's rates are for prompts up to 272K tokens. Gemini 4 Argon's rates double on 1 December 2026. Claude Sonnet 5.5's cache reads cost $0.10 per million from 7 October 2026 (the vendor's release notes, read 26 October 2026), and its runs from that day are priced at that rate. Sources: OpenAI's pricing, Anthropic's pricing, Anthropic's caching docs, Meta's pricing, Google's pricing and xAI's models page.
Time#
Time to finish is the whole run, from starting the tool to its exit, so it includes start-up. Two runs of the same model can differ by several seconds, and a gap of a second is a tie.
On the deep suite, minutes run from the tool's start to its exit, so a run stopped at the time limit can read a little over it. Output speed is an answer's tokens over the seconds from its first text to its last.