Changes
Every correction, new run and method change, newest first.
- Home: the lede now says every number is public, since the remeasured models' runs are published as totals.
- Correction: deep suite v4's tasks were merged before Claude Fable 5.2 came out, not after.
- Deep suite v4: the page and its source lines name the version.
- Deep suite v4: the Limits line names only the models at each setting and states both caps.
- Deep suite v1 to v3: each page names its version and says it is frozen.
- Home: Updated now counts the remeasured deep suite runs, so it shows the newest run day.
- Deep suite v1 to v3 moved to one page of earlier versions, linked once from the deep suite page.
- Deep suite v4: remeasured models' runs have a 2-hour cap, and a run that reaches it counts as a fail.
- Deep suite v4: chart sources and cap notes now count the remeasured models' runs too.
- Deep suite v4: Claude Opus 5.5 remeasured with 3 runs per task, 89% at medium and 93% at xhigh.
- Deep suite v4: GPT-6 Luna at high effort, 3 runs per task, solved 43%.
- Deep suite v4: Claude Haiku 5.5 remeasured with 3 runs per task, 70% at medium and 85% at xhigh.
- The top bar and link previews now show the three-bar mark beside the WorkloadBench name.
- Deep suite v4: every run regraded without its own test files, so hidden tests grade only the model's code.
- Deep suite v4: Claude Fable 5.2 at xhigh now solved 51 of 54.
- Deep suite v4: GPT-6.1 Terra now solved 41, 63 and 73 of 90 at medium, xhigh and max.
- GPT-6 Luna: a page of its own, with its deep suite runs by kind of task.
- Deep suite v4: GPT-6 Luna remeasured with 5 runs per task, 24% at medium and 62% at xhigh.
- Deep suite: Claude Fable 5.2 at xhigh, 3 runs per task.
- Effort findings now show tasks better, worse and the same.
- Correction: GPT-6.1 Terra's max gain over xhigh could be chance. Effort comparisons now use an exact paired test.
- Deep suite charts now include GPT-6.1 Terra at max effort, and its page charts max by kind of task.
- GPT-6.1 Terra at max effort on the deep suite: 84% against 72% at xhigh.
- Quick suite: the explain board shows each model's newest timed sitting and its day, and six models were retimed.
- Model pages: each model's own results now sit on its page, and the suite pages show shared results only.
- GPT-6.1 Terra: a chart by kind of task on its own page, big changes against small fixes.
- Claude Fable 5.2: a new page with its quick suite runs from release day.
- GPT-6.1 Terra is back in every deep suite chart, averaged over 5 runs.
- Deep suite v4: a model remeasured under the new method now shows only in the new-method table.
- Deep suite v4: GPT-6.1 Terra remeasured with 5 runs per task. xhigh 72% against medium 47%, a difference beyond chance.
- Deep suite v4: effort comparisons were single runs. I'm remeasuring them with 3 to 5 runs per task.
- Deep suite: in some rounds the 30-minute cap ran up to 2 minutes long.
- Deep suite v4: the 22 runs under the earlier 20-minute cap were all Claude runs.
- Deep suite v4: one Claude Opus 5.5 xhigh run was graded without work it had stashed. I reran it.
- Deep suite v4: the rerun solves Task 12, so Claude Opus 5.5 at xhigh solves 18 tasks, up from 17.
- Deep suite: a setup fault on my side sometimes kept GPT models in Codex from running build and test tools.
- Deep suite: I fixed it and reran the 64 GPT runs on the affected tasks. Only the reruns count.
- Deep suite v4 at medium: GPT-6.1 Sol solves 13 tasks, down from 14, and GPT-6.1 Terra 7, down from 8.
- Deep suite v4 at xhigh: GPT-6.1 Terra solves 11 tasks, up from 9, and GPT-6 Luna 11, down from 12.
- Deep suite v1: GPT-6 Luna now passes 11 of 24 runs, up from 9. GPT-6.1 Sol's reruns changed nothing.
- Deep suite v3: model averages are within 0.01 of before. No Claude result changed. Rerun times moved slightly.
- Deep suite: 5 tasks dropped. Their hidden tests asked for things the instructions didn't state. 20 tasks remain.
- Deep suite: all kept runs were re-graded with the current grader, and no result changed.
- Deep suite: Claude Opus 5.5 at medium run again on the same 25 tasks. It resolved the same 18.
- Deep suite: Claude Opus 5.5 at high effort, beside its medium and xhigh runs on the same 25 tasks.
- Every chart and table the trim removed is back, long ones in folds. The text cuts stay.
- Every page is shorter: one finding per chart, method on its own page, launch pages share one layout.
- New page: Method, with how runs are graded, priced and timed, and the price table.
- Changes move to this page, and the full run lists move to the data file.
- Claude Code 2.1.318 recheck: Claude Opus 5.5 and Claude Sonnet 5.5 kept their typical scores.
- Deep suite grows to 25 tasks, each run by all six models at medium and xhigh.
- I dropped a new deep suite task whose hidden tests contradicted its description, and replaced it.
- Deep suite drops model and tool minutes, which compared unlike measures.
- Claude Haiku 5.5 joins deep suite v4 on all 19 tasks.
- Deep suite v4 also runs at xhigh effort, the same 19 tasks for all six models.
- Deep suite correction, GPT runs redone: GPT-6.1 Sol was 12 of 19, now 11. GPT-6.1 Terra was 6, now 7.
- GPT-6 Luna joins deep suite v4 on all 19 tasks.
- Deep suite v4 is out, and the Deep suite tab opens on it.
- GPT-6 Astra joins the Quick suite's bug hunt and tickets.
- New page: GPT-6.1 Terra on launch day in Codex CLI, with speed, bug hunt and tickets.
- New page: Grok 4.7, xAI's model, in Cursor CLI through the quick suite.
- Correction: Claude Opus 5.5's typical bug hunt cost was $0.782, now $0.783.
- Correction: Claude Opus 5.5's 22 September runs were listed on Claude Code 2.1.284, now 2.1.280.
- New page: History, merging New models and Rechecks into one list.
- Quick suite real app work adds Claude Haiku 5.5, GPT-6 Luna, Gemini 4 Argon and Muse Wonder.
- Correction: the explain task's cost spread was 96 to 116x, now 110 to 116x.
- Correction: deep suite axis labels read 0.3, 0.8 and 1.3, now 0.25, 0.75 and 1.25.
- Correction: on the deep suite Web app task, Claude Opus 5.5 tied GPT-6.1 Sol at 1.00.
- New page: Models, listing every model on the site.
- The Claude Opus 5.5 recheck adds Claude Code 2.1.308 beside 2.1.284, three runs each.
- Correction: the GPT-6.1 Sol recheck said three of four small tasks changed setup, now all four.
- Correction: earlier runs used Claude Code 2.1.278 to 2.1.293, not only 2.1.284.
- New section, Rechecks: Claude Opus 5.5 after posts said it got worse, and GPT-6.1 Sol.
- Correction: the Claude Opus 5.5 recheck no longer says its thinking fell to a third.
- The site is now WorkloadBench, at workloadbench.brenhq.com.
- Correction: real app costs use passed runs only. Gemini 4 Argon's bug hunt cost was $0.52, now $0.44.
- New page: deep suite cache use, and the cost with nothing cached.
- Claude Sonnet 5.5's cache reads now list at $0.10. Its runs stay priced at $0.20.
- Launch days became New models, and the Models tab became Quick suite.
- Deep suite v3: six of my own tasks in six languages, two runs each.
- New page: Muse Wonder, Meta's model, in Cursor CLI through the launch suite.
- Gemini 4 Argon's runs now carry costs at Google's published prices.
- Deep suite v2: my own tasks in one real app, three runs each.
- New page: the Deep suite, eight harder tasks from three public benchmarks, for four models.
- GPT costs are now a range, from OpenAI's input rate to its cache write rate.
- New page: Gemini 4 Argon on launch day in Cursor CLI.
Earlier
- New page: Claude Haiku 5.5 against GPT-6 Luna on launch day, at medium and xhigh.
- Haiku vs Luna TPS now uses a long answer: GPT-6 Luna 122 and 126, Claude Haiku 5.5 239 and 297.
- GPT-6.1 Sol: four runs on 6 October took 10.3 to 15.7 s.
- Beside GPT-6.1 Sol, Claude Sonnet 5.5 took 7.8 to 9.8 s and Claude Opus 5.5 15.2 to 17.7 s.
- GPT-6.1 Sol: four sets of four runs on 5 October, before and after OpenAI's speed change.
- Correction: the GPT-6.1 Sol page's tokens a second was not TPS. True TPS now uses OpenAI's token count.
- Correction: a brief TPS column timed OpenAI's engine, 3.7 to 3.9 times the delivered rate. I removed it.
- Headlines claim a cut or a win only when runs do not overlap. Three headlines changed.
- GPT-6.1 Sol: four runs on 4 October took 13.2 to 22.9 s.
- GPT-6.1 Sol: four runs on 3 October took 15.6 to 26.6 s.
- New page: Karpathy's ASD-STE100 tip, measured on six models.
- New page: GPT-6.1 Sol timed three days running. Runs on 1 October took 13.5 to 16.8 s.
- Correction: some Claude Code runs started before its extras loaded and read low. They now carry a star.
- Correction: the Claude run count in the 29 September cost note was 254, now 194.
- I reran every model's explain task, real app work and timed runs with the extras off.
- With extras off, caveman's cut on Claude Sonnet 5.5 was 13% at launch, now 28%.
- GPT runs on the Skills page are now priced at OpenAI's published rates.
- Correction: Claude Sonnet 5.5's release day was 30 September, now 28 September.
- Correction: caveman on Claude Sonnet 5 showed three sets, now all four at 66%, 62%, 64% and 72%.
- GPT-6.1 Sol ran on launch day at medium effort, then joined the skill runs.
- Every run shows its cost at published API prices.
- New real app work: a bug hunt with 20 planted bugs and tickets with 17 hidden tests.
- Correction: caveman's README dropped its 65% figure on 8 September. I now check its GitHub description.
- New pages: Skills and Models. Models shows every model's tokens, time and output speed.
- Six more models, including Claude Fable 5.1 and GPT-6 Astra, ran with and without caveman.
- Codex CLI runs moved from low to medium effort. Earlier runs stay as they ran.
- Codex CLI token counts now include its hidden first request, 9,751 to 12,371 tokens a run.
- Claude Sonnet 5.5 ran every task and skill on launch day.
- Every model was timed on the explain task, two runs each.
- I changed the explain task's answer check after two trial runs, before any counted run.
- First skill tests: 144 runs of caveman, ponytail, karpathy-skills and my control file. Every run passed.
- Correction: the first write-up pooled the tasks and misquoted caveman's claim.
- Claude Opus 5.5 and GPT-6 Luna ran 108 runs. GPT-6 Luna failed 9 of 15 explain runs.
- Correction: ponytail's token claim was checked against output tokens, now against total tokens.