Deep suite
Version 4: 20 tasks from pull requests merged between GPT-6.1 Terra's release and Claude Fable 5.2's.
Share of tasks resolved
Higher is better. Each range is a 95% interval. Limits: 20 tasks, one run each (GPT-6.1 Terra and GPT-6 Luna: 18 tasks, 5 runs each, Claude Opus 5.5 and Claude Haiku 5.5: 18 tasks, 3 runs each). Single runs had a 30-minute cap, remeasured runs 2 hours. No gap between models is called significant here.
Source: WorkloadBench at workloadbench.brenhq.com, updated 11 November 2026. Deep suite v4, 328 runs at medium effort, run 31 October to 11 November 2026.
Notes
The values of the models remeasured 3 or 5 times leave out the 2 tasks that failed validation.
Tasks resolved against cost per task
20 tasks, one run per task, model and setting (GPT-6.1 Terra and GPT-6 Luna: 18 tasks, 5 runs each, Claude Fable 5.2, Claude Opus 5.5, Claude Haiku 5.5 and GPT-6 Luna at high: 18 tasks, 3 runs each). Higher is better, and left is cheaper. Each share's 95% interval is about 30 points wide.
- Medium
- High, GPT-6 Luna only
- xhigh
- Max
- Pareto frontier
Source: WorkloadBench at workloadbench.brenhq.com, updated 11 November 2026. Deep suite v4, 854 runs at medium, high, xhigh and max effort, run 31 October to 11 November 2026.
Notes
A task is resolved when the hidden tests for its change pass and the tests near it keep passing. Hidden tests are tests the model never sees.
A circle is a model at medium effort, a square GPT-6 Luna at high, a diamond at xhigh and a triangle at max, joined by a thin line. Each point's 95% Wilson interval is in its hover.
The dotted line is the Pareto frontier. No other point resolved at least as many tasks for less, or more for the same.
The values of the models remeasured 3 or 5 times leave out the 2 tasks that failed validation.
I dropped 5 tasks because their hidden tests asked for things the instructions didn't state.
Cost and minutes per task
Source: WorkloadBench at workloadbench.brenhq.com, updated 11 November 2026. Deep suite v4, 328 runs at medium effort, run 31 October to 11 November 2026.
Notes
The values of the models remeasured 3 or 5 times leave out the 2 tasks that failed validation.
Show as a table
| Model | GPT-6.1 Sol | Claude Sonnet 5.5 |
|---|---|---|
| Tasks resolved, medium | 13 of 20 | 14 of 20 |
| Tasks resolved, xhigh | 17 of 20 | 16 of 20 |
| 95% interval on the share, medium | 43% to 82% | 48% to 85% |
| 95% interval on the share, xhigh | 64% to 95% | 58% to 92% |
| Runs that hit the cap, medium | 0 | 0 |
| Runs that hit the cap, xhigh | 0 | 9 |
| Average cost per task, medium | $0.55 | $0.96 |
| Average cost per task, xhigh | $0.75 | $3.01 |
| Average minutes per task, medium | 7.8 | 10.0 |
| Average minutes per task, xhigh | 11.2 | 22.7 |
How I tested#
Show the method
Each task is a bug fix or a feature from a public open-source project, merged between GPT-6.1 Terra's release and Claude Fable 5.2's, and changes at least 4 files and 250 lines outside its tests. Hidden tests taken from the change grade it, less a few that call names the change itself added. The tasks are in Python (5), C++ (3), Java (3), Rust (3), TypeScript (3), JavaScript (2) and Go (1). Agents built the tasks, and I checked each without a model. The code before the change fails its tests, and the merged change passes them.
Single runs: one per task, model and setting, Codex CLI 0.159.0 and Claude Code 2.1.284, at medium effort and again at xhigh, with a 30-minute cap. 0 medium and 9 xhigh runs hit it, and I graded them on the code at that point. A capped run can read up to 32.0 minutes, because the coding tool kept running for a while after the cap told it to stop. I reran the 3 runs that stopped for a reason outside the model. Those runs are not results. The runs were 31 October to 7 November 2026. The remeasured models ran each of the 18 tasks 3 times per setting, 5 for GPT-6.1 Terra and GPT-6 Luna at medium and xhigh, with a 2-hour cap, and a run that hit it counts as a fail. I also ran Claude Sonnet 5.5 and Claude Haiku 5.5 at max and don't show them. Max means something different for each maker, and the cap cut off many of those runs.
30 runs were replaced and run again, and only the new ones count. Each is in results.json under deep_v4_suite.replaced.
Costs are at list prices as the vendors listed them on 26 October 2026. Neither tool logs usage for a run stopped at the cap, so for 9 Claude runs I added up each request's usage. GPT costs are the low end of what OpenAI could bill, because Codex logs no cache writes.
Every single run is in results.json, under deep_v4_runs and deep_v4_suite, and the remeasured models' totals under deep_v4_p1. Method covers grading, prices and time.