Physical-Engineering’s Last Exam Leaderboard

Comparing frontier models and agent harnesses on all 85 tasks by score, cost, and runtime.

Tasks: 85 in 4 disciplines
Runs: Three per pair
Time limit: 6 hours per task
Rubric judge: Claude Opus 5.5

Benchmark Score vs. cost per task on all 85 tasks

Score

Efficient Inefficient 0.000.050.100.150.200.250.300.350.400.450.500.55 $0.002$0.005$0.01$0.02$0.05$0.1$0.2$0.5$1$2$5$10$20$50$100 Opus 5.5 (claude-code) GPT 6 Astra (codex) Sonnet 5.5 (claude-code) Gemini 4 Argon (agy) Opus 5 (claude-code) Fable 5 (claude-code) Grok 4.7 (grok-build) Gemini 3.8 Flash (agy) DeepSeek-V4.1-Flash (dsh) Gemini 3.1 Pro (agy) ≈ Gemma 4 26B (agy) gpt-oss 120B (codex) GLM 5.2 (codex) Qwen3 Coder 480B (hermes) Kimi K2 Thinking (opencode) MiniMax M2 (claude-code) Qwen3-Next 80B (openhands)^† Llama 4 Maverick (openhands)^†

Cost per task (US$, log scale)

Harness’s Effect How far the harness moves each model’s score on all 85 tasks

ModelRange

0.00 0.10 0.20 0.30 0.40 0.50

Claude Opus 5.5

pi0.473 claude-code0.506

pi 0.473 to claude-code 0.506, average 0.488, native claude-code 0.506 range 0.033

Claude Sonnet 5.5

openhands0.422 agy0.452 claude-code0.436

openhands 0.422 to agy 0.452, average 0.433, native claude-code 0.436 range 0.030

Gemini 4 Argon

openhands0.311 claude-code0.432 agy0.417

openhands 0.311 to claude-code 0.432, average 0.394, native agy 0.417 range 0.121

Claude Opus 5

codex0.388 agy0.426 claude-code0.408

codex 0.388 to agy 0.426, average 0.406, native claude-code 0.408 range 0.038

Claude Fable 5

hermes0.328 claude-code0.349

hermes 0.328 to claude-code 0.349, average 0.339, native claude-code 0.349 range 0.021

Grok 4.7

codex0.262 pi0.319 grok-build0.320

codex 0.262 to pi 0.319, average 0.282, native grok-build 0.320 range 0.057

Gemini 3.8 Flash

hermes0.178 claude-code0.282 agy0.264

hermes 0.178 to claude-code 0.282, average 0.258, native agy 0.264 range 0.104

GLM 5.2

claude-code0.223 codex0.253

claude-code 0.223 to codex 0.253, average 0.237 range 0.030

DeepSeek-V4.1-Flash

codex0.047 opencode0.224 dsh0.153

codex 0.047 to opencode 0.224, average 0.115, native dsh 0.153 range 0.177

Gemini 3.1 Pro

opencode0.133 hermes0.142 agy0.137

opencode 0.133 to hermes 0.142, average 0.139, native agy 0.137 range 0.009

Qwen3 Coder 480B

opencode0.041 hermes0.062

opencode 0.041 to hermes 0.062, average 0.050 range 0.021

Kimi K2 Thinking

codex0.030 opencode0.059

codex 0.030 to opencode 0.059, average 0.044 range 0.029

MiniMax M2

agy0.037 claude-code0.053

agy 0.037 to claude-code 0.053, average 0.045 range 0.016

Gemma 4 26B

codex0.007 openhands0.047 agy0.045

codex 0.007 to openhands 0.047, average 0.032, native agy 0.045 range 0.040

gpt-oss 120B

opencode0.037 claude-code0.042 codex0.038

opencode 0.037 to claude-code 0.042, average 0.040, native codex 0.038 range 0.005

Qwen3-Next 80B

agy0.013 openhands0.037

agy 0.013 to openhands 0.037, average 0.022 range 0.024

Llama 4 Maverick

codex0.000 openhands0.007

codex 0.000 to openhands 0.007, average 0.003 range 0.007

Key takeaways

Claude Opus 5.5 (claude-code) leads all 85 tasks at 0.506. On the same model, openhands (0.499) and codex (0.493) are within run-to-run variation.
The model matters most. Averaged across all seven harnesses, model scores range from 0.003 (Llama 4 Maverick) to 0.488 (Claude Opus 5.5), whereas switching harnesses on a single model shifts its score by up to 0.177 (DeepSeek-V4.1-Flash, codex to opencode).
Cost varies far more than score on one model. On Claude Opus 5.5, a task costs 3× as much with agy ($42.32) as with opencode ($12.53). The top-scoring pair, Claude Opus 5.5 (claude-code), costs $17.18 per task.
Different pairs lead different disciplines: Opus 5.5 (claude-code) in structural and civil, Opus 5.5 (codex) in electrical and GPT 6 Astra (codex) in mechanical.

Rankings Each model on each harness, averaged over three runs

Each model's score with each harness, models ranked by their best score: All 85 tasks, overall
#	Model	pi	claude-code	opencode	agy	openhands	codex	hermes	Additional
1	Claude Opus 5.5Opus 5.5	0.473$13.3369 min	native harness: 0.506$17.1863 min	0.474$12.5346 min	0.484≈$42.3244 min	0.499$14.8265 min	0.493$35.15123 min	0.486$16.4267 min	–
2	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.483$9.1525 min	–	–
3	Claude Sonnet 5.5Sonnet 5.5	0.435$11.6892 min	native harness: 0.436$20.02100 min	0.423$10.4862 min	0.452≈$33.8952 min	0.422$15.28111 min	0.427$41.12236 min	0.437$14.7586 min	–
4	Gemini 4 ArgonGemini 4 Argon	0.402$6.4175 min	0.432$8.2166 min	0.384$6.8653 min	native harness: 0.417$7.2470 min	0.311$7.4382 min	0.403$7.0984 min	0.411$7.6159 min	–
5	Claude Opus 5Opus 5	0.404$15.0658 min	native harness: 0.408$17.2149 min	0.398$13.0038 min	0.426≈$29.5942 min	0.419$17.8269 min	0.388$22.3371 min	0.396$14.2952 min	–
6	Claude Fable 5Fable 5	0.330$20.2943 min	native harness: 0.349$26.8745 min	0.345$22.9237 min	0.338≈$54.1937 min	0.334$26.1143 min	0.347$29.2557 min	0.328$24.9356 min	–
7	Grok 4.7Grok 4.7	0.319$7.1038 min	0.268$12.6647 min	0.292$10.2733 min	0.287≈$16.81^†34 min	0.274$9.5133 min	0.262$6.9638 min	0.275$10.3848 min	grok-buildnative harness: 0.320$10.0652 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.269$2.5343 min	0.282$2.5534 min	0.260$2.3225 min	native harness: 0.264$2.6340 min	0.276$3.2132 min	0.276$3.5946 min	0.178$2.3523 min	–
9	GLM 5.2GLM 5.2	0.238$14.55116 min	0.223$16.28100 min	0.244–98 min	0.230≈$12.7186 min	0.229$16.62108 min	0.253$13.11121 min	0.239$11.1371 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.218$4.38166 min	0.120$3.73238 min	0.224$3.27171 min	0.088≈$3.50228 min	0.058$1.77196 min	0.047$2.09301 min	0.051$2.89246 min	dshnative harness: 0.153$5.90241 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.136$2.9130 min	0.141$3.9819 min	0.133$2.7515 min	native harness: 0.137$2.5516 min	0.140$5.4417 min	0.141$3.3518 min	0.142$2.3813 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.043$0.1410 min	0.056$0.139 min	0.041$0.207 min	0.045≈$0.497 min	0.057$0.2030 min	0.044$0.116 min	0.062$0.138 min	–
13	Kimi K2 ThinkingKimi K2 Thinking	0.042$0.219 min	0.044$0.257 min	0.059$0.299 min	0.035≈$0.2918 min	0.047$0.2511 min	0.030$0.278 min	0.052$0.298 min	–
14	MiniMax M2MiniMax M2	0.047$0.09414 min	0.053$0.1113 min	0.039$0.07814 min	0.037≈$0.3312 min	0.045$0.1214 min	0.044$0.09014 min	0.053$0.09910 min	–
15	Gemma 4 26BGemma 4 26B	0.037$0.2982 min	0.037$0.5292 min	0.032$0.4442 min	native harness: 0.045≈$0.2195 min	0.047$0.68111 min	0.007$0.1822 min	0.022$0.2823 min	–
16	gpt-oss 120Bgpt-oss 120B	0.041$0.1229 min	0.042$0.1146 min	0.037$0.1646 min	0.040–37 min	0.042$0.1555 min	native harness: 0.038$0.1948 min	0.039$0.1652 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.022$1.03^†56 min	0.025$0.53^†41 min	0.022$0.45^†66 min	0.013≈$0.40^†40 min	0.037$0.68^†82 min	0.015$0.52^†36 min	0.018$0.42^†50 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.007$0.26^†1 min	0.004$0.070^†1 min	0.000$0.004^†<1 min	0.001≈$0.060^†3 min	0.007$0.18^†2 min	0.000$0.093^†2 min	0.001$0.051^†1 min	–
	Harness average	0.204$5.9055 min	0.202$7.6757 min	0.200$5.3845 min	0.196$12.9551 min	0.191$7.0862 min	0.189$9.7373 min	0.188$6.3951 min
1	Claude Opus 5.5Opus 5.5	0.450$13.3369 min	native harness: 0.466$17.1863 min	0.444$12.5346 min	0.416≈$42.3244 min	0.466$14.8265 min	0.438$35.15123 min	0.408$16.4267 min	–
2	Claude Sonnet 5.5Sonnet 5.5	0.424$11.6892 min	native harness: 0.411$20.02100 min	0.440$10.4862 min	0.431≈$33.8952 min	0.438$15.28111 min	0.405$41.12236 min	0.426$14.7586 min	–
3	Claude Opus 5Opus 5	0.427$15.0658 min	native harness: 0.401$17.2149 min	0.411$13.0038 min	0.425≈$29.5942 min	0.428$17.8269 min	0.434$22.3371 min	0.418$14.2952 min	–
4	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.415$9.1525 min	–	–
5	Claude Fable 5Fable 5	0.397$20.2943 min	native harness: 0.350$26.8745 min	0.349$22.9237 min	0.334≈$54.1937 min	0.407$26.1143 min	0.398$29.2557 min	0.377$24.9356 min	–
6	Gemini 4 ArgonGemini 4 Argon	0.354$6.4175 min	0.395$8.2166 min	0.393$6.8653 min	native harness: 0.404$7.2470 min	0.295$7.4382 min	0.366$7.0984 min	0.348$7.6159 min	–
7	Gemini 3.8 FlashGemini 3.8 Flash	0.240$2.5343 min	0.287$2.5534 min	0.244$2.3225 min	native harness: 0.253$2.6340 min	0.291$3.2132 min	0.267$3.5946 min	0.085$2.3523 min	–
8	Grok 4.7Grok 4.7	0.289$7.1038 min	0.249$12.6647 min	0.265$10.2733 min	0.255≈$16.81^†34 min	0.256$9.5133 min	0.255$6.9638 min	0.231$10.3848 min	grok-buildnative harness: 0.263$10.0652 min
9	GLM 5.2GLM 5.2	0.226$14.55116 min	0.230$16.28100 min	0.234–98 min	0.213≈$12.7186 min	0.213$16.62108 min	0.250$13.11121 min	0.229$11.1371 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.174$4.38166 min	0.020$3.73238 min	0.226$3.27171 min	0.010≈$3.50228 min	0.006$1.77196 min	0.012$2.09301 min	0.000$2.89246 min	dshnative harness: 0.018$5.90241 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.097$2.9130 min	0.108$3.9819 min	0.100$2.7515 min	native harness: 0.070$2.5516 min	0.095$5.4417 min	0.098$3.3518 min	0.100$2.3813 min	–
12	Kimi K2 ThinkingKimi K2 Thinking	0.007$0.219 min	0.006$0.257 min	0.037$0.299 min	0.009≈$0.2918 min	0.004$0.2511 min	0.016$0.278 min	0.012$0.298 min	–
13	Qwen3 Coder 480BQwen3 Coder 480B	0.022$0.1410 min	0.019$0.139 min	0.018$0.207 min	0.013≈$0.497 min	0.028$0.2030 min	0.022$0.116 min	0.036$0.138 min	–
14	MiniMax M2MiniMax M2	0.013$0.09414 min	0.024$0.1113 min	0.002$0.07814 min	0.005≈$0.3312 min	0.004$0.1214 min	0.033$0.09014 min	0.018$0.09910 min	–
15	gpt-oss 120Bgpt-oss 120B	0.023$0.1229 min	0.012$0.1146 min	0.019$0.1646 min	0.020–37 min	0.021$0.1555 min	native harness: 0.011$0.1948 min	0.022$0.1652 min	–
16	Qwen3-Next 80BQwen3-Next 80B	0.001$1.03^†56 min	0.011$0.53^†41 min	0.000$0.45^†66 min	0.002≈$0.40^†40 min	0.007$0.68^†82 min	0.015$0.52^†36 min	0.005$0.42^†50 min	–
17	Gemma 4 26BGemma 4 26B	0.001$0.2982 min	0.000$0.5292 min	0.000$0.4442 min	native harness: 0.008≈$0.2195 min	0.006$0.68111 min	0.000$0.1822 min	0.000$0.2823 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.002$0.26^†1 min	0.000$0.070^†1 min	0.000$0.004^†<1 min	0.000≈$0.060^†3 min	0.002$0.18^†2 min	0.000$0.093^†2 min	0.000$0.051^†1 min	–
	Harness average	0.185$5.9055 min	0.176$7.6757 min	0.187$5.3845 min	0.169$12.9551 min	0.175$7.0862 min	0.178$9.7373 min	0.160$6.3951 min
1	Claude Opus 5.5Opus 5.5	0.607$13.3369 min	native harness: 0.699$17.1863 min	0.621$12.5346 min	0.621≈$42.3244 min	0.670$14.8265 min	0.678$35.15123 min	0.651$16.4267 min	–
2	Claude Sonnet 5.5Sonnet 5.5	0.538$11.6892 min	native harness: 0.531$20.02100 min	0.500$10.4862 min	0.556≈$33.8952 min	0.517$15.28111 min	0.531$41.12236 min	0.544$14.7586 min	–
3	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.555$9.1525 min	–	–
4	Claude Opus 5Opus 5	0.501$15.0658 min	native harness: 0.516$17.2149 min	0.451$13.0038 min	0.552≈$29.5942 min	0.531$17.8269 min	0.468$22.3371 min	0.471$14.2952 min	–
5	Gemini 4 ArgonGemini 4 Argon	0.509$6.4175 min	0.544$8.2166 min	0.480$6.8653 min	native harness: 0.526$7.2470 min	0.347$7.4382 min	0.518$7.0984 min	0.515$7.6159 min	–
6	Claude Fable 5Fable 5	0.370$20.2943 min	native harness: 0.418$26.8745 min	0.403$22.9237 min	0.410≈$54.1937 min	0.391$26.1143 min	0.426$29.2557 min	0.373$24.9356 min	–
7	Grok 4.7Grok 4.7	0.420$7.1038 min	0.281$12.6647 min	0.386$10.2733 min	0.375≈$16.81^†34 min	0.387$9.5133 min	0.372$6.9638 min	0.361$10.3848 min	grok-buildnative harness: 0.421$10.0652 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.315$2.5343 min	0.348$2.5534 min	0.282$2.3225 min	native harness: 0.305$2.6340 min	0.340$3.2132 min	0.331$3.5946 min	0.245$2.3523 min	–
9	GLM 5.2GLM 5.2	0.264$14.55116 min	0.250$16.28100 min	0.266–98 min	0.235≈$12.7186 min	0.255$16.62108 min	0.293$13.11121 min	0.273$11.1371 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.265$4.38166 min	0.157$3.73238 min	0.284$3.27171 min	0.095≈$3.50228 min	0.069$1.77196 min	0.055$2.09301 min	0.068$2.89246 min	dshnative harness: 0.176$5.90241 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.149$2.9130 min	0.140$3.9819 min	0.144$2.7515 min	native harness: 0.152$2.5516 min	0.147$5.4417 min	0.158$3.3518 min	0.166$2.3813 min	–
12	Gemma 4 26BGemma 4 26B	0.045$0.2982 min	0.040$0.5292 min	0.046$0.4442 min	native harness: 0.063≈$0.2195 min	0.054$0.68111 min	0.012$0.1822 min	0.033$0.2823 min	–
13	MiniMax M2MiniMax M2	0.055$0.09414 min	0.047$0.1113 min	0.033$0.07814 min	0.041≈$0.3312 min	0.044$0.1214 min	0.042$0.09014 min	0.052$0.09910 min	–
14	Kimi K2 ThinkingKimi K2 Thinking	0.048$0.219 min	0.042$0.257 min	0.044$0.299 min	0.027≈$0.2918 min	0.046$0.2511 min	0.020$0.278 min	0.050$0.298 min	–
15	Qwen3 Coder 480BQwen3 Coder 480B	0.034$0.1410 min	0.041$0.139 min	0.021$0.207 min	0.034≈$0.497 min	0.045$0.2030 min	0.044$0.116 min	0.048$0.138 min	–
16	gpt-oss 120Bgpt-oss 120B	0.038$0.1229 min	0.043$0.1146 min	0.031$0.1646 min	0.032–37 min	0.034$0.1555 min	native harness: 0.032$0.1948 min	0.046$0.1652 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.021$1.03^†56 min	0.021$0.53^†41 min	0.021$0.45^†66 min	0.016≈$0.40^†40 min	0.032$0.68^†82 min	0.021$0.52^†36 min	0.015$0.42^†50 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.004$0.26^†1 min	0.003$0.070^†1 min	0.000$0.004^†<1 min	0.001≈$0.060^†3 min	0.007$0.18^†2 min	0.000$0.093^†2 min	0.003$0.051^†1 min	–
	Harness average	0.246$5.9055 min	0.242$7.6757 min	0.236$5.3845 min	0.238$12.9551 min	0.230$7.0862 min	0.235$9.7373 min	0.230$6.3951 min
1	Claude Opus 5.5Opus 5.5	0.467$13.3369 min	native harness: 0.507$17.1863 min	0.479$12.5346 min	0.495≈$42.3244 min	0.506$14.8265 min	0.531$35.15123 min	0.519$16.4267 min	–
2	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.496$9.1525 min	–	–
3	Gemini 4 ArgonGemini 4 Argon	0.412$6.4175 min	0.453$8.2166 min	0.370$6.8653 min	native harness: 0.444$7.2470 min	0.335$7.4382 min	0.394$7.0984 min	0.436$7.6159 min	–
4	Claude Sonnet 5.5Sonnet 5.5	0.429$11.6892 min	native harness: 0.431$20.02100 min	0.416$10.4862 min	0.443≈$33.8952 min	0.401$15.28111 min	0.424$41.12236 min	0.424$14.7586 min	–
5	Claude Opus 5Opus 5	0.388$15.0658 min	native harness: 0.397$17.2149 min	0.396$13.0038 min	0.375≈$29.5942 min	0.393$17.8269 min	0.377$22.3371 min	0.382$14.2952 min	–
6	Claude Fable 5Fable 5	0.298$20.2943 min	native harness: 0.337$26.8745 min	0.329$22.9237 min	0.316≈$54.1937 min	0.309$26.1143 min	0.305$29.2557 min	0.296$24.9356 min	–
7	Grok 4.7Grok 4.7	0.319$7.1038 min	0.293$12.6647 min	0.280$10.2733 min	0.298≈$16.81^†34 min	0.257$9.5133 min	0.237$6.9638 min	0.277$10.3848 min	grok-buildnative harness: 0.334$10.0652 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.258$2.5343 min	0.225$2.5534 min	0.238$2.3225 min	native harness: 0.248$2.6340 min	0.249$3.2132 min	0.244$3.5946 min	0.172$2.3523 min	–
9	GLM 5.2GLM 5.2	0.210$14.55116 min	0.219$16.28100 min	0.237–98 min	0.225≈$12.7186 min	0.217$16.62108 min	0.241$13.11121 min	0.221$11.1371 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.208$4.38166 min	0.149$3.73238 min	0.207$3.27171 min	0.093≈$3.50228 min	0.051$1.77196 min	0.048$2.09301 min	0.049$2.89246 min	dshnative harness: 0.157$5.90241 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.139$2.9130 min	0.155$3.9819 min	0.145$2.7515 min	native harness: 0.159$2.5516 min	0.150$5.4417 min	0.151$3.3518 min	0.149$2.3813 min	–
12	Kimi K2 ThinkingKimi K2 Thinking	0.056$0.219 min	0.071$0.257 min	0.087$0.299 min	0.047≈$0.2918 min	0.067$0.2511 min	0.040$0.278 min	0.080$0.298 min	–
13	Qwen3 Coder 480BQwen3 Coder 480B	0.064$0.1410 min	0.083$0.139 min	0.065$0.207 min	0.070≈$0.497 min	0.074$0.2030 min	0.059$0.116 min	0.083$0.138 min	–
14	MiniMax M2MiniMax M2	0.069$0.09414 min	0.078$0.1113 min	0.072$0.07814 min	0.057≈$0.3312 min	0.067$0.1214 min	0.059$0.09014 min	0.063$0.09910 min	–
15	gpt-oss 120Bgpt-oss 120B	0.059$0.1229 min	0.048$0.1146 min	0.052$0.1646 min	0.052–37 min	0.049$0.1555 min	native harness: 0.040$0.1948 min	0.048$0.1652 min	–
16	Qwen3-Next 80BQwen3-Next 80B	0.039$1.03^†56 min	0.040$0.53^†41 min	0.039$0.45^†66 min	0.023≈$0.40^†40 min	0.055$0.68^†82 min	0.020$0.52^†36 min	0.027$0.42^†50 min	–
17	Gemma 4 26BGemma 4 26B	0.044$0.2982 min	0.044$0.5292 min	0.032$0.4442 min	native harness: 0.041≈$0.2195 min	0.053$0.68111 min	0.005$0.1822 min	0.028$0.2823 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.008$0.26^†1 min	0.004$0.070^†1 min	0.000$0.004^†<1 min	0.001≈$0.060^†3 min	0.010$0.18^†2 min	0.000$0.093^†2 min	0.000$0.051^†1 min	–
	Harness average	0.204$5.9055 min	0.208$7.6757 min	0.203$5.3845 min	0.199$12.9551 min	0.191$7.0862 min	0.187$9.7373 min	0.191$6.3951 min
1	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.450$9.1525 min	–	–
2	Claude Opus 5.5Opus 5.5	0.405$13.3369 min	native harness: 0.405$17.1863 min	0.392$12.5346 min	0.414≈$42.3244 min	0.402$14.8265 min	0.366$35.15123 min	0.384$16.4267 min	–
3	Claude Sonnet 5.5Sonnet 5.5	0.383$11.6892 min	native harness: 0.392$20.02100 min	0.376$10.4862 min	0.406≈$33.8952 min	0.378$15.28111 min	0.374$41.12236 min	0.387$14.7586 min	–
4	Claude Opus 5Opus 5	0.352$15.0658 min	native harness: 0.355$17.2149 min	0.364$13.0038 min	0.396≈$29.5942 min	0.371$17.8269 min	0.336$22.3371 min	0.356$14.2952 min	–
5	Gemini 4 ArgonGemini 4 Argon	0.344$6.4175 min	0.359$8.2166 min	0.336$6.8653 min	native harness: 0.332$7.2470 min	0.273$7.4382 min	0.355$7.0984 min	0.347$7.6159 min	–
6	Claude Fable 5Fable 5	0.314$20.2943 min	native harness: 0.317$26.8745 min	0.322$22.9237 min	0.317≈$54.1937 min	0.298$26.1143 min	0.320$29.2557 min	0.314$24.9356 min	–
7	Gemini 3.8 FlashGemini 3.8 Flash	0.261$2.5343 min	0.282$2.5534 min	0.272$2.3225 min	native harness: 0.256$2.6340 min	0.257$3.2132 min	0.275$3.5946 min	0.175$2.3523 min	–
8	Grok 4.7Grok 4.7	0.269$7.1038 min	0.245$12.6647 min	0.255$10.2733 min	0.235≈$16.81^†34 min	0.228$9.5133 min	0.220$6.9638 min	0.236$10.3848 min	grok-buildnative harness: 0.267$10.0652 min
9	GLM 5.2GLM 5.2	0.251$14.55116 min	0.207$16.28100 min	0.241–98 min	0.235≈$12.7186 min	0.231$16.62108 min	0.242$13.11121 min	0.239$11.1371 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.212$4.38166 min	0.108$3.73238 min	0.202$3.27171 min	0.104≈$3.50228 min	0.076$1.77196 min	0.052$2.09301 min	0.062$2.89246 min	dshnative harness: 0.180$5.90241 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.138$2.9130 min	0.139$3.9819 min	0.128$2.7515 min	native harness: 0.131$2.5516 min	0.142$5.4417 min	0.136$3.3518 min	0.137$2.3813 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.037$0.1410 min	0.055$0.139 min	0.040$0.207 min	0.040≈$0.497 min	0.059$0.2030 min	0.039$0.116 min	0.060$0.138 min	–
13	MiniMax M2MiniMax M2	0.032$0.09414 min	0.045$0.1113 min	0.026$0.07814 min	0.029≈$0.3312 min	0.039$0.1214 min	0.036$0.09014 min	0.055$0.09910 min	–
14	Gemma 4 26BGemma 4 26B	0.037$0.2982 min	0.043$0.5292 min	0.033$0.4442 min	native harness: 0.051≈$0.2195 min	0.052$0.68111 min	0.008$0.1822 min	0.017$0.2823 min	–
15	Kimi K2 ThinkingKimi K2 Thinking	0.038$0.219 min	0.033$0.257 min	0.050$0.299 min	0.037≈$0.2918 min	0.043$0.2511 min	0.033$0.278 min	0.041$0.298 min	–
16	gpt-oss 120Bgpt-oss 120B	0.031$0.1229 min	0.047$0.1146 min	0.034$0.1646 min	0.041–37 min	0.049$0.1555 min	native harness: 0.050$0.1948 min	0.034$0.1652 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.015$1.03^†56 min	0.017$0.53^†41 min	0.016$0.45^†66 min	0.005≈$0.40^†40 min	0.035$0.68^†82 min	0.007$0.52^†36 min	0.017$0.42^†50 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.009$0.26^†1 min	0.004$0.070^†1 min	0.000$0.004^†<1 min	0.001≈$0.060^†3 min	0.007$0.18^†2 min	0.001$0.093^†2 min	0.001$0.051^†1 min	–
	Harness average	0.184$5.9055 min	0.180$7.6757 min	0.182$5.3845 min	0.178$12.9551 min	0.173$7.0862 min	0.168$9.7373 min	0.168$6.3951 min

A stronger shade marks a higher score, cost, or time.
Models rank by their best harness (bold). marks a model's native harness. The harness average covers the 17 models scored on every harness.
85 tasks (0 to 1 scale). Claude Opus 5.5 grades the tasks that use a rubric.
Additional, a harness only that model runs: Grok 4.7 (grok-build), DeepSeek-V4.1-Flash (dsh); codex only: GPT 6 Astra.
Cost and time are per-task averages (≈ estimated cost; † no prompt caching).

How the models were run

Every model runs all 85 tasks on each of its harnesses (6 hours per task). Each pair runs three times, and its score averages its complete runs. Each model runs at its highest available reasoning effort.

Harnesses run at fixed releases: agy 1.3.3, claude-code 2.1.296, codex 0.162.1, hermes 0.22.2, opencode 2.0.26, openhands 1.54.0, pi 1.1.0, grok-build 1.0.52 and dsh 0.2.1-alpha.4.

Cost per task is calculated from token usage at list prices, including cached reads.

Reasoning effortper model

max: adaptive thinking at effort max (Opus 5.5, Sonnet 5.5, Opus 5, Fable 5); reasoning effort max (GLM 5.2, DeepSeek-V4.1-Flash, GPT 6 Astra)
high: thinking level HIGH (Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.1 Pro); reasoning effort high (Grok 4.7, MiniMax M2, Kimi K2 Thinking, gpt-oss 120B)
n/a: no reasoning setting (Qwen3 Coder 480B, Qwen3-Next 80B, Llama 4 Maverick)
thinking on: thinking enabled (Gemma 4 26B)

How grading works

Each task runs in an isolated container with the project's drawings, spreadsheets, specs, and local services.

Automated scripts check quantitative outputs, while an LLM judge grades open-ended engineering work against an expert rubric.

No internet access
6 hours per task
Three runs per harness and model