Physical-Engineering’s Last Exam Leaderboard

Comparing frontier models and agent harnesses on the 29 public dev tasks by score, cost, and runtime.

Tasks: 29 in 4 disciplines
Runs: Three per pair
Time limit: 6 hours per task
Rubric judge: Claude Opus 5.5

Benchmark Score vs. cost per task on the 29 public dev tasks

Score

Efficient Inefficient 0.000.050.100.150.200.250.300.350.400.450.500.550.60 $0.002$0.005$0.01$0.02$0.05$0.1$0.2$0.5$1$2$5$10$20$50$100 Opus 5.5 (claude-code) GPT 6 Astra (codex) Gemini 4 Argon (agy) Sonnet 5.5 (claude-code) Opus 5 (claude-code) Grok 4.7 (grok-build) Fable 5 (claude-code) Gemini 3.8 Flash (agy) DeepSeek-V4.1-Flash (dsh) Gemini 3.1 Pro (agy) ≈ Gemma 4 26B (agy) gpt-oss 120B (codex) GLM 5.2 (codex) Qwen3 Coder 480B (hermes) Kimi K2 Thinking (hermes) MiniMax M2 (pi) Qwen3-Next 80B (openhands)^† Llama 4 Maverick (openhands)^†

Cost per task (US$, log scale)

Harness’s Effect How far the harness moves each model’s score on the 29 public dev tasks

ModelRange

0.00 0.10 0.20 0.30 0.40 0.50 0.60

Claude Opus 5.5

pi0.474 claude-code0.551

pi 0.474 to claude-code 0.551, average 0.516, native claude-code 0.551 range 0.077

Gemini 4 Argon

openhands0.350 claude-code0.473 agy0.439

openhands 0.350 to claude-code 0.473, average 0.432, native agy 0.439 range 0.123

Claude Sonnet 5.5

hermes0.410 agy0.460 claude-code0.438

hermes 0.410 to agy 0.460, average 0.431, native claude-code 0.438 range 0.050

Claude Opus 5

pi0.377 agy0.446 claude-code0.422

pi 0.377 to agy 0.446, average 0.401, native claude-code 0.422 range 0.069

Claude Fable 5

pi0.307 agy0.365 claude-code0.356

pi 0.307 to agy 0.365, average 0.341, native claude-code 0.356 range 0.058

Grok 4.7

codex0.267 pi0.353 grok-build0.357

codex 0.267 to pi 0.353, average 0.298, native grok-build 0.357 range 0.086

Gemini 3.8 Flash

hermes0.186 claude-code0.303 agy0.243

hermes 0.186 to claude-code 0.303, average 0.257, native agy 0.243 range 0.117

GLM 5.2

pi0.213 codex0.260

pi 0.213 to codex 0.260, average 0.233 range 0.047

DeepSeek-V4.1-Flash

hermes0.047 opencode0.221 dsh0.170

hermes 0.047 to opencode 0.221, average 0.113, native dsh 0.170 range 0.174

Gemini 3.1 Pro

opencode0.118 agy0.135

opencode 0.118 to agy 0.135, average 0.129, native agy 0.135 range 0.017

Qwen3 Coder 480B

opencode0.034 hermes0.061

opencode 0.034 to hermes 0.061, average 0.046 range 0.027

Kimi K2 Thinking

codex0.027 hermes0.057

codex 0.027 to hermes 0.057, average 0.045 range 0.030

Gemma 4 26B

codex0.007 openhands0.053 agy0.052

codex 0.007 to openhands 0.053, average 0.036, native agy 0.052 range 0.046

MiniMax M2

openhands0.030 pi0.048

openhands 0.030 to pi 0.048, average 0.041 range 0.018

gpt-oss 120B

agy0.036 hermes0.045 codex0.038

agy 0.036 to hermes 0.045, average 0.040, native codex 0.038 range 0.009

Qwen3-Next 80B

agy0.009 openhands0.026

agy 0.009 to openhands 0.026, average 0.018 range 0.017

Llama 4 Maverick

codex0.000 openhands0.007

codex 0.000 to openhands 0.007, average 0.003 range 0.007

Key takeaways

Claude Opus 5.5 (claude-code) leads the 29 public dev tasks at 0.551. Claude Opus 5.5 (codex) at 0.549 and GPT 6 Astra (codex) at 0.530 are within run-to-run variation.
The model matters most. Averaged across all seven harnesses, model scores range from 0.003 (Llama 4 Maverick) to 0.516 (Claude Opus 5.5), whereas switching harnesses on a single model shifts its score by up to 0.174 (DeepSeek-V4.1-Flash, hermes to opencode).
Cost varies far more than score on one model. On Claude Opus 5.5, a task costs 4× as much with agy ($42.32) as with opencode ($11.87). The top-scoring pair, Claude Opus 5.5 (claude-code), costs $17.30 per task.
Different pairs lead different disciplines: Sonnet 5.5 (pi) in structural, Opus 5.5 (codex) in civil and GPT 6 Astra (codex) in electrical and mechanical.

Rankings Each model on each harness, averaged over three runs

Each model's score with each harness, models ranked by their best score: The 29 public dev tasks, overall
#	Model	claude-code	pi	opencode	agy	codex	hermes	openhands	Additional
1	Claude Opus 5.5Opus 5.5	native harness: 0.551$17.3063 min	0.474$13.2971 min	0.492$11.8741 min	0.496≈$42.3242 min	0.549$30.38110 min	0.526$17.1770 min	0.525$14.7264 min	–
2	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	native harness: 0.530$8.4224 min	–	–	–
3	Gemini 4 ArgonGemini 4 Argon	0.473$7.6262 min	0.471$5.2970 min	0.406$6.4950 min	native harness: 0.439$6.3663 min	0.431$7.0085 min	0.454$7.0055 min	0.350$7.2378 min	–
4	Claude Sonnet 5.5Sonnet 5.5	native harness: 0.438$22.61101 min	0.423$11.5184 min	0.418$10.4060 min	0.460≈$33.8952 min	0.419$38.34224 min	0.410$14.8283 min	0.449$14.43109 min	–
5	Claude Opus 5Opus 5	native harness: 0.422$18.5351 min	0.377$14.6557 min	0.387$13.9638 min	0.446≈$29.5940 min	0.388$25.3676 min	0.381$15.0553 min	0.405$19.1474 min	–
6	Claude Fable 5Fable 5	native harness: 0.356$27.6645 min	0.307$18.5934 min	0.353$22.8736 min	0.365≈$54.1935 min	0.350$30.1951 min	0.318$26.1559 min	0.337$26.6440 min	–
7	Grok 4.7Grok 4.7	0.274$12.4945 min	0.353$5.3535 min	0.307$10.3032 min	0.304≈$16.81^†35 min	0.267$6.8534 min	0.311$10.9551 min	0.272$10.7233 min	grok-buildnative harness: 0.357$9.8250 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.303$2.4332 min	0.270$2.2445 min	0.261$2.0624 min	native harness: 0.243$2.6839 min	0.272$3.5648 min	0.186$2.2623 min	0.265$3.2031 min	–
9	GLM 5.2GLM 5.2	0.231$19.17110 min	0.213$16.25129 min	0.252–103 min	0.214≈$12.7187 min	0.260$14.14121 min	0.238$11.7173 min	0.221$18.77114 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.128$3.71231 min	0.203$4.07160 min	0.221$3.00172 min	0.080≈$3.50228 min	0.059$2.10314 min	0.047$2.71230 min	0.050$2.00201 min	dshnative harness: 0.170$5.12243 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.135$4.6121 min	0.126$3.1940 min	0.118$2.9114 min	native harness: 0.135$2.7017 min	0.133$4.4024 min	0.131$2.9216 min	0.126$6.9320 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.046$0.118 min	0.045$0.128 min	0.034$0.115 min	0.048≈$0.498 min	0.039$0.116 min	0.061$0.168 min	0.051$0.1834 min	–
13	Kimi K2 ThinkingKimi K2 Thinking	0.041$0.267 min	0.046$0.2210 min	0.054$0.309 min	0.040≈$0.2918 min	0.027$0.287 min	0.057$0.318 min	0.052$0.259 min	–
14	Gemma 4 26BGemma 4 26B	0.035$0.50101 min	0.033$0.2878 min	0.040$0.4237 min	native harness: 0.052≈$0.2195 min	0.007$0.1419 min	0.029$0.2621 min	0.053$0.84113 min	–
15	MiniMax M2MiniMax M2	0.046$0.1214 min	0.048$0.09618 min	0.034$0.07514 min	0.042≈$0.3312 min	0.040$0.08412 min	0.044$0.1010 min	0.030$0.1214 min	–
16	gpt-oss 120Bgpt-oss 120B	0.042$0.09648 min	0.043$0.1333 min	0.039$0.1544 min	0.036–38 min	native harness: 0.038$0.2155 min	0.045$0.1758 min	0.036$0.1560 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.022$0.51^†36 min	0.014$0.99^†52 min	0.016$0.27^†58 min	0.009≈$0.40^†39 min	0.025$0.43^†36 min	0.017$0.45^†55 min	0.026$0.50^†61 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.004$0.084^†2 min	0.007$0.090^†<1 min	0.000$0.004^†<1 min	0.001≈$0.060^†4 min	0.000$0.086^†2 min	0.002$0.061^†1 min	0.007$0.19^†2 min	–
	Harness average	0.209$8.1157 min	0.203$5.6754 min	0.202$5.3243 min	0.201$12.9150 min	0.194$9.6372 min	0.192$6.6051 min	0.191$7.4162 min
1	Claude Sonnet 5.5Sonnet 5.5	native harness: 0.317$22.61101 min	0.326$11.5184 min	0.317$10.4060 min	0.325≈$33.8952 min	0.314$38.34224 min	0.289$14.8283 min	0.320$14.43109 min	–
2	Gemini 4 ArgonGemini 4 Argon	0.314$7.6262 min	0.256$5.2970 min	0.325$6.4950 min	native harness: 0.323$6.3663 min	0.316$7.0085 min	0.295$7.0055 min	0.255$7.2378 min	–
3	Claude Opus 5.5Opus 5.5	native harness: 0.320$17.3063 min	0.307$13.2971 min	0.302$11.8741 min	0.297≈$42.3242 min	0.320$30.38110 min	0.311$17.1770 min	0.300$14.7264 min	–
4	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	native harness: 0.311$8.4224 min	–	–	–
5	Claude Opus 5Opus 5	native harness: 0.292$18.5351 min	0.299$14.6557 min	0.296$13.9638 min	0.283≈$29.5940 min	0.301$25.3676 min	0.300$15.0553 min	0.288$19.1474 min	–
6	Claude Fable 5Fable 5	native harness: 0.268$27.6645 min	0.289$18.5934 min	0.278$22.8736 min	0.271≈$54.1935 min	0.276$30.1951 min	0.274$26.1559 min	0.287$26.6440 min	–
7	Grok 4.7Grok 4.7	0.225$12.4945 min	0.236$5.3535 min	0.229$10.3032 min	0.242≈$16.81^†35 min	0.243$6.8534 min	0.208$10.9551 min	0.230$10.7233 min	grok-buildnative harness: 0.233$9.8250 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.228$2.4332 min	0.213$2.2445 min	0.229$2.0624 min	native harness: 0.226$2.6839 min	0.242$3.5648 min	0.038$2.2623 min	0.218$3.2031 min	–
9	GLM 5.2GLM 5.2	0.227$19.17110 min	0.172$16.25129 min	0.179–103 min	0.199≈$12.7187 min	0.224$14.14121 min	0.199$11.7173 min	0.217$18.77114 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.000$3.71231 min	0.114$4.07160 min	0.123$3.00172 min	0.000≈$3.50228 min	0.000$2.10314 min	0.000$2.71230 min	0.016$2.00201 min	dshnative harness: 0.021$5.12243 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.077$4.6121 min	0.065$3.1940 min	0.078$2.9114 min	native harness: 0.057$2.7017 min	0.074$4.4024 min	0.074$2.9216 min	0.091$6.9320 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.016$0.118 min	0.041$0.128 min	0.042$0.115 min	0.023≈$0.498 min	0.024$0.116 min	0.050$0.168 min	0.024$0.1834 min	–
13	MiniMax M2MiniMax M2	0.019$0.1214 min	0.014$0.09618 min	0.006$0.07514 min	0.008≈$0.3312 min	0.040$0.08412 min	0.011$0.1010 min	0.007$0.1214 min	–
14	gpt-oss 120Bgpt-oss 120B	0.022$0.09648 min	0.035$0.1333 min	0.032$0.1544 min	0.034–38 min	native harness: 0.004$0.2155 min	0.028$0.1758 min	0.026$0.1560 min	–
15	Qwen3-Next 80BQwen3-Next 80B	0.012$0.51^†36 min	0.004$0.99^†52 min	0.000$0.27^†58 min	0.005≈$0.40^†39 min	0.028$0.43^†36 min	0.016$0.45^†55 min	0.001$0.50^†61 min	–
16	Kimi K2 ThinkingKimi K2 Thinking	0.012$0.267 min	0.018$0.2210 min	0.023$0.309 min	0.022≈$0.2918 min	0.025$0.287 min	0.013$0.318 min	0.010$0.259 min	–
17	Gemma 4 26BGemma 4 26B	0.000$0.50101 min	0.000$0.2878 min	0.000$0.4237 min	native harness: 0.010≈$0.2195 min	0.000$0.1419 min	0.000$0.2621 min	0.014$0.84113 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.000$0.084^†2 min	0.005$0.090^†<1 min	0.000$0.004^†<1 min	0.000≈$0.060^†4 min	0.000$0.086^†2 min	0.000$0.061^†1 min	0.005$0.19^†2 min	–
	Harness average	0.138$8.1157 min	0.141$5.6754 min	0.145$5.3243 min	0.137$12.9150 min	0.143$9.6372 min	0.124$6.6051 min	0.136$7.4162 min
1	Claude Opus 5.5Opus 5.5	native harness: 0.695$17.3063 min	0.590$13.2971 min	0.619$11.8741 min	0.621≈$42.3242 min	0.720$30.38110 min	0.682$17.1770 min	0.701$14.7264 min	–
2	Claude Opus 5Opus 5	native harness: 0.570$18.5351 min	0.572$14.6557 min	0.446$13.9638 min	0.649≈$29.5940 min	0.542$25.3676 min	0.499$15.0553 min	0.537$19.1474 min	–
3	Gemini 4 ArgonGemini 4 Argon	0.573$7.6262 min	0.515$5.2970 min	0.490$6.4950 min	native harness: 0.514$6.3663 min	0.491$7.0085 min	0.549$7.0055 min	0.403$7.2378 min	–
4	Claude Sonnet 5.5Sonnet 5.5	native harness: 0.542$22.61101 min	0.498$11.5184 min	0.480$10.4060 min	0.549≈$33.8952 min	0.547$38.34224 min	0.531$14.8283 min	0.564$14.43109 min	–
5	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	native harness: 0.534$8.4224 min	–	–	–
6	Claude Fable 5Fable 5	native harness: 0.444$27.6645 min	0.425$18.5934 min	0.481$22.8736 min	0.501≈$54.1935 min	0.501$30.1951 min	0.433$26.1559 min	0.456$26.6440 min	–
7	Grok 4.7Grok 4.7	0.301$12.4945 min	0.495$5.3535 min	0.412$10.3032 min	0.429≈$16.81^†35 min	0.430$6.8534 min	0.424$10.9551 min	0.487$10.7233 min	grok-buildnative harness: 0.470$9.8250 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.411$2.4332 min	0.342$2.2445 min	0.308$2.0624 min	native harness: 0.293$2.6839 min	0.361$3.5648 min	0.305$2.2623 min	0.371$3.2031 min	–
9	GLM 5.2GLM 5.2	0.295$19.17110 min	0.265$16.25129 min	0.321–103 min	0.237≈$12.7187 min	0.348$14.14121 min	0.312$11.7173 min	0.276$18.77114 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.198$3.71231 min	0.263$4.07160 min	0.327$3.00172 min	0.129≈$3.50228 min	0.074$2.10314 min	0.101$2.71230 min	0.079$2.00201 min	dshnative harness: 0.262$5.12243 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.147$4.6121 min	0.177$3.1940 min	0.152$2.9114 min	native harness: 0.173$2.7017 min	0.164$4.4024 min	0.184$2.9216 min	0.149$6.9320 min	–
12	Gemma 4 26BGemma 4 26B	0.054$0.50101 min	0.056$0.2878 min	0.080$0.4237 min	native harness: 0.099≈$0.2195 min	0.015$0.1419 min	0.041$0.2621 min	0.083$0.84113 min	–
13	MiniMax M2MiniMax M2	0.065$0.1214 min	0.080$0.09618 min	0.036$0.07514 min	0.064≈$0.3312 min	0.062$0.08412 min	0.063$0.1010 min	0.066$0.1214 min	–
14	Qwen3 Coder 480BQwen3 Coder 480B	0.046$0.118 min	0.044$0.128 min	0.024$0.115 min	0.033≈$0.498 min	0.058$0.116 min	0.075$0.168 min	0.068$0.1834 min	–
15	gpt-oss 120Bgpt-oss 120B	0.066$0.09648 min	0.044$0.1333 min	0.042$0.1544 min	0.052–38 min	native harness: 0.050$0.2155 min	0.068$0.1758 min	0.048$0.1560 min	–
16	Kimi K2 ThinkingKimi K2 Thinking	0.057$0.267 min	0.066$0.2210 min	0.056$0.309 min	0.039≈$0.2918 min	0.026$0.287 min	0.066$0.318 min	0.064$0.259 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.022$0.51^†36 min	0.034$0.99^†52 min	0.026$0.27^†58 min	0.017≈$0.40^†39 min	0.036$0.43^†36 min	0.016$0.45^†55 min	0.045$0.50^†61 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.006$0.084^†2 min	0.008$0.090^†<1 min	0.000$0.004^†<1 min	0.001≈$0.060^†4 min	0.001$0.086^†2 min	0.006$0.061^†1 min	0.011$0.19^†2 min	–
	Harness average	0.264$8.1157 min	0.263$5.6754 min	0.253$5.3243 min	0.259$12.9150 min	0.260$9.6372 min	0.256$6.6051 min	0.259$7.4162 min
1	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	native harness: 0.682$8.4224 min	–	–	–
2	Claude Opus 5.5Opus 5.5	native harness: 0.654$17.3063 min	0.496$13.2971 min	0.561$11.8741 min	0.547≈$42.3242 min	0.624$30.38110 min	0.577$17.1770 min	0.558$14.7264 min	–
3	Gemini 4 ArgonGemini 4 Argon	0.540$7.6262 min	0.589$5.2970 min	0.430$6.4950 min	native harness: 0.539$6.3663 min	0.490$7.0085 min	0.495$7.0055 min	0.369$7.2378 min	–
4	Claude Sonnet 5.5Sonnet 5.5	native harness: 0.481$22.61101 min	0.488$11.5184 min	0.468$10.4060 min	0.538≈$33.8952 min	0.429$38.34224 min	0.455$14.8283 min	0.480$14.43109 min	–
5	Claude Opus 5Opus 5	native harness: 0.457$18.5351 min	0.354$14.6557 min	0.470$13.9638 min	0.379≈$29.5940 min	0.401$25.3676 min	0.388$15.0553 min	0.425$19.1474 min	–
6	Claude Fable 5Fable 5	native harness: 0.376$27.6645 min	0.192$18.5934 min	0.322$22.8736 min	0.302≈$54.1935 min	0.289$30.1951 min	0.250$26.1559 min	0.299$26.6440 min	–
7	Grok 4.7Grok 4.7	0.335$12.4945 min	0.338$5.3535 min	0.300$10.3032 min	0.302≈$16.81^†35 min	0.150$6.8534 min	0.321$10.9551 min	0.113$10.7233 min	grok-buildnative harness: 0.376$9.8250 min
8	GLM 5.2GLM 5.2	0.214$19.17110 min	0.183$16.25129 min	0.254–103 min	0.211≈$12.7187 min	0.237$14.14121 min	0.211$11.7173 min	0.196$18.77114 min	–
9	Gemini 3.8 FlashGemini 3.8 Flash	0.197$2.4332 min	0.231$2.2445 min	0.214$2.0624 min	native harness: 0.222$2.6839 min	0.213$3.5648 min	0.145$2.2623 min	0.224$3.2031 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.162$3.71231 min	0.203$4.07160 min	0.222$3.00172 min	0.079≈$3.50228 min	0.084$2.10314 min	0.046$2.71230 min	0.068$2.00201 min	dshnative harness: 0.190$5.12243 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.148$4.6121 min	0.113$3.1940 min	0.128$2.9114 min	native harness: 0.159$2.7017 min	0.133$4.4024 min	0.123$2.9216 min	0.130$6.9320 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.070$0.118 min	0.052$0.128 min	0.052$0.115 min	0.094≈$0.498 min	0.041$0.116 min	0.079$0.168 min	0.052$0.1834 min	–
13	Kimi K2 ThinkingKimi K2 Thinking	0.061$0.267 min	0.063$0.2210 min	0.078$0.309 min	0.065≈$0.2918 min	0.033$0.287 min	0.091$0.318 min	0.086$0.259 min	–
14	Gemma 4 26BGemma 4 26B	0.045$0.50101 min	0.038$0.2878 min	0.030$0.4237 min	native harness: 0.040≈$0.2195 min	0.006$0.1419 min	0.031$0.2621 min	0.068$0.84113 min	–
15	gpt-oss 120Bgpt-oss 120B	0.030$0.09648 min	0.065$0.1333 min	0.048$0.1544 min	0.034–38 min	native harness: 0.051$0.2155 min	0.045$0.1758 min	0.038$0.1560 min	–
16	MiniMax M2MiniMax M2	0.058$0.1214 min	0.051$0.09618 min	0.062$0.07514 min	0.057≈$0.3312 min	0.042$0.08412 min	0.052$0.1010 min	0.008$0.1214 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.038$0.51^†36 min	0.003$0.99^†52 min	0.020$0.27^†58 min	0.009≈$0.40^†39 min	0.027$0.43^†36 min	0.032$0.45^†55 min	0.035$0.50^†61 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.006$0.084^†2 min	0.011$0.090^†<1 min	0.000$0.004^†<1 min	0.002≈$0.060^†4 min	0.000$0.086^†2 min	0.000$0.061^†1 min	0.010$0.19^†2 min	–
	Harness average	0.228$8.1157 min	0.204$5.6754 min	0.215$5.3243 min	0.211$12.9150 min	0.191$9.6372 min	0.197$6.6051 min	0.186$7.4162 min
1	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	native harness: 0.453$8.4224 min	–	–	–
2	Claude Opus 5.5Opus 5.5	native harness: 0.367$17.3063 min	0.393$13.2971 min	0.350$11.8741 min	0.384≈$42.3242 min	0.363$30.38110 min	0.381$17.1770 min	0.384$14.7264 min	–
3	Gemini 4 ArgonGemini 4 Argon	0.349$7.6262 min	0.385$5.2970 min	0.315$6.4950 min	native harness: 0.281$6.3663 min	0.342$7.0085 min	0.370$7.0055 min	0.313$7.2378 min	–
4	Claude Opus 5Opus 5	native harness: 0.263$18.5351 min	0.202$14.6557 min	0.257$13.9638 min	0.366≈$29.5940 min	0.222$25.3676 min	0.266$15.0553 min	0.276$19.1474 min	–
5	Claude Sonnet 5.5Sonnet 5.5	native harness: 0.319$22.61101 min	0.297$11.5184 min	0.333$10.4060 min	0.321≈$33.8952 min	0.302$38.34224 min	0.264$14.8283 min	0.334$14.43109 min	–
6	Claude Fable 5Fable 5	native harness: 0.268$27.6645 min	0.315$18.5934 min	0.269$22.8736 min	0.327≈$54.1935 min	0.276$30.1951 min	0.283$26.1559 min	0.261$26.6440 min	–
7	Gemini 3.8 FlashGemini 3.8 Flash	0.275$2.4332 min	0.261$2.2445 min	0.279$2.0624 min	native harness: 0.216$2.6839 min	0.250$3.5648 min	0.170$2.2623 min	0.210$3.2031 min	–
8	Grok 4.7Grok 4.7	0.187$12.4945 min	0.257$5.3535 min	0.223$10.3032 min	0.183≈$16.81^†35 min	0.221$6.8534 min	0.212$10.9551 min	0.223$10.7233 min	grok-buildnative harness: 0.260$9.8250 min
9	GLM 5.2GLM 5.2	0.175$19.17110 min	0.209$16.25129 min	0.202–103 min	0.196≈$12.7187 min	0.196$14.14121 min	0.202$11.7173 min	0.184$18.77114 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.086$3.71231 min	0.178$4.07160 min	0.119$3.00172 min	0.063≈$3.50228 min	0.043$2.10314 min	0.007$2.71230 min	0.009$2.00201 min	dshnative harness: 0.109$5.12243 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.135$4.6121 min	0.113$3.1940 min	0.085$2.9114 min	native harness: 0.100$2.7017 min	0.124$4.4024 min	0.104$2.9216 min	0.112$6.9320 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.034$0.118 min	0.038$0.128 min	0.017$0.115 min	0.023≈$0.498 min	0.020$0.116 min	0.027$0.168 min	0.043$0.1834 min	–
13	gpt-oss 120Bgpt-oss 120B	0.039$0.09648 min	0.019$0.1333 min	0.028$0.1544 min	0.021–38 min	native harness: 0.027$0.2155 min	0.025$0.1758 min	0.024$0.1560 min	–
14	Kimi K2 ThinkingKimi K2 Thinking	0.012$0.267 min	0.012$0.2210 min	0.034$0.309 min	0.021≈$0.2918 min	0.020$0.287 min	0.026$0.318 min	0.018$0.259 min	–
15	MiniMax M2MiniMax M2	0.023$0.1214 min	0.019$0.09618 min	0.012$0.07514 min	0.013≈$0.3312 min	0.011$0.08412 min	0.029$0.1010 min	0.024$0.1214 min	–
16	Gemma 4 26BGemma 4 26B	0.021$0.50101 min	0.016$0.2878 min	0.023$0.4237 min	native harness: 0.029≈$0.2195 min	0.003$0.1419 min	0.029$0.2621 min	0.017$0.84113 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.007$0.51^†36 min	0.007$0.99^†52 min	0.006$0.27^†58 min	0.003≈$0.40^†39 min	0.008$0.43^†36 min	0.005$0.45^†55 min	0.004$0.50^†61 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.000$0.084^†2 min	0.001$0.090^†<1 min	0.000$0.004^†<1 min	0.000≈$0.060^†4 min	0.000$0.086^†2 min	0.000$0.061^†1 min	0.000$0.19^†2 min	–
	Harness average	0.151$8.1157 min	0.160$5.6754 min	0.150$5.3243 min	0.150$12.9150 min	0.143$9.6372 min	0.141$6.6051 min	0.143$7.4162 min

A stronger shade marks a higher score, cost, or time.
Models rank by their best harness (bold). marks a model's native harness. The harness average covers the 17 models scored on every harness.
29 tasks (0 to 1 scale). Claude Opus 5.5 grades the tasks that use a rubric.
Additional, a harness only that model runs: Grok 4.7 (grok-build), DeepSeek-V4.1-Flash (dsh); codex only: GPT 6 Astra.
Cost and time are per-task averages (≈ estimated cost; † no prompt caching).

How the models were run

Every model runs all 85 tasks on each of its harnesses (6 hours per task). Each pair runs three times, and its score averages its complete runs. Each model runs at its highest available reasoning effort.

Harnesses run at fixed releases: agy 1.3.3, claude-code 2.1.296, codex 0.162.1, hermes 0.22.2, opencode 2.0.26, openhands 1.54.0, pi 1.1.0, grok-build 1.0.52 and dsh 0.2.1-alpha.4.

Cost per task is calculated from token usage at list prices, including cached reads.

Reasoning effortper model

max: adaptive thinking at effort max (Opus 5.5, Sonnet 5.5, Opus 5, Fable 5); reasoning effort max (GLM 5.2, DeepSeek-V4.1-Flash, GPT 6 Astra)
high: thinking level HIGH (Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.1 Pro); reasoning effort high (Grok 4.7, Kimi K2 Thinking, MiniMax M2, gpt-oss 120B)
n/a: no reasoning setting (Qwen3 Coder 480B, Qwen3-Next 80B, Llama 4 Maverick)
thinking on: thinking enabled (Gemma 4 26B)

How grading works

Each task runs in an isolated container with the project's drawings, spreadsheets, specs, and local services.

Automated scripts check quantitative outputs, while an LLM judge grades open-ended engineering work against an expert rubric.

No internet access
6 hours per task
Three runs per harness and model