Physical-Engineering’s Last Exam Leaderboard

Comparing frontier models and agent harnesses on the 56 hidden test tasks by score, cost, and runtime.

Tasks: 56 in 4 disciplines
Runs: Three per pair
Time limit: 6 hours per task
Rubric judge: Claude Opus 5.5

Benchmark Score vs. cost per task on the 56 hidden test tasks

Score

Efficient Inefficient 0.000.050.100.150.200.250.300.350.400.450.500.55 $0.002$0.005$0.01$0.02$0.05$0.1$0.2$0.5$1$2$5$10$20$50$100 Opus 5.5 (claude-code) GPT 6 Astra (codex) Sonnet 5.5 (claude-code) Gemini 4 Argon (agy) Opus 5 (claude-code) Fable 5 (claude-code) Grok 4.7 (grok-build) Gemini 3.8 Flash (agy) DeepSeek-V4.1-Flash (dsh) Gemini 3.1 Pro (agy) ≈ Gemma 4 26B (agy) gpt-oss 120B (codex) GLM 5.2 (codex) Qwen3 Coder 480B (claude-code) Kimi K2 Thinking (opencode) MiniMax M2 (claude-code) Qwen3-Next 80B (openhands)^† Llama 4 Maverick (openhands)^†

Cost per task (US$, log scale)

Harness’s Effect How far the harness moves each model’s score on the 56 hidden test tasks

ModelRange

0.00 0.10 0.20 0.30 0.40 0.50

Claude Opus 5.5

codex0.464 openhands0.486 claude-code0.483

codex 0.464 to openhands 0.486, average 0.473, native claude-code 0.483 range 0.022

Claude Sonnet 5.5

openhands0.407 hermes0.451 claude-code0.435

openhands 0.407 to hermes 0.451, average 0.434, native claude-code 0.435 range 0.044

Claude Opus 5

codex0.389 openhands0.426 claude-code0.400

codex 0.389 to openhands 0.426, average 0.408, native claude-code 0.400 range 0.037

Gemini 4 Argon

openhands0.290 claude-code0.411 agy0.406

openhands 0.290 to claude-code 0.411, average 0.375, native agy 0.406 range 0.121

Claude Fable 5

agy0.325 claude-code0.345

agy 0.325 to claude-code 0.345, average 0.338, native claude-code 0.345 range 0.020

Grok 4.7

hermes0.256 pi0.302 grok-build0.301

hermes 0.256 to pi 0.302, average 0.274, native grok-build 0.301 range 0.046

Gemini 3.8 Flash

hermes0.174 openhands0.282 agy0.275

hermes 0.174 to openhands 0.282, average 0.258, native agy 0.275 range 0.108

GLM 5.2

claude-code0.218 codex0.250

claude-code 0.218 to codex 0.250, average 0.239 range 0.032

DeepSeek-V4.1-Flash

codex0.041 opencode0.225 dsh0.144

codex 0.041 to opencode 0.225, average 0.117, native dsh 0.144 range 0.184

Gemini 3.1 Pro

agy0.138 hermes0.148

agy 0.138 to hermes 0.148, average 0.143, native agy 0.138 range 0.010

Qwen3 Coder 480B

pi0.042 claude-code0.062

pi 0.042 to claude-code 0.062, average 0.052 range 0.020

Kimi K2 Thinking

agy0.032 opencode0.062

agy 0.032 to opencode 0.062, average 0.044 range 0.030

MiniMax M2

agy0.035 claude-code0.057

agy 0.035 to claude-code 0.057, average 0.048 range 0.022

gpt-oss 120B

hermes0.037 openhands0.046 codex0.038

hermes 0.037 to openhands 0.046, average 0.040, native codex 0.038 range 0.009

Gemma 4 26B

codex0.007 openhands0.045 agy0.042

codex 0.007 to openhands 0.045, average 0.031, native agy 0.042 range 0.038

Qwen3-Next 80B

codex0.010 openhands0.043

codex 0.010 to openhands 0.043, average 0.024 range 0.033

Llama 4 Maverick

codex0.000 openhands0.007

codex 0.000 to openhands 0.007, average 0.003 range 0.007

Key takeaways

Claude Opus 5.5 (openhands) leads the 56 hidden test tasks at 0.486. On the same model, claude-code (0.483) and agy (0.477) are within run-to-run variation.
The model matters most. Averaged across all seven harnesses, model scores range from 0.003 (Llama 4 Maverick) to 0.473 (Claude Opus 5.5), whereas switching harnesses on a single model shifts its score by up to 0.184 (DeepSeek-V4.1-Flash, codex to opencode).
Cost varies far more than score on one model. On Claude Opus 5.5, a task costs 3× as much with agy ($42.32) as with opencode ($12.87). The top-scoring pair, Claude Opus 5.5 (openhands), costs $14.87 per task.
Different pairs lead different disciplines: Opus 5.5 (openhands) in structural, Opus 5.5 (claude-code) in civil, Opus 5.5 (hermes) in electrical and GPT 6 Astra (codex) in mechanical.

Rankings Each model on each harness, averaged over three runs

Each model's score with each harness, models ranked by their best score: The 56 hidden test tasks, overall
#	Model	pi	opencode	claude-code	agy	openhands	codex	hermes	Additional
1	Claude Opus 5.5Opus 5.5	0.472$13.3568 min	0.465$12.8749 min	native harness: 0.483$17.1263 min	0.477≈$42.3245 min	0.486$14.8766 min	0.464$37.62130 min	0.466$16.0366 min	–
2	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.458$9.5326 min	–	–
3	Claude Sonnet 5.5Sonnet 5.5	0.442$11.7796 min	0.425$10.5363 min	native harness: 0.435$18.6899 min	0.449≈$33.8953 min	0.407$15.72112 min	0.431$42.56243 min	0.451$14.7187 min	–
4	Claude Opus 5Opus 5	0.418$15.2658 min	0.404$12.5138 min	native harness: 0.400$16.5248 min	0.415≈$29.5942 min	0.426$17.1366 min	0.389$20.7769 min	0.404$13.8952 min	–
5	Gemini 4 ArgonGemini 4 Argon	0.366$7.0078 min	0.373$7.0455 min	0.411$8.5167 min	native harness: 0.406$7.7073 min	0.290$7.5483 min	0.389$7.1383 min	0.389$7.9261 min	–
6	Claude Fable 5Fable 5	0.342$21.1747 min	0.341$22.9538 min	native harness: 0.345$26.4645 min	0.325≈$54.1937 min	0.332$25.8344 min	0.345$28.7760 min	0.333$24.3055 min	–
7	Grok 4.7Grok 4.7	0.302$8.0140 min	0.284$10.2534 min	0.266$12.7548 min	0.278≈$16.81^†33 min	0.275$8.8934 min	0.259$7.0239 min	0.256$10.0947 min	grok-buildnative harness: 0.301$10.1852 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.268$2.6842 min	0.259$2.4525 min	0.272$2.6134 min	native harness: 0.275$2.6040 min	0.282$3.2133 min	0.278$3.6145 min	0.174$2.3924 min	–
9	GLM 5.2GLM 5.2	0.250$13.66109 min	0.240–94 min	0.218$14.7894 min	0.238≈$12.7186 min	0.234$15.50104 min	0.250$12.57121 min	0.240$10.8370 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.225$4.54169 min	0.225$3.46172 min	0.117$3.74241 min	0.092≈$3.50229 min	0.063$1.65194 min	0.041$2.08295 min	0.053$2.98255 min	dshnative harness: 0.144$6.30240 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.141$2.7624 min	0.141$2.6715 min	0.144$3.6517 min	native harness: 0.138$2.4715 min	0.147$4.6716 min	0.145$2.8016 min	0.148$2.0912 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.042$0.1510 min	0.045$0.258 min	0.062$0.1310 min	0.044≈$0.496 min	0.060$0.2128 min	0.047$0.116 min	0.062$0.128 min	–
13	Kimi K2 ThinkingKimi K2 Thinking	0.040$0.208 min	0.062$0.288 min	0.046$0.257 min	0.032≈$0.2918 min	0.044$0.2512 min	0.032$0.269 min	0.050$0.288 min	–
14	MiniMax M2MiniMax M2	0.046$0.09412 min	0.042$0.07914 min	0.057$0.1012 min	0.035≈$0.3313 min	0.053$0.1314 min	0.046$0.09314 min	0.057$0.09710 min	–
15	gpt-oss 120Bgpt-oss 120B	0.040$0.1226 min	0.037$0.1647 min	0.042$0.1146 min	0.042–36 min	0.046$0.1452 min	native harness: 0.038$0.1845 min	0.037$0.1648 min	–
16	Gemma 4 26BGemma 4 26B	0.038$0.2984 min	0.027$0.4546 min	0.038$0.5388 min	native harness: 0.042≈$0.2195 min	0.045$0.61109 min	0.007$0.2024 min	0.018$0.2924 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.027$1.05^†59 min	0.026$0.54^†70 min	0.026$0.54^†44 min	0.014≈$0.40^†41 min	0.043$0.77^†92 min	0.010$0.56^†36 min	0.019$0.41^†47 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.007$0.34^†1 min	0.000$0.004^†<1 min	0.004$0.063^†1 min	0.001≈$0.060^†3 min	0.007$0.18^†2 min	0.000$0.097^†2 min	0.000$0.046^†1 min	–
	Harness average	0.204$6.0355 min	0.200$5.4146 min	0.198$7.4457 min	0.194$12.9751 min	0.191$6.9062 min	0.187$9.7973 min	0.186$6.2751 min
1	Claude Opus 5.5Opus 5.5	0.546$13.3568 min	0.539$12.8749 min	native harness: 0.563$17.1263 min	0.496≈$42.3245 min	0.576$14.8766 min	0.516$37.62130 min	0.472$16.0366 min	–
2	Claude Opus 5Opus 5	0.513$15.2658 min	0.487$12.5138 min	native harness: 0.473$16.5248 min	0.521≈$29.5942 min	0.520$17.1366 min	0.523$20.7769 min	0.497$13.8952 min	–
3	Claude Sonnet 5.5Sonnet 5.5	0.489$11.7796 min	0.522$10.5363 min	native harness: 0.474$18.6899 min	0.501≈$33.8953 min	0.517$15.72112 min	0.467$42.56243 min	0.518$14.7187 min	–
4	Claude Fable 5Fable 5	0.469$21.1747 min	0.396$22.9538 min	native harness: 0.406$26.4645 min	0.376≈$54.1937 min	0.487$25.8344 min	0.480$28.7760 min	0.446$24.3055 min	–
5	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.484$9.5326 min	–	–
6	Gemini 4 ArgonGemini 4 Argon	0.420$7.0078 min	0.438$7.0455 min	0.449$8.5167 min	native harness: 0.458$7.7073 min	0.322$7.5483 min	0.399$7.1383 min	0.382$7.9261 min	–
7	Gemini 3.8 FlashGemini 3.8 Flash	0.256$2.6842 min	0.253$2.4525 min	0.322$2.6134 min	native harness: 0.271$2.6040 min	0.339$3.2133 min	0.284$3.6145 min	0.117$2.3924 min	–
8	Grok 4.7Grok 4.7	0.325$8.0140 min	0.289$10.2534 min	0.265$12.7548 min	0.264≈$16.81^†33 min	0.273$8.8934 min	0.264$7.0239 min	0.245$10.0947 min	grok-buildnative harness: 0.283$10.1852 min
9	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.214$4.54169 min	0.321$3.46172 min	0.033$3.74241 min	0.017≈$3.50229 min	0.000$1.65194 min	0.020$2.08295 min	0.000$2.98255 min	dshnative harness: 0.017$6.30240 min
10	GLM 5.2GLM 5.2	0.263$13.66109 min	0.276–94 min	0.231$14.7894 min	0.223≈$12.7186 min	0.211$15.50104 min	0.268$12.57121 min	0.249$10.8370 min	–
11	Gemini 3.1 ProGemini 3.1 Pro	0.119$2.7624 min	0.114$2.6715 min	0.129$3.6517 min	native harness: 0.078$2.4715 min	0.098$4.6716 min	0.114$2.8016 min	0.117$2.0912 min	–
12	Kimi K2 ThinkingKimi K2 Thinking	0.000$0.208 min	0.045$0.288 min	0.002$0.257 min	0.001≈$0.2918 min	0.001$0.2512 min	0.010$0.269 min	0.010$0.288 min	–
13	Qwen3 Coder 480BQwen3 Coder 480B	0.009$0.1510 min	0.001$0.258 min	0.022$0.1310 min	0.007≈$0.496 min	0.031$0.2128 min	0.020$0.116 min	0.027$0.128 min	–
14	MiniMax M2MiniMax M2	0.012$0.09412 min	0.000$0.07914 min	0.027$0.1012 min	0.003≈$0.3313 min	0.002$0.1314 min	0.028$0.09314 min	0.023$0.09710 min	–
15	gpt-oss 120Bgpt-oss 120B	0.015$0.1226 min	0.010$0.1647 min	0.005$0.1146 min	0.011–36 min	0.017$0.1452 min	native harness: 0.015$0.1845 min	0.018$0.1648 min	–
16	Qwen3-Next 80BQwen3-Next 80B	0.000$1.05^†59 min	0.000$0.54^†70 min	0.011$0.54^†44 min	0.000≈$0.40^†41 min	0.011$0.77^†92 min	0.007$0.56^†36 min	0.001$0.41^†47 min	–
17	Gemma 4 26BGemma 4 26B	0.002$0.2984 min	0.001$0.4546 min	0.001$0.5388 min	native harness: 0.007≈$0.2195 min	0.000$0.61109 min	0.000$0.2024 min	0.000$0.2924 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.000$0.34^†1 min	0.000$0.004^†<1 min	0.000$0.063^†1 min	0.000≈$0.060^†3 min	0.000$0.18^†2 min	0.000$0.097^†2 min	0.000$0.046^†1 min	–
	Harness average	0.215$6.0355 min	0.217$5.4146 min	0.201$7.4457 min	0.190$12.9751 min	0.200$6.9062 min	0.201$9.7973 min	0.184$6.2751 min
1	Claude Opus 5.5Opus 5.5	0.624$13.3568 min	0.623$12.8749 min	native harness: 0.703$17.1263 min	0.621≈$42.3245 min	0.640$14.8766 min	0.635$37.62130 min	0.620$16.0366 min	–
2	Claude Sonnet 5.5Sonnet 5.5	0.579$11.7796 min	0.521$10.5363 min	native harness: 0.520$18.6899 min	0.562≈$33.8953 min	0.470$15.72112 min	0.514$42.56243 min	0.557$14.7187 min	–
3	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.576$9.5326 min	–	–
4	Gemini 4 ArgonGemini 4 Argon	0.503$7.0078 min	0.470$7.0455 min	0.515$8.5167 min	native harness: 0.538$7.7073 min	0.291$7.5483 min	0.545$7.1383 min	0.481$7.9261 min	–
5	Claude Opus 5Opus 5	0.431$15.2658 min	0.456$12.5138 min	native harness: 0.461$16.5248 min	0.455≈$29.5942 min	0.525$17.1366 min	0.394$20.7769 min	0.443$13.8952 min	–
6	Claude Fable 5Fable 5	0.315$21.1747 min	0.326$22.9538 min	native harness: 0.391$26.4645 min	0.319≈$54.1937 min	0.326$25.8344 min	0.351$28.7760 min	0.312$24.3055 min	–
7	Grok 4.7Grok 4.7	0.346$8.0140 min	0.359$10.2534 min	0.260$12.7548 min	0.321≈$16.81^†33 min	0.286$8.8934 min	0.314$7.0239 min	0.297$10.0947 min	grok-buildnative harness: 0.371$10.1852 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.289$2.6842 min	0.255$2.4525 min	0.286$2.6134 min	native harness: 0.318$2.6040 min	0.309$3.2133 min	0.301$3.6145 min	0.184$2.3924 min	–
9	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.268$4.54169 min	0.217$3.46172 min	0.116$3.74241 min	0.062≈$3.50229 min	0.058$1.65194 min	0.037$2.08295 min	0.032$2.98255 min	dshnative harness: 0.090$6.30240 min
10	GLM 5.2GLM 5.2	0.263$13.66109 min	0.209–94 min	0.206$14.7894 min	0.234≈$12.7186 min	0.233$15.50104 min	0.238$12.57121 min	0.234$10.8370 min	–
11	Gemini 3.1 ProGemini 3.1 Pro	0.121$2.7624 min	0.135$2.6715 min	0.133$3.6517 min	native harness: 0.132$2.4715 min	0.144$4.6716 min	0.152$2.8016 min	0.148$2.0912 min	–
12	MiniMax M2MiniMax M2	0.030$0.09412 min	0.031$0.07914 min	0.029$0.1012 min	0.018≈$0.3313 min	0.023$0.1314 min	0.022$0.09314 min	0.040$0.09710 min	–
13	Qwen3 Coder 480BQwen3 Coder 480B	0.023$0.1510 min	0.019$0.258 min	0.036$0.1310 min	0.036≈$0.496 min	0.021$0.2128 min	0.031$0.116 min	0.022$0.128 min	–
14	Kimi K2 ThinkingKimi K2 Thinking	0.030$0.208 min	0.032$0.288 min	0.028$0.257 min	0.015≈$0.2918 min	0.027$0.2512 min	0.013$0.269 min	0.034$0.288 min	–
15	gpt-oss 120Bgpt-oss 120B	0.033$0.1226 min	0.020$0.1647 min	0.021$0.1146 min	0.011–36 min	0.020$0.1452 min	native harness: 0.013$0.1845 min	0.024$0.1648 min	–
16	Gemma 4 26BGemma 4 26B	0.033$0.2984 min	0.013$0.4546 min	0.025$0.5388 min	native harness: 0.024≈$0.2195 min	0.026$0.61109 min	0.010$0.2024 min	0.026$0.2924 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.008$1.05^†59 min	0.016$0.54^†70 min	0.021$0.54^†44 min	0.015≈$0.40^†41 min	0.020$0.77^†92 min	0.005$0.56^†36 min	0.014$0.41^†47 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.001$0.34^†1 min	0.000$0.004^†<1 min	0.001$0.063^†1 min	0.000≈$0.060^†3 min	0.002$0.18^†2 min	0.000$0.097^†2 min	0.000$0.046^†1 min	–
	Harness average	0.229$6.0355 min	0.218$5.4146 min	0.221$7.4457 min	0.217$12.9751 min	0.201$6.9062 min	0.210$9.7973 min	0.204$6.2751 min
1	Claude Opus 5.5Opus 5.5	0.453$13.3568 min	0.438$12.8749 min	native harness: 0.433$17.1263 min	0.468≈$42.3245 min	0.479$14.8766 min	0.485$37.62130 min	0.490$16.0366 min	–
2	Claude Sonnet 5.5Sonnet 5.5	0.399$11.7796 min	0.390$10.5363 min	native harness: 0.406$18.6899 min	0.395≈$33.8953 min	0.360$15.72112 min	0.421$42.56243 min	0.408$14.7187 min	–
3	Gemini 4 ArgonGemini 4 Argon	0.323$7.0078 min	0.341$7.0455 min	0.410$8.5167 min	native harness: 0.397$7.7073 min	0.318$7.5483 min	0.346$7.1383 min	0.407$7.9261 min	–
4	Claude Opus 5Opus 5	0.406$15.2658 min	0.358$12.5138 min	native harness: 0.367$16.5248 min	0.374≈$29.5942 min	0.378$17.1366 min	0.365$20.7769 min	0.379$13.8952 min	–
5	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.404$9.5326 min	–	–
6	Claude Fable 5Fable 5	0.350$21.1747 min	0.332$22.9538 min	native harness: 0.317$26.4645 min	0.323≈$54.1937 min	0.314$25.8344 min	0.312$28.7760 min	0.320$24.3055 min	–
7	Grok 4.7Grok 4.7	0.309$8.0140 min	0.269$10.2534 min	0.271$12.7548 min	0.296≈$16.81^†33 min	0.328$8.8934 min	0.281$7.0239 min	0.256$10.0947 min	grok-buildnative harness: 0.312$10.1852 min
8	Gemini 3.8 FlashGemini 3.8 Flash	0.271$2.6842 min	0.250$2.4525 min	0.232$2.6134 min	native harness: 0.262$2.6040 min	0.262$3.2133 min	0.259$3.6145 min	0.186$2.3924 min	–
9	GLM 5.2GLM 5.2	0.223$13.66109 min	0.229–94 min	0.222$14.7894 min	0.233≈$12.7186 min	0.227$15.50104 min	0.243$12.57121 min	0.226$10.8370 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.211$4.54169 min	0.201$3.46172 min	0.145$3.74241 min	0.100≈$3.50229 min	0.042$1.65194 min	0.030$2.08295 min	0.051$2.98255 min	dshnative harness: 0.141$6.30240 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.151$2.7624 min	0.153$2.6715 min	0.159$3.6517 min	native harness: 0.158$2.4715 min	0.160$4.6716 min	0.160$2.8016 min	0.162$2.0912 min	–
12	MiniMax M2MiniMax M2	0.078$0.09412 min	0.076$0.07914 min	0.088$0.1012 min	0.057≈$0.3313 min	0.096$0.1314 min	0.069$0.09314 min	0.068$0.09710 min	–
13	Kimi K2 ThinkingKimi K2 Thinking	0.052$0.208 min	0.091$0.288 min	0.076$0.257 min	0.038≈$0.2918 min	0.058$0.2512 min	0.044$0.269 min	0.075$0.288 min	–
14	Qwen3 Coder 480BQwen3 Coder 480B	0.070$0.1510 min	0.072$0.258 min	0.090$0.1310 min	0.058≈$0.496 min	0.085$0.2128 min	0.067$0.116 min	0.085$0.128 min	–
15	Qwen3-Next 80BQwen3-Next 80B	0.058$1.05^†59 min	0.048$0.54^†70 min	0.041$0.54^†44 min	0.030≈$0.40^†41 min	0.065$0.77^†92 min	0.017$0.56^†36 min	0.026$0.41^†47 min	–
16	gpt-oss 120Bgpt-oss 120B	0.056$0.1226 min	0.054$0.1647 min	0.054$0.1146 min	0.061–36 min	0.054$0.1452 min	native harness: 0.035$0.1845 min	0.049$0.1648 min	–
17	Gemma 4 26BGemma 4 26B	0.046$0.2984 min	0.032$0.4546 min	0.044$0.5388 min	native harness: 0.042≈$0.2195 min	0.046$0.61109 min	0.005$0.2024 min	0.027$0.2924 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.006$0.34^†1 min	0.000$0.004^†<1 min	0.004$0.063^†1 min	0.000≈$0.060^†3 min	0.010$0.18^†2 min	0.000$0.097^†2 min	0.000$0.046^†1 min	–
	Harness average	0.204$6.0355 min	0.196$5.4146 min	0.198$7.4457 min	0.194$12.9751 min	0.193$6.9062 min	0.185$9.7973 min	0.189$6.2751 min
1	GPT 6 AstraGPT 6 Astra codex only	–	–	–	–	–	native harness: 0.448$9.5326 min	–	–
2	Claude Sonnet 5.5Sonnet 5.5	0.409$11.7796 min	0.389$10.5363 min	native harness: 0.415$18.6899 min	0.432≈$33.8953 min	0.391$15.72112 min	0.397$42.56243 min	0.425$14.7187 min	–
3	Claude Opus 5.5Opus 5.5	0.409$13.3568 min	0.405$12.8749 min	native harness: 0.416$17.1263 min	0.424≈$42.3245 min	0.408$14.8766 min	0.367$37.62130 min	0.385$16.0366 min	–
4	Claude Opus 5Opus 5	0.398$15.2658 min	0.396$12.5138 min	native harness: 0.383$16.5248 min	0.405≈$29.5942 min	0.400$17.1366 min	0.370$20.7769 min	0.384$13.8952 min	–
5	Gemini 4 ArgonGemini 4 Argon	0.331$7.0078 min	0.343$7.0455 min	0.361$8.5167 min	native harness: 0.348$7.7073 min	0.261$7.5483 min	0.359$7.1383 min	0.339$7.9261 min	–
6	Claude Fable 5Fable 5	0.314$21.1747 min	0.339$22.9538 min	native harness: 0.332$26.4645 min	0.315≈$54.1937 min	0.309$25.8344 min	0.333$28.7760 min	0.323$24.3055 min	–
7	Gemini 3.8 FlashGemini 3.8 Flash	0.261$2.6842 min	0.269$2.4525 min	0.285$2.6134 min	native harness: 0.269$2.6040 min	0.271$3.2133 min	0.282$3.6145 min	0.177$2.3924 min	–
8	Grok 4.7Grok 4.7	0.273$8.0140 min	0.265$10.2534 min	0.263$12.7548 min	0.251≈$16.81^†33 min	0.230$8.8934 min	0.219$7.0239 min	0.243$10.0947 min	grok-buildnative harness: 0.270$10.1852 min
9	GLM 5.2GLM 5.2	0.263$13.66109 min	0.252–94 min	0.217$14.7894 min	0.247≈$12.7186 min	0.244$15.50104 min	0.256$12.57121 min	0.251$10.8370 min	–
10	DeepSeek-V4.1-FlashDeepSeek-V4.1-Flash	0.223$4.54169 min	0.225$3.46172 min	0.115$3.74241 min	0.117≈$3.50229 min	0.097$1.65194 min	0.055$2.08295 min	0.077$2.98255 min	dshnative harness: 0.201$6.30240 min
11	Gemini 3.1 ProGemini 3.1 Pro	0.146$2.7624 min	0.141$2.6715 min	0.140$3.6517 min	native harness: 0.141$2.4715 min	0.152$4.6716 min	0.139$2.8016 min	0.146$2.0912 min	–
12	Qwen3 Coder 480BQwen3 Coder 480B	0.037$0.1510 min	0.046$0.258 min	0.062$0.1310 min	0.046≈$0.496 min	0.064$0.2128 min	0.044$0.116 min	0.069$0.128 min	–
13	MiniMax M2MiniMax M2	0.035$0.09412 min	0.030$0.07914 min	0.051$0.1012 min	0.034≈$0.3313 min	0.044$0.1314 min	0.043$0.09314 min	0.063$0.09710 min	–
14	Gemma 4 26BGemma 4 26B	0.044$0.2984 min	0.036$0.4546 min	0.050$0.5388 min	native harness: 0.058≈$0.2195 min	0.063$0.61109 min	0.010$0.2024 min	0.013$0.2924 min	–
15	gpt-oss 120Bgpt-oss 120B	0.035$0.1226 min	0.036$0.1647 min	0.049$0.1146 min	0.047–36 min	0.057$0.1452 min	native harness: 0.057$0.1845 min	0.037$0.1648 min	–
16	Kimi K2 ThinkingKimi K2 Thinking	0.045$0.208 min	0.054$0.288 min	0.040$0.257 min	0.042≈$0.2918 min	0.051$0.2512 min	0.037$0.269 min	0.046$0.288 min	–
17	Qwen3-Next 80BQwen3-Next 80B	0.017$1.05^†59 min	0.019$0.54^†70 min	0.020$0.54^†44 min	0.006≈$0.40^†41 min	0.044$0.77^†92 min	0.006$0.56^†36 min	0.021$0.41^†47 min	–
18	Llama 4 MaverickLlama 4 Maverick	0.012$0.34^†1 min	0.000$0.004^†<1 min	0.006$0.063^†1 min	0.002≈$0.060^†3 min	0.009$0.18^†2 min	0.001$0.097^†2 min	0.001$0.046^†1 min	–
	Harness average	0.191$6.0355 min	0.191$5.4146 min	0.189$7.4457 min	0.187$12.9751 min	0.182$6.9062 min	0.175$9.7973 min	0.176$6.2751 min

A stronger shade marks a higher score, cost, or time.
Models rank by their best harness (bold). marks a model's native harness. The harness average covers the 17 models scored on every harness.
56 tasks (0 to 1 scale). Claude Opus 5.5 grades the tasks that use a rubric.
Additional, a harness only that model runs: Grok 4.7 (grok-build), DeepSeek-V4.1-Flash (dsh); codex only: GPT 6 Astra.
Cost and time are per-task averages (≈ estimated cost; † no prompt caching).

How the models were run

Every model runs all 85 tasks on each of its harnesses (6 hours per task). Each pair runs three times, and its score averages its complete runs. Each model runs at its highest available reasoning effort.

Harnesses run at fixed releases: agy 1.3.3, claude-code 2.1.296, codex 0.162.1, hermes 0.22.2, opencode 2.0.26, openhands 1.54.0, pi 1.1.0, grok-build 1.0.52 and dsh 0.2.1-alpha.4.

Cost per task is calculated from token usage at list prices, including cached reads.

Reasoning effortper model

max: adaptive thinking at effort max (Opus 5.5, Sonnet 5.5, Opus 5, Fable 5); reasoning effort max (GLM 5.2, DeepSeek-V4.1-Flash, GPT 6 Astra)
high: thinking level HIGH (Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.1 Pro); reasoning effort high (Grok 4.7, MiniMax M2, Kimi K2 Thinking, gpt-oss 120B)
n/a: no reasoning setting (Qwen3 Coder 480B, Qwen3-Next 80B, Llama 4 Maverick)
thinking on: thinking enabled (Gemma 4 26B)

How grading works

Each task runs in an isolated container with the project's drawings, spreadsheets, specs, and local services.

Automated scripts check quantitative outputs, while an LLM judge grades open-ended engineering work against an expert rubric.

No internet access
6 hours per task
Three runs per harness and model