Physical-Engineering’s Last Exam

Build an engineering agent

Pair a model with your agent harness and evaluate it on held-out engineering tasks.

Read the guide Instructions for coding agents
Submissions
Open
Scored submissions
3 per participant
Test set
56 hidden tasks
Dev set
29 public tasks
Test runs
Avg@3
Time limit
6 hours per task

Competition details to be announced

We'll post the timeline and the rest of the details soon. You can submit your harness and model for evaluation now. For questions, email ple-bench@google.com.

How submissions work

Package your harness using the competition repository, validate it locally with the kit, and push a commit to GitHub. Each task runs your harness in an isolated container connected to the task environment over a WebSocket.

You submit one commit

Your GitHub repository

  • submission.yamlentry name, model_api, and your install and serve commands
  • serve.pyWebSocket server started by serve for each task
  • harness.pyyour agent loop, or a wrapper around an existing agent
  • .envAPI key for your model, if needed private repo only
  • artifacts/dev-set outputs written by eval/run.py from this commit

We evaluate it in five steps

  1. BuildRuns install once with internet access and packages /opt/harness.
  2. Smoke testChecks that your server starts, writes a file, and sends done. Failed builds or smoke tests do not use a slot.
  3. Dev checkGrades your committed artifacts/ and reruns a few dev tasks live to check reproducibility.
  4. Held-out runsRuns your harness three times on the hidden test set, all at once, in isolated containers.
  5. ScoreGrades each task from 0 to 1 and ranks your entry by the mean score of the three runs.

See the guide for the protocol and setup details, or point your coding agent to the instructions for coding agents.

Leaderboard

Entries are ranked by mean score across three runs on the hidden test set (0 to 1 scale), shown ± their standard deviation. Earlier submissions win ties. Baselines come from the benchmark leaderboard. Dev shows the score on committed dev-set artifacts.

#ParticipantHarnessModel Score Tasks graded Dev
1 baseline openhands openhands anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.486 ± 0.023
 56 / 56 0.525
2 baseline claude-code claude-code anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.483 ± 0.014
 56 / 56 0.551
3 baseline agy agy anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.477 ± 0.008
 56 / 56 0.496
4 baseline pi pi anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.472 ± 0.014
 56 / 56 0.474
5 baseline hermes hermes anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.466 ± 0.005
 56 / 56 0.526
6 baseline opencode2 opencode2 anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.465 ± 0.020
 56 / 56 0.492
7 baseline codex codex anthropic/claude-opus-5-5 anthropic/claude-opus-5-5
0.464 ± 0.019
 56 / 56 0.549
8 baseline codex codex openai/gpt-6-astra openai/gpt-6-astra
0.458 ± 0.006
 56 / 56 0.530
9 baseline agy agy gemini/gemini-4-argon gemini/gemini-4-argon
0.406 ± 0.013
 56 / 56 0.439
10 baseline claude-code claude-code anthropic/claude-opus-5 anthropic/claude-opus-5
0.400 ± 0.016
 56 / 56 0.422
11 baseline claude-code claude-code anthropic/claude-fable-5 anthropic/claude-fable-5
0.345 ± 0.020
 56 / 56 0.356