Physical-Engineering’s Last Exam

Guide

Submit a GitHub repository with your harness server (serve.py), submission.yaml, and artifacts/ from a dev-set run. We evaluate your harness and model on the hidden test set.

  1. Copy a sample

    The competition repository includes working samples, the validation kit, and the dev set. Copy a sample into its own git repository (for example, ../my-harness).

  2. Check it

    Run python -m kit check ../my-harness to test your container and smoke task, then eval/run.py to run the dev set and write artifacts/. Both require Docker.

  3. Submit

    Before your first submission, sign in with GitHub and register on this site. Commit your code and artifacts/ together, push to GitHub, and select your repository on the submission page. You get 3 scored submissions (one active evaluation at a time).

Using a coding agent? Point it to the instructions for coding agents.

Your repository

submission.yaml    entry name, model_api, and your install and serve commands
serve.py           WebSocket server started by serve for each task
harness.py         your agent loop, or a wrapper around an existing agent
.env               API keys, if needed (private repository only)
artifacts/         dev-set outputs and manifest written by eval/run.py

This layout matches the pi-opus sample. Smaller entries like minimal keep the entire loop in serve.py.

submission.yaml

name: my-harness                          # display name on the leaderboard
model_api: google-cloud/claude-opus-5-5   # see Models below
install: >-                               # runs once at image build time (as root, with internet)
  uv venv --python 3.12 /opt/harness/venv
  && uv pip install --python /opt/harness/venv/bin/python -r {src}/requirements.txt
serve: /opt/harness/venv/bin/python {src}/serve.py    # starts your server on $PORT
env:                                      # non-sensitive settings, quoted as strings (optional)
  LOG_LEVEL: "info"

{src} expands to /opt/harness/src and {port} to $PORT. Only /opt/harness is kept after the build step, and runtime package downloads are blocked.

Models

model_apiServed byIn .env
google-cloud/<id>Google sponsored modelsNone
openai/<model>OpenAI, on your accountOPENAI_API_KEY=...
anthropic/<model>Anthropic, on your accountANTHROPIC_API_KEY=...
gemini/<model>Google AI Studio, on your accountGEMINI_API_KEY=...
openai-compatible/<model> + model_base_urlYour own server, at an https:// address on a public DNS nameMODEL_API_KEY=...

Available Google sponsored models:

.env

OPENAI_API_KEY=sk-...

Dev-set artifacts

Commit artifacts/ in the same commit as the harness code that generated it. eval/run.py writes and hashes this folder automatically; do not edit it by hand.

uv run python eval/run.py ../my-harness              # runs the dev set, writes ../my-harness/artifacts/
uv run python -m kit artifacts ../my-harness         # checks the folder with the server's rules
uv run python eval/run.py --artifacts ../my-harness  # grades the folder as the server does (calls only the judge)
git -C ../my-harness add -A && git -C ../my-harness commit -m "dev artifacts"   # everything: the manifest records the harness

After you submit:

  1. We build your harness and run the smoke task. Failures here do not use a slot.
  2. We grade your artifacts to compute your dev score. Missing or unusable tasks score 0.
  3. We run your harness on a few sampled dev tasks to verify reproducibility.
  4. Your harness runs three times on the hidden test set, all at once, to set your leaderboard rank.

The submission page checks artifacts/ before queueing (Git LFS is not supported, and at least one dev task must have usable outputs). See docs/artifacts.md for full validation rules.

What your server receives

VariableValue
PORTTCP port your WebSocket server listens on
MODEL_API, MODEL_NAMEYour model_api string and the model name to pass in API requests
MODEL_BASE_URL, MODEL_API_KEYOpenAI-compatible base URL and key, set for every model (also mirrored as OPENAI_BASE_URL and OPENAI_API_KEY)
ANTHROPIC_BASE_URL, ANTHROPIC_API_KEYNative Anthropic endpoint and key (for Claude models)
GOOGLE_GEMINI_BASE_URL, GEMINI_API_KEYNative Gemini endpoint and key (for Gemini models)
Your env and .envPassed through unchanged

How a task runs

For each task, the evaluator starts your harness in an isolated container (read-only root filesystem, uid 1000, 2 CPUs, 4 GB RAM) and connects to 0.0.0.0:$PORT over a WebSocket. Your harness queries the model and sends WebSocket requests to run commands, read and write files, and call MCP tools inside the task container, ending with a done message.

Your container

Your harness

  • serve listens on 0.0.0.0:$PORT
  • 2 CPUs, 4 GB RAM, read-only /opt/harness
  • runs your loop or an existing agent via taskbridge.py
  1. taskprompt, workdir, time limits, MCP servers
  2. exec read_file write_file httpshell commands, file I/O, MCP tool calls
  3. resultcommand output or file data for each request
  4. doneyour closing summary, passed to the graders

Managed by us

The evaluator

  • one WebSocket session per task
  • enforces time and resource limits
  1. executes each request as the task user
  2. returns command output and file contents

Task container no network

The task

  • project files in workdir
  • bash shell as the task user
  • local MCP app servers when needed (mail, calendar, docs)

Via egress proxy only

Your model

  • model_api: Google sponsored model, your own API key, or self-hosted server
  • only your model endpoint is reachable; web search and outside APIs are blocked

After the task finishes

The graders

  • automated verification scripts on deliverable files
  • LLM rubric judge reviewing deliverables and your closing summary
All commands and file operations pass through the evaluator. Your harness never accesses the task container or grading scripts directly.

Bring an existing agent

To wrap an existing CLI agent without writing a WebSocket client, use taskbridge.py from the sample submissions. It runs a local HTTP relay so your subprocess agent can execute commands, read and write files, and call MCP tools inside the task container.

  1. Pi
  2. task-tools.ts
  3. taskbridge.py (local relay)
  4. WebSocket
  5. evaluator
  6. task container

See pi-opus, hermes-opus, and agy-gemini in the competition repository for working examples.

Requirements

Ship as one commit
Commit submission.yaml, your harness code, and the artifacts/ folder generated from it. Keep API keys in .env inside a private repository.
Build once
install runs once at build time with internet access. Only /opt/harness is kept (up to 2 GiB compressed); runtime package downloads are blocked.
Listen in time
Accept a WebSocket connection on 0.0.0.0:$PORT within 120 seconds of startup.
Act through the protocol
Exchange UTF-8 JSON frames up to 16 MiB (up to 8 concurrent requests). File reads and writes use base64.
Reach only your model
Outbound traffic goes through HTTPS_PROXY, which permits only your declared model endpoint.
Deliver, then say done
Write all requested files to the exact paths in the prompt and send your summary in done before the task timeout.

Rules

Scoring