Build, check and submit a harness

You are helping a participant enter the harness competition.

Each entry is an agent harness that runs as a WebSocket server (serve.py) and solves held-out engineering tasks over the competition protocol. Participants pair their harness with a Google-sponsored model, a commercial model on their own API key (OpenAI, Anthropic, or Gemini), or a self-hosted model at an OpenAI-compatible endpoint. The leaderboard ranks each harness + model pair together.

Your goal is to build an entry the evaluation server can run, verify it locally with the kit, and run it on the dev set to generate artifacts/. The participant submits the harness and its dev-set artifacts on the portal.

Follow these steps in order and do not skip any checks:

1. Start from one of the sample submissions (Hermes Agent or Pi on Claude Opus 5.5, the Antigravity CLI on Gemini 3.8 Flash, or a custom harness on an OpenAI key or vLLM server) under sample_submissions/ in the competition repository that the organizers shared with the participant, or from the participant's existing repository.
2. Write the harness and submission.yaml (Section 2) so they satisfy Sections 3 through 7.
3. Verify the entry locally with python -m kit check (Section 8), then run the dev set with eval/run.py to write artifacts/ (Section 8). Fix any issues until every check passes.
4. Put any required API keys in .env (Section 6). Stage and commit everything (git add -A) so the harness and artifacts/ share one commit, then push to GitHub.
5. Tell the participant the entry is ready to submit (Section 9). Each participant has 3 scored submissions in total, one in the queue or under evaluation at a time.

1. The entry

An entry is a single commit in a GitHub repository with this layout at its root:

submission.yaml      how to install and start the harness, and which model it calls
serve.py             the harness: a WebSocket server (any language works; the samples use Python)
.env                 API keys, if the model or harness needs any (private repository only)
artifacts/           what this harness produced on the dev set, written by eval/run.py (Section 8)
...                  anything else it needs: requirements.txt, prompts, or a vendored agent

When you submit, the evaluation server clones your commit, runs install inside a build container, and packages /opt/harness as the runtime bundle. For each task, it starts your serve command from that bundle in an isolated container alongside the task container and connects over WebSocket. Your harness calls its model and interacts with the task environment exclusively through the competition protocol (Section 7) to run commands, read and write files, and call MCP tools inside the task container.

Requirements:

• Use a public repository, or a private repository owned by the participant with the competition's GitHub App installed (granting read-only access to selected repositories). Any repository containing a .env file must be private. Private git submodules are not supported, and submodules are fetched over https only.
• The submit form reads submission.yaml at the submitted commit, falling back to recipe.yaml if submission.yaml is absent. A recipe.yaml written in the submission.yaml format is read and reviewed like submission.yaml; an earlier recipe format is refused for participant submissions because it does not include dev-set artifacts.
• The submit form inspects artifacts/ at the submitted commit and refuses commits without usable artifacts (Section 8). The build step excludes artifacts/ from /opt/harness, so your running harness never sees it.
• The serve container can reach only its model endpoint. No other host is reachable, so web search and outside APIs are unavailable. The task container has no network access at all.
• Every task's prompt and files (including hidden test tasks) are sent to the model endpoint your harness calls. Make sure the participant is comfortable with their endpoint's logging policy, and never retain copies of task content.

2. submission.yaml

name: my-harness                          # leaderboard entry name: 2-39 chars of a-z, 0-9, and -
model_api: google-cloud/claude-opus-5-5   # the model your harness calls (Section 3)
install: >-                               # optional; runs once at build time as root with internet access
  uv venv --python 3.12 /opt/harness/venv
  && uv pip install --python /opt/harness/venv/bin/python -r {src}/requirements.txt
serve: /opt/harness/venv/bin/python {src}/serve.py    # required; starts your server on $PORT
env:                                      # optional; non-secret settings as quoted strings
  LOG_LEVEL: "info"

Environment variables managed by the evaluator cannot be set in env (or in .env, other than your model API key): PORT, HOME, USER, proxy variables (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY, NODE_USE_ENV_PROXY), and the model variables listed in Section 3. Any {...} placeholders other than {src} and {port} are passed through untouched to the shell. Unknown top-level keys are rejected, and submission.yaml must be at most 16 KiB. Do not include repository or commit fields; the portal records those automatically.

3. The model

model_api declares which model your harness uses and how it is billed:

Available Google-sponsored models (all support tool calling):

At runtime, the serve container receives the same set of environment variables regardless of provider:

If you use openai-compatible/, your endpoint must accept Authorization: Bearer <MODEL_API_KEY>, support tools and tool_calls on POST /v1/chat/completions (for vLLM, pass --enable-auto-tool-choice and the appropriate --tool-call-parser), handle both streaming and non-streaming requests, and serve concurrent tasks. The egress proxy blocks hostnames that resolve to private IP ranges.

Model call tips:

• Stream long responses or set generous timeouts, since extended reasoning turns can stay quiet for several minutes.
• Retry 429 rate limits, network timeouts, and 5xx errors with exponential backoff. Do not retry 400 errors.
• Google-sponsored models default to their highest reasoning level (max for Claude and GLM; high for Gemini, gpt-oss, Kimi, and MiniMax) on both the evaluation server and kit/model_proxy. Pass reasoning_effort explicitly to use a lower level.
• Claude Opus 5.5 rejects temperature, top_p, top_k, and thinking blocks with budget_tokens, so omit them. Set a generous max_tokens (32,000 is a good default).
• Claude caches a prompt prefix only where the request marks it: put "cache_control": {"type": "ephemeral"} on the system prompt and on the newest message (Claude allows 4 such marks). Without the marks every turn pays the full input price for the whole conversation, which is many times the cost; the minimal, plan-verify and node-protocol samples show how. Gemini, OpenAI and vLLM cache on their own.
• Gemini 4 Argon accepts a thinking level (low, medium, or high via reasoning_effort) and returns HTTP 400 if given a thinkingBudget. Keep max_tokens at or below 65,536.
• Track elapsed time against limits.agent_timeout_sec and wrap up model calls before the deadline.

4. The build container (install)

• Runs once in a clean debian:bookworm container as root, with internet access and no task data, subject to a 30-minute timeout.
• Your repository is checked out at /opt/harness/src (the initial working directory). $HOME is a temporary scratch directory.
• Preinstalled tools include uv (/opt/harness/bin/uv), Node 22 (/opt/harness/node, where npm -g installs packages), git, curl, standard build tools, and python3.
• Only /opt/harness is archived into the runtime bundle (up to 2 GiB compressed / 8 GiB unpacked), and all symlinks must resolve inside /opt/harness. System packages installed outside /opt/harness (e.g., via apt-get) are discarded after the build. Your git repository (including full history) must be at most 1 GiB: every committed version of artifacts/ stays in git history, so keep artifacts compact or squash history if it grows.
• Pin dependency versions for reproducible builds. The serve container uses the same base image, so binaries built during install run identically at evaluation time.

5. The serve container (serve)

• Each task starts a fresh container from the base image with your /opt/harness bundle mounted read-only. Your serve command runs in /opt/harness/src as uid 1000 with writable /tmp and $HOME tmpfs mounts (1 GB each), 2 CPUs, 4 GB RAM, and a 512-process limit.
• Listen on 0.0.0.0:$PORT within 120 seconds. If your server fails to accept a WebSocket connection in time, the task fails and counts against your score. Each container serves one WebSocket connection for one task.
• Route all outbound traffic through HTTPS_PROXY / HTTP_PROXY / ALL_PROXY (NODE_USE_ENV_PROXY=1 is pre-set for Node.js). Standard Python (requests, httpx), OpenAI, Anthropic, and Google SDKs use these environment variables automatically. Direct DNS resolution is disabled; the proxy resolves hostnames for you. Runtime package downloads are blocked.
• Standard output and standard error are captured as the serve log.

6. Keys: .env

Place all API keys in a .env file at the root of your repository, with one NAME=value per line:

OPENAI_API_KEY=sk-...

• Include the model API key Section 3 requires (none for google-cloud/). Variables are injected into the serve container's environment at runtime; the .env file itself is never copied into the bundle.
• Commit .env to a private repository with the competition's GitHub App installed. Public repositories containing a .env file are rejected (if you ever push a key to a public repo, revoke it immediately).
• If .env is gitignored, force-add it with git add -f .env.
• Keep each value on a single line without variable interpolation (${...}). Evaluator-reserved variable names (Section 2) are rejected except for the required model key. Limits: at most 32 variables and 16 KiB.

7. The protocol

Each task communicates over a single WebSocket connection using UTF-8 JSON text frames (up to 16 MiB per frame, one JSON object per frame). The evaluator sends the opening message:

{"type": "task", "protocol": 1, "session": "3f2c9a1e0b7d4c55", "instruction": "<the task, verbatim>",
 "workdir": "/app",
 "limits": {"agent_timeout_sec": 21600, "exec_timeout_sec": 600, "max_output_bytes": 100000,
            "max_write_bytes": 10000000},
 "mcp_servers": [{"name": "apps", "url": "http://127.0.0.1:8000/mcp/", "transport": "streamable-http"}]}

Your server then sends action requests, each with a unique id of your choice (up to 8 run concurrently and the rest queue), and receives a matching {"type": "result", "id": ...} response:

Protocol details:

• exec runs bash -c inside the task container starting in workdir as the task user. Only filesystem state persists across calls (working directory changes via cd and shell variables do not). Commands time out at min(timeout_sec, exec_timeout_sec, remaining_task_time) (default timeout_sec: 120), returning timed_out: true and exit_code: 124.
• File paths may be absolute or relative to workdir, and file contents are base64-encoded in both directions. To write files larger than 10 MB, split them across multiple write_file calls with append: true.
• http reaches only the task's declared mcp_servers (if any) inside the task container. These servers use Streamable HTTP MCP; tools/list responses on some tasks include hundreds of tools and exceed 500 KB.
• Failed operations return ok: false with an error message. Malformed frames also return an error without aborting the task, but more than 20 malformed frames end the session.

One-way messages from your harness (no reply):

• {"type": "log", "text": "..."}: appends a line to the trial log.
• {"type": "usage", "model": "...", "input_tokens": N, "output_tokens": N, "cache_read_tokens": N}: reports token usage for a model call.
• {"type": "done", "final_message": "..."}: finishes the task cleanly.
• {"type": "error", "message": "..."}: aborts the task early (any files already written in the container are still graded).

When the task time limit expires, the evaluator sends {"type": "cancel", "reason": "timeout"} and closes the WebSocket 10 seconds later. On receiving cancel, stop in-flight model calls immediately and send done with your best summary; both final_message and any files on disk still count. If the socket closes before done or error (or if your server never listens), the task fails and counts against your harness.

Failures on the evaluator's side are retried, up to three attempts per task; a task still without a result counts 0 in that test run, and a test run in which no task could be graded runs again, keeping the runs already scored. For Google-sponsored models, endpoint outages, timeouts, and rate limits are evaluator faults; for self-hosted or BYO-key models, endpoint errors and outages count against your entry. For any model, an invalid request from your harness (such as a 400 error) counts against your entry. A harness that exits with an error before its model produces any output is treated as not started and run again, up to three attempts in all; if it never starts, the task counts 0.

See sample_submissions/taskbridge.py in the competition repository for a ready-to-use implementation (WebSocket server, per-task session handling, cancellation, and a local subprocess relay), or docs/protocol.md for the full specification.

8. Local checks

How tasks are graded:

• Graders inspect the exact deliverable file paths, names, and formats requested in the task instruction. Deliverables account for the vast majority of every score.
• Rubric judges also read done.final_message alongside the deliverable files. Always finish with a concise summary of what you produced and where each file is saved.

High-scoring harnesses read the full instruction carefully before acting, inspect the starting files in workdir, work in small verified steps, write an initial deliverable early and refine it, and budget time so they always send done before agent_timeout_sec.

The local kit in the competition repository runs the same checks as the evaluation server. You need uv and a local Docker daemon (remote DOCKER_HOST endpoints are rejected because the kit bind-mounts local directories). Containers run as linux/amd64 (emulated on Apple Silicon). Running the dev set also requires Harbor (uv tool install harbor==0.20.0 --with websockets). Keep your entry in its own git repository beside the competition repository; inside the clone, kit check warns and never reports it ready to submit. From the competition repository root:

cp -r sample_submissions/pi-opus ../my-harness && git -C ../my-harness init   # a starting point of its own
uv run python -m kit check ../my-harness [--cancel-test] [--rebuild]
uv run python eval/run.py ../my-harness       # run and score the dev set, writing ../my-harness/artifacts/

If your harness files and install command have not changed since the last build, the cached bundle is reused automatically (pass --rebuild to force a fresh build). If the LLM judge fails during a dev-set run, eval/run.py reports the error and exits non-zero; after fixing your judge setup, re-run grading without repeating the trials via uv run python eval/run.py --regrade runs/<name>-<time>.

Detailed logs are written to ~/.cache/plebench/<name>/: build.log, serve.log, egress.log (allowed and blocked outbound connections), relay.log (protocol requests and responses), and check.json. To test a google-cloud/ model locally, start the local Vertex AI proxy in your own Google Cloud project with kit/model_proxy/start.sh (VERTEXAI_PROJECT, MODEL_PROXY_URL, MODEL_PROXY_KEY). Additional kit subcommands: python -m kit validate, build, up, and models.

Dev-set artifacts

Every submission must include an artifacts/ directory in the same commit as the harness, holding the outputs your harness produced on the dev set. Running eval/run.py <entry> generates <entry>/artifacts/ automatically (one subfolder per dev task with files/<container path>, final_message.md, and a top-level manifest.json recording the run and harness state). Never create or edit artifacts/ by hand.

uv run python eval/run.py ../my-harness                  # runs the dev set and writes ../my-harness/artifacts/
uv run python eval/run.py ../my-harness --tasks <id>     # reruns a single task, keeping the other task folders
uv run python -m kit artifacts ../my-harness             # validates artifacts/ against the evaluation server's rules
uv run python eval/run.py --artifacts ../my-harness      # grades committed artifacts as the server does; only the judge is called
git -C ../my-harness add -A && git -C ../my-harness commit -m "dev artifacts"   # commit everything together

When you submit, the evaluation server runs four steps in order: it builds the harness and runs the smoke task; grades the committed artifacts with the dev-set graders to compute the dev score (any task without usable artifacts counts 0); runs the harness live on a few randomly sampled dev tasks to verify that its scores reproduce (entries scoring well below their committed artifacts are flagged for organizer review); and runs the harness on the hidden test set, whose score ranks the leaderboard. Your harness runs three times on the hidden test set, all at once, and your test score is the mean of the three runs. A submission that fails the build or smoke step stops there and is not graded.

At submit time, the submit form refuses any commit that:

• Lacks an artifacts/ folder,
• Substitutes an archive (artifacts.zip, .tar.gz, etc.) instead of an unpacked directory,
• Tracks artifact files with Git LFS,
• Exceeds 512 MiB or 10,000 files in artifacts/, or
• Submits a folder in which no dev task has usable artifacts.

Individual dev tasks with a missing folder, a symlink, a file over 95 MiB, or no gradable deliverables score 0. Whenever you modify your harness, rerun the dev set and commit everything together (git add -A): manifest.json hashes all harness files, and the server checks that your submitted code matches the harness that generated artifacts/ (a change to README, LICENSE or CHANGELOG files alone gives a note instead of a warning). See docs/artifacts.md in the competition repository for the full rules.

Before submitting, confirm that every kit check stage passes, eval/run.py completed with deliverables in the expected paths, and kit artifacts accepts your committed artifacts/ directory.

9. Submit and track

1. Commit and push all changes to GitHub. Record the commit SHA via git rev-parse HEAD.
2. Before their first submission, the participant signs in to the portal with GitHub and registers there. The participant opens the portal's Submit page (/competition/submit), selects the repository and branch, reviews the checks, and submits.
3. The submission page tracks progress through queued, validating (commit and artifact structure checks), building, smoke, running (the dev artifacts are graded, then the reproduction, then the test set), grading, and finally scored or failed. It also shows the dev score and reproduction verdict (pending, consistent, flagged, unmeasured, or not_run). If a submission fails, inspect the failure reason and build/smoke logs, fix the issue, push a new commit, and resubmit. After a failure on the evaluator's side (failed, reason error) with test runs already in (the submission page says they are kept), do not resubmit: an organizer re-queues it, and only the missing runs run.

A submission reserves a slot as soon as it is queued. If it fails before reaching running (invalid, build_failed, or smoke_failed) or fails due to an evaluator error, the slot is returned. A participant can have one submission in the queue or under evaluation at a time, and can submit the next one once it is scored or failed. An evaluation takes about a day once it starts, and up to about three days when a harness uses most of the 6-hour limit on many tasks; time in the queue comes on top.

If the entry uses the participant's own API key or model server, the participant pays for every model call the evaluation server makes: the smoke task, the reproduction check on a few dev tasks, and the hidden test set three times, including any task or run repeated after a fault on the evaluator's side. The key's rate limits or the server must sustain about 36 tasks at once; 429s and timeouts from that key or server count against your entry.

10. Troubleshooting

11. Fair play

Do not hard-code task solutions or write logic that fingerprints specific tasks. Committed artifacts/ must be the unaltered output of your submitted harness, without precomputed answers to dev or test tasks. Do not bypass network restrictions, inspect grading scripts, or tamper with the evaluation container. Your declared model_api must match the model your harness calls, and you must not retain or share task contents. Organizers audit top-ranking code and artifacts before publishing final results.