Official rules

The Physical-Engineering’s Last Exam (PLE) harness competition. Free to enter.

1. Binding agreement

   To enter the competition, you must agree to these Official Rules ("Rules"). Registering on this site or submitting an entry means that you agree to them. These Rules form a binding legal agreement between you and Google with respect to the competition. If you do not agree, do not register or submit an entry.

2. Sponsor

   The competition is sponsored by Google LLC ("Google"), 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA.

3. Competition period

   The Competition Period starts when the competition opens on this site and runs until an end date that will be announced on this site. Submissions are accepted during the Competition Period only. Every submission made during the Competition Period completes all of its test runs before the final standings are published.

4. How to enter

   Sign in to this site with your GitHub account and register with your name, email address, and affiliation. Each person may register once, with one GitHub account. Then submit an entry as the guide describes: a single commit of a GitHub repository with your harness, its submission.yaml, and the dev-set artifacts the harness produced. You have 3 scored submissions for the whole competition, with one in the queue or under evaluation at a time. Entries that are incomplete, altered, or obtained by fraud are void.

5. Evaluation

   Entries are evaluated on 85 engineering tasks: 29 public dev tasks and 56 hidden test tasks. Each scored submission runs three times on the test set, with a limit of 6 hours per task, and its test score is the mean of those runs, on a 0 to 1 scale. The leaderboard ranks each entrant by their best scored submission; ties go to the earlier submission. The guide describes the scoring in full. Google's decisions on evaluation, scoring, and ranking are final.

6. Fair play and disqualification

   Google may disqualify, at its sole discretion, any entry or entrant that:

   • hard-codes task answers or detects specific tasks, or commits dev-set artifacts that the submitted harness did not produce;
   • tries to access the hidden test tasks, the graders, or the evaluation systems, or to get around their network restrictions;
   • abuses the sponsored model access the competition provides, or uses it for anything other than the entry;
   • enters through more than one account, or keeps or shares the contents of evaluation tasks; or
   • otherwise cheats, deceives, or undermines the fair running of the competition.

   The organizers review the code and artifacts of top-ranking entries before results are published.

7. Submissions and intellectual property

   As between you and Google, you keep ownership of your entry, including your harness code. By submitting, you grant Google and its affiliates a worldwide, royalty-free, non-exclusive license to copy, build, run, and evaluate your entry for the competition, and to publish its entry name, its scores, and results derived from its evaluation. You warrant that the entry is your own work, or that you have the right to submit any third-party code it contains under that code's license, and that it does not infringe anyone's rights or break any law.

8. Privacy

   Google collects personal data when you register and enter. The privacy notice below says what is collected, how it is used, and how to access, correct, or delete it.

9. Publicity

   The public leaderboard shows each entry's GitHub username, entry name, and scores. By entering, you agree that Google may show them there and when it announces the results, without compensation, unless the law prohibits it.

10. Right to cancel or modify

    If the competition cannot run as planned for any reason, including tampering, fraud, technical failures, or anything else that affects its administration, security, fairness, or integrity, Google may cancel, end, modify, or suspend it at its sole discretion.

11. Warranty, indemnity, and limitation of liability

    To the maximum extent permitted by law, you agree to indemnify Google against any claims, losses, and costs that arise from your entry or from your breach of these Rules. Google is not responsible for any malfunction of this site or of the evaluation, or for errors in collecting, processing, or keeping entries. To the maximum extent permitted by law, Google and its affiliates are not liable for any damages arising from your participation in the competition, and the competition, including its evaluation and results, is provided "as is", without warranty of any kind. Some jurisdictions do not allow these limitations, so some of them may not apply to you.

Questions about these Rules: ple-bench@google.com.