Most engineering benchmarks test whether a model can answer a technical question; professional work requires reconciling drawings, specifications, calculations, and code requirements into a deliverable someone can review. PLE targets that larger gap: agents must work through realistic project files and produce engineering outputs, not just solve isolated problems.
Key Findings
- The full benchmark contains 85 tasks across four disciplines; the public development set includes 29. This gives researchers a concrete way to test long-horizon work while leaving the rest for a competition and private evaluation.
- Tasks use project workspaces with materials such as drawings, spreadsheets, specifications, and solver scripts. The agent has up to six hours per task, making planning and file-level reasoning part of the evaluation.
- Grading combines quantitative checks and expert rubrics. This captures both verifiable calculations and less easily automated deliverables, though rubric-based scores can be less reproducible than exact checks.
- The paper reports a mean score of 50.6% for the strongest evaluated configuration. The result suggests that strong general-purpose benchmark performance does not yet translate into reliable end-to-end engineering work.
Who It’s For
A good fit for researchers building or evaluating agents for complex professional tasks, especially those interested in engineering workflows and long-context tool use. It is less suitable for testing basic engineering knowledge or comparing models on unrestricted tasks: access to the development set is gated, the tasks are discipline-specific, and scores reflect performance in a controlled environment rather than licensed engineering approval.
