Requirement interpretation
Turn a specification, its constraints and an objective J into a concrete device plan.
Methodology
PhysDesignBench tests whether an AI agent can go from a specification to a validated physical design: the agent designs with real simulators under a fixed budget, and an independent evaluator rebuilds and scores what it submits.
A complete design workflow
Tool-guided iteration
Use simulators and a limited number of evaluator probes to decide what to change next.
Artifact correctness
Submit design source that the evaluator can rebuild into a layout that passes every gate.
Verified performance
Score well on J once the rebuilt design has passed design-rule, connectivity, electrical, thermal and numerical checks.
Two kinds of engineering work
Start from functional requirements
Open-ended tasks, from single components to full electrothermal circuits. The agent chooses the device concept and how to get there. Each task has an expert reference design that passes every gate.
Start from a published device
The agent rebuilds a device from the literature and tries to improve on it. The published device's objective is the reference, so matching it scores S = 0.5.
Scoring
Objective J first, then score S
Objective
J is task-specific and computed by the evaluator; lower is better. A design that fails any gate has no J and scores S = 0.
Score
S runs from 0 to 1, and S = 0.5 when J equals the reference c. For Design, c is a fixed engineering scale per task; for Reimplementation, c is the published device's objective, so S above 0.5 beats the published device.
The leaderboard follows Table 3 of the paper: the best eligible evaluated design for Design tasks and the final submission for Reimplementation, with invalid results at S = 0, averaged per task family with 95% bootstrap intervals. The RF preview uses a diagnostic S = 1 / (1 + J) and is not part of the ranking.
What the results do and do not show
How 11 model–agent configurations compare on 26 photonic and multiphysics design tasks, with uncertainty; how often each one reaches a valid design; and every run's probes, scores and final layout.
Fabricated performance: every score comes from simulation. Nor equal cost: tokens and time are recorded as each harness reports them. Difficulty levels are relative to these configurations, and other physics domains are outside this release.