From real engineering work
A design problem you know well from your own research or practice: photonics, RF and microwave, multiphysics, or another area of physical design.
Help build PhysDesignBench.
We invite researchers and engineers from every area of physics-based design to be part of this benchmark. Bring a design problem from your own work, review tasks from others, or run new agents. The benchmark starts with photonic, multiphysics and RF design, and we want it to grow with the community.
01 · What we look for
A good task is real, checkable and hard.
Checkable by simulation
A clear objective J and validity gates that an independent evaluator can compute from the submitted design alone.
Solvable, but not easy
A reference design that passes every gate, and a problem that today's agents cannot solve by recall or a single simulation.
Fits the run budget
An agent can make real progress within 120 minutes and up to 40 evaluator probes on standard simulation tools.
02 · Parts of a task
Browse the task folders →One folder, five parts.
Every task lives in its own folder in the repository. The agent and the evaluator run in separate containers, so the agent never grades its own work.
instruction.md- What the agent reads: the specification, the constraints, what to submit and how it is scored.
environment/- The agent's container: simulators, design rules, task files and a starting baseline.
evaluator/- A separate container the agent never sees. It rebuilds the submitted design, applies every gate and computes J.
solution/- Your reference design. It must pass every gate, which shows the task is solvable and sets the score's reference scale.
task.toml · tests/- Task metadata, budgets and the verifier entry point, in the Harbor task format used by every existing task.
03 · From idea to merge
How a task joins the benchmark.
- 1
Propose
Open an issue describing the device, its objective and how a design would be checked.
- 2
Agree on scope
We discuss the gates, the reference and the budget with you before you build.
- 3
Build
Write the task package in tasks/<task-id>/, using an existing task as a template.
- 4
Test locally
Your reference passes every gate, and at least one agent runs end to end with Harbor.
- 5
Review
Open a pull request. Maintainers and domain experts check the evaluator, gates and reference.
- 6
Merge
The task joins the benchmark, and its results appear on this site.
04 · Other ways to help
Not every contribution is a new task.
Review a task
Domain experts help most by checking that a specification is sound, its gates are fair and its reference is right.
Run new agents
Evaluate another model or agent harness on the tasks with Harbor and share the results.
How to run the benchmark →Report a problem
An evaluator bug, an unclear instruction or a mistake on this site: tell us in an issue.
Open the issue tracker →Have a design problem in mind?
Open an issue with a short description. We will reply there and help shape it into a task.