Data version baa9a1af5f18

Data & citation

Every number on this site comes from the files below, exported from the paper's own results. Download them to check a figure, reuse the scores, or cite the benchmark.

results.json

Paper results

Table 3: each configuration's Design, Reimplementation and Overall means with 95% intervals, and the per-task score of all 286 runs in the paper. The leaderboard and Explore page read this file.

Openresources.json

Effort per configuration

Mean agent time, probes and tokens per task, for each configuration and task family, as used by the Explore page.

Opentask-analysis.json

Task scores

The paper's task-by-configuration score table, with each cell's status (valid or invalid) and run identifier.

Openlayouts/index.json

Final layouts

Index of the same-scale final layouts on the task pages: each run's SVG, the task's shared view and scale bar, and the hash of the GDS it was drawn from.

Opencatalog.json

All 318 published runs with links to their full records: every probe, the final evaluation and provenance hashes. These records keep their archived values, so a Reimplementation reporting_score there is 2 / (1 + J / J_baseline), twice the paper's S.

Openrf/catalog.json

RF preview

The 4 RF preview tasks and 9 runs, with diagnostic S = 1 / (1 + J). Not part of the paper's ranking.

Opencitation.bib

Citation

BibTeX entry for PhysDesignBench.

DownloadGitHub

Source repository

Task environments, evaluators, the agent harnesses and the scripts that produce these files.

View repository

Run records

What a published run contains

Published

Task, model, agent and run identifiers
Objective J and score S of every probe and of the final evaluation
The final layout, bound by hash to the archived run
Hashes of the submission, evaluation and archive

Not published

The evaluator's private checks and internals
Credentials, environment values and infrastructure details
Agent transcripts and intermediate files

This release

What the files cover

26 tasks: 16 Design and 10 Reimplementation
11 model–agent configurations, one run each per task: 286 runs in the paper
32 further published runs from two mini-SWE-agent framework swaps, shown on task pages but not in the ranking
4 RF preview tasks with diagnostic scores

Cite

PhysDesignBench

@misc{momeni2026physdesignbench,
  author = {Ali Momeni},
  title = {PhysDesignBench: Evaluating AI Agents on End-to-End Physics-Based Engineering Design},
  year = {2026},
  url = {https://physdesignbench.com}
}

Reuse

License and attribution

The PICBench repository is released under the MIT License. Please cite PhysDesignBench when you use its task definitions, evaluation framework, published results or figures.