PhysDesignBench
GitHub
11 model–agent configurations · 16 Design + 10 Reimplementation tasks

Leaderboard

Mean normalized scores with 95% confidence intervals, the same numbers as the paper's Table 3. Switch between task families, sort by any column, or filter by agent framework.

Leaderboard table

Agent:
#Model · agent% · 95% CIOverall %of 16mean / taskmin / taskM / task
1GPT-6 AstraCodex · OpenAI47.45±9.0447.431625.312023.4
2Opus 5.5Claude Code · Anthropic39.92±8.8339.721523.49210.6
3GPT-6.1 SolCodex · OpenAI39.52±9.4439.521425.012020.3
4Fable 5.1Claude Code · Anthropic33.45±8.4533.451320.69011.6
5Sonnet 5.5 (high)Claude Code · Anthropic32.18±8.5532.101124.68114.9
6Gemini 4 Argonmini-SWE-agent · Google25.18±9.0425.1877.912014.8
7Muse Wonder (high)Muse Code · Meta24.17±8.9124.17726.09129.4‡
8GPT-5.6 SolCodex · OpenAI23.31±8.0823.311030.512024.2
9Kimi K3.1 ProClaude Code · Moonshot AI · open weights22.22±8.0722.22823.711724.6
10GLM 5.3 Flash (max)mini-SWE-agent · Z.ai · open weights15.08±7.4015.08313.79032.3
11GLM 5.3 Flash (high)Claude Code · Z.ai · open weights12.54±6.2512.54210.010310.7

Scores are mean normalized scores over all 26 tasks, with invalid outcomes scored zero, exactly as in Table 3 of the paper. Intervals are 95% task-bootstrap intervals; they reflect task sampling, not run-to-run variation. Final submission scores Design from the final submission only; grey means it equals the headline score. ‡ Lower bound: incomplete usage records. Ranks always follow the selected family's score.

Score against resources

What each point of score costs.

Similar scores can come at very different prices, times and probe counts. Switch the x-axis to compare.

Score against API costOverall score (%) · dashed line = Pareto frontier
0102030405060$1$2$5$10$20$50Mean inference cost per task (USD, log scale)Overall score (%)GPT-6 AstraOpus 5.5GPT-6.1 SolSonnet 5.5 (high)GLM 5.3 Flash (max) · mini-SWE-agent

On the frontier: GLM 5.3 Flash (max) · mini-SWE-agent, Sonnet 5.5 (high), GPT-6.1 Sol, Opus 5.5, GPT-6 Astra. Hover or focus a point for details.

Validity and quality

Finding a valid design is the first hurdle.

One square per task. In Design, many runs never produce a design that passes every gate. In Reimplementation, every submission is valid, but few beat the published device.

Task outcomes per configuration
ConfigurationDesign · 16 tasksReimplementation · 10 tasks
GPT-6 Astra16/16 eligible3/10 at or above baseline
Opus 5.515/16 eligible3/10 at or above baseline
GPT-6.1 Sol14/16 eligible3/10 at or above baseline
Fable 5.113/16 eligible1/10 at or above baseline
Sonnet 5.5 (high)11/16 eligible2/10 at or above baseline
Gemini 4 Argon7/16 eligible2/10 at or above baseline
Muse Wonder (high)7/16 eligible3/10 at or above baseline
GPT-5.6 Sol10/16 eligible2/10 at or above baseline
Kimi K3.1 Pro8/16 eligible1/10 at or above baseline
GLM 5.3 Flash (max) · mini-SWE-agent3/16 eligible1/10 at or above baseline
GLM 5.3 Flash (high) · Claude Code2/16 eligible1/10 at or above baseline
Every published run

Results by task

Each marker is one published run, at the score of its final submission; select it to open its task page. The table above uses the paper's headline score, which for Design tasks is the best eligible design a run produced, so a few Design markers sit below the table's values. RF component tasks have their own panel: their results are diagnostic and not part of the paper's 26 tasks.

Design · 16 tasks

Reporting score S from 0 to 1 · one marker per published run · S = 0.5 is the fixed reference budget

Showing 13 of 13 systems · toggle systems to compare

Vertical dot plot for all 16 design and discovery tasks. Tasks are on the x-axis and reporting score S is on the y-axis from zero to one. Each system is a filled circle in its leaderboard colour. Only published runs are plotted. Published runs without a valid score are shown at S = 0.Tasks are on the horizontal axis and reporting score S is on the vertical axis from zero to one. All system markers for a task share exactly the same x-position. Each system is a filled circle in its leaderboard colour; the tooltip lists published runs for the selected systems. Published runs without a valid score are shown at S = 0.0.000.250.500.751.00Reporting score SFour-Channel Electrothermal Dual-Ring WDM ChannelizerWDM-v1Eight-Channel Electrothermal Dual-Ring WDM ChannelizerWDM-v2Single-Ring O-Band Microresonator ModulatorMicroresonator-v1Five-Channel C-Band WDM-PAM4 Microresonator ModulatorMicroresonator-v2Broadband 4x4 Rectangular MZI MeshMZI-v1Broadband 6x6 Rectangular MZI MeshMZI-v2MZI-Coupled Microring Add-Drop Filter Electrothermal DesignMZI-Ring-v1MZI-coupled microring add-drop filter electrothermal design v2MZI-Ring-v2Four-Mode Quantum Projector Electrothermal DesignQuantum-4Eight-Mode Quantum Projector Electrothermal DesignQuantum-8Compact PDK-Physical Eight-Delay Sparse-Spectrum MZI SpectrometerMZI-Spec-8Microring-Assisted Fourier Spectrometer Electrothermal DesignRAFT-SpecFour-Wavelength Two-Port Ring Selective Cross-ConnectRing-XC-v1Four-Wavelength Four-Port Ring Selective Cross-ConnectRing-XC-v2Serial Dual-Ring-Assisted MZI Elliptic Bandpass Electrothermal Design v1Serial MZI-v190 Degree SiN Waveguide Bend Design90SiN Bend-v0Task

Scroll horizontally to view all tasks.

Design S = 1/(1+J/J_scale), from 0–1; higher is better. S = 0.5 marks the fixed external reference budget, not specification acceptance. Only published runs are plotted. Published runs without a valid score are shown at S = 0. Select a marker for run details.

Reimplementation · 10 tasks

Score S from 0 to 1, as in the paper · S = 0.5 matches the published device · shaded band is at or above it

At or above the published device · S = 0.50–1.00

Showing 11 of 11 systems with results · toggle systems to compare

Vertical dot plot for all 10 reimplementation tasks. Tasks are on the x-axis and score S is on the y-axis from zero to one, as in the paper. Only published runs are plotted. Published runs without a valid score are shown at S = 0. The shaded range from 0.5 to 1 is at or above the published device.Tasks are on the horizontal axis and reporting score S is on the vertical axis from zero to one. All system markers for a task share exactly the same x-position. Each system is a filled circle in its leaderboard colour; the tooltip lists published runs for the selected systems. Published runs without a valid score are shown at S = 0.0.000.250.500.751.00Reporting score SAnisotropic SWG 2 x 2 Beamsplitter Implementation/OptimizationAnisotropic SWG-v0α-Si/Si3N4 SWG Grating Coupler Implementation/Optimizationα-Si/Si₃N₄ Grating-v0Dual-Trident SWG Edge Coupler Implementation/OptimizationDual-Trident SWG-v0HMDM Polarization Splitter–Rotator Implementation/OptimizationHMDM polarizationsplitter–rotatorHMDM Six-Channel WDM Binary OptimizationHMDM six-channel WDMSb2Se3/Si Reconfigurable Polarization Rotator Implementation/OptimizationSb₂Se₃/Si Rotator-v0SRSN Grating Coupler Implementation/OptimizationSRSN Grating-v0Three-Direction TFLN MMI Crossing Implementation/OptimizationTFLN Crossing-v0Polarization-Insensitive TFLN 1x2 MMI Implementation/OptimizationTFLN PI-MMI-v0Trapezoidal SWG 2 x 2 3-dB Coupler Implementation/OptimizationTrapezoidal SWG-v0Task

Reimplementation S = 1/(1+J/J_baseline), from 0–1 as in the paper; S = 0.5 matches the published device. Only published runs are plotted. Published runs without a valid score are shown at S = 0.

Individually approved single-task runs are not official campaign results or task-release certifications.

RF component tasks · 4 tasks

HFSS-evaluated RF design · normalized diagnostic results · preview, not in the paper's table

S = 1/(1+J) · diagnostic reporting · range 0–1

Showing 6 of 6 systems · toggle systems to compare

RF component tasks: vertical dot plot with diagnostic score S from zero to one. Waveguide filters and antennas. S=1/(1+J); higher is better. Only published runs are plotted. Published runs without a valid score have unscored markers at zero. Official qualification remains pending. Select a result for its score versus time, geometry, HFSS response and measurement provenance.Tasks are on the horizontal axis and reporting score S is on the vertical axis from zero to one. All system markers for a task share exactly the same x-position. Each system is a filled circle in its leaderboard colour; the tooltip lists published runs for the selected systems. Published runs without a valid score are shown at S = 0.0.000.250.500.751.00Diagnostic score SKu-band waveguide filter designKu-band Filter-v2Dual-band waveguide filter designDual-band Filter-v0Compact antenna designCompact Antenna v1Compact waveguide filtenna designFiltena v1Task

RF S = 1/(1+J), from 0–1; higher is better. Valid scores come from completed, native-converged diagnostic measurements, with final-measurement provenance on its task page. Only published runs are plotted. Published runs without a valid score are shown at S = 0. Official scoring and release qualification remain pending. Each task has its own objective; select a marker to view the run.