Beyond the average

Explore results

Every configuration on every task, two configurations side by side, how consistently each one scores, and what its score costs in tokens, time and probes. Scores are the paper's: the best eligible design for Design tasks and the final submission for Reimplementation, with invalid results at S = 0.

Every task

Score by configuration and task.

Rows follow the leaderboard; columns run from the easiest task to the hardest, as ranked in the paper. Hatched cells are invalid results, scored zero. Select a cell to open its task.

Score S by configuration (rows) and task (columns)
Configuration	90SiN Bend-v0	Microresonator-v1	MZI-Spec-8	Quantum-4	WDM-v1	RAFT-Spec	Microresonator-v2	MZI-v1	MZI-v2	MZI-Ring-v1	WDM-v2	Quantum-8	Ring-XC-v1	Serial MZI-v1	Ring-XC-v2	MZI-Ring-v2	Anisotropic SWG-v0	SRSN Grating-v0	Sb₂Se₃ rotator	Trapezoidal SWG-v0	Dual-trident	TFLN PI-MMI-v0	α-Si/SiN grating	TFLN Crossing-v0	HMDM WDM	HMDM PSR	Mean S
	Design · 16																Reimplementation · 10
	Easy				Medium				Hard			Expert					Easy			Medium			Hard			Expert
GPT-6 AstraCodex	0.72	0.75	0.51	0.57	0.85	0.44	0.59	0.54	0.79	0.40	0.71	0.48	0.31	0.05	0.43	0.02	0.68	0.59	0.93	0.40	0.35	0.33	0.27	0.22	0.31	0.11	0.47
Opus 5.5Claude Code	0.74	0.78	0.48	0.49	0.50	0.43	0.53	0.67	0.41	0.38	0.69	0.38	0.37	0.04		0.01	0.63	0.52	0.73	0.31	0.37	0.33	0.26	0.17	0.05	0.09	0.40
GPT-6.1 SolCodex	0.65	0.78	0.50	0.58	0.81	0.44	0.60	0.72	0.41	0.28	0.06	0.34		0.10		0.01	0.64	0.57	0.65	0.47	0.35	0.32	0.26	0.29	0.31	0.12	0.40
Fable 5.1Claude Code	0.76	0.63	0.49	0.50	0.61	0.42	0.57	0.32	0.53	0.24		0.10		0.04		0.01	0.39	0.56	0.50	0.36	0.36	0.34	0.27	0.36	0.25	0.09	0.33
Sonnet 5.5 (high)Claude Code	0.61	0.65	0.47	0.52	0.13	0.41	0.51	0.68	0.39	0.35	0.36						0.60	0.53	0.47	0.35	0.35	0.31	0.27	0.30	0.03	0.10	0.32
Gemini 4 Argonmini-SWE-agent	0.70		0.50	0.51		0.44	0.59			0.41				0.30			0.71	0.52	0.26	0.28	0.35	0.22	0.22	0.19	0.24	0.09	0.25
Muse Wonder (high)Muse Code	0.67	0.49	0.48	0.46	0.60									0.04		0.01	0.67	0.53	0.52	0.37	0.28	0.31	0.23	0.29	0.25	0.09	0.24
GPT-5.6 SolCodex	0.64	0.62	0.34	0.21	0.10	0.42		0.33		0.27	0.07	0.12					0.64	0.52	0.50	0.26	0.14	0.30	0.22	0.20	0.06	0.10	0.23
Kimi K3.1 ProClaude Code	0.64	0.61	0.47	0.29		0.41				0.04			0.37	0.10			0.37	0.52	0.50	0.25	0.32	0.27	0.21	0.15	0.15	0.09	0.22
GLM 5.3 Flash (max)mini-SWE-agent	0.62	0.46	0.48														0.31	0.52	0.15	0.34	0.18	0.12	0.22	0.12	0.31	0.09	0.15
GLM 5.3 Flash (high)Claude Code	0.52		0.45														0.32	0.52	0.26	0.04	0.11	0.25	0.21	0.27	0.21	0.09	0.13
Task mean	0.66	0.53	0.47	0.37	0.33	0.31	0.31	0.30	0.23	0.22	0.17	0.13	0.10	0.06	0.04	0.00	0.54	0.54	0.50	0.31	0.29	0.28	0.24	0.23	0.20	0.10

Hover or focus a cell for its exact score.

Head-to-head

Two configurations, task by task.

Each dot is a task. Below the diagonal the configuration on the horizontal axis scores higher; above it, the one on the vertical axis. Invalid results count as S = 0 and are drawn as rings.

Horizontal axisVertical axis

GPT-6 Astra higher: 19
Opus 5.5 higher: 5
Ties (within 0.005): 2

26 tasks. Hover a dot or a gap for its scores.

Largest gaps

Consistency

How often each configuration reaches a score.

The share of tasks on which each configuration scores at least the threshold. Invalid results count as S = 0, so every share is out of all 26 tasks. Select configurations to compare their full curves.

ThresholdS ≥ 0.50

Effort

Score against tokens, time and probes.

Each dot is a configuration: its mean score against what it spent per task. The dashed line joins the configurations that no other one beats on both score and effort.

Mean S against tokensOverall · 11 configurations

Hover a dot or a configuration for its numbers.

Tokens are recorded input plus output per task, cached input counted once; ≥ marks a lower bound where some usage was not recorded. Equal tokens do not mean equal cost.