HiPhO: High School Physics Olympiad Benchmark

• HiPhO is a benchmark built on 13 recent Olympiad exams, offering a detailed evaluation framework with medal-based scoring aligned to official marking schemes.
• It addresses prior deficits by incorporating multimodal problems, step-level grading, and mapping model performance directly to human contest benchmarks.
• The system enables the assessment of models on theoretical derivations and diagram-based tasks, highlighting strengths and persistent challenges in advanced physics reasoning.

HiPhO is a High School Physics Olympiad benchmark built from recent, real Olympiad exams and designed for human-aligned evaluation of multimodal, olympiad-grade physics reasoning. It is described as the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation, compiling 13 latest Olympiad exams from 2024–2025, using official marking schemes, and mapping model scores to gold, silver, and bronze medal thresholds so that models can be compared directly with human contestants (Yu et al., 9 Sep 2025). In later work, HiPhO is also treated as a high-end scientific-reasoning benchmark, including by systems that target long-horizon reasoning, multimodal perception, reinforcement learning, and multi-agent verification (Bai et al., 29 Jun 2026).

1. Origin and benchmark rationale

HiPhO was introduced to address four deficits identified in prior physics evaluation suites: outdated Olympiad coverage, insufficient multimodal content, coarse answer-only evaluation, and the lack of direct comparison to human contestants (Yu et al., 9 Sep 2025). Earlier datasets were described as either stopping well before the most recent Olympiads, omitting diagram-heavy problems, relying on answer accuracy without official stepwise partial credit, or reporting scores in forms that did not correspond to how Olympiad medals are actually awarded.

The benchmark therefore adopts recent competition material, professional grading criteria, and medal-aligned reporting. Its central research question is how far current LLMs and multimodal LLMs are from human contestants on the latest high-school physics Olympiads, not only in final-answer accuracy but also in sustained derivational reasoning and exam-level performance (Yu et al., 9 Sep 2025).

This framing places HiPhO closer to contest-grade scientific evaluation than to conventional benchmark QA. Later papers explicitly use it as a stress test for “Olympiad-level” or “science-grade” reasoning, especially in settings where diagrams are not merely decorative but encode essential constraints that are absent from the text (Luo et al., 10 Feb 2026).

2. Corpus, modalities, and task structure

HiPhO compiles 13 exam papers from 2024–2025 spanning seven competition series: IPhO, APhO, EuPhO, NBPhO, PanPhO, PanMechanics, and F=MA (Yu et al., 9 Sep 2025). In the benchmark paper’s dataset statistics, this yields 360 problems and 519 subquestions across five physics fields—Mechanics, Electromagnetism, Thermodynamics, Optics, and Modern Physics—with six answer types and four modality types. The corpus contains 308 problems in English and 52 in Chinese (Yu et al., 9 Sep 2025).

The modality taxonomy is explicit. Problems are grouped into Text-Only (TO), Text + Illustration Figure (TI), Text + Variable Figure (TV), and Text + Data Figure (TD) (Yu et al., 9 Sep 2025). Subsequent work on vision-LLMs emphasizes that, at Olympiad level, diagrams are often “constitutive rather than illustrative,” carrying boundary conditions, circuit topology, spatial symmetry, or quantitative data that the text does not restate (Luo et al., 10 Feb 2026).

HiPhO focuses on theoretical and visualization-based reasoning. Experimental labs, diagram-drawing tasks, and plot-sketching tasks are excluded, because current models are not evaluated as embodied experimental agents or robust diagram generators in the benchmark’s core protocol (Yu et al., 9 Sep 2025). As a result, some papers distinguish between Full Mark (Human) and Full Mark (Model), with the latter restricted to the theoretical parts available to the model (Luo et al., 10 Feb 2026).

The problems themselves are open-ended and multi-step. The benchmark includes long theoretical derivations, semi-experimental or data-analysis tasks, short-answer subparts embedded inside larger solutions, and question chains in which intermediate quantities feed later parts (Yu et al., 9 Sep 2025). This structure is important because official marking schemes assign credit to intermediate reasoning, not only to terminal boxed answers.

3. Evaluation formalism and human-aligned scoring

HiPhO’s defining methodological feature is its combination of answer-level and step-level grading. For each problem, the benchmark computes an answer-level score by checking final answers and a step-level score by comparing the full solution against the official marking scheme; the final problem score is then

$$\text{Problem Score} = \max(\text{answer-level score}, \text{step-level score}).$$

This mirrors Olympiad grading practice: a correct final answer can receive full credit even if exposition is terse, while a wrong final answer can still earn substantial partial credit through correct intermediate reasoning (Yu et al., 9 Sep 2025).

Step-level grading is rubric-based rather than synthetic. Official marking schemes are structured into machine-readable criteria, and when multiple valid official solutions exist, the benchmark evaluates all available schemes and takes the maximum step-level score (Yu et al., 9 Sep 2025). For modality-wise or field-wise analysis, HiPhO uses the mean normalized score

$$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$

where $M$ denotes a modality class or physics field (Yu et al., 9 Sep 2025).

The benchmark also supports exam-level aggregation. Later papers using the 13-exam suite report an average score of the form

$$\text{Avg} = \frac{1}{13}\sum_{e=1}^{13} \text{Score}_e,$$

and reproduce average medal thresholds over the 13 exams of $\bar{G} = 29.3$, $\bar{S} = 19.7$, and $\bar{B} = 11.4$ (Luo et al., 10 Feb 2026).

HiPhO’s medal mapping is central to its human alignment. Model exam scores are compared against official or benchmark-reconstructed gold, silver, and bronze thresholds, allowing statements of the form “gold on IPhO 2025” rather than only raw percentage accuracy (Yu et al., 9 Sep 2025). In the core benchmark setup, inference is implemented with VLMEvalKit, uses 8 independent runs per problem at temperature $0.6$, and expects structured outputs with full reasoning and final boxed answers (Yu et al., 9 Sep 2025).

4. Empirical findings and difficulty profile

The initial large-scale study evaluates 30 state-of-the-art models and reports a sharply stratified landscape (Yu et al., 9 Sep 2025). Across the 13 exams, open-source MLLMs mostly remain at or below the bronze level; open-source LLMs show promising progress with occasional golds; closed-source reasoning MLLMs can achieve 6 to 12 gold medals; and most models still have a significant gap from full marks (Yu et al., 9 Sep 2025).

The benchmark paper also shows that the best models remain well below top human contestants on most exams. One nuance is that performance is heterogeneous across contests: on easier or more mechanics-heavy settings, strong models can approach or even exceed the top reported human score for a specific exam, while still lagging substantially on more difficult multimodal Olympiads (Yu et al., 9 Sep 2025). A plausible implication is that HiPhO measures not a single monolithic “physics ability,” but an interaction among symbolic derivation, visual interpretation, long-horizon consistency, and competition-specific grading compliance.

The difficulty structure is systematic. Performance declines as visual complexity increases, with the ordering TO $>$ TI $>$ TV $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$0 TD across model classes (Yu et al., 9 Sep 2025). Field-wise, optics is consistently the most challenging area; even strong closed-source models that score around 70–80% normalized performance in mechanics, electromagnetism, thermodynamics, and modern physics are markedly weaker in optics (Yu et al., 9 Sep 2025). The benchmark’s error analyses attribute failures to diagram misinterpretation, fragile algebraic execution, unit and dimensional mistakes, graph-reasoning errors, and conceptual misunderstandings in applying physical laws (Yu et al., 9 Sep 2025).

In this sense, HiPhO functions not only as a leaderboard but also as a diagnostic instrument. It distinguishes models that can retrieve familiar formulas from models that can sustain olympiad-style derivations under multimodal constraints and partial-credit grading.

5. HiPhO in subsequent model development

HiPhO rapidly became a primary target for systems built explicitly for Olympiad physics. “PhysicsMinions” is a coevolutionary multimodal multi-agent system organized into a Visual Studio, Logic Studio, and Review Studio, and evaluated on the seven latest physics Olympiads under the HiPhO setup (Yu et al., 29 Sep 2025). It reports three major outcomes: consistent gains over single-model baselines, the first-ever open-source gold medal in the latest IPhO under the average-score metric, and a Pass@32 score of $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$1 on the latest IPhO, corresponding to 4th of 406 contestants (Yu et al., 29 Sep 2025). Its design explicitly reflects HiPhO’s step-level grading and multimodal demands: structured figure extraction, stepwise LaTeX derivations, physics-specific verification, and iterative self-correction.

The P1 family extends this direction through reinforcement learning. “P1: Mastering Physics Olympiads with Reinforcement Learning” trains open-source models entirely through RL on curated Olympiad-style physics problems and evaluates them on HiPhO, where P1-235B-A22B earns 12 gold and 1 silver, while P1-235B-A22B + PhysicsMinions reaches an average of $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$2 and achieves the top reported IPhO 2025 score of $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$3 in that paper’s comparison (Chen et al., 17 Nov 2025). The work treats HiPhO as a contest-grade evaluation target rather than a generic physics QA dataset.

P1-VL adds multimodal RL post-training to this line. “P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiads” presents open-source vision-LLMs evaluated on the 13-exam HiPhO suite, where P1-VL-235B-A22B reports an average score of $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$4 with 12 gold and 1 silver, and P1-VL-235B-A22B + PhysicsMinions reports $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$5, ranking second overall in that paper’s global comparison (Luo et al., 10 Feb 2026). The paper argues that HiPhO is a genuine VLM benchmark because many Olympiad figures encode indispensable physical constraints rather than optional visual context.

HiPhO has also been incorporated into broader agentic evaluation. “Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent” treats HiPhO as one of four representative scientific-research benchmarks and reports that Agents-A1, a 35B Mixture-of-Experts agentic model, achieves $$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$6 on HiPhO, ahead of several larger comparison models in that evaluation table (Bai et al., 29 Jun 2026). This suggests that HiPhO now serves two roles simultaneously: a specialized Olympiad benchmark and a general stress test for long-horizon scientific reasoning.

6. Limitations, controversies, and open directions

HiPhO’s scope is deliberately constrained. It excludes experimental labs, embodied measurement, and diagram-generation tasks, so model full marks can be below human full marks on some exams (Yu et al., 9 Sep 2025). The benchmark is also biased toward elite contest settings rather than ordinary classroom physics, and its coverage remains concentrated in English and Chinese with some major competitions omitted because public score distributions are unavailable (Yu et al., 9 Sep 2025).

A second limitation is that multimodality remains unresolved even when benchmark quality is high. Later systems report substantial progress, but their own analyses continue to expose bottlenecks. PhysicsMinions notes that fine-grained data extraction from dense plots remains difficult (Yu et al., 29 Sep 2025). P1-VL identifies persistent challenges in image-heavy reasoning and relies on frozen vision encoders during RL post-training (Luo et al., 10 Feb 2026). Agents-A1 shows that a unified policy can fall slightly below a domain-specialized science teacher on HiPhO, indicating a tradeoff between cross-domain deployment and per-domain peak performance (Bai et al., 29 Jun 2026).

A further issue is protocol heterogeneity. The benchmark paper specifies one evaluation recipe, whereas later papers sometimes adopt different sampling or aggregation schemes, such as multi-run averages, Pass@$$\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,$$7, or maxima over multiple attempts (Yu et al., 9 Sep 2025). This does not invalidate HiPhO, but it means that cross-paper comparisons require attention to inference settings and scoring conventions.

The benchmark authors propose broader exam coverage, dynamic annual updates, stronger multimodal reasoning, support for generated diagrams or plots, and eventually more embodied forms of physical evaluation (Yu et al., 9 Sep 2025). Subsequent work reinforces those directions. A plausible implication is that future HiPhO-like benchmarks will need to integrate visual perception, symbolic derivation, tool use, and verification in a single evaluation framework if they are to approximate the full skill profile of human Olympiad problem solving.