---
title: 'HiPhO: High School Physics Olympiad Benchmark'
url: https://www.emergentmind.com/topics/hipho
type: topic
---

# HiPhO: High School Physics Olympiad Benchmark

HiPhO is a High School Physics Olympiad benchmark built from recent, real Olympiad exams and designed for human-aligned evaluation of multimodal, olympiad-grade physics reasoning. It is described as the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation, compiling 13 latest Olympiad exams from 2024–2025, using official marking schemes, and mapping model scores to gold, silver, and bronze medal thresholds so that models can be compared directly with human contestants [2509.07894]. In later work, HiPhO is also treated as a high-end scientific-reasoning benchmark, including by systems that target long-horizon reasoning, multimodal perception, reinforcement learning, and multi-agent verification [2606.30616].

## 1. Origin and benchmark rationale

HiPhO was introduced to address four deficits identified in prior physics evaluation suites: outdated Olympiad coverage, insufficient multimodal content, coarse answer-only evaluation, and the lack of direct comparison to human contestants [2509.07894]. Earlier datasets were described as either stopping well before the most recent Olympiads, omitting diagram-heavy problems, relying on answer accuracy without official stepwise partial credit, or reporting scores in forms that did not correspond to how Olympiad medals are actually awarded.

The benchmark therefore adopts recent competition material, professional grading criteria, and medal-aligned reporting. Its central research question is how far current large language models and multimodal language models are from human contestants on the latest high-school physics Olympiads, not only in final-answer accuracy but also in sustained derivational reasoning and exam-level performance [2509.07894].

This framing places HiPhO closer to contest-grade scientific evaluation than to conventional benchmark QA. Later papers explicitly use it as a stress test for “Olympiad-level” or “science-grade” reasoning, especially in settings where diagrams are not merely decorative but encode essential constraints that are absent from the text [2602.09443].

## 2. Corpus, modalities, and task structure

HiPhO compiles 13 exam papers from 2024–2025 spanning seven competition series: IPhO, APhO, EuPhO, NBPhO, PanPhO, PanMechanics, and F=MA [2509.07894]. In the benchmark paper’s dataset statistics, this yields 360 problems and 519 subquestions across five physics fields—Mechanics, Electromagnetism, Thermodynamics, Optics, and Modern Physics—with six answer types and four modality types. The corpus contains 308 problems in English and 52 in Chinese [2509.07894].

The modality taxonomy is explicit. Problems are grouped into Text-Only (TO), Text + Illustration Figure (TI), Text + Variable Figure (TV), and Text + Data Figure (TD) [2509.07894]. Subsequent work on vision-language models emphasizes that, at Olympiad level, diagrams are often “constitutive rather than illustrative,” carrying boundary conditions, circuit topology, spatial symmetry, or quantitative data that the text does not restate [2602.09443].

HiPhO focuses on theoretical and visualization-based reasoning. Experimental labs, diagram-drawing tasks, and plot-sketching tasks are excluded, because current models are not evaluated as embodied experimental agents or robust diagram generators in the benchmark’s core protocol [2509.07894]. As a result, some papers distinguish between **Full Mark (Human)** and **Full Mark (Model)**, with the latter restricted to the theoretical parts available to the model [2602.09443].

The problems themselves are open-ended and multi-step. The benchmark includes long theoretical derivations, semi-experimental or data-analysis tasks, short-answer subparts embedded inside larger solutions, and question chains in which intermediate quantities feed later parts [2509.07894]. This structure is important because official marking schemes assign credit to intermediate reasoning, not only to terminal boxed answers.

## 3. Evaluation formalism and human-aligned scoring

HiPhO’s defining methodological feature is its combination of answer-level and step-level grading. For each problem, the benchmark computes an answer-level score by checking final answers and a step-level score by comparing the full solution against the official marking scheme; the final problem score is then

\[
\text{Problem Score} = \max(\text{answer-level score}, \text{step-level score}).
\]

This mirrors Olympiad grading practice: a correct final answer can receive full credit even if exposition is terse, while a wrong final answer can still earn substantial partial credit through correct intermediate reasoning [2509.07894].

Step-level grading is rubric-based rather than synthetic. Official marking schemes are structured into machine-readable criteria, and when multiple valid official solutions exist, the benchmark evaluates all available schemes and takes the maximum step-level score [2509.07894]. For modality-wise or field-wise analysis, HiPhO uses the mean normalized score

\[
\text{MNS}(M) = \frac{1}{N_M} \sum_{Q \in M} \frac{\text{Exam Score}(Q)}{\text{Full Mark}(Q)} \times 100\%,
\]

where \(M\) denotes a modality class or physics field [2509.07894].

The benchmark also supports exam-level aggregation. Later papers using the 13-exam suite report an average score of the form

\[
\text{Avg} = \frac{1}{13}\sum_{e=1}^{13} \text{Score}_e,
\]

and reproduce average medal thresholds over the 13 exams of \(\bar{G} = 29.3\), \(\bar{S} = 19.7\), and \(\bar{B} = 11.4\) [2602.09443].

HiPhO’s medal mapping is central to its human alignment. Model exam scores are compared against official or benchmark-reconstructed gold, silver, and bronze thresholds, allowing statements of the form “gold on IPhO 2025” rather than only raw percentage accuracy [2509.07894]. In the core benchmark setup, inference is implemented with VLMEvalKit, uses 8 independent runs per problem at temperature \(0.6\), and expects structured outputs with full reasoning and final boxed answers [2509.07894].

## 4. Empirical findings and difficulty profile

The initial large-scale study evaluates 30 state-of-the-art models and reports a sharply stratified landscape [2509.07894]. Across the 13 exams, open-source MLLMs mostly remain at or below the bronze level; open-source LLMs show promising progress with occasional golds; closed-source reasoning MLLMs can achieve 6 to 12 gold medals; and most models still have a significant gap from full marks [2509.07894].

The benchmark paper also shows that the best models remain well below top human contestants on most exams. One nuance is that performance is heterogeneous across contests: on easier or more mechanics-heavy settings, strong models can approach or even exceed the top reported human score for a specific exam, while still lagging substantially on more difficult multimodal Olympiads [2509.07894]. A plausible implication is that HiPhO measures not a single monolithic “physics ability,” but an interaction among symbolic derivation, visual interpretation, long-horizon consistency, and competition-specific grading compliance.

The difficulty structure is systematic. Performance declines as visual complexity increases, with the ordering TO \(>\) TI \(>\) TV \(>\) TD across model classes [2509.07894]. Field-wise, optics is consistently the most challenging area; even strong closed-source models that score around 70–80% normalized performance in mechanics, electromagnetism, thermodynamics, and modern physics are markedly weaker in optics [2509.07894]. The benchmark’s error analyses attribute failures to diagram misinterpretation, fragile algebraic execution, unit and dimensional mistakes, graph-reasoning errors, and conceptual misunderstandings in applying physical laws [2509.07894].

In this sense, HiPhO functions not only as a leaderboard but also as a diagnostic instrument. It distinguishes models that can retrieve familiar formulas from models that can sustain olympiad-style derivations under multimodal constraints and partial-credit grading.

## 5. HiPhO in subsequent model development

HiPhO rapidly became a primary target for systems built explicitly for Olympiad physics. “PhysicsMinions” is a coevolutionary multimodal multi-agent system organized into a Visual Studio, Logic Studio, and Review Studio, and evaluated on the seven latest physics Olympiads under the HiPhO setup [2509.24855]. It reports three major outcomes: consistent gains over single-model baselines, the first-ever open-source gold medal in the latest IPhO under the average-score metric, and a Pass@32 score of \(26.8/30\) on the latest IPhO, corresponding to 4th of 406 contestants [2509.24855]. Its design explicitly reflects HiPhO’s step-level grading and multimodal demands: structured figure extraction, stepwise LaTeX derivations, physics-specific verification, and iterative self-correction.

The P1 family extends this direction through reinforcement learning. “P1: Mastering Physics Olympiads with Reinforcement Learning” trains open-source models entirely through RL on curated Olympiad-style physics problems and evaluates them on HiPhO, where P1-235B-A22B earns 12 gold and 1 silver, while P1-235B-A22B + PhysicsMinions reaches an average of \(38.4\) and achieves the top reported IPhO 2025 score of \(23.2/30\) in that paper’s comparison [2511.13612]. The work treats HiPhO as a contest-grade evaluation target rather than a generic physics QA dataset.

P1-VL adds multimodal RL post-training to this line. “P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiads” presents open-source vision-language models evaluated on the 13-exam HiPhO suite, where P1-VL-235B-A22B reports an average score of \(39.3\) with 12 gold and 1 silver, and P1-VL-235B-A22B + PhysicsMinions reports \(40.9\), ranking second overall in that paper’s global comparison [2602.09443]. The paper argues that HiPhO is a genuine VLM benchmark because many Olympiad figures encode indispensable physical constraints rather than optional visual context.

HiPhO has also been incorporated into broader agentic evaluation. “Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent” treats HiPhO as one of four representative scientific-research benchmarks and reports that Agents-A1, a 35B Mixture-of-Experts agentic model, achieves \(46.4\) on HiPhO, ahead of several larger comparison models in that evaluation table [2606.30616]. This suggests that HiPhO now serves two roles simultaneously: a specialized Olympiad benchmark and a general stress test for long-horizon scientific reasoning.

## 6. Limitations, controversies, and open directions

HiPhO’s scope is deliberately constrained. It excludes experimental labs, embodied measurement, and diagram-generation tasks, so model full marks can be below human full marks on some exams [2509.07894]. The benchmark is also biased toward elite contest settings rather than ordinary classroom physics, and its coverage remains concentrated in English and Chinese with some major competitions omitted because public score distributions are unavailable [2509.07894].

A second limitation is that multimodality remains unresolved even when benchmark quality is high. Later systems report substantial progress, but their own analyses continue to expose bottlenecks. PhysicsMinions notes that fine-grained data extraction from dense plots remains difficult [2509.24855]. P1-VL identifies persistent challenges in image-heavy reasoning and relies on frozen vision encoders during RL post-training [2602.09443]. Agents-A1 shows that a unified policy can fall slightly below a domain-specialized science teacher on HiPhO, indicating a tradeoff between cross-domain deployment and per-domain peak performance [2606.30616].

A further issue is protocol heterogeneity. The benchmark paper specifies one evaluation recipe, whereas later papers sometimes adopt different sampling or aggregation schemes, such as multi-run averages, Pass@\(k\), or maxima over multiple attempts [2509.07894]. This does not invalidate HiPhO, but it means that cross-paper comparisons require attention to inference settings and scoring conventions.

The benchmark authors propose broader exam coverage, dynamic annual updates, stronger multimodal reasoning, support for generated diagrams or plots, and eventually more embodied forms of physical evaluation [2509.07894]. Subsequent work reinforces those directions. A plausible implication is that future HiPhO-like benchmarks will need to integrate visual perception, symbolic derivation, tool use, and verification in a single evaluation framework if they are to approximate the full skill profile of human Olympiad problem solving.

Source: https://www.emergentmind.com/topics/hipho