---
title: 'HiPhO Benchmark: Multimodal Physics Evaluation'
url: https://www.emergentmind.com/topics/hipho-benchmark
type: topic
---

# HiPhO Benchmark: Multimodal Physics Evaluation

HiPhO is the first open-source benchmark designed to rigorously evaluate the multimodal physical reasoning abilities of language models and multimodal language models (LLMs/MLLMs) using real high-school physics Olympiad exams from 2024–2025. It uniquely enables direct, human-aligned comparison by scoring models and humans with the same official answer- and step-level marking schemes, then mapping results to gold, silver, and bronze medal thresholds identical to those in Olympiad competitions. HiPhO comprises 13 recent theoretical Olympiad exams spanning a diverse set of contests and modalities, setting a new standard for domain-specific, multi-modal AI evaluation[2509.07894].

## 1. Dataset Composition

HiPhO aggregates 13 theoretical exam papers from seven major Olympiad events conducted during 2024–2025: International Physics Olympiad (IPhO), Asian Physics Olympiad (APhO), European Physics Olympiad (EuPhO), Nordic-Baltic Physics Olympiad (NBPhO), Pan Pearl-River-Delta Physics Olympiad (PanPhO), Pan Pearl-River-Delta Mechanics Test (PanMechanics), and F=MA (U.S.). The dataset contains 360 problems and 519 subquestions, manually extracted, OCR-corrected, QA-matched to official solutions, and expert-verified.

Problems are classified along three orthogonal axes:

- **Physics Field**: Mechanics (8%), Electromagnetism (9%), Thermodynamics (62%), Optics (7%), Modern Physics (8%).
- **Modality**:
    - Text-only (TO): 39%
    - Text+Illustration Figure (TI): 24%
    - Text+Variable Figure (TV): 28%
    - Text+Data Figure (TD): 9%
- **Difficulty (by exam)**:
    - Hard: IPhO, APhO, EuPhO
    - Medium: NBPhO, PanPhO
    - Easy: PanMechanics, F=MA

| Olympiad Event | Modality Coverage | Difficulty |
|----------------|------------------|------------|
| IPhO           | TO, TI, TV, TD   | Hard       |
| APhO           | TO, TI, TV, TD   | Hard       |
| EuPhO          | TO, TI, TV, TD   | Hard       |
| NBPhO          | TO, TI, TV, TD   | Medium     |
| PanPhO         | TO, TI, TV, TD   | Medium     |
| PanMechanics   | TO, TI           | Easy       |
| F=MA           | TO               | Easy       |

This coverage allows for both international benchmarking and fine-grained analysis of performance across physics subfields, modalities, and problem complexity.

## 2. Evaluation Methodology

HiPhO adopts domain-specific professional evaluation protocols utilizing official Olympiad marking schemes to assign both answer-level (coarse) and step-level (fine-grained) scores.

- **Answer-Level Scoring**: Each model’s boxed final answer is verified using a rule-based numeric/symbolic math checker (“Kydlíček”). If exact matching fails, Gemini-2.5-Flash is invoked for equivalence judgment. The answer score $A_i \in \{0,1\}$ per subquestion $i$.

- **Step-Level Scoring**: Official marking schemes define $n$ independent criteria per subquestion, each weighted by $w_{ij}$. Gemini-2.5-Flash grades each step $s_{ij}\in[0,1]$. The step score: $S_i = \sum_{j=1}^n w_{ij}s_{ij}$.

- **Aggregation**: Per question, score is $\max(A_i, S_i)$, in line with Olympiad conventions—full credit for correct final answer, partial otherwise. Exam scores sum over all questions.

- **Modality Normalization**: For modality $M$, Mean Normalized Score (MNS) is reported:
$$ 
MNS(M) = \frac{1}{|Q_M|} \sum_{Q \in M} \frac{Score(Q)}{FullMark(Q)} \times 100\%
$$
where $Q_M$ is the modality-specific question set.

This rigorous protocol is fully aligned with human examiner practices, supporting fine-grained and holistic performance analysis.

## 3. Human vs. Model Comparison Procedure

HiPhO implements medal-based comparisons directly reusing the exact cutoff scores for gold, silver, and bronze as defined by Olympiad organizers. For each contest:

- Gold medal threshold $G$ is the lowest score achieved by a gold-winning human, $G = \min_{h \in H_g} Score_h$.
- Silver: $S = \min_{h\in H_s} Score_h$, including gold and silver medalists.
- Bronze: $B = \min_{h\in H_b} Score_h$, including all medaled competitors.

Model assignment:
- $ExamScore \geq G$: Gold
- $S \leq ExamScore < G$: Silver
- $B \leq ExamScore < S$: Bronze
- Otherwise: No medal

This mechanism allows for direct and transparent assessment of model performance relative to top human candidates.

## 4. Benchmark Results

Thirty state-of-the-art models—including 11 closed-source MLLMs, 11 open-source MLLMs, and 8 open-source LLMs—were assessed on the HiPhO suite.

Key findings:

- **Closed-source reasoning MLLMs** consistently outperformed, winning 6–12 gold medals (e.g., Gemini-2.5-Pro: 12 gold; GPT-5: 11 gold, 2 silver; o3: 11 gold, 2 silver; Grok-4: 10 gold, 3 silver).
- **Open-source MLLMs** predominantly scored at or below bronze; Intern-S1 was the highest with 6 gold medals.
- **Open-source LLMs** were competitive on easy contests (F=MA, PanMechanics), each earning 4–5 golds (GPT-OSS-120B, DeepSeek-R1, Qwen3-235B-A22B), but lag significantly on more difficult exams.
- No current model matches absolute top human contestants across all events. For example, in IPhO 2025, the highest human score was 29.2/30, while Gemini-2.5-Pro scored 22.7/29.4.

Performance on visual modalities shows a pronounced decline in Mean Normalized Score as complexity increases:
- TO: 86% (Gemini-2.5-Pro)
- TI: 81%
- TV: 75%
- TD: 67%

*This suggests that variable-based and data-figure reasoning remain key bottlenecks, especially for open-source models.*

## 5. Contributions

HiPhO introduces several methodological and practical innovations:

1. **Olympiad Focus & Timeliness**: Integration of 13 up-to-date exams from recent contests ensures challenge relevance for both global and regional physics communities.
2. **Mixed-Modal Coverage**: Problems encompass a spectrum from text-only to complex data-figure modalities, supporting nuanced exploration of multimodal reasoning.
3. **Human-Aligned Scoring**: Step-level partial credit via official schemes, exam-level scoring—not just accuracy—reflect actual contest standards.
4. **Medal-Based Comparison**: Direct mapping of scores to contest medal thresholds fully aligns model evaluation with human competitive outcomes.

## 6. Limitations and Future Directions

Identified limitations include:

- **Experimental-lab and Diagram Generation Omission**: Current evaluation excludes experimental practicals and requires no drawing, reflecting the present inability of models to physically interact or generate graphical outputs.
- **Coverage Expansion Constraints**: Some major contests (e.g., USAPhO, CPhO) are absent due to incomplete medal data.
- **Update Mechanisms**: The need for automated ingestion of new exams and scoring thresholds is recognized to mitigate staleness and possible training contamination.
- **Embodied and Generative Capabilities**: Achieving true human-level parity will necessitate models capable of producing graphical sketches (trajectories, waveforms) and reasoning about embodied physical experimentation.

*A plausible implication is that continued progress in both multimodal input processing and embodied output generation will be crucial for closing the gap with human contestants, particularly on vision-intensive and creative reasoning tasks.*

HiPhO provides a comprehensive and rigorous foundation for benchmarking multimodal physics reasoning, establishing direct comparability with high-performing humans and promoting transparent, human-aligned continual progress in AI physical problem-solving[2509.07894].

Source: https://www.emergentmind.com/topics/hipho-benchmark