---
title: Humanity's Last Exam (HLE) Benchmark
url: https://www.emergentmind.com/topics/humanity-s-last-exam-hle-benchmark
type: topic
---

# Humanity's Last Exam (HLE) Benchmark

Humanity’s Last Exam (HLE) is an advanced closed-book academic benchmark designed as a definitive yardstick for evaluating the frontier capabilities of large language models (LLMs) and research agents. Consisting of 2,500 expert-curated short-answer questions spanning more than 100 subdomains—mathematics, sciences, humanities, and engineering—HLE establishes a rigorous challenge at or above graduate level. Unlike earlier benchmarks that have become saturated (e.g., MMLU), HLE’s question design explicitly resists trivial internet lookup, focusing on questions answerable by domain experts but typically beyond current LLM generalization and reasoning abilities. Scores on HLE remain substantially below human expert levels, with the best contemporary systems approaching 30–50% accuracy under tool-augmented or multi-agent protocols, while individual human experts exceed 98% in controlled settings.

## 1. Benchmark Motivation and Historical Context

The inception of HLE was motivated by the rapid rise in LLM performance, which rendered legacy benchmarks like MMLU ineffective for measuring incremental advances, as SOTA models routinely surpass 90% accuracy. HLE was developed as a “final” closed-ended academic benchmark prioritizing:

- Broad subject representation, with 2,500 questions sourced from mathematics (450), physics (220), computer science (210), chemistry (160), biology (140), humanities (180), engineering (200), law/policy (100), trivia/puzzles (120), and over 90 other specialist areas [2501.14249].
- Graduate-level or specialist difficulty: Each item vetted by ≥1,000 subject-matter experts from 500+ institutions, ensuring human-expert “ceiling” (≳98% accuracy).
- Resistance to superficial retrieval: Solutions are constructed so that answers cannot be trivially found online.

The benchmark's release fostered a paradigm shift toward high-difficulty, expert-only benchmarks, providing a robust test bed as reflected in subsequent agentic and reasoning LLM research.

## 2. Dataset Composition and Question Design

HLE’s question bank comprises 2,500 standalone items, available primarily in short-answer (exact-match string) format (~80%), with the remainder in multiple-choice (≥5 options, ~20%) and ∼10% supporting multimodal input (text + image) [2501.14249, 2508.15842]. Each question was filtered to ensure:

- Unambiguous, single-answer solutions.
- Eligibility for automated grading—i.e., string matching, numeric normalization (fractions to decimals), and LLM judge verification.
- Task formulation requiring one or more steps of domain-knowledge reasoning (“Prove that every subgroup of index 2 is normal,” “What is the social impact of the 1954 Brown v. Board of Education decision?”).
- Explicit resistance to trivial lookup (content not indexed online at release).

For agent-research evaluations, each question is issued verbatim to the agent; all search queries and retrieval artifacts are logged for verification [2508.13180]. A subset (“Bio/Chem Gold,” N=149) is used for advanced biological/chemical reasoning challenges [2509.21193, 2508.02789].

## 3. Evaluation Protocols and Metrics

HLE evaluation relies on automated, LLM-powered judgment with strict matching and normalization protocols. The primary metric is pass@1 accuracy:

\[
\text{Accuracy} = \frac{1}{N} \sum_{i=1}^N c_i
\]

where \(c_i \in \{0,1\}\) is the correctness indicator by the “HLE Judge”—typically OpenAI o3-mini (temperature=1.0, max_completion_tokens=4096) or analogous automated agent [2501.14249, 2508.13180, 2510.24695]. Confidence intervals are computed via bootstrapping. For comparative studies, Bayesian credible intervals (Beta priors/posteriors) are employed [2508.10173]. Calibration is assessed using root-mean-square calibration error (RMSCE):

\[
\mathrm{RMSCE} = \sqrt{\frac{1}{N} \sum_{i=1}^N (c_i - y_i)^2} \times 100\%
\]

Additionally, categories such as chain-of-thought length, intra-chain sentiment volatility, and lexicographic hedging signals are analyzed as post-hoc calibration proxies [2508.15842].

## 4. Empirical Results and Performance Trends

Performance on the public text-only HLE split (N≈2,250) demonstrates persistent capability gaps between LLMs and expert humans:

| Model              | Accuracy (%) | RMSCE (%) |
|--------------------|-------------|-----------|
| GPT-4o             | 2.9         | 90.4      |
| Grok 2             | 3.9         | 92.5      |
| Claude 3.5 Sonnet  | 4.2         | 87        |
| Gemini 1.5 Pro     | 4.8         | 91.1      |
| DeepSeek–R1        | 9.4         | 81.8      |
| o1                 | 8.9         | 92        |
| AgentFrontier-30B-A3B | 28.6     | n.a.      |
| SFR-DR-20B         | 28.7        | n.a.      |
| X-Masters          | 32.1        | n.a.      |
| Eigen-1 (Bio/Chem) | 48.3 (N=149)| n.a.      |

Human expert accuracy (time-controlled) reliably exceeds 98%. Agentic and multi-agent pipelines (X-Masters, Eigen-1) and curriculum fine-tuning approaches established new state-of-the-art results, with continual improvement (from <10% to >30%) attributed to explicit tool-use, multi-agent orchestration, and zone-of-proximal-development guided training [2510.24695, 2509.21193, 2509.06283, 2507.05241]. However, calibration error on HLE remains high (>78%), indicating overconfidence even with incorrect answers [2508.15842].

## 5. Agentic Methods, Multi-Agent Workflows, and Scaling Effects

Test-time workflows reflect the complexity of HLE’s reasoning demands:

- **Multi-agent orchestration:** Scattering (diversity via parallel solver agents) and stacking (depth via critic/rewriter/selector agents) boost pass@1 by up to 32.1% (X-Masters) [2507.05241].
- **Self-adaptive reasoning frameworks:** CLIO (Cognitive Loop via In-Situ Optimization) equips vanilla LLMs with recursive reasoning, uncertainty monitoring, and belief-graph aggregation, yielding 22.37% on Bio/Med questions and up to ≈30.27% with further ensemble “MoreThinking” [2508.02789].
- **Benchmark-driven selection/curriculum effects:** Models fine-tuned or reinforced on HLE tasks improve substantially relative to non-exposed counterparts, suggesting that benchmarks themselves function as adaptive curricula [2508.10173, 2510.24695].
- **Tool integration:** Candidates use Python libraries, web search, and custom code interpreters to enhance reasoning [2507.05241, 2509.06283, 2510.24695]. Usage statistics indicate more tool invocations and refined reasoning for higher-performing agents.

Scaling studies (pass@N, best-of-N sampling) show increased accuracy at linear compute cost growth, peaking at ~37% with best-of-3 strategies under o3-high [2502.09955]. However, inference costs and question complexity limit practical scaling.

## 6. Data Contamination, Integrity and Auditing

Search-time data contamination (STC) emerges as an acute concern for benchmark validity. STC occurs when web-retrieval agents surface exact or near-duplicate benchmark items with answers—most often from open repositories like HuggingFace—enabling agents to copy instead of genuinely infer [2508.13180]. Empirical results:

| Agent                     | Contamination Rate | Accuracy (contaminated) | Accuracy (clean) | Accuracy Drop ΔA (blocked) |
|---------------------------|--------------------|-------------------------|------------------|-----------------------------|
| Sonar Pro                 | 3.36%              | 55%                     | 43%              | ≈12%                       |
| Sonar Deep Research       | 3.36%              | 60%                     | 38%              | ≈22%                       |

Blocking contaminated sources (e.g., huggingface.co) yields a ∼15% accuracy reduction on previously contaminated items. Recommended mitigation includes multi-stage filters, canary strings, time-cutoff enforcement, full trajectory logging, and real-time substring audits. It is necessary to report contamination rates and post-mitigation accuracy to maintain benchmark integrity.

## 7. Limitations, Recommended Practices, and Future Directions

HLE is strictly closed-ended and does not assess a system’s open-ended research creativity, interactive learning, or adaptive teaming. Limitations include:

- Saturation of “closed-book” question types may eventually raise obsolescence risks as model generalization increases [2501.14249, 2508.13180].
- Persistent calibration errors suggest the need for improved uncertainty measures.
- Contamination risks necessitate transparent filtering, retrieval audits, and centralized logging for trustworthy benchmarking.

Best practices advocated across sources:

1. *Editor’s term*: “Swiss cheese filtering”—layered domain/date/content filters on retrieval sources.
2. Explicit reporting of search configurations and mitigation experiments.
3. Public release of retrieval logs for external auditing.
4. Post-hoc error analysis differentiating reasoning vs. knowledge gaps (Eigen-1: failure overlap >85%) [2509.21193].

Future benchmarks are expected to couple HLE’s static closed-ended difficulty with dynamic, open-ended tasks—interactive theorem proving, code synthesis, longitudinal research workflows—providing more holistic evaluations of agentic intelligence [2501.14249]. Standardizing agentic monitoring, calibration-by-lexical-hint detection, and hybrid human–AI steering protocols remain active areas for further research.

---

In summary, Humanity's Last Exam delivers a technically rigorous, high-difficulty challenge that remains unsolved by existing LLMs and agents. It serves as both an evaluation suite and—through its curricular influence—a driver of frontier model and agent development. Its explicit focus on closed-ended reasoning, rigorous data integrity, and comprehensive subject coverage make HLE a core asset for advancing and safely assessing state-of-the-art AI capabilities.

Source: https://www.emergentmind.com/topics/humanity-s-last-exam-hle-benchmark