---
title: HLE-Verified Gold Benchmark
url: https://www.emergentmind.com/topics/hle-verified-gold
type: topic
---

# HLE-Verified Gold Benchmark

Searching arXiv for the cited benchmark and ATLAS paper to ground the article in current primary sources.
HLE-Verified Gold denotes the subset of “Humanity’s Last Exam” (HLE) items that passed an independent, component-wise verification protocol without any modification, within HLE-Verified, a verified and revised version of the original benchmark [2602.13964]. In downstream evaluation, the same label is also used more narrowly: the ATLAS study reports that all of its HLE-Verified results refer to the first 100 examples of the Gold subset, which it calls “HLE-Verified Gold” [2606.01667]. The term therefore names both a benchmark partition and, in at least one prominent test-time-scaling study, a specific evaluation slice.

## 1. Definition and scope

HLE-Verified is not a new task, but a reliability-enhanced release of the original 2,500-item HLE benchmark, exposing which items are truly “gold” versus those that required expert revision or remain open for future adjudication [2602.13964]. In the HLE-Verified paper, the benchmark is partitioned into a Gold subset (641 items), a Revision subset (1,170 items), and an Uncertain subset (689 items). Gold consists of items whose problem statements and reference answers were judged well-posed and correct “as is.”

The ATLAS paper uses the official “skylenage/HLE-Verified” release and describes the full release as 2,500 items split into Gold (668 items), Revision (1,143 items), and Uncertain (689 items); it further states that all reported HLE-Verified numbers refer to the first 100 examples of the Gold subset \((n=100)\), which it calls “HLE-Verified Gold” [2606.01667]. *This suggests that the label has a dataset-level meaning in benchmark construction and a task-slice meaning in downstream evaluation.*

At the item level, HLE-Verified Gold remains a closed-book, answer-equivalence benchmark. Each item is a standalone problem statement plus a single “final answer” in free-form text, and sometimes an author rationale. The evaluation target is the final answer, not the rationale.

## 2. Verification and subset formation

HLE-Verified Gold is the output of Stage I of a two-stage validation-and-repair workflow. Stage I performs component-wise binary verification with the goal of certifying items that are correct “out of the box” without any edits [2602.13964]. The components are assessed independently: the problem statement and images must be well-posed, unambiguous, and fully specified; the final answer must be numerically or logically correct, with proper units and format; and the rationale is used as an auxiliary diagnostic for internal consistency but does not by itself determine inclusion.

The evidence model in Stage I is explicitly multi-source. External domain-expert screening supplies binary valid/invalid/uncertain judgments on each component, with short notes. Model-assisted replication uses pass@8 with multiple state-of-the-art LLMs under a standardized solver prompt; answers are extracted, normalized, and compared to the reference under a fixed equivalence protocol. Disagreements flag potential underspecification or answer-key errors. Internal expert adjudication then synthesizes expert labels and model signals, accepting items only if both Problem and Answer are affirmed valid and no high-risk ambiguity is detected. The outcome of this stage is the Gold subset.

Stage II exists to handle items that are flawed but repairable under strict “no-intent-change” constraints. Its presence is important for understanding Gold: Gold is the set that required no repair. In Stage II, Team A and Team B each propose corrections in the order Problem Fix \(\rightarrow\) Solution Fix \(\rightarrow\) Answer Fix, model-assisted auxiliary proposals are used for stability checking, and final expert adjudication selects or synthesizes a canonical repaired version under the principles of objective preservation, correctness, and minimal edits. Ambiguous cases are diverted to Uncertain rather than promoted to Gold.

## 3. Task format and grading semantics

The task is answer-equivalence, not rationale scoring. The HLE-Verified release defines the evaluation as: given the prompt, the model must output exactly the final answer. Final scoring is based solely on answer equivalence, with rationales used only in the dataset curation pipeline [2606.01667]. This distinction matters because the benchmark includes rationales only intermittently, and even when present they are auxiliary to evaluation.

In the HLE-Verified construction pipeline, rationales are diagnostic artifacts. They can expose internal inconsistency, missing prerequisites, circularity, or domain misapplication, but they do not by themselves determine Gold inclusion unless they signal a problem with the problem statement or final answer. A common misunderstanding is therefore incorrect: Gold certification is not awarded because a rationale looks convincing, but because the problem and answer are affirmed valid under the verification protocol.

The ATLAS evaluation adds a separate grading layer for its 100-item HLE-Gold slice. It employs an LLM-as-judge setup: the model’s predicted answer and the gold answer are fed to Claude Haiku 4.5, using a fixed judge prompt from the HLE release to decide semantic equivalence. This preserves the benchmark’s answer-equivalence semantics while operationalizing grading for free-form outputs.

## 4. Error taxonomy and the rationale for a Gold subset

The HLE-Verified paper defines 19 defect categories distributed across the Problem, Rationale, and Answer components [2602.13964]. Problem-level defects are Q1–Q5: Semantic Error, Knowledge Error, Missing Information, Theoretical Invalidity, and Format Semantic Error (Problem). Rationale-level defects are S1–S10, including Non-Redundancy Violation, Circular Reasoning, Empirical Soundness Violation, Step Inconsistency, Domain Misapplication, Overconfidence Bias, Missing Prerequisite, Deceptive Similarity, Multi-Solution Inconsistency, and Format Semantic Error (Rationale). Answer-level defects are A1–A4: Incorrect Answer, Incomplete Answer, Ambiguous/Ill-defined Answer, and Format Semantic Error (Answer).

These categories explain why a Gold partition is methodologically necessary. On the 1,859 non-gold items, the problem component is reported as approximately \(40\%\) valid, approximately \(35\%\) invalid, and approximately \(25\%\) uncertain; the answer component has a substantially higher invalid share of approximately \(45\%\) and moderate uncertainty of approximately \(20\%\); and the rationale component is the most error-prone, with a large invalid proportion of approximately \(50\%\) and high uncertainty of approximately \(30\%\). In the Revision subset, Incorrect Answer dominates approximately \(90\%\) of answer defects; Missing Prerequisite and Format Semantic Error are the top two rationale defects; and Format Semantic Error and Missing Information are the most common problem defects.

The paper also gives domain-specific top defects. In Mathematics, the top-1 defect by component is Q5 \((40\%)\), S7 \((40.1\%)\), and A1 \((89.4\%)\). In Physics it is Q3 \((57.1\%)\), S7 \((42.9\%)\), and A1 \((80\%)\). In Chemistry it is Q5 \((58.3\%)\), S10 \((40.5\%)\), and A1 \((69.4\%)\). In Biomedicine it is Q2 \((33.3\%)\), S7 \((41.9\%)\), and A1 \((97.2\%)\). In CS/AI it is Q5 \((92.3\%)\), S10 \((63.4\%)\), and A1 \((78.1\%)\). Gold is therefore best understood as the benchmark partition in which these defect pathways have already been screened out to the extent certified by the protocol.

## 5. Metrics and benchmark effects

The HLE-Verified paper defines benchmark-wide accuracy and calibration error as
$$
\mathrm{Acc}=\tfrac{\#\text{correct}}{\#\text{total}}
$$
and
$$
\mathrm{CaliErr}=100 \times \mathrm{calib\_err}(c,y;\,p=2,\beta=100).
$$
It evaluates GPT-5.2-Thinking, Gemini3-Pro-Preview, Claude-Opus4.5, Claude-Opus4.6, Grok-4.1 (fast), DeepSeek-V3.2-Thinking, and Qwen3-Max-Thinking, all using avg5 accuracy [2602.13964].

The reported effect of verification is substantial. On all 2,500 text-only items, every model gains \(+7\) to \(+11\) points in accuracy after verification. On the 624 items whose problem or answer changed, accuracy jumps by \(+30\) to \(+40\) points, and calibration error also drops systematically. The paper further defines
$$
\Delta \mathrm{Conf} = \mathbb{E}[\,c_{\mathrm{Verified}} - c_{\mathrm{Raw}}\,],
$$
and reports that on the Problem-Error Subset all models exhibit positive \(\Delta \mathrm{Conf}\) of \(+1.8\) to \(+11.1\) points. These results motivate the use of Gold as a high-reliability substrate for capability evaluation.

For the ATLAS HLE-Gold slice, the metric is standard answer-equivalence accuracy on the Gold subset. With \(N=100\), predicted answer \(\hat y_i\), and gold answer \(y_i^{\mathrm{gold}}\),
$$
\mathrm{Accuracy}_{\mathrm{HLE\small{-}Gold}}
= \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}\{\hat y_i = y_i^{\mathrm{gold}}\}.
$$
The original HLE paper also reported calibration error, but the ATLAS study focuses exclusively on accuracy.

## 6. Use in agentic test-time scaling

In the ATLAS framework, HLE-Verified Gold serves as one of four benchmarks for evaluating agentic test-time scaling under a Claude Sonnet 4.6 backbone [2606.01667]. The Base ATLAS variant used on HLE-Gold exposes only two orchestrator actions: `explore[]`, which calls a fresh solver on the original problem, and `stop[<final_answer>]`, which terminates exploration and synthesizes a final answer. Each turn produces \(\mathrm{Thought}_t\), \(\mathrm{Action}_t\), and \(\mathrm{Observation}_t\), where the observation is a new candidate \(c_t=(\text{answer}, \text{reasoning}, \text{approach}, \text{confidence})\). Because each explore sees only the problem statement \(x\), not prior candidates, all \(c_t\) are independent given \(x\); stop and synthesis are conditioned on the full Thought/Action/Observation history, which the paper describes as “stateful evidence management.”

On HLE-Gold, single-model ATLAS uses Claude Sonnet 4.6 for both orchestrator and explorer, with a per-question exploration budget of \(T=8\) and direct orchestrator synthesis rather than a separate integrate step. The orchestrator uses the default Medium explore-effort level, which biases it to require at least two agreeing candidates plus one cross-method check before stopping. Its prompt is benchmark-agnostic and encodes the role constraint (“cannot solve itself, only orchestrate”), four high-level principles, and the explore-effort injection; the explorer prompt asks Sonnet 4.6 to solve \(x\) from scratch and return \(\{\text{approach}, \text{reasoning}, \text{answer}, \text{confidence}\}\).

| Method | Accuracy | Cost |
|---|---:|---:|
| Pass@1 | 48.00% | — |
| Self-Refine (early stop) | 53.00% | — |
| Self-Refine (no early stop) | 58.00% | — |
| Budget Forcing | 51.00% | — |
| Reward-model reranking (Skywork-Reward-V2) | 52.00% | — |
| ATLAS (single-model) | 56.00% | \$1.59/q |
| ATLAS-MM (solver choice exposed) | 60.00% | \$1.50/q |

ATLAS is reported as \(+8\) percentage points over Pass@1 and \(+3\) percentage points over the best fixed-budget reranking baseline. ATLAS-MM improves HLE-Verified to \(60.00\%\), exceeding Self-Refine (no early stop) by \(+2\) percentage points and any single-model variant by \(+4\) percentage points. An ablation with a dedicated “integrate” tool achieves \(57.00\%\) accuracy on HLE-Verified but increases cost to \$2.35/q, whereas direct synthesis by the orchestrator achieves \(56.00\%\) at \$1.59/q. The ATLAS paper interprets these findings as consistent with the role of stateful evidence management: the orchestrator can detect straightforward convergence through diverse agreement, but can also identify minority-candidate correctness when majority candidates share a reasoning flaw. On HLE-Gold, the reported result is robust gains with fewer API calls than fixed-workflow baselines.

Source: https://www.emergentmind.com/topics/hle-verified-gold