HLE-Verified Gold Benchmark

• HLE-Verified Gold is a validated benchmark partition of HLE items confirmed as correct without modifications through independent, component-wise checks.
• The dataset is formed via a rigorous two-stage process that integrates expert screening and model-assisted replication to eliminate over 19 defect categories.
• This benchmark underpins agentic test-time scaling studies, exemplified by ATLAS, which uses answer-equivalence and stateful evidence management for improved accuracy.

Searching arXiv for the cited benchmark and ATLAS paper to ground the article in current primary sources. HLE-Verified Gold denotes the subset of “Humanity’s Last Exam” (HLE) items that passed an independent, component-wise verification protocol without any modification, within HLE-Verified, a verified and revised version of the original benchmark (Zhai et al., 15 Feb 2026). In downstream evaluation, the same label is also used more narrowly: the ATLAS study reports that all of its HLE-Verified results refer to the first 100 examples of the Gold subset, which it calls “HLE-Verified Gold” (Qin et al., 1 Jun 2026). The term therefore names both a benchmark partition and, in at least one prominent test-time-scaling study, a specific evaluation slice.

1. Definition and scope

HLE-Verified is not a new task, but a reliability-enhanced release of the original 2,500-item HLE benchmark, exposing which items are truly “gold” versus those that required expert revision or remain open for future adjudication (Zhai et al., 15 Feb 2026). In the HLE-Verified paper, the benchmark is partitioned into a Gold subset (641 items), a Revision subset (1,170 items), and an Uncertain subset (689 items). Gold consists of items whose problem statements and reference answers were judged well-posed and correct “as is.”

The ATLAS paper uses the official “skylenage/HLE-Verified” release and describes the full release as 2,500 items split into Gold (668 items), Revision (1,143 items), and Uncertain (689 items); it further states that all reported HLE-Verified numbers refer to the first 100 examples of the Gold subset $(n=100)$, which it calls “HLE-Verified Gold” (Qin et al., 1 Jun 2026). This suggests that the label has a dataset-level meaning in benchmark construction and a task-slice meaning in downstream evaluation.

At the item level, HLE-Verified Gold remains a closed-book, answer-equivalence benchmark. Each item is a standalone problem statement plus a single “final answer” in free-form text, and sometimes an author rationale. The evaluation target is the final answer, not the rationale.

2. Verification and subset formation

HLE-Verified Gold is the output of Stage I of a two-stage validation-and-repair workflow. Stage I performs component-wise binary verification with the goal of certifying items that are correct “out of the box” without any edits (Zhai et al., 15 Feb 2026). The components are assessed independently: the problem statement and images must be well-posed, unambiguous, and fully specified; the final answer must be numerically or logically correct, with proper units and format; and the rationale is used as an auxiliary diagnostic for internal consistency but does not by itself determine inclusion.

The evidence model in Stage I is explicitly multi-source. External domain-expert screening supplies binary valid/invalid/uncertain judgments on each component, with short notes. Model-assisted replication uses pass@8 with multiple state-of-the-art LLMs under a standardized solver prompt; answers are extracted, normalized, and compared to the reference under a fixed equivalence protocol. Disagreements flag potential underspecification or answer-key errors. Internal expert adjudication then synthesizes expert labels and model signals, accepting items only if both Problem and Answer are affirmed valid and no high-risk ambiguity is detected. The outcome of this stage is the Gold subset.

Stage II exists to handle items that are flawed but repairable under strict “no-intent-change” constraints. Its presence is important for understanding Gold: Gold is the set that required no repair. In Stage II, Team A and Team B each propose corrections in the order Problem Fix $\rightarrow$ Solution Fix $\rightarrow$ Answer Fix, model-assisted auxiliary proposals are used for stability checking, and final expert adjudication selects or synthesizes a canonical repaired version under the principles of objective preservation, correctness, and minimal edits. Ambiguous cases are diverted to Uncertain rather than promoted to Gold.

3. Task format and grading semantics

The task is answer-equivalence, not rationale scoring. The HLE-Verified release defines the evaluation as: given the prompt, the model must output exactly the final answer. Final scoring is based solely on answer equivalence, with rationales used only in the dataset curation pipeline (Qin et al., 1 Jun 2026). This distinction matters because the benchmark includes rationales only intermittently, and even when present they are auxiliary to evaluation.

In the HLE-Verified construction pipeline, rationales are diagnostic artifacts. They can expose internal inconsistency, missing prerequisites, circularity, or domain misapplication, but they do not by themselves determine Gold inclusion unless they signal a problem with the problem statement or final answer. A common misunderstanding is therefore incorrect: Gold certification is not awarded because a rationale looks convincing, but because the problem and answer are affirmed valid under the verification protocol.

The ATLAS evaluation adds a separate grading layer for its 100-item HLE-Gold slice. It employs an LLM-as-judge setup: the model’s predicted answer and the gold answer are fed to Claude Haiku 4.5, using a fixed judge prompt from the HLE release to decide semantic equivalence. This preserves the benchmark’s answer-equivalence semantics while operationalizing grading for free-form outputs.

4. Error taxonomy and the rationale for a Gold subset

The HLE-Verified paper defines 19 defect categories distributed across the Problem, Rationale, and Answer components (Zhai et al., 15 Feb 2026). Problem-level defects are Q1–Q5: Semantic Error, Knowledge Error, Missing Information, Theoretical Invalidity, and Format Semantic Error (Problem). Rationale-level defects are S1–S10, including Non-Redundancy Violation, Circular Reasoning, Empirical Soundness Violation, Step Inconsistency, Domain Misapplication, Overconfidence Bias, Missing Prerequisite, Deceptive Similarity, Multi-Solution Inconsistency, and Format Semantic Error (Rationale). Answer-level defects are A1–A4: Incorrect Answer, Incomplete Answer, Ambiguous/Ill-defined Answer, and Format Semantic Error (Answer).

These categories explain why a Gold partition is methodologically necessary. On the 1,859 non-gold items, the problem component is reported as approximately $40\%$ valid, approximately $35\%$ invalid, and approximately $25\%$ uncertain; the answer component has a substantially higher invalid share of approximately $45\%$ and moderate uncertainty of approximately $20\%$; and the rationale component is the most error-prone, with a large invalid proportion of approximately $50\%$ and high uncertainty of approximately $30\%$. In the Revision subset, Incorrect Answer dominates approximately $\rightarrow$0 of answer defects; Missing Prerequisite and Format Semantic Error are the top two rationale defects; and Format Semantic Error and Missing Information are the most common problem defects.

The paper also gives domain-specific top defects. In Mathematics, the top-1 defect by component is Q5 $\rightarrow$1, S7 $\rightarrow$2, and A1 $\rightarrow$3. In Physics it is Q3 $\rightarrow$4, S7 $\rightarrow$5, and A1 $\rightarrow$6. In Chemistry it is Q5 $\rightarrow$7, S10 $\rightarrow$8, and A1 $\rightarrow$9. In Biomedicine it is Q2 $\rightarrow$0, S7 $\rightarrow$1, and A1 $\rightarrow$2. In CS/AI it is Q5 $\rightarrow$3, S10 $\rightarrow$4, and A1 $\rightarrow$5. Gold is therefore best understood as the benchmark partition in which these defect pathways have already been screened out to the extent certified by the protocol.

5. Metrics and benchmark effects

The HLE-Verified paper defines benchmark-wide accuracy and calibration error as

$\rightarrow$6

and

$\rightarrow$7

It evaluates GPT-5.2-Thinking, Gemini3-Pro-Preview, Claude-Opus4.5, Claude-Opus4.6, Grok-4.1 (fast), DeepSeek-V3.2-Thinking, and Qwen3-Max-Thinking, all using avg5 accuracy (Zhai et al., 15 Feb 2026).

The reported effect of verification is substantial. On all 2,500 text-only items, every model gains $\rightarrow$8 to $\rightarrow$9 points in accuracy after verification. On the 624 items whose problem or answer changed, accuracy jumps by $40\%$0 to $40\%$1 points, and calibration error also drops systematically. The paper further defines

$40\%$2

and reports that on the Problem-Error Subset all models exhibit positive $40\%$3 of $40\%$4 to $40\%$5 points. These results motivate the use of Gold as a high-reliability substrate for capability evaluation.

For the ATLAS HLE-Gold slice, the metric is standard answer-equivalence accuracy on the Gold subset. With $40\%$6, predicted answer $40\%$7, and gold answer $40\%$8,

$40\%$9

The original HLE paper also reported calibration error, but the ATLAS study focuses exclusively on accuracy.

6. Use in agentic test-time scaling

In the ATLAS framework, HLE-Verified Gold serves as one of four benchmarks for evaluating agentic test-time scaling under a Claude Sonnet 4.6 backbone (Qin et al., 1 Jun 2026). The Base ATLAS variant used on HLE-Gold exposes only two orchestrator actions: explore[], which calls a fresh solver on the original problem, and stop[<final_answer>], which terminates exploration and synthesizes a final answer. Each turn produces $35\%$0, $35\%$1, and $35\%$2, where the observation is a new candidate $35\%$3. Because each explore sees only the problem statement $35\%$4, not prior candidates, all $35\%$5 are independent given $35\%$6; stop and synthesis are conditioned on the full Thought/Action/Observation history, which the paper describes as “stateful evidence management.”

On HLE-Gold, single-model ATLAS uses Claude Sonnet 4.6 for both orchestrator and explorer, with a per-question exploration budget of $35\%$7 and direct orchestrator synthesis rather than a separate integrate step. The orchestrator uses the default Medium explore-effort level, which biases it to require at least two agreeing candidates plus one cross-method check before stopping. Its prompt is benchmark-agnostic and encodes the role constraint (“cannot solve itself, only orchestrate”), four high-level principles, and the explore-effort injection; the explorer prompt asks Sonnet 4.6 to solve $35\%$8 from scratch and return $35\%$9.

Method	Accuracy	Cost
Pass@1	48.00%	—
Self-Refine (early stop)	53.00%	—
Self-Refine (no early stop)	58.00%	—
Budget Forcing	51.00%	—
Reward-model reranking (Skywork-Reward-V2)	52.00%	—
ATLAS (single-model)	56.00%	\$1.59/q
ATLAS-MM (solver choice exposed)	60.00%	\$1.50/q

ATLAS is reported as $25\%$0 percentage points over Pass@1 and $25\%$1 percentage points over the best fixed-budget reranking baseline. ATLAS-MM improves HLE-Verified to $25\%$2, exceeding Self-Refine (no early stop) by $25\%$3 percentage points and any single-model variant by $25\%$4 percentage points. An ablation with a dedicated “integrate” tool achieves $25\%$5 accuracy on HLE-Verified but increases cost to \$25\%$61.59/q. The ATLAS paper interprets these findings as consistent with the role of stateful evidence management: the orchestrator can detect straightforward convergence through diverse agreement, but can also identify minority-candidate correctness when majority candidates share a reasoning flaw. On HLE-Gold, the reported result is robust gains with fewer API calls than fixed-workflow baselines.