MATH-500 Problem Set Overview

Updated 8 October 2026

MATH-500 is a 500-problem evaluation subset derived from the MATH benchmark, focusing on high-school competition mathematics problems across multiple subjects.
Evaluation is typically conducted via exact or normalized matching of a model-generated final answer, with scores not directly comparable across studies due to varied decoding budgets and metrics.
Baseline results using GPT-2 models showed low accuracy, while later models like Qwen-2.5-Math-ORM + FANS achieved significantly higher accuracy, highlighting advancements in mathematical reasoning capabilities.

MATH-500 is a 500-problem mathematical-reasoning evaluation subset drawn from the MATH benchmark, which was introduced as the Mathematics Aptitude Test of Heuristics. The original MATH dataset contains 12,500 high-school competition mathematics problems, including 7,500 training and 5,000 test problems, whereas MATH-500 is documented in later work as a derivative 500-problem evaluation set rather than in the original MATH paper itself. Its problems span prealgebra, algebra, number theory, counting and probability, geometry, intermediate algebra, and precalculus. MATH-500 is commonly evaluated by exact or normalized matching of a model-generated final answer, often extracted from a LaTeX \boxed{...} expression. Because later studies use different decoding budgets, answer verifiers, adaptation protocols, and metrics, MATH-500 scores are not automatically directly comparable across papers.

1. Origin and benchmark scope

The original MATH benchmark was introduced by Hendrycks and collaborators in “Measuring Mathematical Problem Solving With the MATH Dataset” (Hendrycks et al., 2021). It was designed to evaluate mathematical problem solving expressed in ordinary written mathematics rather than formal theorem proving. The problems are drawn from high-school mathematics competitions, including AMC 10, AMC 12, AIME, and other competitions available through sources such as the Art of Problem Solving competition archive.

MATH differs from elementary arithmetic and algorithmically generated calculation datasets because many of its problems require selecting and chaining non-obvious techniques or heuristics. It is also distinguished from formal theorem-proving datasets such as HOList, HOLStep, Coq, and Metamath, as well as from multiple-choice mathematics subsets, symbolic integration tasks, and other narrowly structured evaluations.

Every MATH problem is assigned one of seven subjects:

Prealgebra
Algebra
Number Theory
Counting and Probability
Geometry
Intermediate Algebra
Precalculus

Each problem also receives a difficulty level from 1 through 5 following difficulty conventions associated with the Art of Problem Solving. Level 1 denotes the easiest problems within a subject, whereas level 5 denotes the hardest. Early AMC 8 problems are described as often being level 1, while AIME problems are generally level 5. These levels are relative to subject and competition context rather than universal mathematical-grade classifications.

MATH problems and solutions use natural language, LaTeX mathematical notation, and Asymptote vector-graphics code for diagrams. Each example contains a problem statement, a full step-by-step solution, and a final answer enclosed in a LaTeX \boxed{...} expression. Geometry problems therefore need not be represented by raster images; some visual information can be conveyed indirectly through diagram-generating code.

The original paper does not define MATH-500. It establishes the full MATH dataset and its 7,500-example training and 5,000-example test split, but does not specify which 500 examples constitute MATH-500, whether they were sampled randomly, stratified by subject or difficulty, manually curated, drawn from the original test set, or selected to avoid contamination. It also does not establish whether MATH-500 preserves the original MATH distribution or uses exactly the original normalization code.

2. Answer representation and evaluation

The intended evaluation paradigm for MATH is accuracy based on comparison between a generated answer and a reference answer after normalization:

$\operatorname{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1} \left[ \operatorname{Normalize}(\hat{a}_i) = \operatorname{Normalize}(a_i) \right].$

Here, $a_i$ is the reference answer, $\hat{a}_i$ is the generated answer, and normalization applies the benchmark’s equivalence and formatting rules. The answers are designed to be unique after normalization, allowing exact-match evaluation rather than a heuristic metric such as BLEU.

The original MATH evaluation principle extracts the content inside the model’s final \boxed{...} expression. A model may therefore generate a complete derivation before its final answer. The intended output structure is either:

$a_i$ 8

or:

$a_i$ 9

The normalization conventions described for MATH include simplified fractions for probabilities, x/y for matrix-entry fractions, omission of explicit multiplication symbols in coefficients, alphabetization of variables in multivariable expressions, and decreasing-degree ordering of polynomial terms. Spaces are ignored; units may be included or omitted; and forms such as $\frac{x}{y}$ , $\dfrac{x}{y}$ , and $x/y$ may be treated as equivalent where appropriate. Numerically equivalent forms such as $0.5$ and $\frac12$ are accepted, as are permutations of factors such as

$4(x+1)(x-1) \equiv 4(x-1)(x+1).$

These rules do not constitute unrestricted theorem-prover-level equivalence. The original paper does not fully specify how every trigonometric identity, radical simplification, rationalization, or arbitrary algebraic rearrangement is handled. Later MATH-500 papers likewise differ in the extent to which they disclose answer extraction and normalization.

Exact-answer evaluation has several consequences. It rewards obtaining the reference answer but does not independently verify the validity of an explanation. A model can produce a correct answer with an invalid derivation, provide a valid derivation but fail through formatting, or produce an incorrect answer with high confidence. The original MATH study explicitly shows that generated solutions may be mathematically formatted and coherent-looking while containing serious logical or arithmetic errors.

3. Baseline difficulty and early model results

The original MATH experiments used autoregressive LLMs, principally GPT-2 and GPT-3. GPT-2 models were pretrained on AMPS, the Auxiliary Mathematics Problems and Solutions corpus, and fine-tuned on MATH. AMPS combines more than 100,000 Khan Academy problems with approximately five million Mathematica-generated problems and is reported as approximately 23 GB. GPT-3 models did not receive AMPS pretraining because of API limitations.

The original full-MATH test results were low. The best reported average in the principal table was 6.9% for GPT-2 1.5B. The reported GPT-2 sequence was:

Model	Average accuracy on full MATH
GPT-2 0.1B	5.4%
GPT-2 0.3B	6.2%
GPT-2 0.7B	6.4%
GPT-2 1.5B	6.9%
GPT-3 13B, few-shot	3.0%
GPT-3 13B, fine-tuned	5.6%
GPT-3 175B, few-shot	5.2%

These values are full-MATH results, not MATH-500 results. The original paper reported approximately 3.0%–6.9% across the listed configurations and found that scaling parameter count improved accuracy only modestly. For GPT-2, increasing parameters from 0.1B to 1.5B raised average accuracy from 5.4% to 6.9%. The paper’s extrapolative log-linear analysis suggested that achieving approximately 40% accuracy under the observed trend would require around $10^{35}$ parameters. This estimate was presented as an extrapolation rather than a fully reproducible equation because the supplied material does not give the fitted constants.

The original experiments identified several asymmetries in mathematical reasoning. AMPS pretraining improved performance substantially; a 0.1B GPT-2 model with AMPS pretraining reached 5.4%, compared with 5.2% for a 13B GPT-3 model fine-tuned on MATH without AMPS. Training on human-written step-by-step solutions also helped: GPT-2 1.5B accuracy decreased from 6.9% to 6.3% when fine-tuning used only final answers. By contrast, asking the model to generate a full solution before its answer lowered accuracy to 5.3%, a result associated with a hypothesized “snowballing” effect in which an early derivation error derails subsequent reasoning.

For GPT-2 1.5B, accuracy was approximately 15% on level-1 problems and approximately 4% on level-5 problems. Machine and human difficulty tracked one another, while longer problems and longer reference solutions tended to be associated with lower accuracy. The model was also overconfident: its answer-confidence AUROC was 68.8%, substantially above random ranking but insufficient for reliable error detection.

These results establish the original benchmark as difficult for early LLMs, but they should not be interpreted as showing that LLMs cannot perform mathematics in general. They concern a particular competition-problem distribution and an exact-answer evaluation protocol.

4. Modern MATH-500 evaluation regimes

Later research uses MATH-500 under substantially stronger models and varied evaluation settings. Consequently, a MATH-500 score must be interpreted together with the model, sampling budget, decoding configuration, answer verifier, and whether the metric is pass@1, pass@8, pass@16, pass@64, or another aggregation.

Formal answer selection

FANS—Formal Answer Selection for Natural Language Math Reasoning Using Lean4—uses Lean4 to select among multiple natural-language candidate solutions (Yao et al., 5 Mar 2025). For each MATH-500 problem, a base model generally generates eight candidates. The candidates are evaluated through pass@1, pass@8, majority vote, an outcome reward model, FANS, or ORM combined with FANS.

FANS translates the question together with a candidate answer into a Lean4 theorem whose conclusion asserts that the candidate answer is correct. LeanTranslator, based on LoT-Solver, generates the formal statement. DeepSeek-Prover-v1.5 and Goedel-Prover then search for proofs, and Lean4’s kernel checks whether the translated theorem and generated proof compile. Verified candidates can be prioritized in answer selection, with majority vote or ORM used as fallback when too few candidates are verified.

The main reported MATH-500 results were:

Model and method	MATH-500 accuracy
Mistral-MV	33.80%
Mistral-FANS	36.40%
DeepSeek-Math-ORM	62.60%
DeepSeek-Math-ORM + FANS	63.80%
Qwen-2.5-Math-ORM	80.80%
Qwen-2.5-Math-ORM + FANS	81.80%

The largest reported overall MATH-500 gain was 1.91% relative for DeepSeek-Math-ORM + FANS, corresponding to an absolute increase from 62.60% to 63.80%, or 1.20 percentage points. Gains were strongest in number theory and algebra, where Lean4 and Mathlib have comparatively mature support. Formalization was more difficult for geometry, combinatorics, and some precalculus problems.

Lean verification does not guarantee semantic correctness with respect to the original problem. A verified proof establishes the translated theorem; if the translation drops a condition, weakens a constraint, or misrepresents an optimization statement, a false positive remains possible.

Distillation, preference optimization, and verifiable-reward training

Phi-4-Mini-Reasoning applies a four-stage recipe to the 3.8B-parameter Phi-4-Mini: large-scale distilled long-CoT mid-training, supervised fine-tuning on high-quality long-CoT data, Rollout DPO, and reinforcement learning with verifiable reward (Xu et al., 30 Apr 2025). Under temperature 0.6, top- $a_i$ 0 0.95, maximum sequence length 32K, and three evaluation runs, it achieved 94.6 Pass@1 on MATH-500.

The stage-by-stage results were:

Training stage	MATH-500
Phi-4-Mini	71.8%
Distilled mid-training	82.9%
Distilled fine-tuning	89.3%
Rollout DPO	93.6%
RL with verifiable reward	94.6%

The final model exceeded DeepSeek-R1-Distill-Qwen-7B at 91.4% by 3.2 points and DeepSeek-R1-Distill-Llama-8B at 86.9% by 7.7 points. The largest cumulative gains came from the two distillation stages, while Rollout DPO added 4.3 points and the final RL stage added 1.0 point on MATH-500.

The result is reported as Pass@1 rather than majority vote or pass@ $a_i$ 1. The paper specifies three evaluation runs but does not report the exact prompt, answer-normalization implementation, or MATH-500-specific sampling aggregation beyond the Pass@1 label.

Natural-formal hybrid reasoning

NL-FL HybridReasoning combines natural-language reasoning with formal Lean reasoning through problem alignment, mixed problem input, and answer extraction (Wang et al., 29 May 2025). It converts a natural-language question-answering problem into an existence problem before autoformalizing it. The formal reasoner receives both the original question and the Lean existence theorem, and a final Qwen3-8B extractor converts the prover’s output into a boxed answer.

Using pass@16, the framework achieved 89.80% on MATH-500 compared with 85.20% for its natural-language baseline, an absolute improvement of 4.60 percentage points. The largest subject-level gain occurred in geometry, where performance increased from 65.85% to 80.49%. Precalculus increased from 76.79% to 83.93%, while algebra increased from 94.35% to 95.16%.

The authors identified 23 MATH-500 problems solved by HybridReasoning but not by the natural-language baseline, even when the baseline received 64 attempts. This is a subset analysis rather than an overall pass@64 result. Removing existence alignment reduced MATH-500 performance to 88.20%, while replacing the specialized Kimina-Prover-Preview-7B with Qwen3-8B reduced it to 88.33%.

Cross-domain reinforcement learning

NEMOTRON-CROSSTHINK applies Group Relative Policy Optimization to mixtures of mathematical and general-purpose reasoning data (Akter et al., 15 Apr 2025). Its principal 7B baseline is Qwen2.5-7B. Training data include NuminaMath, PersonaSkill-math, MATH, MMLU[Train], Natural Reasoning, and Syn-qa. The main blends vary the ratio of mathematical to general-purpose reasoning data.

The reported MATH-500 results were:

Model or blend	MATH-500
Qwen2.5-7B baseline	48.30%
Open-Reasoner-Zero-7B	81.40%
NEMOTRON-CROSSTHINK, natural distribution	77.00%
NEMOTRON-CROSSTHINK, 2:1 mathematical reasoning	77.80%
NEMOTRON-CROSSTHINK, 2:1 general-purpose reasoning	77.60%
NEMOTRON-CROSSTHINK, 2:1 multiple-choice	78.00%
NEMOTRON-CROSSTHINK, 2:1 open-ended	78.40%
NEMOTRON-CROSSTHINK, math-only	78.60%

The headline “+30.1%” improvement refers numerically to an absolute increase from 48.30% to 78.40%, associated with the 2:1 open-ended blend. The highest listed proposed-model MATH-500 score was 78.60% for math-only training, while the 2:1 general-purpose blend achieved the best overall average across seven benchmarks. This distinction illustrates the difference between optimizing MATH-500 alone and optimizing broader reasoning performance.

At 32B scale, the filtered blend reached 84.00% on MATH-500, compared with 82.40% for the unfiltered blend and 60.55% for the Qwen2.5-32B baseline. The filtering procedure retained examples answered incorrectly by Qwen2.5-7B in a zero-shot setting, although the paper does not provide an equal-size random-subsample control.

5. Test-time adaptation and pseudo-label reliability

Test-time reinforcement learning methods use model-generated pseudo-labels rather than ground-truth answers. This creates a distinctive failure mode: an incorrect response can become the majority pseudo-label, while a correct minority response can be assigned negative reward. DDRL—Debiased and Denoised test-time Reinforcement Learning—targets this problem through balanced confidence-aware sampling, debiased fixed advantages, and consensus-based off-policy refinement (Yu et al., 23 Apr 2026).

The paper evaluates MATH-500 with 64 pseudo-label rollouts per prompt, maximum generation length 3,072 tokens, 10 RL episodes, and final evaluation labeled pass@1. The principal MATH-500 results were:

Model	No adaptation	TTRL	ETMR	DDRL
Qwen2.5-Math-1.5B	32.7%	73.0%	76.9%	80.7%
Qwen2.5-Base-3B	53.2%	72.2%	71.7%	72.7%
Llama-3.1-8B-Instruct	48.6%	63.7%	59.5%	67.6%
Qwen2.5-Math-7B	46.7%	83.4%	—	86.7%

For Qwen2.5-Math-1.5B, DDRL improved over TTRL by 7.7 percentage points and over ETMR by 3.8 points. For Qwen2.5-Base-3B, the gain over ETMR was 1.0 point; for Llama-3.1-8B-Instruct, it was 8.1 points.

The method distinguishes high-frequency, low-frequency, and medium-frequency responses. High-frequency answers are predominantly correct, low-frequency answers predominantly incorrect, and medium-frequency answers form an ambiguity region with unstable correctness. DDRL discards medium-frequency responses, balances positive and negative trajectories, and uses fixed advantages of $a_i$ 2 for the pseudo-label and $a_i$ 3 for other selected responses rather than group-relative normalization.

The largest MATH-500 ablation gain for Qwen2.5-Math-1.5B came from consensus-based off-policy refinement: performance increased from 74.6% after balanced sampling and debiased advantage estimation to 80.7%. Additional on-policy RL decreased performance from 74.6% to 74.4%, whereas five epochs of consensus refinement required approximately five minutes and reached 80.7%, compared with approximately 72 minutes for the additional RL comparison.

The paper does not specify a MATH-500 release identifier, answer-normalization algorithm, or precise reduction from the 16 evaluation responses to the reported pass@1 value. It also does not report an oracle-label adaptation condition or a separate MATH-500 majority-vote score.

6. Saturation, contamination, and the search for harder evaluations

MATH-500 is increasingly used as a high-performance mathematical reasoning test, but later work argues that its interpretability depends strongly on the sampling budget. “MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model” (Mayilvahanan et al., 13 Oct 2025) presents MATH-500 as an example of a benchmark that may become saturated under sufficiently large budgets such as pass@1024.

The paper distinguishes sharpening from expansion. Sharpening increases the probability of already available solution modes, while expansion means solving problems outside the base model’s reachable set. For a base model $a_i$ 4 and post-trained model $a_i$ 5, the expansion set is defined as the problems solved by $a_i$ 6 at a specified sampling budget but not by $a_i$ 7. The Expansion Rate is the fraction of the evaluation set belonging to this set.

The argument is that a high pass@1 or pass@1024 score does not alone establish that RL discovered new reasoning strategies. A model may improve by concentrating probability on solutions that were already present in the base model’s sampling distribution. If MATH-500 is nearly solved at pass@1024, it contains little remaining room for measuring boundary expansion.

MATH-Beyond was constructed to provide this missing condition. Its problems were drawn from DAPO-Math-17K and DeepScaleR, filtered for integer-valued answers and other answer-format constraints, screened with DeepSeek-R1-Distill-Qwen2.5-7B, verified with o4-mini-high and GPT-5-Mini, deduplicated against MATH-500 and other benchmarks, and evaluated using 1,024 samples per model. The resulting MATH-Beyond-Union contains 181 problems, while the MATH-Beyond-Intersection contains 41 problems that all considered base models failed to solve at pass@1024.

The authors report that RL-trained models achieved modest expansion rates on MATH-Beyond: 7.83% for Nemotron v1, 9.57% for Nemotron v2, 5.22% for DeepScaleR, and 21.21% for Skywork-OR1-7B. Qwen3 models produced through long-CoT distillation achieved 58.93% and 66.38% expansion for Qwen3-4B and Qwen3-8B, respectively. These are not MATH-500 results; they are presented as a complementary evaluation designed to measure capabilities beyond repeated sampling of base-model solution modes.

The original MATH study also identified contamination concerns. MATH problems derive from public competitions and AoPS archives, and large web-trained models may have encountered original questions, discussions, worked solutions, or derivative copies. The original paper did not provide systematic memorization tests. Later fixed subsets such as MATH-500 may become increasingly familiar through repeated model development and evaluation, making contamination audits particularly relevant.

MATH-500 is also structurally narrow. It emphasizes high-school competition culture, English-language presentation, short-answer problems with unique final answers, and written symbolic reasoning. It does not measure mathematical competence in a universal sense, nor does it directly evaluate experimentation, proof discovery, mathematical modeling in broader settings, or unrestricted theorem proving.

7. Comparative interpretation and limitations

MATH-500 results span a wide range: approximately 33.80% for Mistral majority vote in FANS experiments, 63.80% for DeepSeek-Math with ORM plus FANS, 78.60% for a math-only NEMOTRON-CROSSTHINK configuration, 89.80% pass@16 for NL-FL HybridReasoning, and 94.6 Pass@1 for Phi-4-Mini-Reasoning. These values should not be interpreted as a single chronological leaderboard because the studies differ in model family, training data, sampling budget, temperature, prompt, answer extraction, verifier, aggregation rule, and whether results were reproduced or imported.

Several distinctions are especially important:

Pass@1 versus pass@16 or pass@1024: pass@16 counts a problem as solved if at least one of 16 candidates is correct, whereas pass@1 concerns a single reported attempt or a metric labeled as such. Large sampling budgets can substantially raise success without demonstrating a new reasoning method.
Answer selection versus problem solving: FANS primarily selects among generated candidates using formal verification. Its Lean proof validates a translated theorem and does not necessarily establish that the natural-language interpretation was faithful.
Formal reasoning versus formal verification: NL-FL HybridReasoning uses formal reasoning as an auxiliary capability and extracts an answer from the formal reasoner’s output. The paper does not claim that every reported answer is independently verified end-to-end in Lean.
Specialization versus generalization: Math-only training can be strongest on MATH-500, while mixed-domain training can produce a better average across mathematical and non-mathematical benchmarks.
Relative gains versus percentage points: Several papers describe absolute accuracy differences as percentage improvements. For example, the increase from 48.30% to 78.40% is 30.1 percentage points, not a 30.1% relative increase.
Benchmark construction uncertainty: The original MATH paper does not define MATH-500’s exact composition, split, subject balance, difficulty distribution, contamination controls, or evaluator. Those properties must be verified from the particular MATH-500 release or evaluation harness.

MATH-500 remains useful as a standardized evaluation of high-school mathematical problem solving, particularly when accompanied by complete methodological disclosure. Its interpretation is strongest when reports specify the exact benchmark version, prompt, decoding parameters, number of candidates, answer-normalization implementation, verifier behavior, aggregation metric, and contamination controls. For claims about reinforcement-learning capability expansion, MATH-Beyond proposes a complementary design in which base-model failure at a large sampling budget is established before evaluating post-training gains.