---
title: RealMath Benchmark for Research Math
url: https://www.emergentmind.com/topics/realmath-benchmark
type: topic
---

# RealMath Benchmark for Research Math

RealMath is a continuous benchmark for evaluating language models on research-level mathematics, designed to measure something closer to the actual work mathematicians do than standard competition or proof benchmarks. It is derived directly from research papers and mathematical forums, emphasizes authentic mathematical statements with a single exact answer, usually a numerical or symbolic closed-form response, and is intended to be continually refreshable so that newly published material can be incorporated after model releases [2505.12575]. In a later line of work, RealMath also became the main evaluation target for SageMath-augmented LLM agents, which introduced a computable subset and a multi-stage validation pipeline for symbolic and numerical tasks [2607.06820].

## 1. Definition, scope, and benchmark philosophy

RealMath is introduced as a **continuous benchmark for evaluating LLMs on research-level mathematics**. Its central motivation is that existing mathematical benchmarks are often not representative of research practice: competition-style benchmarks emphasize contest problems, formal theorem-proving benchmarks test proof generation inside proof assistants, and expert-crafted frontier benchmarks are labor-intensive and narrow [2505.12575]. RealMath is intended to fill a different niche by evaluating models on authentic mathematical statements drawn from the literature and forums, spanning the broader range of “math in the wild,” including routine research lemmas, exact formulas, and derivable facts that researchers actually encounter.

The benchmark focuses on tasks that have a **single exact answer**, usually a **numerical or symbolic closed-form response**, with clear verification criteria. It explicitly avoids open-ended proof tasks, inequalities and asymptotic-only claims, statements with multiple valid answers, non-unique existence results, and claims where the interesting part is only the proof rather than the final answer [2505.12575]. This design makes large-scale automated evaluation possible while preserving a connection to authentic mathematical content.

A later study sharpened this framing by describing RealMath as “a Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics” whose original instances are triples of the form \((\text{context}, \text{question}, \text{answer})\). In that study, the benchmark was characterized as especially oriented toward research-level problems, particularly constructive theorems and tasks whose answers can be checked algorithmically [2607.06820]. The same study also noted an important practical limitation of the original format: the answer field is plain LaTeX, which makes automatic comparison difficult, and many items are not immediately suitable for symbolic computation.

## 2. Source corpora and construction pipeline

RealMath draws from two main source types: **research papers**, mainly from **arXiv**, and **Mathematics Stack Exchange**. The arXiv component includes mathematical papers in categories such as `math.CO`, `math.NT`, `math.AP`, and `math.RT`, together with some `CS.arXiv` papers when they contain math-heavy constructive statements. The forum component uses Stack Exchange questions with tags such as limits, definite integrals, and integration, which are processed into theorem-like statements plus question-answer items [2505.12575].

The original paper describes a five-stage automatic pipeline. In **Stage 1**, papers are retrieved from the arXiv API for a chosen time window; an example used in the paper collects about **4,000 Math papers from May to September 2022**. In **Stage 2**, the original LaTeX source is downloaded so that notation is preserved faithfully; in the example, **3,922** papers remained usable after source extraction. In **Stage 3**, an LLM judge identifies constructive theorems with a **single, exact, verifiable answer**. The judge prompt accepts exact numerical answers, unique existence or uniqueness statements, exact formulas, exact complexity equalities such as \(\Theta(n^2)\), and identities or equations that imply a fixed answer, while rejecting inequalities, approximations, non-unique existence statements, ambiguous results, and statements whose main content is a proof rather than a fixed answer. In the example pipeline, **14,747** theorems were detected and **407** were classified as high-quality constructive theorems. In **Stage 4**, another LLM converts each retained theorem into a question-answer pair; **401** QA pairs were generated from the **407** selected theorems. In **Stage 5**, trivial or low-quality questions are filtered out, leaving **280** QA pairs in the illustrated example [2505.12575].

The benchmark’s theorem-to-QA transformation preserves exact mathematical content while changing the task form. The paper’s canonical example is a theorem about the number of cliques of order \(3\) in a Peisert graph. It is converted into the question “What is the number of cliques of order 3 in \(P^*(q)\) as a function of \(q\)?”, with answer \(\frac{q(q-1)(q-5)}{48}\) [2505.12575]. This conversion mechanism is the basis on which RealMath turns research-level mathematical statements into automatically verifiable benchmark items.

## 3. Dataset composition, input format, and evaluation

The released benchmark is evaluated on three datasets, and a later SageMath-oriented study extracted a narrower executable subset from the arXiv portion [2505.12575] [2607.06820].

| Component | Content | Size |
|---|---|---:|
| Math.arXiv | QA pairs from arXiv-derived mathematics papers | 633 |
| CS.arXiv | QA pairs from CS.arXiv papers | 111 |
| Math.StackExchange | QA pairs from forum posts | 542 |
| Refined SageMath subset | Numerical and symbolic-expression tasks sampled from RealMath | 133 |

For **Math.arXiv** and **CS.arXiv**, the model receives the generated question together with the full relevant paper context preceding the theorem. For **Math.StackExchange**, the model receives only the question, because the forum format is already self-contained [2505.12575]. The original correctness criterion is exact-answer verification: a response is marked correct only if it exactly matches the ground-truth answer, although the evaluation judge is instructed to accept mathematically equivalent expressions. The answer must be mathematically correct, equivalent to the ground truth, and renderable in standard LaTeX without custom definitions. The metric is **accuracy**,
\[
\text{Accuracy} = \frac{\#\text{correct answers}}{\#\text{total questions}}.
\]

The original paper also defines **Easy / Medium / Hard** difficulty buckets using several older models. A problem is **Hard** if none of the older models answer correctly, **Medium** if fewer than half answer correctly, and **Easy** if more than half answer correctly [2505.12575]. Manual review is part of the curation process: reported manual removal is **5.2\%** for Math.arXiv and **7.5\%** for CS.arXiv, with discarded samples consisting mostly of theorems without a unique answer or QA pairs that earlier filters failed to reject. The paper further notes that arXiv statements are assumed correct but not guaranteed, since arXiv is not fully peer reviewed.

The later SageMath-oriented refinement operates under stricter computability constraints. From the **633 RealMath problems** in Math.arXiv, answers were normalized from free-form LaTeX into executable SymPy-parsable strings, context was compressed into the definitions and information needed to interpret the question, answer type was annotated as number, expression, or formula, a CAS-suitability check was applied, and **133** problems were then sampled for evaluation, of which **73** are expression-typed and **60** are numerical-typed [2607.06820].

## 4. Empirical findings on the original RealMath benchmark

The central empirical claim of the original RealMath paper is that frontier models perform **surprisingly well** on research mathematics compared with harder competition-style or frontier benchmarks [2505.12575]. On **Math.arXiv**, reported accuracies include **49.1\%** for **o3**, **43.4\%** for **o4-mini**, **32.5\%** for **Gemini 2.5 Pro**, and **30.5\%** for **DeepSeek-R1**. On **CS.arXiv**, the corresponding values are **44.1\%**, **42.3\%**, **25.2\%**, and **31.5\%**. On **Math.StackExchange**, the same models achieve **70.7\%**, **70.8\%**, **60.9\%**, and **62.2\%**, respectively.

Performance is highly non-uniform across difficulty levels. For example, on Math.arXiv, **o3** scores **97.5\%** on easy problems, **81.4\%** on medium problems, and **27.9\%** on hard problems [2505.12575]. The benchmark therefore contains both an accessible subset and a materially difficult subset. Topic-specific variation is also substantial: **o3** is reported as especially strong on theoretical categories such as **Representation Theory**, **Number Theory**, and **Analysis of PDEs**, whereas **Gemini 2.5 Pro** performs better on more applied domains such as **Machine Learning**, **Optimization and Control**, and **Probability**.

The paper reports that context helps, but is not always essential. For arXiv-derived tasks, preceding paper context can be important for notation or proof dependencies, yet many items remain solvable without the full context because the statements are self-contained enough or inferable from the question alone [2505.12575]. Error analysis indicates that the dominant failures are reasoning failures: flawed reasoning, missing key insights, conceptual misunderstandings, and arithmetic mistakes.

Two further findings are methodologically significant. First, fine-tuning **GPT-4o-mini** on **500** Math.arXiv samples and testing on the remaining **133** produced **no statistically significant improvement** [2505.12575]. Second, a contamination analysis found that models often do better on **newer 2025 samples** than on **older 2022** ones, which the authors interpret as evidence that current training data may include more recent mathematical content near the cutoff period.

## 5. Executable refinement and SageMath-augmented RealMath

A 2026 study used RealMath as the principal target for evaluating a **ReAct-style** agentic setup that combines LLM reasoning with **SageMath** and **Context7** [2607.06820]. In this setup, the model alternates between reasoning and acting in a multi-turn loop. **SageMath** serves as the core computer algebra system: the model can generate Sage code, execute it in an isolated sandbox, inspect outputs, and iterate. **Context7** provides up-to-date SageMath documentation and code examples, with the stated role of reducing hallucinations about module names, paths, and outdated API signatures.

The paper first refines RealMath into a computable subset. Its high-level selection procedure is: retain only problems that can be normalized into SymPy objects suitable for automatic verification; use LLM-assisted filtering to identify problems for which symbolic experimentation or computational verification is feasible within SageMath; and sample from the filtered pool while extending it with problems from underrepresented arXiv categories [2607.06820]. The appendix gives the concrete construction. Starting from **633** RealMath problems, the authors enrich the data with arXiv categories, normalize answers from free-form LaTeX into executable SymPy-parsable strings, compress context, annotate answer type, perform a CAS-suitability check, and then sample **133** problems to reduce category skew. The post-processing stages reduce the set to **300** problems after answer normalization, then to **219** after the CAS-suitability stage, and finally to **133** after sampling. A task is deemed suitable if objects mentioned in the question can be represented by SageMath, candidate answers can be explored via computational experiment, and intermediate computation steps can be executed in SageMath.

The same study introduces a **multi-stage validation pipeline**. In **Stage 1**, a symbolic validator parses model output and ground truth into SymPy objects, simplifies their difference, and accepts the answer if the result is zero; multi-component answers are compared component-wise. In **Stage 2**, any symbolic rejection is sent to three LLM judges—**Claude Opus 4.7**, **GPT-5.5**, and **Gemini 3.5 Flash**—which each return a verdict and justification; the final decision is by majority vote. The authors manually inspect **30\%** of the cases reaching this stage and use those human annotations as ground truth. Quantitatively, the symbolic checker directly accepted **39.2\%** of all **3,990** predictions; the remaining **60.8\%** went to the judge panel; the judges overturned the symbolic rejection in **30.8\%** of those cases, corresponding to **747** answers; the panel had **97.2\%** unanimous verdicts; and, taking the final pipeline verdict as reference, the symbolic checker alone was correct in **81.3\%** of cases [2607.06820].

The experimental study evaluates **15 models** under a **tool-free baseline** and a **tool-augmented agentic setting** with SageMath and Context7: Claude Opus 4.7, Claude Opus 4.8, Sonnet 5, DeepSeek-V3.2, DeepSeek-V4-Flash, DeepSeek-V4-Pro, Qwen 3.7-Max, GLM 5.2, Kimi 2.7, MiniMax-M3, GPT-5.5, Gemini-3.5-Flash, Gemini-3.1-Pro-preview, Grok 4.3, and Fugu-Ultra. The main metric is **solve rate**, defined as the fraction of problems whose final answer is accepted by the validation pipeline, with **95\% Wilson score confidence intervals**.

The headline result is that SageMath access improves every model. The average gain across models is **+9.7 percentage points**, with gains ranging from **+1.5 pp** to **+27.8 pp** [2607.06820]. Among the largest improvements are **Qwen 3.7-Max: 42.1\% \rightarrow 69.9\%**, **MiniMax: 38.3\% \rightarrow 55.6\%**, and **DeepSeek 3.2: 33.1\% \rightarrow 48.9\%**. Closed frontier models also improve: **GPT-5.5: 67.7\% \rightarrow 75.2\%**, **Opus 4.8: 66.2\% \rightarrow 73.7\%**, **Opus 4.7: 61.7\% \rightarrow 72.9\%**, and **Fugu-Ultra: 66.9\% \rightarrow 72.9\%**. The paper emphasizes that tool access narrows the open-vs-closed gap: open-weight models gain **+15.3 pp on average**, while closed models gain **+6.5 pp on average**. **GPT-5.5** is reported as the best overall model, with **75.2\%** solve rate, the lowest token usage among tool-enabled configurations, and the strongest efficiency on the accuracy-cost plane.

The analysis of traces and failures is unusually detailed. Tool access raises token usage by factors from **4.5×** to **99×**, with tool-enabled runs spanning roughly **34k to 556k tokens per problem**, but higher token use does **not** imply higher accuracy [2607.06820]. Failure analysis shows that the dominant Sage exceptions are **TypeError, NameError, AttributeError, SyntaxError**, which together account for **87\%** of all exceptions. Recovery after failed calls varies sharply: some models recover successfully **69\%–77\%** of the time, whereas weaker ones are as low as **16\%–32\%**. Trace length is strongly bimodal: a problem is often solved within the first **3–4 tool interactions**, or the agent exhausts the full **15-call budget**. The study argues that SageMath helps most when models use it as a verifier or experimental probe rather than as a brute-force search engine. It also presents a case study on twisted torus knots in which **GPT-5.5** computes intermediate objects, extracts exponent gaps from Alexander polynomials, infers staircase lengths, builds Smith normal forms over \(\mathbb{F}_2[U]\), and uses these computations to formulate and test conjectures. The authors explicitly describe this as a step toward **automated conjecture discovery**.

The Sage sandbox itself is tightly specified: code runs in an ephemeral Docker container; each call starts in a fresh interpreter; a **45s** wall-clock timeout and **50s** CPU-time limit are enforced; **2 CPU cores** and **4 GB RAM** are allocated; network access is disallowed; and outputs larger than **262,144 bytes** are rejected [2607.06820].

## 6. Related benchmarks, adjacent directions, and common confusions

RealMath occupies a specific position within a broader landscape of mathematical evaluation. It should be distinguished from **ReliableMath**, which studies reliability on both solvable and unsolvable mathematical reasoning tasks and defines success in terms of solving solvable problems correctly while explicitly recognizing unsolvable problems [2507.03133]. It is also distinct from **RealMath-Eval**, which evaluates LLM judges on **224** real high-school exam responses and finds that state-of-the-art judges struggle on authentic human reasoning, with a high Mean Squared Error of approximately **2.96** against expert grading [2606.10254]. Likewise, **MathReal** is a real-scene multimodal benchmark of **2,000** K–12 questions from handheld mobile-device images rather than a research-level symbolic benchmark [2508.06009].

Within research-level mathematics, RealMath is one of several attempts to move beyond competition-style evaluation, but the neighboring benchmarks make different design choices. **Riemann-Bench** is a **private benchmark of 25 expert-curated problems** for “moonshot mathematics,” evaluated as unconstrained research agents with **100 independent runs per problem**, and all frontier models score below **10\%** [2604.06802]. **LiveMathematicianBench** is a live benchmark built from post-cutoff arXiv papers that uses a **thirteen-category logical taxonomy** and **proof-sketch-guided distractors**, with best standard-setting accuracy **43.5\%** and a substitution-resistant regime in which **GPT-5.4** reaches **30.6\%** [2604.01754]. **EternalMath** explicitly cites RealMath as related work but positions itself as a stronger theorem-grounded and execution-verified living benchmark derived from recent peer-reviewed mathematical literature [2601.01400].

The benchmark has also influenced adjacent evaluation ideas. **MathArena** is presented as a dynamic public benchmark based on newly released competitions and is described as a concrete realization of a “RealMath Benchmark” idea oriented toward uncontaminated, live evaluation rather than literature-derived theorem completion [2505.23281]. A plausible implication is that the phrase “RealMath Benchmark” now refers not only to the specific dataset introduced in 2025, but also to a wider methodological shift toward authentic, contamination-resistant, and practically verifiable mathematical evaluation. Within that shift, RealMath remains the benchmark most explicitly centered on authentic research papers and mathematical forums, verifiable fixed-answer tasks, and continuous refreshability [2505.12575].

Source: https://www.emergentmind.com/topics/realmath-benchmark