RealMath Benchmark for Research Math

Updated 14 July 2026

RealMath Benchmark is a continuous evaluation framework that tests language models on research-level math problems featuring single exact answers.
It employs an automatic, multi-stage pipeline to extract and transform theorem-to-QA pairs from arXiv papers and Mathematics Stack Exchange posts.
The design supports both tool-free and SageMath-augmented evaluations, enabling precise, automated verification of numerical and symbolic results.

RealMath is a continuous benchmark for evaluating LLMs on research-level mathematics, designed to measure something closer to the actual work mathematicians do than standard competition or proof benchmarks. It is derived directly from research papers and mathematical forums, emphasizes authentic mathematical statements with a single exact answer, usually a numerical or symbolic closed-form response, and is intended to be continually refreshable so that newly published material can be incorporated after model releases (Zhang et al., 18 May 2025). In a later line of work, RealMath also became the main evaluation target for SageMath-augmented LLM agents, which introduced a computable subset and a multi-stage validation pipeline for symbolic and numerical tasks (Snopov et al., 7 Jul 2026).

1. Definition, scope, and benchmark philosophy

RealMath is introduced as a continuous benchmark for evaluating LLMs on research-level mathematics. Its central motivation is that existing mathematical benchmarks are often not representative of research practice: competition-style benchmarks emphasize contest problems, formal theorem-proving benchmarks test proof generation inside proof assistants, and expert-crafted frontier benchmarks are labor-intensive and narrow (Zhang et al., 18 May 2025). RealMath is intended to fill a different niche by evaluating models on authentic mathematical statements drawn from the literature and forums, spanning the broader range of “math in the wild,” including routine research lemmas, exact formulas, and derivable facts that researchers actually encounter.

The benchmark focuses on tasks that have a single exact answer, usually a numerical or symbolic closed-form response, with clear verification criteria. It explicitly avoids open-ended proof tasks, inequalities and asymptotic-only claims, statements with multiple valid answers, non-unique existence results, and claims where the interesting part is only the proof rather than the final answer (Zhang et al., 18 May 2025). This design makes large-scale automated evaluation possible while preserving a connection to authentic mathematical content.

A later study sharpened this framing by describing RealMath as “a Continuous Benchmark for Evaluating LLMs on Research-Level Mathematics” whose original instances are triples of the form $(\text{context}, \text{question}, \text{answer})$ . In that study, the benchmark was characterized as especially oriented toward research-level problems, particularly constructive theorems and tasks whose answers can be checked algorithmically (Snopov et al., 7 Jul 2026). The same study also noted an important practical limitation of the original format: the answer field is plain LaTeX, which makes automatic comparison difficult, and many items are not immediately suitable for symbolic computation.

2. Source corpora and construction pipeline

RealMath draws from two main source types: research papers, mainly from arXiv, and Mathematics Stack Exchange. The arXiv component includes mathematical papers in categories such as math.CO, math.NT, math.AP, and math.RT, together with some CS.arXiv papers when they contain math-heavy constructive statements. The forum component uses Stack Exchange questions with tags such as limits, definite integrals, and integration, which are processed into theorem-like statements plus question-answer items (Zhang et al., 18 May 2025).

The original paper describes a five-stage automatic pipeline. In Stage 1, papers are retrieved from the arXiv API for a chosen time window; an example used in the paper collects about 4,000 Math papers from May to September 2022. In Stage 2, the original LaTeX source is downloaded so that notation is preserved faithfully; in the example, 3,922 papers remained usable after source extraction. In Stage 3, an LLM judge identifies constructive theorems with a single, exact, verifiable answer. The judge prompt accepts exact numerical answers, unique existence or uniqueness statements, exact formulas, exact complexity equalities such as $\Theta(n^2)$ , and identities or equations that imply a fixed answer, while rejecting inequalities, approximations, non-unique existence statements, ambiguous results, and statements whose main content is a proof rather than a fixed answer. In the example pipeline, 14,747 theorems were detected and 407 were classified as high-quality constructive theorems. In Stage 4, another LLM converts each retained theorem into a question-answer pair; 401 QA pairs were generated from the 407 selected theorems. In Stage 5, trivial or low-quality questions are filtered out, leaving 280 QA pairs in the illustrated example (Zhang et al., 18 May 2025).

The benchmark’s theorem-to-QA transformation preserves exact mathematical content while changing the task form. The paper’s canonical example is a theorem about the number of cliques of order $3$ in a Peisert graph. It is converted into the question “What is the number of cliques of order 3 in $P^*(q)$ as a function of $q$ ?”, with answer $\frac{q(q-1)(q-5)}{48}$ (Zhang et al., 18 May 2025). This conversion mechanism is the basis on which RealMath turns research-level mathematical statements into automatically verifiable benchmark items.

3. Dataset composition, input format, and evaluation

The released benchmark is evaluated on three datasets, and a later SageMath-oriented study extracted a narrower executable subset from the arXiv portion (Zhang et al., 18 May 2025, Snopov et al., 7 Jul 2026).

Component	Content	Size
Math.arXiv	QA pairs from arXiv-derived mathematics papers	633
CS.arXiv	QA pairs from CS.arXiv papers	111
Math.StackExchange	QA pairs from forum posts	542
Refined SageMath subset	Numerical and symbolic-expression tasks sampled from RealMath	133

For Math.arXiv and CS.arXiv, the model receives the generated question together with the full relevant paper context preceding the theorem. For Math.StackExchange, the model receives only the question, because the forum format is already self-contained (Zhang et al., 18 May 2025). The original correctness criterion is exact-answer verification: a response is marked correct only if it exactly matches the ground-truth answer, although the evaluation judge is instructed to accept mathematically equivalent expressions. The answer must be mathematically correct, equivalent to the ground truth, and renderable in standard LaTeX without custom definitions. The metric is accuracy,

$\text{Accuracy} = \frac{\#\text{correct answers}}{\#\text{total questions}}.$

The original paper also defines Easy / Medium / Hard difficulty buckets using several older models. A problem is Hard if none of the older models answer correctly, Medium if fewer than half answer correctly, and Easy if more than half answer correctly (Zhang et al., 18 May 2025). Manual review is part of the curation process: reported manual removal is 5.2\% for Math.arXiv and 7.5\% for CS.arXiv, with discarded samples consisting mostly of theorems without a unique answer or QA pairs that earlier filters failed to reject. The paper further notes that arXiv statements are assumed correct but not guaranteed, since arXiv is not fully peer reviewed.

The later SageMath-oriented refinement operates under stricter computability constraints. From the 633 RealMath problems in Math.arXiv, answers were normalized from free-form LaTeX into executable SymPy-parsable strings, context was compressed into the definitions and information needed to interpret the question, answer type was annotated as number, expression, or formula, a CAS-suitability check was applied, and 133 problems were then sampled for evaluation, of which 73 are expression-typed and 60 are numerical-typed (Snopov et al., 7 Jul 2026).

4. Empirical findings on the original RealMath benchmark

The central empirical claim of the original RealMath paper is that frontier models perform surprisingly well on research mathematics compared with harder competition-style or frontier benchmarks (Zhang et al., 18 May 2025). On Math.arXiv, reported accuracies include 49.1\% for o3, 43.4\% for o4-mini, 32.5\% for Gemini 2.5 Pro, and 30.5\% for DeepSeek-R1. On CS.arXiv, the corresponding values are 44.1\%, 42.3\%, 25.2\%, and 31.5\%. On Math.StackExchange, the same models achieve 70.7\%, 70.8\%, 60.9\%, and 62.2\%, respectively.

Performance is highly non-uniform across difficulty levels. For example, on Math.arXiv, o3 scores 97.5\% on easy problems, 81.4\% on medium problems, and 27.9\% on hard problems (Zhang et al., 18 May 2025). The benchmark therefore contains both an accessible subset and a materially difficult subset. Topic-specific variation is also substantial: o3 is reported as especially strong on theoretical categories such as Representation Theory, Number Theory, and Analysis of PDEs, whereas Gemini 2.5 Pro performs better on more applied domains such as Machine Learning, Optimization and Control, and Probability.

The paper reports that context helps, but is not always essential. For arXiv-derived tasks, preceding paper context can be important for notation or proof dependencies, yet many items remain solvable without the full context because the statements are self-contained enough or inferable from the question alone (Zhang et al., 18 May 2025). Error analysis indicates that the dominant failures are reasoning failures: flawed reasoning, missing key insights, conceptual misunderstandings, and arithmetic mistakes.

Two further findings are methodologically significant. First, fine-tuning GPT-4o-mini on 500 Math.arXiv samples and testing on the remaining 133 produced no statistically significant improvement (Zhang et al., 18 May 2025). Second, a contamination analysis found that models often do better on newer 2025 samples than on older 2022 ones, which the authors interpret as evidence that current training data may include more recent mathematical content near the cutoff period.

A 2026 study used RealMath as the principal target for evaluating a ReAct-style agentic setup that combines LLM reasoning with SageMath and Context7 (Snopov et al., 7 Jul 2026). In this setup, the model alternates between reasoning and acting in a multi-turn loop. SageMath serves as the core computer algebra system: the model can generate Sage code, execute it in an isolated sandbox, inspect outputs, and iterate. Context7 provides up-to-date SageMath documentation and code examples, with the stated role of reducing hallucinations about module names, paths, and outdated API signatures.

The paper first refines RealMath into a computable subset. Its high-level selection procedure is: retain only problems that can be normalized into SymPy objects suitable for automatic verification; use LLM-assisted filtering to identify problems for which symbolic experimentation or computational verification is feasible within SageMath; and sample from the filtered pool while extending it with problems from underrepresented arXiv categories (Snopov et al., 7 Jul 2026). The appendix gives the concrete construction. Starting from 633 RealMath problems, the authors enrich the data with arXiv categories, normalize answers from free-form LaTeX into executable SymPy-parsable strings, compress context, annotate answer type, perform a CAS-suitability check, and then sample 133 problems to reduce category skew. The post-processing stages reduce the set to 300 problems after answer normalization, then to 219 after the CAS-suitability stage, and finally to 133 after sampling. A task is deemed suitable if objects mentioned in the question can be represented by SageMath, candidate answers can be explored via computational experiment, and intermediate computation steps can be executed in SageMath.

The same study introduces a multi-stage validation pipeline. In Stage 1, a symbolic validator parses model output and ground truth into SymPy objects, simplifies their difference, and accepts the answer if the result is zero; multi-component answers are compared component-wise. In Stage 2, any symbolic rejection is sent to three LLM judges—Claude Opus 4.7, GPT-5.5, and Gemini 3.5 Flash—which each return a verdict and justification; the final decision is by majority vote. The authors manually inspect 30\% of the cases reaching this stage and use those human annotations as ground truth. Quantitatively, the symbolic checker directly accepted 39.2\% of all 3,990 predictions; the remaining 60.8\% went to the judge panel; the judges overturned the symbolic rejection in 30.8\% of those cases, corresponding to 747 answers; the panel had 97.2\% unanimous verdicts; and, taking the final pipeline verdict as reference, the symbolic checker alone was correct in 81.3\% of cases (Snopov et al., 7 Jul 2026).

The experimental study evaluates 15 models under a tool-free baseline and a tool-augmented agentic setting with SageMath and Context7: Claude Opus 4.7, Claude Opus 4.8, Sonnet 5, DeepSeek-V3.2, DeepSeek-V4-Flash, DeepSeek-V4-Pro, Qwen 3.7-Max, GLM 5.2, Kimi 2.7, MiniMax-M3, GPT-5.5, Gemini-3.5-Flash, Gemini-3.1-Pro-preview, Grok 4.3, and Fugu-Ultra. The main metric is solve rate, defined as the fraction of problems whose final answer is accepted by the validation pipeline, with 95\% Wilson score confidence intervals.

The headline result is that SageMath access improves every model. The average gain across models is +9.7 percentage points, with gains ranging from +1.5 pp to +27.8 pp (Snopov et al., 7 Jul 2026). Among the largest improvements are Qwen 3.7-Max: 42.1\% \rightarrow 69.9\%, MiniMax: 38.3\% \rightarrow 55.6\%, and DeepSeek 3.2: 33.1\% \rightarrow 48.9\%. Closed frontier models also improve: GPT-5.5: 67.7\% \rightarrow 75.2\%, Opus 4.8: 66.2\% \rightarrow 73.7\%, Opus 4.7: 61.7\% \rightarrow 72.9\%, and Fugu-Ultra: 66.9\% \rightarrow 72.9\%. The paper emphasizes that tool access narrows the open-vs-closed gap: open-weight models gain +15.3 pp on average, while closed models gain +6.5 pp on average. GPT-5.5 is reported as the best overall model, with 75.2\% solve rate, the lowest token usage among tool-enabled configurations, and the strongest efficiency on the accuracy-cost plane.

The analysis of traces and failures is unusually detailed. Tool access raises token usage by factors from 4.5× to 99×, with tool-enabled runs spanning roughly 34k to 556k tokens per problem, but higher token use does not imply higher accuracy (Snopov et al., 7 Jul 2026). Failure analysis shows that the dominant Sage exceptions are TypeError, NameError, AttributeError, SyntaxError, which together account for 87\% of all exceptions. Recovery after failed calls varies sharply: some models recover successfully 69\%–77\% of the time, whereas weaker ones are as low as 16\%–32\%. Trace length is strongly bimodal: a problem is often solved within the first 3–4 tool interactions, or the agent exhausts the full 15-call budget. The study argues that SageMath helps most when models use it as a verifier or experimental probe rather than as a brute-force search engine. It also presents a case study on twisted torus knots in which GPT-5.5 computes intermediate objects, extracts exponent gaps from Alexander polynomials, infers staircase lengths, builds Smith normal forms over $\mathbb{F}_2[U]$ , and uses these computations to formulate and test conjectures. The authors explicitly describe this as a step toward automated conjecture discovery.

The Sage sandbox itself is tightly specified: code runs in an ephemeral Docker container; each call starts in a fresh interpreter; a 45s wall-clock timeout and 50s CPU-time limit are enforced; 2 CPU cores and 4 GB RAM are allocated; network access is disallowed; and outputs larger than 262,144 bytes are rejected (Snopov et al., 7 Jul 2026).

RealMath occupies a specific position within a broader landscape of mathematical evaluation. It should be distinguished from ReliableMath, which studies reliability on both solvable and unsolvable mathematical reasoning tasks and defines success in terms of solving solvable problems correctly while explicitly recognizing unsolvable problems (Xue et al., 3 Jul 2025). It is also distinct from RealMath-Eval, which evaluates LLM judges on 224 real high-school exam responses and finds that state-of-the-art judges struggle on authentic human reasoning, with a high Mean Squared Error of approximately 2.96 against expert grading (Mao et al., 8 Jun 2026). Likewise, MathReal is a real-scene multimodal benchmark of 2,000 K–12 questions from handheld mobile-device images rather than a research-level symbolic benchmark (Feng et al., 8 Aug 2025).

Within research-level mathematics, RealMath is one of several attempts to move beyond competition-style evaluation, but the neighboring benchmarks make different design choices. Riemann-Bench is a private benchmark of 25 expert-curated problems for “moonshot mathematics,” evaluated as unconstrained research agents with 100 independent runs per problem, and all frontier models score below 10\% (Garre et al., 8 Apr 2026). LiveMathematicianBench is a live benchmark built from post-cutoff arXiv papers that uses a thirteen-category logical taxonomy and proof-sketch-guided distractors, with best standard-setting accuracy 43.5\% and a substitution-resistant regime in which GPT-5.4 reaches 30.6\% (He et al., 2 Apr 2026). EternalMath explicitly cites RealMath as related work but positions itself as a stronger theorem-grounded and execution-verified living benchmark derived from recent peer-reviewed mathematical literature (Ma et al., 4 Jan 2026).

The benchmark has also influenced adjacent evaluation ideas. MathArena is presented as a dynamic public benchmark based on newly released competitions and is described as a concrete realization of a “RealMath Benchmark” idea oriented toward uncontaminated, live evaluation rather than literature-derived theorem completion (Balunović et al., 29 May 2025). A plausible implication is that the phrase “RealMath Benchmark” now refers not only to the specific dataset introduced in 2025, but also to a wider methodological shift toward authentic, contamination-resistant, and practically verifiable mathematical evaluation. Within that shift, RealMath remains the benchmark most explicitly centered on authentic research papers and mathematical forums, verifiable fixed-answer tasks, and continuous refreshability (Zhang et al., 18 May 2025).