RMBench: Benchmark for Research-Level Math

• RMBench is a private benchmark containing 25 research-level math problems that require advanced theoretical reasoning and sustained, multi-step argumentation.
• It employs a rigorous double-blind verification and programmatic answer checking process to ensure precise evaluation of AI systems using advanced mathematical machinery.
• Empirical results show frontier models score below 10% pass@1, highlighting the gap between high performance in competition math and genuine research-level problem solving.

RMBench, introduced in "Riemann-Bench: A Benchmark for Moonshot Mathematics" (Garre et al., 8 Apr 2026), is a private, expert-curated benchmark built to measure AI systems on research-level mathematical reasoning rather than olympiad-style puzzle solving. It contains 25 problems designed to probe deep theoretical reasoning, multi-step argument construction, and the use of advanced mathematical machinery across PhD-level domains. The benchmark evaluates frontier models as unconstrained research agents with access to coding tools, search, and open-ended reasoning, and reports that all tested frontier models currently score below 10\% pass@1, highlighting a substantial gap between olympiad-level performance and genuine research-level mathematical reasoning (Garre et al., 8 Apr 2026).

1. Motivation and conceptual scope

RMBench was motivated by a growing gap between success on competition mathematics and performance on sustained theoretical work. In the benchmark’s framing, competition mathematics deliberately targets four domains—algebra, combinatorics, geometry, and number theory—and excludes advanced machinery such as calculus. Problems are designed to be solvable in a few hours and often reward a single, clever insight. Research mathematics is presented as a fundamentally different regime: solving a single problem can take weeks, requires specialized theory, and demands long chains of interlocking arguments (Garre et al., 8 Apr 2026).

The benchmark therefore targets what the authors call “moonshot mathematics.” In this usage, the term refers not merely to clever tricks, but to deep theoretical reasoning that pulls from advanced areas such as measure theory, variational principles, stability analysis, manifolds, and advanced algebraic structures. RMBench is positioned as a necessary intermediate milestone on the path to truly open-ended “moonshots,” where even the existence of a solution may be unknown. Reliable performance on research-level problems with known solutions is treated as a prerequisite for AI systems that aspire to contribute to open research.

A central distinction from olympiad-style benchmarks follows from this framing. IMO/MATH-style datasets emphasize cleverness within narrow, elementary toolsets and fixed domains, whereas RMBench is explicitly built around PhD-level theory, multi-hour-to-multi-week solution efforts, and multi-step chains of arguments that often draw on multiple advanced areas simultaneously. This suggests that RMBench is intended less as a contest benchmark than as an evaluation of theory acquisition, framework selection, and long-horizon deductive control.

2. Benchmark composition and problem design

RMBench contains 25 problems. They were authored and curated by Ivy League mathematics professors, graduate students, and PhD-holding IMO medalists. Contributors drew on problems from their own research—problems that routinely took them weeks to solve independently—and noted that even their graduate students and colleagues would struggle to solve them (Garre et al., 8 Apr 2026).

The problem set spans diverse areas of PhD-level mathematics, including variational principles, measure theory, stability analysis, manifolds, and advanced algebraic structures. In contrast to the four foundational IMO domains, the benchmark explicitly probes theory-heavy topics and multi-step reasoning that go substantially beyond competition settings. Every problem admits a unique, closed-form solution. There is no partial credit or subjective judgment; answers are either correct or incorrect. Where multiple equivalent solution representations exist, correctness is assessed via programmatic verifiers.

The paper includes an illustrative example involving multibasic $A$-modules over the ring of Hahn series with real-valued valuation and residue field $\mathbf F_2$. Let $K$ be the field of Hahn series in indeterminate $t$ with value group $\mathbf R$, and $A$ the subring of elements with non-negative valuation. “Basic” $A$-modules are quotients of $A$-submodules of $K$; “multibasic” modules are finite direct sums of basic modules, and every multibasic $A$-module has a unique decomposition into basic submodules. A core insight is that submodules of $\mathbf F_2$0 are determined by which powers $\mathbf F_2$1 they contain, so each submodule corresponds to a cut in $\mathbf F_2$2; this severely constrains the canonical forms of basic modules $\mathbf F_2$3 (Garre et al., 8 Apr 2026).

Because multibasic modules decompose uniquely, the classification reduces to counting allowed combinations of these building blocks under three structural conditions: constraints on the endomorphism ring of $\mathbf F_2$4 and a dimension function on associated $\mathbf F_2$5-vector spaces. Solving the problem requires techniques spanning valuation theory, tensor-hom adjunction, ring-theoretic restrictions on summands, and a final combinatorial count. The example is included to illustrate the kind of deep theoretical machinery and sustained reasoning RMBench targets, without revealing private test content.

3. Verification protocol, answer checking, and privacy

Each RMBench problem undergoes double-blind verification: two independent domain experts who are not shown the author’s solution must solve the problem from scratch and confirm its validity. Verifiers also check for ambiguity, underspecification, and appropriate difficulty; problems that fail verification are revised or excluded (Garre et al., 8 Apr 2026). This procedure is central to the benchmark’s claim that the tasks are both mathematically sound and appropriately calibrated for research-level evaluation.

Answer checking is designed to eliminate subjective grading. Because every problem has a unique, closed-form solution, correctness is binary. Where multiple equivalent representations exist, programmatic verifiers determine equivalence. This removes the need for partial credit and avoids evaluator discretion at scoring time.

Privacy is treated as a methodological requirement rather than a distributional inconvenience. RMBench is kept fully private. The authors argue that public benchmarks—even with contamination controls—are vulnerable to leakage and memorization; a benchmark that has been seen is, in effect, compromised. To ensure unbiased evaluation unaffected by training-data exposure, models are tested via a controlled evaluation service. A plausible implication is that RMBench treats contamination resistance as part of the benchmark definition itself, not as an auxiliary hygiene measure.

4. Evaluation protocol and statistical reporting

Models are evaluated as unconstrained research agents. They have full access to coding tools, specifically a Python interpreter, search, and open-ended reasoning, with no artificial constraints on interaction format or token budget. The stated objective is to mirror a realistic research workflow rather than a rigid prompt-response loop (Garre et al., 8 Apr 2026).

For each problem, every model is run 100 independent times. Pass rates are computed using the unbiased statistical estimator introduced by Chen et al. (2021). The paper defines the estimator in LaTeX as

$\mathbf F_2$6

where $\mathbf F_2$7 is the total number of samples ($\mathbf F_2$8), and $\mathbf F_2$9 is the number of correct samples among those. Intuitively, this estimates the probability that at least one of $K$0 sampled attempts would succeed, given the observed success count $K$1 in $K$2 attempts.

Scores are reported across all 25 problems as pass@1, that is, the single-attempt success rate estimated from the 100-run pool. The paper does not introduce additional variance estimators or confidence interval formulas, and none are reported. It also does not report per-domain breakdowns.

5. Empirical results and characteristic failure modes

Under this protocol, all frontier models currently score below 10\% on RMBench, even with tool use and search (Garre et al., 8 Apr 2026).

Model	pass@1
Gemini 3.1 Pro (Google)	6%
Claude Opus 4.6 (Anthropic)	6%
Gemini 3 Pro (Google)	4%
Kimi K2.5 (Moonshot AI)	4%
DeepSeek V3.2 (DeepSeek)	3%
GPT 5.2 (OpenAI)	2%
Claude Opus 4.5 (Anthropic)	2%

The paper emphasizes the contrast with the same generation of systems on competition mathematics, where near-perfect AIME accuracy and gold-medal-level IMO performance have been achieved. The drop from $K$3 on AIME to $K$4 on RMBench is presented as evidence of a qualitative leap from “insightful trick” problems to research-level reasoning.

The reported qualitative failure mode is not simple arithmetic error or shallow symbolic failure. In a representative example, when confronted with specialized theory involving Hahn series and multibasic $K$5-modules, the model substituted a superficially related but inapplicable framework (“generalized scales”), fabricated a non-existent theorem (“M. Getz, Theorem 4.14”), and produced a numerically huge but incorrect answer. The paper states that this pattern—plausible-sounding but fundamentally misgrounded chains of reasoning—recurs across RMBench. The recurring issue is therefore a failure of grounding in unfamiliar machinery, together with misapplication or fabrication of theoretical infrastructure.

This failure profile is significant because it differs from the dominant narrative around mathematical benchmark progress. The benchmark’s results suggest that success on elementary or competition-style mathematics does not straightforwardly transfer to environments where the main bottleneck is choosing the correct advanced framework and maintaining validity over many dependent inferential steps.

6. Significance, accessibility, and limitations

RMBench’s stated contributions are fourfold: a private, research-level mathematical benchmark of 25 problems that probes deep theoretical reasoning beyond olympiad frontiers; a rigorous double-blind, from-scratch expert verification protocol for each problem, plus programmatic verifiers for solution equivalence; unconstrained agent evaluation with tool use and search, and pass rates computed via the unbiased estimator over 100 runs per problem; and a contamination-resistant design through full privacy to ensure measurements reflect genuine capability rather than memorization (Garre et al., 8 Apr 2026).

The principal implication drawn in the paper is that current frontier models remain far from reliable autonomous research mathematicians. Below-10\% performance points to major gaps in theory acquisition, framework selection, and long-horizon reasoning. A near-term path forward is AI-assisted research: targeted tool use by human mathematicians who verify outputs, rather than fully autonomous agents. Progress on RMBench is presented as a critical milestone toward systems capable of contributing to open research.

The benchmark is not publicly released; it is available only via a controlled evaluation service to prevent leakage and contamination. Evaluations use the model providers’ APIs and provide access to a Python interpreter and search in an unconstrained agent setup. Licensing or broader access details are not specified beyond the private service.

RMBench also has explicit limitations. It focuses on problems with known, verifiable answers, and the authors describe this as a necessary intermediate evaluation rather than the endpoint of “moonshot mathematics.” The privacy of the benchmark protects uncontaminated measurement but limits community access. The paper does not report per-domain breakdowns, confidence intervals, or variance estimators, and does not specify plans for expanding the benchmark. Methodological caveats are addressed chiefly by the double-blind verification protocol and the decision to keep the benchmark fully private.