FrontierMath: Benchmarking Statistical & AI Frontiers

• FrontierMath is a dual-domain initiative that benchmarks advanced nonparametric frontier estimation and AI mathematical reasoning.
• It employs kernel-based linear programming and extreme-value techniques to recover boundary functions with proven sparsity and convergence properties.
• The benchmark rigorously tests large language models with expert-curated, research-level math problems and automated solution verification protocols.

FrontierMath measures two distinct but convergent mathematical research frontiers: (1) advanced nonparametric and extreme-value estimation of boundary functions (frontiers) in statistics and econometrics; and (2) research-level mathematical reasoning benchmarks designed to quantify and stress-test the highest capabilities of LLMs and agentic artificial intelligence in mathematics. In both domains, “FrontierMath” refers to mathematically rigorous frameworks and datasets for benchmarking the edge of methodological or algorithmic prowess, ensuring strict verifiability, and catalyzing progress at the hardest unsolved problems.

1. Statistical and Nonparametric Frontier Estimation

Frontier estimation addresses the recovery of an unknown boundary function $f$ that constrains an observed sample, formalized as recovering the upper envelope of points $(X_i, Y_i)$ drawn from $$S = \{ (x, y): 0 \leq x \leq 1,\, 0 \leq y \leq f(x) \}$$, with $f$ assumed Lipschitz and bounded away from zero. The estimator $\hat{f}_N$ must satisfy the covering constraint ($\hat{f}_N(X_i) \geq Y_i$, $\forall i$) and minimize the Lebesgue “surface” $\int_0^1 \hat{f}_N(x) dx$. The kernel-based linear programming formulation (Bouchard et al., 2011) casts $\hat{f}_N$ in a finite-dimensional span of nonnegative kernel translates: $\hat{f}_N(x) = \sum_{i=1}^N \alpha_i K_h(x - X_i),$ with bandwidth parameter $(X_i, Y_i)$0 and nonnegative coefficients $(X_i, Y_i)$1 solved by the LP: $(X_i, Y_i)$2 The solution is typically sparse; only a minority of $(X_i, Y_i)$3 are nonzero—these “support vectors” directly determine the estimator, mirroring sparsity in SVMs. The estimator achieves almost-sure $(X_i, Y_i)$4 consistency with explicit rates, attaining (for optimally tuned $(X_i, Y_i)$5) convergence $(X_i, Y_i)$6 a.s. (Bouchard et al., 2011).

Relatedly, extreme-value theory approaches estimate monotone frontiers by leveraging the asymptotics of sample maxima (free disposability hull [FDH]). The FDH estimator for multivariate $(X_i, Y_i)$7 samples seeks the lowest step-function covering all points. Under suitable regular variation (domain-of-attraction) and smoothness, its limit distribution is Weibull-type, with convergence rate $(X_i, Y_i)$8, and robust asymptotic Gaussian estimators can be constructed via Pickands- and moment-type methods yielding confidence bands for the frontier (Daouia et al., 2010).

Finite-sample studies show that kernel-LP estimators achieve superior sparsity and error stability on moderate $(X_i, Y_i)$9; EVT-based frontiers provide robust inference and are stable to outliers, particularly when combined with careful bandwidth and threshold selection in applications such as efficiency analysis in economics or outlier detection in high-dimensional data (Bouchard et al., 2011, Daouia et al., 2010).

2. The FrontierMath Benchmark: Definition and Rationale

The “FrontierMath” benchmark (Glazer et al., 2024) is a large-scale, expert-curated testbed consisting of hundreds of unpublished, research-level mathematics problems spanning the 2020 MSC. FrontierMath is motivated by the near-saturation of existing benchmarks (MATH, GSM8K, AIME) for LLMs and the need for evaluation sets that remain beyond the feasible reach of current AI systems.

Problems are sourced and reviewed by mathematical experts (Fields Medalists, IMO gold medalists, research faculty) and are “guessproof”: answers are unique, typically large integers or symbolic expressions, and each is verifiable by automated code (usually SymPy scripts) that accept only correctly derived solutions. The coverage approaches 70% of the MSC subject hierarchy, with largest representation in number theory, combinatorics, algebraic geometry, group theory, and analysis.

Difficulty is multifactorial: each problem is tagged with background level (1–5, high-school through research), estimated “creativity” (hours to find key ideas), and “execution” (hours to carry out complete proof or calculation), with the median problem demanding expert-level effort. Problem statements, answers, automated verifiers, and tagging metadata undergo strict peer review—including anti-contamination/anti-plagiarism protocols and controlled review channels—to ensure integrity.

Sample problems require advanced techniques and often demand assembling non-elementary arguments (e.g., applying Chebotarev density, Galois cohomology, or delicate analytic number theory lemmas) and multi-stage computational pipelines (Glazer et al., 2024).

3. Evaluation Protocols and Observed Capabilities

FrontierMath employs automated validation: the model submits code (usually Python) to produce a numeric or symbolic object as answer, which is then checked by a problem-specific verification function $$S = \{ (x, y): 0 \leq x \leq 1,\, 0 \leq y \leq f(x) \}$$0. For numerical responses, verification is exact equality; for symbolic, algebraic simplification; for combinatorial structures, property-specific scripts.

First public evaluations used a 10,000-token cap per submission and allowed exploratory code synthesis. State-of-the-art LLMs (OpenAI, Anthropic, Google) in 2024 solved under 2% of problems. In follow-up, hierarchical agentic systems (notably the AI co-mathematician) achieved 48% on Tier 4 (48 out of 50 research-level problems), utilizing asynchronous multi-agent workstreams, automatic review cycles, and persistent “negative space” management (Zheng et al., 7 May 2026). Table 1 summarizes model progress over time:

System	Year	Tier 4 %	Strategy Highlights
GPT-4o, Gemini 1.5	2024	<2	Chain-of-thought + code
Gemini-3.1 Pro	2026	19	Chain-of-thought, strict token budget
AI Co-Mathematician	2026	48	Multi-agent, review cycles, async search

The gap between top models and expert baseline remains significant, and no system is saturated: newer models show gains mostly by unlocking harder tasks and increasing reliability rather than by using fewer tokens per solution (McFadyen et al., 16 Jun 2026).

4. Protocol Sensitivity, Scaling, and Limitations

FrontierMath accuracy is highly sensitive to inference compute—the total token budget and the opportunity for repeated submissions or wider exploration. Increasing token caps from 1M to 10M raises mean success by ≈12 points; nearly all reachable accuracy is unlocked above the 1M threshold. Iterative resubmission (serial depth) yields additional but smaller gains, and substantial parallel width (multiple independent runs) shows only marginal further improvement (Δ_parallel=+0.028 at 10M tokens) (McFadyen et al., 16 Jun 2026). Later model generations require deeper serial scaling to reveal their full capability.

Formal reporting under any fixed compute regime risks underestimating the underlying capability of advanced models. Benchmark documentation for new LLMs should plot performance as a function of compute ($$S = \{ (x, y): 0 \leq x \leq 1,\, 0 \leq y \leq f(x) \}$$1), specify iteration and feedback protocols, and advocate standardized trajectories as per (McFadyen et al., 16 Jun 2026).

Limitations are not just technical. Even with powerful agents, successful performance may hinge on, for example, the degree of mathematical or computational tool integration (Python, SymPy), the system’s orchestration mechanisms, and resistance to specification gaming or partial solutions—underscoring the need for robust standardization.

5. Influence and Successor Benchmarks

FrontierMath’s design catalyzed a second generation of research-level evaluation suites, most notably Soohak (439 problems authored by 68 mathematicians) (Son et al., 9 May 2026). Soohak advances the field along three axes: (1) breadth (440+ new problems), (2) data contamination resistance (fresh authoring, strict review), and (3) refusal evaluation—a capability requiring models to withhold answers on ill-posed, ambiguous, or non-unique prompts.

On the Soohak Challenge subset, leading models plateau at ≈30% (Gemini-3-Pro) with large headroom; on refusal, no model reliably exceeds ≈50%. By comparison, on previous research-level benchmarks (e.g., Riemann-Bench), most problems have been “unlocked” by leading models, but Soohak retains unsolved items across nearly every represented field.

The impact is methodological as well as empirical: future benchmarks now emphasize automated verification (moving beyond integer outputs to symbolic, proof, or program-level validation), robust “refusal” subsets, and stateful management of solution audit trails and failure cases.

6. Connections to AI-Driven Mathematical Discovery

The FrontierMath paradigm anchors the emerging science-of-mathematical-AI. Research in agentic systems—Aletheia (Feng et al., 10 Feb 2026), the AI co-mathematician (Zheng et al., 7 May 2026), and structural hypergraph approaches (Barkeshli et al., 7 Apr 2026)—emphasizes agent architectures capable of multi-agent orchestration, review/revise proof loops, robust tool integration, and autonomous research-level conjecturing and proof. These systems encode and advance “autonomy” and “novelty” levels (e.g., Level A autonomy: the agent produces new publishable results end to end), and explicitly target metrics such as reliability, creative insight time, and resilience to failure.

FrontierMath problems serve as community reference points for evaluating scientific AI at the edge of current mathematical practice, shaping both technical (tooling, validation) and strategic (what constitutes progress) directions for the field.

7. Outlook and Research Directions

FrontierMath has catalyzed research in both mathematical statistics (optimal nonparametric estimation, sparsity guarantees, extremal theory) and AI mathematics (benchmarking, agent design, protocol sensitivity). Key directions include:

• Expansion of benchmark coverage to broader mathematical domains, more nuanced verification formats (including formal proofs and structured outputs), and explicit evaluation of “refusal” and calibration capabilities.
• Integration of advanced retrieval and formal proof-checking tools into AI agents, combining natural-language reasoning with rigorous symbolic backends.
• Methodological development for benchmarking that plots full compute-to-performance curves, tracks stateful audit trails, and measures collaborative efficacy and provenance metadata in AI-human workflows.
• Statistical advances in adaptive bandwidth and threshold choice for frontier estimators, extension to high-dimensional and multivariate settings, and minimax rate analyses under additional regularity or adversarial noise assumptions.

In sum, FrontierMath designates both a class of mathematical estimation problems at distributional extremes and the canonical testbed for measuring and advancing AI systems at the frontiers of mathematical reasoning, with methodologies and tools now deeply shaping both domains (Glazer et al., 2024, Bouchard et al., 2011, Daouia et al., 2010, Feng et al., 10 Feb 2026, Barkeshli et al., 7 Apr 2026, Zheng et al., 7 May 2026, Son et al., 9 May 2026, McFadyen et al., 16 Jun 2026).