HorizonMath Benchmark

• HorizonMath Benchmark is a robust open-source suite that assesses AI's capacity for genuine mathematical discovery through unsolved problems across eight math subfields.
• It leverages a generator–verifier gap to demand inventive constructs while ensuring solutions can be efficiently and deterministically validated.
• The framework employs automated compliance checking, precise performance metrics, and state-of-the-art model evaluations to advance scalable AI-driven math research.

HorizonMath is an open-source benchmark and automated evaluation framework designed to rigorously measure AI progress toward genuine mathematical discovery. It consists of over 100 predominantly unsolved problems spanning eight areas within computational and applied mathematics, specifically targeting challenges where mathematical insight is necessary for discovery, yet computational solution verification is efficient and deterministic. HorizonMath seeks to fill the gap left by existing benchmarks—most of which either focus on problems with known answers or require human-in-the-loop evaluation—by providing a scalable, contamination-proof platform for assessing the capability of LLMs and related systems to produce novel results in mathematical research (Wang et al., 16 Mar 2026).

1. Motivation and Foundational Goals

HorizonMath was motivated by the observation that frontier LLMs can solve standard mathematics benchmarks, such as GSM8K and MATH, at near-100% accuracy. However, these benchmarks are limited to problems with known answers, failing to distinguish memorization from genuine mathematical discovery. Prior research agents (e.g., FunSearch, AlphaEvolve, Aletheia) have made authentic progress on specific open problems, but no large-scale, public, reproducible benchmark suite of unsolved problems existed with the capacity for automated solution checking. HorizonMath addresses this by:

• Focusing exclusively on predominantly unsolved problems originating in current research; any correct solution would therefore constitute a novel mathematical contribution.
• Selecting problem classes with a substantial "generator–verifier gap": these are problems where inventive mathematical construction is hard, yet solutions (once obtained) are easy to check.
• Providing a community resource encompassing more than 100 such problems with automated verification scripts, thereby eliminating the risk of data contamination and enabling reproducible, scalable evaluation.

The aim is to produce a benchmark immune to training data leakage, capable of quantifying substantive AI-driven advances in mathematical research without relying on formal proof assistants or labor-intensive human judgment.

2. Benchmark Suite Construction

HorizonMath comprises 101 problems selected from eight research-active mathematical subfields:

• Analysis (special-function integrals, lattice sums)
• Mathematical physics (power-series constants, spectral values)
• Geometry (packing and Kakeya-type constructions)
• Number theory (constant discovery, zeta-function identities)
• Combinatorics (difference sets, Ramsey-type bounds)
• Coding theory (optimal codes, threshold constants)
• Algebra (Hadamard matrices, group-theoretic existence)
• Optimization (extremal graph parameters, design-theory scopes)

Problem selection follows a two-stage filtering protocol:

1. Automatable Evaluation: The problem requires a solution that is a concrete mathematical object (number, symbolic expression, finite combinatorial construction) and can be deterministically checked by a computer.
2. Meaningful Discovery: The problem resists brute-force or algorithmic solution and is rooted in contemporary research rather than being artifically manufactured.

No problem in the benchmark admits a known closed-form or proven-optimal solution. As a result, any correct answer generated for a HorizonMath problem is, by construction, a novel achievement.

3. Automated Verification and Compliance Infrastructure

The automated evaluation pipeline consists of the following core components:

• Model Output Format: Each model submission must produce a complete, self-contained Python function capable of returning the proposed solution, e.g.,

def proposed_solution(): from mpmath import mp mp.dps = 100 # ...construct closed form from mp.pi, mp.gamma, mp.hyper, etc. return expression

• Compliance Checking: A dedicated LLM-based compliance checker reviews each submission, rigorously rejecting solutions containing unevaluated integrals, infinite series, root-finding, or invocations of external solvers. Only those composed of a finite combination of allowed constants and special functions (e.g., rational numbers, elementary functions, π, e, Euler’s γ, Catalan’s G, and mpmath-accessible special functions) pass to evaluation.
• Evaluation Modes: Each problem is assigned one of three fully automated evaluation modes:
  • ground_truth_computable: For closed-form discovery, numerical high-precision evaluation (using mpmath) of the expression is compared with a high-precision reference value. Acceptance requires matching at a minimum of 20 decimal digits.
  • benchmark_best_known: For optimization problems, a problem-specific validator checks whether the submission strictly improves upon the published best-known metric.
  • new_construction: For existence questions, an evaluator checks all algebraic or combinatorial properties exactly, yielding pass/fail.

Pseudocode for the evaluation loop is given explicitly in the dataset and is strictly adhered to in execution.

4. Performance Metrics and Reporting Protocol

Multiple metrics are used to quantify model performance on HorizonMath:

• Overall accuracy: $$\mathrm{Acc} = \frac{|\{\text{problems passed}\}|}{N}$$, where $N = 101$.
• Tier-wise accuracy: Focused reporting is done for the 10 "solvability 0" calibration problems (with known solutions), providing $\mathrm{Acc}_0 = \frac{|\text{passed solv0}|}{10}$.
• Novel improvement count: The number of unsolved problems (tiers 1–3) for which the model achieves a result strictly better than the published baseline.
• Relative improvement: For optimization problems, the fractional gain over the best-known result is tracked per problem via $$\Delta_{\mathrm{rel}} = \frac{m_{\mathrm{best}} - m(\mathrm{model})}{m_{\mathrm{best}}}$$.

These metrics allow for both raw capability measurement and nuanced tracking of research-level progress.

5. Empirical Results with State-of-the-Art Models

Three advanced LLMs—Claude Opus 4.6, Gemini 3.1 Pro, and GPT 5.4 Pro—were evaluated on the full HorizonMath suite under "high-effort reasoning" configurations. Experimental findings are as follows:

• Calibration tasks: On the 10 solvability 0 problems, GPT 5.4 Pro achieved 5/10, Gemini 3.1 Pro and Claude 4.6 each scored 3/10.
• Unsolved research problems: On the 91 benchmark tasks with no known solution, all models were effectively at 0% accuracy, with the exception of GPT 5.4 Pro, which produced 2 novel improvements.
• Aggregate accuracy: Out of all 101 problems, scores were:
  • GPT 5.4 Pro: 7/101 (6.9%)
  • Gemini 3.1 Pro: 3/101 (3.0%)
  • Claude Opus 4.6: 3/101 (3.0%)
• Novel research contributions: Only GPT 5.4 Pro produced strict improvements over literature baselines.

Model	Full Suite (101)	Solv0 (10)	New Optima
GPT 5.4 Pro	7 / 101 (6.9%)	5 / 10	2
Gemini 3.1 Pro	3 / 101 (3.0%)	3 / 10	0
Claude Opus 4.6	3 / 101 (3.0%)	3 / 10	0

Detailed Descriptions of Novel Improvements:

• Thin-Triangle Kakeya (128 slopes): GPT 5.4 Pro identified an intercept clustering solution yielding a union area $A_{\mathrm{model}}\approx 0.1091479892$ versus the previous best $A_{\mathrm{baseline}}\approx 0.1148103258$ ($\Delta_{\mathrm{rel}}$ ≈ 4.93% area reduction), validated via exact piecewise-linear integration.
• Diagonal Ramsey Number Bound: A quintic correction for the exponent base in the Ramsey bound reduced $c$ from approximately 3.7992 to $c_{\mathrm{model}}\approx 3.6961$, passing all formalized numeric checks and exhibiting a $\Delta_{\mathrm{rel}}$ ≈ 2.71% reduction over prior art.

Each improvement passes automated evaluation and verification, qualifying as a candidate for a genuine mathematical advance pending human expert review.

6. Strengths, Limitations, and Research Trajectory

Advantages:

• Contamination-proof: The exclusion of any problems with known solutions ensures that models cannot exploit data leakage, validating claims of true mathematical innovation.
• Scalability and Automation: Automated compliance checking and validation eliminate the need for manual grading or reliance on formal proof assistants, enabling large-scale, reproducible experimentation.
• Openness: The resource is extensible; the community can add new problems and potentially incorporate proof-based tasks.

Current limitations:

• Numerical confirmation: Agreement at 20+ digits, while strong empirical evidence, does not constitute a formal mathematical proof.
• Checker brittleness: The compliance checker may not detect every subtle violation in problem constraints or disallowed construction.
• Scope: At present, only three evaluation classes are supported, and proof-generation is not yet directly addressed.

Planned developments:

1. Proof-oriented extension: Integration with proof assistants (e.g., Lean) for problems necessitating formal verification.
2. Expression flexibility: Support for simplified closed forms that advance mathematical understanding even if they fail strict admissibility criteria.
3. Dynamic curation: Expansion into further research domains such as random matrix theory and analytic number theory as the community participates.

7. Impact and Position within the Benchmarking Landscape

HorizonMath represents a new paradigm among mathematical AI benchmarks, focusing exclusively on unsolved research problems with robust, automated evaluation. It stands in contrast to standard math benchmarks characterized by data contamination risks, and to benchmarks reliant on either formal proof verification or human review, both of which lack scalability. By providing a neutral signal of AI-driven progress on authentic mathematical challenges, HorizonMath facilitates empirical assessment of model discovery capabilities and offers a standardized platform for AI-driven mathematical research advancement (Wang et al., 16 Mar 2026). A plausible implication is that the HorizonMath framework could serve as a template for future benchmarks in domains where the generator–verifier gap enables scalable measurement of creative problem-solving.