HorizonMath: AI-Driven Math Discovery Benchmark

• HorizonMath is a machine-verified platform for evaluating AI progress on unsolved, research-level mathematical problems across diverse domains.
• It uses objective, deterministic verification techniques such as high-precision numerical matching and property checks to ensure genuine solution breakthroughs.
• Empirical results demonstrate selective AI successes, with models improving benchmarks like the Thin-Triangle Kakeya area and asymptotic Ramsey constants.

HorizonMath is a machine-verified benchmark for AI-driven mathematical discovery focused on unsolved research-level problems spanning eight domains of computational and applied mathematics. Unlike known-answer benchmarks, HorizonMath is designed to detect genuine progress on open questions, providing a contamination-immune platform for evaluating LLMs’ ability to contribute novel mathematical insights (Wang et al., 16 Mar 2026).

1. Benchmark Motivation and Problem Selection Principles

The overarching objective of HorizonMath is to quantify AI progress on open mathematical problems—those whose solutions are not present in textbooks, pre-existing datasets, or the training data of contemporary LLMs. This distinguishes HorizonMath from benchmarks such as MATH or GSM8K, which measure retrieval, procedural recall, or familiar problem solving where data contamination is a critical concern. HorizonMath targets the generator-verifier gap in mathematics: the discovery phase is insight-intensive and currently unsolved, but automated verification for potential solutions is algorithmically efficient and deterministic.

Problem selection is governed by four principles:

• Explicit output requirement: Each problem requires a canonical answer (number, closed-form expression, discrete construction, or optimization parameter), not a narrative or proof sketch.
• Objective and deterministic verification: Verification can be performed unambiguously by a mechanical checker, eliminating human grading subjectivity.
• Resistance to routine algorithms: Problems are filtered to exclude those tractable by off-the-shelf symbolic or numerical routines.
• Active research relevance: Each item is drawn from recent literature or open repositories, avoiding contrived or trivial instances.

2. Scope: Domains and Problem Typology

The current HorizonMath release comprises 101 problems across eight thematic domains:

Domain	Representative Problems	Output Types
Analysis & special functions	High moments/integrals, e.g., ∫₀^∞ Ai(x)^5 dx	Constants, functions
Mathematical physics	Lattice sums, statistical mechanics integrals	Expressions, constants
Geometry	Packing/covering, Kakeya computations	Explicit constructions
Number theory	Zeta/Stieltjes constants at rationals	Closed forms, constants
Combinatorics	Ramsey numbers, difference sets	Optimized parameters
Coding/information theory	Optimal code/existence constructions	Constructions, parameters
Algebraic constructions	Hadamard matrices, MOLS	Matrices, combinatorial
Optimization constants	Tao–Ivanisvili–Davis bounds	Real-valued parameters

Problems are indexed by output form and a "solvability tier" (0 is reserved for calibration with known closed forms, 1–3 are progressively more difficult open research problems). Each instance specifies a precise answer form, permitted primitives, and a reference for baseline comparison.

3. Automatic Verification Framework

The HorizonMath evaluation pipeline consists of three stages for each submission:

• Compliance checking: The proposed Python function is filtered for banned primitives (unevaluated integrals, non-standard constants, PSLQ, numerical root-finding, infinite series) via an LLM-assisted compliance filter.
• Task-specific verification: Solutions are routed to domain-specific verifiers according to the answer type:
  • ground_truth_computable: Solutions must numerically match reference values to at least D=20 decimal digits, using mpmath evaluation at 100-digit precision.
  • benchmark_best_known: For optimization/bound problems, the submitted value must empirically improve on the best published baseline, with improvement Δ defined as (M_baseline–M_prop)/M_baseline.
  • new_construction: For existence tasks (e.g., explicit constructions), deterministic property checks (orthogonality, unimodularity for matrices, interval arithmetic for Ramsey certificates) determine pass/fail status.
• Aggregation and scoring: Results are reported by absolute pass rate on calibration items (accuracy), and by summed improvement Δ on optimization problems, both stratified by domain and tier.

Verification Mode	Acceptance Criterion	Main Use Cases
ground_truth_computable		v_prop–v_ref
benchmark_best_known	M(prop) < M_baseline	Bounds/improvements on real parameters
new_construction	Deterministic property check	Existence-style combinatorial constructs

The framework enforces that only numerically verifiable or constructively testable solutions are accepted—no free-form reasoning or uncheckable code is permitted.

4. Contamination Immunity: Guarantee of Novelty

HorizonMath eliminates the risk of data contamination by exclusively selecting items whose correct solution is not available in the literature nor in any known AI training corpus. Since "ground-truth" is either a high-precision numerical reference or a currently best-known bound from the literature, no submission can be validated by mere memorization; a "pass" signals new symbolic computation or intelligent search rather than training set artifact reproduction (Wang et al., 16 Mar 2026).

5. Empirical Evaluation and Model Results

Experimental evaluation with three state-of-the-art models—Claude Opus 4.6, Gemini 3.1 Pro, and GPT 5.4 Pro—was performed using elevated reasoning effort settings:

• On solvability-0 calibration (10 known closed forms), GPT 5.4 Pro achieved 5/10, Gemini 3, Opus 3.
• On 91 open problems (tiers 1–3), only GPT 5.4 Pro achieved any nonzero pass rate (7%).
• GPT 5.4 Pro produced two deterministically verified results that improved on published best-known values:
  1. Thin-Triangle Kakeya (128 slopes): area reduced from 0.1148103… (baseline) to 0.1091480… (Δ ≈ 4.93%).
  2. Asymptotic diagonal Ramsey constant: bound improved from c = 3.7992027… to c = 3.6960839… (Δ ≈ 2.71%).

Both were accepted on the basis of deterministic verification—piecewise-linear integration (area) and interval arithmetic on Ramsey certificates—indicating potential novel mathematical advances, subject to further expert assessment.

6. Limitations and Future Developments

Several limitations are recognized:

• Numerical verification is not proof: High-precision numerical matches (e.g., 20 digits) provide strong evidence but do not constitute formal proof of closed-form identities. Human mathematician review remains necessary for full validation.

• Compliance filtering risk: The LLM-based compliance checker can occasionally allow nonconforming code—future iterations may use formal proof assistants (Lean, Isabelle) for enhanced rigor.
• Scope of admissible operations: The current framework prohibits nested radicals and various symbolic manipulations, which may restrict the solution space unduly. Expanding permitted primitives and integrating bounding of approximate solutions is under consideration.
• Benchmark extensibility: As new open problems and improved baselines emerge, HorizonMath is designed to expand and adapt, supported by its open-source infrastructure and community contribution mechanisms (Wang et al., 16 Mar 2026).

7. Significance and Research Context

HorizonMath represents a methodological advance in AI evaluation by replacing memorization-based or formal-proof-centric paradigms with machine-verifiable objective discovery on unsolved problems. Its architecture is explicitly tailored to detect true research-level progress by LLMs. Initial results suggest the advent of AI systems capable of mathematically significant discovery, though the distinction between empirical computational evidence and human-validated proof remains critical. HorizonMath stands as an open, extensible challenge platform for measuring and fostering AI’s mathematical research capabilities.