Papers
Topics
Authors
Recent
Search
2000 character limit reached

Benchmarks in Leipzig

Published 4 Jun 2026 in math.HO, cs.AI, math.AG, math.CO, and math.RT | (2606.05818v1)

Abstract: Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the 3-day workshop Benchmarks in Leipzig with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100 questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive.

Summary

  • The paper presents a benchmark with 100 research-level math problems developed and audited by 49 mathematicians.
  • The multi-phase evaluation reveals high performance variability among LLMs and highlights issues of model non-determinism in technical reasoning.
  • Findings emphasize the need for next-generation benchmarks focused on proof generation and multi-step reasoning rather than traditional exercise-style questions.

Research-Level Mathematical Evaluation: The Leipzig Benchmark

Overview and Motivation

The "Benchmarks in Leipzig" paper (2606.05818) presents a systematically curated dataset for evaluating the mathematical reasoning capabilities of state-of-the-art LLMs at the level of contemporary research mathematics. The benchmark consists of 100 problems, each with a unique and unguessable answer, spanning a broad range of mathematical subfields including Algebraic Geometry, Combinatorics, Representation Theory, Polytope Theory, and more. Questions were collectively developed and audited by 49 mathematicians, emphasizing rigorous submission guidelines to avoid trivial computation, guessable answers, and dependency on unpublished results.

The central aim is to probe the limits and progress of LLMs in problems that extend well beyond undergraduate or classroom-level mathematics benchmarks, instead requiring familiarity with recent literature, technical definitions, and abstract reasoning.

Benchmark Construction and Guidelines

The benchmark was constructed via a three-phase process:

  • Submission Phase: Contributors proposed questions through the ScienceBench platform, ensuring uniqueness and substantial difficulty. Each question required a worked-out solution and references, with the interface supporting collaborative peer-review and audit.
  • Automated Filtering: Before full acceptance, questions underwent trial solution attempts by five advanced LLMs (GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.7, DeepSeek-V4-Pro, Grok 4.3). Only questions that could be solved by three or fewer models were admitted, ensuring non-triviality.
  • Auditing: Community review and LLM-assisted review flagged errors and ambiguities, resulting in substantial iterative refinement and elimination of defective questions. The process illuminated the value of LLMs in supporting error-detection and peer-review within the mathematical community.

The accepted questions cover a distribution of topics, with Algebraic Geometry and Algebraic Combinatorics being particularly represented.

Multi-Step LLM Evaluation Protocol

Evaluation was executed in three distinct stages:

  • Stage 1—Single-Run Attempts: Each of five LLMs attempted every question once. Only 59 questions were solved by at least one model; 41 questions remained unaddressed.
  • Stage 2—Multiple-Run Evaluation: Using Surge AI, three top models were each run 20 times per question to probe stochasticity and variance. GPT-5.5 achieved the highest coverage (solving 75/100), with dramatic variance: many questions were only solved in a minority of runs for weaker models. Notably, model performance was highly sensitive to sampling and prompt stochasticity, underscoring issues in model determinism for mathematical inference.
  • Stage 3—"Heavy Thinking" Models: Strongest available configurations (GPT-5.5 Pro Extended Thinking, Gemini 3.1 Pro Deep Think) attempted every question three times each. Here, GPT-5.5 Pro achieved 88/100 solved, Gemini 3.1 Pro Deep Think 56/100. At this most resource-intensive (and time-consuming) effort level, only two questions remained unsolved by all models.

Strong empirical result: In cumulative evaluation, LLMs—especially with enabled "extended thinking" or high computational budget—were able to solve all but two of the benchmark's 100 research-level questions. The initial filtering approach, favoring hardest questions for model selection, was nearly saturated with the latest models.

A pronounced finding is that the "exercise-style" benchmark model has become less informative for the strongest LLMs: previously discriminative tasks are now tractable for top-tier models, suggesting a saturation point for traditional question curation in this paradigm.

Statistical Observations and Model Behavior

The multi-run protocol exposes substantial non-determinism and volatility in LLM responses even on highly technical mathematical questions. For instance, Claude Opus 4.7 solved 44 questions in at least one run, but for 19 of these it produced a correct answer no more than three times out of 20. GPT-5.5 was significantly more consistent, with a substantial tail of questions solved in nearly every run.

There are also large inter-model differences. While GPT-5.5 solved 88/100 under extended thinking, rivals (Gemini 3.1 Pro, Claude Opus 4.7) lagged far behind both in coverage and consistency. This variance remains, despite all models being considered top of their respective product lines, and despite parity in prompt access.

In audit, AI-assisted cross-verification was instrumental: model-generated solutions led contributors to identify and correct flaws in their own problem statements and in referenced answers, further validating the dual use of LLMs as benchmarking targets and as peer reviewers.

Implications and Theoretical Significance

The empirical results indicate that state-of-the-art LLMs, especially under high resource allocations and extended context reasoning, now routinely solve research-level problems considered 'unguessable' and intractable to prior model generations. This reveals rapid advances in compositionality, technical memory, and mathematical abstraction within LLM architectures.

A striking claim of the paper is that the traditional practice of constructing exercise-based research benchmarks is now "reaching its limits" for top models: the hard questions that could not be solved by LLMs even under adversarial curation are now numerically negligible. For the evaluation of future model generations, novel benchmarks—potentially involving longer proofs, open conjecture synthesis, or adversarial proof mining—will become necessary.

Practically, model stochasticity and response diversity reinforce the requirement for multi-run, multi-judgment evaluation in mathematical LLM benchmarking, as single-run statistics systematically underestimate true model coverage.

The paper also demonstrates the appetite of research mathematicians to engage with LLMs as peer collaborators, as evidenced by substantial ScienceBench chat usage and iterative refinement based on model feedback.

Prospects for Future Mathematical AI Benchmarks

Given this saturation, several implications emerge:

  • More challenging benchmarks must consider currently open problems, proofs of novel results, or tasks that require multi-stage reasoning chains and proof-construction, not merely computation or answer synthesis.
  • There is an emerging argument for models to function as mathematical research assistants—supporting peer review, error detection, conjecture generation—rather than only as question solvers.
  • Cross-model comparisons should focus on aspects of solution explainability, proof generation, and verifiability, not just answer correctness.
  • Experimental protocols should systematically exploit and quantify response variance and model randomness, which are non-trivial in highly technical domains.
  • The relationship between formal verification tools and LLMs in mathematics research suggests a hybrid pathway for rigorous, computer-verified mathematics.

Conclusion

"Benchmarks in Leipzig" (2606.05818) establishes a new empirical benchmark for the mathematical reach of LLMs, demonstrating that current state-of-the-art models are able to solve nearly all research-level questions in a carefully curated, adversarial dataset. The observed performance gradient across models and runs, as well as the increasing utility of LLMs in mathematical peer review, highlights both major progress and the shifting frontier of AI evaluation in mathematics. For future work, benchmark design must adapt by focusing on fundamentally new problem types and incorporating AI as both solver and auditor in the mathematical research ecosystem.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

Explain it Like I'm 14

What this paper is about

This paper describes a big, carefully made “test” for advanced math AIs. Forty‑nine mathematicians created 100 tough, research‑level math questions that each have a single, exact answer (like a specific number, formula, or polynomial). The team then checked how well several top AI models could solve these questions, and they studied where the AIs did well, where they failed, and how consistent they were.

What the researchers wanted to find out

They set out to answer simple but important questions:

  • How good are today’s best AI models at solving hard, research‑style math problems with exact answers?
  • How much do results change if you let the same model try the same problem multiple times?
  • Does giving a model extra “thinking time” help?
  • Can AI help spot mistakes in math questions and answers (like a smart proofreading assistant)?
  • How should we design future math benchmarks now that the strongest models are getting very good?

How the study worked (in everyday language)

Think of this like building a tournament for AIs:

  • A benchmark is a standardized test. Here, it’s a set of 100 hard math questions with known answers.
  • A LLM is an AI that reads and writes text. Some can do advanced reasoning.

Here’s the process they used:

  • Building the questions
    • 49 mathematicians contributed. They used a platform called ScienceBench to write questions.
    • Each question had to have:
    • One unique, “unguessable” answer (so random guessing wouldn’t work).
    • Reasoning at its core (not just heavy number‑crunching).
    • Only publicly available math facts (not private or unpublished results).
    • During submission, AIs tried each question once. If more than three models got it right easily, the question was made harder.
  • Three rounds of AI testing
    • Stage 1: Single try per model
    • Five top models each tried all 100 questions once.
    • Stage 2: Multiple tries per model
    • Three models each tried every question 20 times. (This shows how much luck or randomness matters.)
    • Stage 3: “Heavy thinking”
    • Two models got extra “thinking time” and tried each question three times. (This is like giving them more scratch paper and time to reason.)
  • Tools and fairness
    • Web search, coding, and external computation tools were turned off during most runs to focus on the models’ own reasoning.
    • If a model crashed or timed out, they retried up to three times.
  • Review and cleanup
    • The researchers (and the AIs) helped catch mistakes in the submitted questions and answers. Some questions were corrected, and three were removed.

What they found and why it matters

Here are the main takeaways, explained simply:

  • Overall performance improved with more attempts and more thinking time
    • After Stage 1 (single try): 41 questions were unsolved by all models.
    • After Stage 2 (20 tries/model): only 16 remained unsolved.
    • After Stage 3 (extra thinking): just 2 questions were left unsolved by all models.
    • This shows the best AIs are getting very good at advanced math when given time or multiple chances.
  • Models differ a lot—and are sometimes inconsistent
    • In Stage 1 (single try):
    • GPT‑5.5 solved 44/100.
    • Gemini 3.1 Pro solved 15/100.
    • Claude Opus 4.7 solved 13/100.
    • DeepSeek‑V4‑Pro solved 10/100.
    • Grok 4.3 solved 6/100.
    • In Stage 2 (20 tries each):
    • GPT‑5.5 solved 75/100 at least once.
    • Gemini 3.1 Pro solved 40/100 at least once.
    • Claude Opus 4.7 solved 44/100 at least once.
    • Inconsistency example: Claude solved some questions only a few times out of 20, suggesting it can find the right path but doesn’t do so reliably.
  • Extra “thinking time” helps a lot
    • In Stage 3 (3 runs each, with extra reasoning time):
    • GPT‑5.5 Pro (Extended Thinking) solved 88/100 at least once and produced answers on all attempts.
    • Gemini 3.1 Pro Deep Think solved 56/100 at least once but sometimes didn’t produce any answer in a run.
  • The best models are pushing the limits of current benchmark styles
    • The paper suggests that exercise‑style questions (based on known, public math) may no longer be challenging enough for the very top models. New, more creative or proof‑heavy benchmarks might be needed to measure real progress.
  • AI was helpful as a reviewer
    • AI reviews flagged issues in 16 submissions, leading to corrections and removals. This hints that AIs could help proofread math content and reduce human errors.
  • Mathematicians used the platform and chat tools a lot
    • 39 contributors used the ScienceBench chat, totaling over a thousand messages. This suggests researchers will use AI tools if they trust them and they’re easy to access.

How this was organized (the human side)

  • Most questions were created during a 3‑day workshop at the Max Planck Institute in Leipzig, Germany.
  • Each day blended talks, small‑group work, and sharing results.
  • The 100 questions came from many fields (like algebraic geometry, combinatorics, number theory, and more), making the benchmark broad and realistic.

Why this work matters

  • Shows rapid progress: Today’s AIs can solve many advanced, exact‑answer math problems—especially with multiple tries or extra reasoning time.
  • Highlights evaluation challenges: Because AIs can be inconsistent, using just one try can underestimate their ability. Multi‑run testing gives a fairer picture.
  • Suggests next steps: To keep pushing the frontier, benchmarks will likely need more open‑ended tasks (like producing proofs) and problems that require deeper creativity.
  • Offers practical help: AI can assist mathematicians by checking for errors and suggesting corrections—like a very careful, tireless assistant.

Bottom line

The benchmark built in Leipzig shows that modern AIs are becoming surprisingly strong at solving tough math questions with precise answers. They’re not perfect—results vary by model and run—but with more tries or more thinking time, the best ones perform impressively. This progress means we need harder, smarter tests next and that AI can already be a helpful partner in real mathematical work.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a consolidated list of unresolved issues and concrete avenues for future work suggested by the paper’s scope, methodology, and results.

  • Representativeness of the benchmark: With only 100 questions and a heavy skew toward algebraic geometry and combinatorics, it is unclear how well results generalize across research mathematics. Action: provide balanced, stratified coverage and per-area stratified performance analyses.
  • Selection bias in question curation: The submission workflow filtered for questions solvable by at most three project-active models, potentially biasing difficulty and content. Action: quantify and mitigate selection effects (e.g., curate blind to model performance or include an unfiltered subset).
  • Small-sample statistical uncertainty: No confidence intervals, hypothesis tests, or power analyses are reported for inter-model differences. Action: add statistical uncertainty quantification and significance testing for “solved” and “correct-run” rates.
  • Lack of per-domain performance breakdown: The paper lists topic counts but does not analyze model performance by field, structure type (e.g., combinatorial vs algebraic), or answer type. Action: publish per-area, per-answer-type success rates and difficulty profiles.
  • Non-comparable evaluation settings across stages: Stage 3 gave GPT-5.5 reruns when it failed to answer but did not do the same for Gemini Deep Think; models were run via different interfaces (API vs ChatGPT website vs third-party). Action: standardize interfaces, rerun policies, and retry budgets.
  • Incomplete reporting of sampling controls: Temperatures, top-p, seeds, and other decoding parameters for Stage 2/3 evaluations are not reported; “effort” settings differ across models. Action: disclose and standardize decoding hyperparameters and seeds; release run configs.
  • Judge reliability and evaluation protocol: Answer checking relied on LLM judges without inter-judge agreement, calibration, or adversarial stress tests. Action: report judge accuracy, inter-annotator agreement, and failure cases; use canonicalization/verification pipelines (e.g., CAS checks, formal equivalence for polynomials/structures).
  • Answer normalization for structured math objects: The paper does not specify how equivalent but differently formatted objects (e.g., polynomials, matroid bases, braid words) were canonicalized. Action: define and release canonicalization routines and validators per task type.
  • Tool usage impact: Code/CAS/web tools were disabled due to timeouts and brute-force behavior, but this was not systematically evaluated. Action: design tool-augmented protocols with resource caps and smarter tool-use constraints; compare tool-on vs tool-off conditions.
  • Ground-truth integrity risks: Authors corrected several answers to match a model’s output; verification procedures are not detailed. Action: require independent human verification and/or formal certificates before retroactively changing ground truth.
  • Training contamination assessment: No audit for pretraining exposure or post-release contamination is reported. Action: perform data provenance checks and contamination audits (e.g., n-gram overlap, retrieval probes) vs model training corpora.
  • Failure mode taxonomy: The study reports aggregate success but not common error types (misinterpretations, algebraic slips, combinatorial counting errors, edge-case mishandling). Action: produce a fine-grained error analysis tied to problem features.
  • Hardness frontier characterization: Only two problems remained unsolved, but their features and what makes them hard are not analyzed. Action: characterize the unsolved set; use it to seed next-generation challenge design.
  • Robustness to prompt phrasing: No experiments vary problem wording, notation, or formatting to test stability. Action: introduce controlled paraphrases and notation perturbations to quantify robustness.
  • Guessability and baseline controls: The “unguessable” requirement is qualitative; no naive or heuristic baselines are measured. Action: include random/heuristic baselines and estimate chance-level success for each answer space.
  • Multi-run scoring interpretation: Reporting “solved if any run correct” conflates sample efficiency with reasoning quality; per-question success vs number of runs is not modeled. Action: model per-run success probabilities and sample-efficiency curves; report AUCR (area under correct-vs-runs).
  • Time/cost/compute budgets: No measurements of latency, token usage, or cost per correct solution are reported. Action: add resource-efficiency metrics and Pareto analyses (accuracy vs cost/time).
  • Reproducibility documentation: Full prompts, judge prompts, seeds, outputs, error logs, and rerun policies are not released. Action: open-source all evaluation artifacts with versioning; freeze benchmark revisions and provide checksums.
  • Per-question result transparency: The paper does not include per-question, per-model outcomes or confusion matrices (beyond coarse aggregates). Action: publish a detailed leaderboard with per-question correctness and answer traces.
  • Interface-induced variance: Mixing API and consumer UI runs introduces hidden differences (context lengths, throttling, hidden system prompts). Action: unify the evaluation stack or document interface-induced deltas via A/B runs.
  • Abstention vs failure analysis: “Answered” vs “correct” is reported, but causes of non-answers (timeouts, abstentions, crashes) are not decomposed. Action: categorize non-answers and quantify their prevalence per model and task type.
  • Next-benchmark design is underspecified: The paper notes that exercise-style questions hit limits for top models but proposes no concrete successor formats (e.g., multi-part proofs, formal verification, tool-constrained reasoning). Action: specify next-phase tasks, scoring rubrics, and verification protocols.
  • Human baselines: No comparison to human mathematicians on time-to-solve or accuracy for the same tasks. Action: include expert and advanced-student baselines to calibrate difficulty and practical utility.
  • Cross-benchmark calibration: No correlation analysis with FrontierMath, Soohak, RealMath, etc., to position Leipzig questions on a shared difficulty map. Action: run cross-benchmark evaluations and release joint analytics.
  • Long-context effects: Several problems may require lengthy context, but the impact of context length limits and truncation is not studied. Action: measure accuracy as a function of available context and summarize truncation rates.
  • Security and post-release drift: After public release, models may train on the benchmark, invalidating longitudinal comparisons. Action: institute hidden test sets and staged releases; monitor for data leakage over time.
  • Licensing and usage policy clarity: The paper does not specify licensing for questions/solutions or constraints on use for training. Action: publish dataset licenses, data-use policies, and a governance plan for updates.
  • Researcher adoption and efficacy: Anecdotal chat usage is reported without controlled measurements of productivity or insight gains. Action: conduct user studies on efficacy, trust, and workflow integration with reasoning models.
View Paper Prompt  View All Prompts 

Practical Applications

Practical Applications Derived from “Benchmarks in Leipzig: A collection of questions in research-level mathematics”

The paper introduces a curated, audited benchmark of 100 research-level math questions with unique, verifiable answers; a rigorous, multi-stage evaluation protocol (single-run, multi-run, “heavy-thinking” modes); and an end-to-end platform (ScienceBench) for contribution, auditing, model evaluation, and researcher-facing chat. The study surfaces actionable patterns: high cross-run variance, strong gains from heavy-thinking modes, and concrete value from AI-assisted review (flagging and fixing errors). Below are real-world applications grounded in these findings and methods.

Immediate Applications

  • Benchmark-driven model selection for math-heavy workloads Organizations can use the Leipzig Benchmark (or a domain-adapted fork) as a regression and acceptance test to choose models and settings that reliably solve domain-specific, exact-answer tasks. The paper’s multi-run metrics (“correct runs” vs “solved questions”) and run-variance distributions provide ready-made KPIs. Sectors: software/AI platforms, finance (quant research), engineering (optimization/design), education. Tools/Products/Workflows: CI pipelines that run benchmark suites per model/version; dashboards showing consistency, solve rates, and answerability. Assumptions/Dependencies: access to comparable model endpoints and “effort” settings; budget for multi-run evaluation.
  • AI-assisted mathematical content review (preprint/journal/teaching materials) The AI review loop flagged 16 issues, corrected multiple answers, and removed faulty items—demonstrating immediate value in catching mathematical inconsistencies and typos. Journals, preprint servers, and instructors can integrate an AI pass before human review. Sectors: academia, publishing, education. Tools/Products/Workflows: “math linting” services that check examples, computations, and answer uniqueness; author-side plugins for LaTeX. Assumptions/Dependencies: human-in-the-loop resolution; publicly accessible references for verifiability.
  • Reliability analytics for LLM deployments (multi-run and consistency scoring) The study shows pronounced cross-run variance and large gains from aggregation. Teams can adopt multi-run protocols, compute consistency scores, and mandate minimum per-item success probabilities for production use. Sectors: software/AI, finance, robotics (planning with numeric constraints). Tools/Products/Workflows: reliability dashboards; auto-rerun policies; majority/ensemble voting; thresholds on “answered” vs “solved”. Assumptions/Dependencies: compute budget; careful handling of non-answers/timeouts.
  • Trusted research chat for domain experts Many mathematicians used the project chat when given access to reasoning models in a trusted environment. Research groups can deploy internal, audited chat with disclosure controls and model comparison to support exploration, sanity checks, and worked examples. Sectors: academia, R&D labs, enterprise R&D. Tools/Products/Workflows: multi-model side-by-side chat; transcript retention with privacy guarantees; quick “convert-to-benchmark-item” for discovered subproblems. Assumptions/Dependencies: data governance; access to reasoning-capable models.
  • Exact-answer auto-grading and competitions The “unique, unguessable answer” format is directly usable for grading and contest judging (no rubric ambiguity). Integrate an LLM judge for format normalization and equality checks (as in the paper’s workflow), while keeping solutions hidden. Sectors: education, competitive programming, hiring/assessments. Tools/Products/Workflows: answer canonicalization, tolerance rules for symbolic equality, auto-feedback on inconsistencies. Assumptions/Dependencies: robust equivalence testing; well-designed “unguessable” items to prevent random success.
  • Dataset curation workflows and governance for STEM benchmarks The submission/audit pipeline—guidelines on difficulty, unguessability, public references; LLM pre-checks; community audit; AI-assisted post-collection review—can be adopted by benchmark maintainers to improve quality and reduce error rates. Sectors: AI evaluation, education-tech, research foundations. Tools/Products/Workflows: contributor dashboards; automated guideline checks; audit logs; post-release correction policies (“living benchmark”). Assumptions/Dependencies: community engagement; transparent versioning and scoring updates.
  • Procurement guidance for math-centric applications Given model disparities (e.g., GPT-5.5 vs peers) and sensitivity to “effort”/token budgets, buyers can align RFPs to Leipzig-style metrics and require vendors to report multi-run consistency, non-answer rates, and heavy-thinking improvements. Sectors: finance, energy, engineering consultancies, legal (quant damages), analytics vendors. Tools/Products/Workflows: standardized evaluation attachments to RFPs; bake-offs with Leipzig or domain variants. Assumptions/Dependencies: test-time compute disclosures; comparable API configurations.
  • Instructional design for advanced courses Use benchmark-style items (with known exact answers) to scaffold research-like exercises where students can check answers but must supply proofs/derivations. Pair with controlled use of reasoning LLMs for hints and error-spotting. Sectors: higher education, online learning. Tools/Products/Workflows: course banks of “unguessable” problems; staged release (question → student proof → AI checker of final answer). Assumptions/Dependencies: academic integrity policies; clear boundaries on AI use.
  • Competition/exam integrity enhancements The guidelines (prevent trivial guessing; avoid heavy reliance on brute-force code) can shape assessments that are robust to model exploitation, while still being machine-checkable after the fact. Sectors: education, certification bodies. Tools/Products/Workflows: item design checklists; automated “guessability” probes; bans or caps on code/tool use during evaluation modes. Assumptions/Dependencies: proctoring and environment controls; calibration of difficulty.
  • Internal IP/compliance control in AI-assisted research Requiring problems to rely on public research reduces leakage risks and ensures reproducibility. This can be an immediate policy for organizations using LLMs in mathematically sensitive contexts. Sectors: industry R&D, defense, pharma. Tools/Products/Workflows: preflight checks for citation availability; prompts that ban unpublished dependencies. Assumptions/Dependencies: access to literature; periodic audits to detect inadvertent reference to non-public material.

Long-Term Applications

  • Integrated AI peer reviewer for mathematics (from answers to proofs) Extend beyond exact-answer checks to draft-level theorem/proof audits that catch gaps, suggest counterexamples, and cross-link public references—becoming a standard component of journal and preprint workflows. Sectors: academia, publishing. Tools/Products/Workflows: proof graph analyzers; counterexample search; “explainable flag” reports. Assumptions/Dependencies: advances in proof verification, formal methods integration, and reduced hallucinations.
  • Cross-discipline research benchmarks with Leipzig-style governance Port the unique-answer, multi-run, heavy-thinking protocol to physics, chemistry, economics, and operations research where problems admit deterministic, verifiable outputs. Sectors: scientific research, industry R&D. Tools/Products/Workflows: domain-specific benchmark suites; standardized reporting of “answered vs solved vs consistent”. Assumptions/Dependencies: availability of public references and gold standards; domain expert curation.
  • Safety and reliability certification for reasoning models Regulators and standards bodies could require reporting of run-variance, heavy-thinking deltas, and non-answer rates for math-reliant use-cases (e.g., model-based risk or design). Sectors: policy/regulation, finance, energy, aerospace. Tools/Products/Workflows: conformance tests; third-party auditing (à la Surge AI) with reproducible settings. Assumptions/Dependencies: agreed-upon protocols; access to models under test; reproducibility across vendors.
  • Orchestrators that escalate effort adaptively Build systems that start with low-effort runs, escalate to heavy-thinking only when necessary, and decide when to rerun or ensemble—optimizing cost vs reliability for complex reasoning tasks. Sectors: software/AI infrastructure, analytics platforms. Tools/Products/Workflows: cost-aware policies; “answerability detectors”; hybrid judge+solver loops. Assumptions/Dependencies: reliable answerability and uncertainty signals; latency/compute budgets.
  • Training curricula for advanced reasoning Use Leipzig-style signals (hard, unguessable items; multi-run evaluation) as a training objective to improve models’ mathematical robustness and reduce variance. Sectors: AI model development. Tools/Products/Workflows: RL from human/AI audits; curriculum that emphasizes consistency and exactness. Assumptions/Dependencies: access to high-quality, auditable data; safeguards against benchmark overfitting.
  • Hybrid formal–informal verification pipelines Marry exact-answer checking with proof assistants (Lean, Coq) and CAS backends to certify not only that answers are correct but that derivations meet formal standards, enabling machine-checked research artifacts. Sectors: software verification, academia, critical engineering. Tools/Products/Workflows: auto-translation to formal languages; CAS–PA bridges; artifact repositories. Assumptions/Dependencies: advances in formalization coverage and automation; community standards for certified outputs.
  • Contamination detection and provenance tracking for benchmarks Build tools that estimate the likelihood a model has seen benchmark content (data leakage), maintain provenance of items, and adjust scoring accordingly—preserving benchmark validity as models scale. Sectors: AI evaluation, policy. Tools/Products/Workflows: data fingerprinting; canary items; public audit trails with versioned corrections. Assumptions/Dependencies: cooperation from model providers; red-team access.
  • Human–AI co-discovery environments for mathematics Expand ScienceBench-like platforms into collaborative systems where AI suggests nontrivial examples, stress-tests conjectures with exact-answer subproblems, and helps curate publishable, error-checked results. Sectors: academia, industrial research labs. Tools/Products/Workflows: multi-model ideation; “convert chat to benchmark/checklist”; structured conjecture testing with reversible logs. Assumptions/Dependencies: stronger reasoning models; community norms for attribution and reproducibility.
View Paper Prompt  View All Prompts 

Glossary

  • Artin generators: The standard generators σ_i of the braid group B_n used to present braids algebraically. Example: "the Artin generators of the pure braid group tij:=σiσi+1⋯σj−2σj−12σj−2−1⋯σi+1−1σi−1t_{ij} := \sigma_i \sigma_{i+1} \cdots \sigma_{j-2} \sigma_{j-1}^2 \sigma_{j-2}^{-1} \cdots \sigma_{i+1}^{-1} \sigma_i^{-1}"
  • Auslander algebra: For a finite-dimensional algebra A of finite representation type, the endomorphism algebra of a direct sum of representatives of all indecomposable A-modules; it controls the module category of A. Example: "Let BB be the Auslander algebra of AA."
  • Bergman fan: A polyhedral fan associated to a matroid capturing its tropical linear space; fundamental in tropical geometry of matroids. Example: "on the Bergman fan of its graphic matroid"
  • braid group: The group B_n of n-strand braids with composition given by concatenation; algebraically generated by σ1,…,σ{n−1} with Artin relations. Example: "the braid group on nn strands"
  • Chow class: The cycle class of an algebraic subvariety in the Chow ring, representing its equivalence class under rational equivalence. Example: "The Chow class of MM is defined as the class of the torus orbit closure (C∗)nx‾\overline{(\mathbb C^*)^n x} in the Chow ring of G(k,n)G(k,n)."
  • Chow ring: The graded ring of algebraic cycles modulo rational equivalence with intersection product; a cohomology-like invariant of varieties or polytopes. Example: "the symmetric part of the degree n−rn-r-part of the Chow ring of the permutohedron"
  • codominant dimension: A homological invariant dual to dominant dimension, measuring the initial length of a minimal injective coresolution consisting of projective-injective modules. Example: "codominant dimension at least 2"
  • coarse fan structure: A fan structure that identifies cones differing only by refinements, capturing a “minimal” combinatorial subdivision of a tropical or polyhedral fan. Example: "the coarse fan structure coincides with the minimal nested set structure"
  • coarse flag Hilbert-Poincaré series: A two-variable generating series encoding graded flag data (often for Orlik–Solomon algebras) in a coarsened manner for a hyperplane arrangement. Example: "the coarse flag Hilbert-Poincaré series of A\mathcal{A}"
  • delooping levels: A homological measure of how many iterations of (co)syzygy or loop functors are needed before a module attains a periodic/projective behavior in the stable category. Example: "the sum of all delooping levels of the indecomposable CC-modules"
  • Dyck path: A lattice path from (0,0) to (n,n) using up and right steps that never goes below the diagonal; in combinatorics, encodes Catalan structures. Example: "For a Dyck path dd of semilength nn"
  • Grassmannian: The algebraic variety G(k,n) parameterizing k-dimensional linear subspaces of an n-dimensional vector space. Example: "the complex Grassmannian G(k,n)G(k,n)"
  • homogeneous ideal: An ideal generated by homogeneous polynomials, stable under grading; defines a projective variety in graded contexts. Example: "Consider the following homogeneous ideal in C[z1,…,z5]\mathbb{C}[z_1,\dots,z_5]:"
  • hyperplane arrangement: A finite set of hyperplanes in a vector space; its combinatorics/topology are studied via Orlik–Solomon algebras and characteristic polynomials. Example: "Let A\mathcal{A} be the hyperplane arrangement that consists of the hyperplanes {xi±xj∣1≤i<j≤11}∪{xi∣1≤i≤11}\{x_i \pm x_j \mid 1\le i < j \le 11\}\cup \{ x_i \mid 1\le i \le 11 \}"
  • Kupisch series: A sequence recording the lengths of indecomposable projective modules in a (typically Nakayama) algebra; determines its module structure. Example: "with Kupisch series [2,3][2,3]"
  • lattice path matroid: A matroid constructed from two lattice paths P and Q bounding a Young diagram; bases correspond to certain transversals between paths. Example: "A matroid MM for which there exists P,QP, Q such that M=M[P,Q]M = M[P, Q] is called a lattice path matroid."
  • minimal nested set structure: A canonical fan structure on the Bergman fan built from nested sets of flats using a minimal building set; gives a coarsest meaningful refinement. Example: "the coarse fan structure coincides with the minimal nested set structure"
  • Nakayama algebra: A finite-dimensional algebra in which every indecomposable projective and injective module is uniserial; extensively studied in representation theory. Example: "For a linear Nakayama algebra AA,"
  • permutohedron: A convex polytope whose vertices are permutations of a fixed vector; its combinatorics connects to symmetric groups and Coxeter theory. Example: "the Chow ring of the permutohedron"
  • Picard group: The group of isomorphism classes of line bundles (invertible sheaves) on a variety under tensor product. Example: "Can you tell me the Picard group of the projective variety cut out by this ideal?"
  • pure braid group: The kernel of the projection B_n → S_n; braids whose strand endpoints return to their original positions. Example: "Artin generators of the pure braid group tij:=σiσi+1⋯σj−2σj−12σj−2−1⋯σi+1−1σi−1t_{ij} := \sigma_i \sigma_{i+1} \cdots \sigma_{j-2} \sigma_{j-1}^2 \sigma_{j-2}^{-1} \cdots \sigma_{i+1}^{-1} \sigma_i^{-1}"
  • projective-injective module: A module that is both projective and injective; often central in homological dimensions and Auslander–Reiten theory. Example: "indecomposable projective-injective BB-modules"
  • regular module: The module that is the algebra itself viewed as a left (or right) module over itself; contains structural information about the algebra. Example: "as a submodule of the regular module"
  • Schubert cycles: Fundamental classes of Schubert varieties in the (co)homology/Chow ring of flag varieties or Grassmannians, forming a natural basis. Example: "as a linear combination of Schubert cycles sλs_{\lambda}"
  • signed graphic matroid: A matroid associated to a signed graph, encoding dependencies with respect to signed cycles and cuts. Example: "Compute the signed graphic matroid of this double cover"
  • skew shape: The set difference of two nested Young diagrams, visualized as a skew Ferrers shape; central in symmetric function theory. Example: "Let λ/μ\lambda/\mu be the skew shape enclosed by the region bounded by PP and QQ"
  • socle: The sum of all minimal (simple) submodules of a module; the largest semisimple submodule. Example: "the socle of the direct sum of all indecomposable projective-injective BB-modules"
  • torus orbit closure: The Zariski closure of the orbit of a point under an algebraic torus action; a subvariety with rich combinatorial structure. Example: "the torus orbit closure (C∗)nx‾\overline{(\mathbb C^*)^n x}"
  • Tor functor: The derived functor of tensor product; Tor_i measures how tensoring fails to be exact and captures syzygies/relations. Example: "the dimension of TorR2(I,C)\mathrm{Tor}^2_R(I,\mathbf{C})"
  • transversal matroid: A matroid whose independent sets are partial transversals of a set system; equivalently representable by a bipartite incidence structure. Example: "the transversal matroid on the ground set [n][n] that has the presentation (N1,N2,…,Nr).(N_1,N_2,\ldots,N_r)."
  • weight enumerator: A polynomial that counts codewords of a linear code by Hamming weight, encoding its weight distribution. Example: "What is the weight enumerator of a code generated by such a GG?"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 2 likes about this paper.

Sign Up for Free

HackerNews

  1. Benchmarks in Leipzig (138 points, 49 comments) 

Reddit

  1. Benchmarks in Leipzig (1 point, 1 comment)