Soohak: Math Benchmark for Research-Level LLMs

• Soohak is a mathematician-curated, bilingual benchmark comprising 439 research-adjacent mathematical problems designed to test advanced research-level reasoning in LLMs.
• It introduces a novel refusal subset that requires models to diagnose and explicitly refuse ill-posed or underspecified queries, addressing key limitations in standard benchmarks.
• The evaluation employs a rigorous multi-stage curation and scoring protocol, revealing significant progress gaps and scaling challenges in current LLM architectures.

Soohak (SH²) is a mathematician-curated, contamination-resistant benchmark comprising 439 original, bilingual (English/Korean), research-adjacent mathematical problems. It is designed to assess the capabilities of LLMs on tasks requiring research-level mathematical reasoning, extending evaluation beyond contest and textbook-style benchmarks saturated by recent LLM progress. Soohak introduces a unique refusal subset, compelling models not only to solve advanced problems but also to diagnose and refuse ill-posed or underspecified queries—an ability intrinsic to genuine mathematical expertise. The benchmark will be publicly released in late 2026, with interim model evaluations available upon request (Son et al., 9 May 2026).

1. Motivation and Distinctiveness

Soohak addresses critical limitations of prior math benchmarks such as GSM8K, MATH, and OMNIMATH, which target high-school or undergraduate-level contest problems, often sourced from public data. Two pervasive issues prompted the creation of Soohak:

• Contamination Risk: Benchmarks based on publicly available problems are susceptible to overlap with LLM training data, leading to inflated accuracy assessments and masking true generalization ability.
• Scope Limitations: Contest problems emphasize isolated tricks or short-horizon logical chains, whereas research mathematics demands long-form reasoning, managing folklore-level knowledge, and original idea synthesis.

To overcome these barriers, Soohak provides a suite of graduate-level and research-adjacent problems authored de novo by professional mathematicians, introducing the refusal axis to evaluate a critical dimension of research reasoning: whether models recognize and appropriately respond to ill-posed or unsolvable problems.

2. Construction and Curation Protocol

The construction of Soohak follows a rigorous, multi-stage curation protocol emphasizing originality, difficulty, and resistance to contamination:

• Contributors: 64 professional mathematicians (38 faculty, 25 PhD students or postdocs, 5 IMO medalists) independently authored 439 new problems without AI assistance, under NDA and full intellectual property transfer.
• Five-Stage Pipeline:

1. Submission: Problems submitted with declarations of no AI assistance.
2. Automated Gating: Problems screened through three model panels (small, mid-size, and large open-weight LLMs) to eliminate items solvable by existing systems.
3. Manual Review: Two independent expert auditors compare LLM-generated solutions to author-provided ones, prompting author revisions or rejections for flagged cases (87 items revised; AI-generated submissions led to contributor bans).
4. Author Feedback Loop: Contributors review feedback and may opt-in for final submission.
5. Final Inclusion: Problems allocated to either the Challenge or Refusal subset based on content and outcomes of prior steps.

Withdrawn or declined problems are expunged from all records with a maximum of two reviewers involved, minimizing contamination potential.

3. Benchmark Composition and Subset Specification

Soohak is partitioned as follows:

Subset	Number of Problems	Defining Criterion
Challenge	340	Graduate-level or research-adjacent; unsolved by all large open-weight LLMs
Refusal	99	Ill-posed or underspecified; the correct action is to recognize and explicitly refuse an answer

Formal definitions:

• For each question $q$ and model $m\in M_{\mathrm{open}}$, let $\mathrm{Sol}(q, m)\in\{0,1\}$ indicate solution status.
• $$\mathrm{Challenge} = \{q\ |\ \forall m\in M_{\mathrm{open}},\ \mathrm{Sol}(q, m)=0\}$$
• $$\mathrm{Refusal} = \{q\ |\ q\ \text{is logically or semantically ill-posed}\}$$
• Correct scoring: $q\in\mathrm{Challenge} \wedge r$ is a correct solution, or $q\in\mathrm{Refusal} \wedge r$ is a refusal diagnosing the flaw.

The bilingual nature (English/Korean) further broadens accessibility and scope.

4. Refusal Subset: Novelty and Significance

A distinguishing innovation in Soohak is the explicit inclusion of a refusal axis, reflecting a higher-order research skill: the recognition of ill-posed or non-unique problems. Unlike prior benchmarks, which only test problem-solving, Soohak requires that models, in the presence of insufficiently specified statements (e.g., missing regularity conditions for a Cauchy functional equation), must refuse to answer and provide a correct diagnosis.

Refusal problems are scored such that confident numeric or incorrect answers lead to a zero score; only explicit refusal recognizing the flaw is judged correct. Empirical results indicate that no model currently exceeds 50% accuracy on this split, establishing refusal recognition as a unique optimization target not directly addressed by existing LLM architectures.

5. Evaluation Metrics and Comparative Results

Soohak employs rigorous multi-sample evaluation procedures:

• Primary Metrics:
  • $$\mathrm{avg@3} = \frac{1}{N} \sum\limits_{i=1}^N \Big(\frac{1}{3}\sum\limits_{j=1}^3 c_{i,j}\Big)$$, with $c_{i,j} \in \{0,1\}$ indicating correctness.
  • $$\mathrm{pass@3} = \frac{1}{N} \sum\limits_{i=1}^N \mathbb{I}[\max_j c_{i,j}=1]$$
• Carefulness Metric: Penalty for overconfident mistakes on the Refusal split, encouraging models to avoid spurious answers.

Empirical results on the Challenge subset (avg@3):

Model	Avg@3 (Challenge)
Gemini-3-Pro	30.39%
GPT-5	26.37%
Claude-Opus-4.5	10.39%
Qwen3-235B	8.04%
GPT-OSS-120B	11.27%
Kimi-2.5	13.87%

Refusal subset (avg@3): highest model GLM-5 at 49.49%; GPT-5 and Gemini-3-Flash at approximately 43%. No model exceeds 50%.

Key observations:

• Substantial unsolved headroom remains: 124 of 340 Challenge items remain open to all evaluated models.
• The performance gap between open-weight and closed-weight models on the Challenge split is roughly double that reported on other tasks.
• Refusal accuracy does not improve with scale or compute to the same extent as standard problem-solving accuracy, indicating distinct error modes.

6. Scaling Properties and Research Implications

Scaling analysis using the Qwen3 model family shows:

• Challenge accuracy: $m\in M_{\mathrm{open}}$0 increases nearly linearly with model size (from 1.18% at 0.6B to 8.63% at 32B).
• Refusal accuracy: Non-monotonic scaling, demonstrating that improvements in general reasoning do not transfer directly to metacognitive refusal skills.
• Test-time compute: For GPT-OSS-120B, extending the reasoning horizon or increasing context size increases Challenge pass@3 significantly (from 18.53% at medium context to 29.71% at hard/extensive context), yet has little impact on refusal accuracy.

This suggests that research-level reasoning and appropriate refusal require partially independent development, implying the need for targeted methods and fine-tuning strategies.

7. Dataset Release, Access Control, and Contamination Prevention

Extensive procedural safeguards are in place to ensure originality and utility:

• Authorship protocol: All problems written under NDA with IP transfer and explicit declarations of no AI assistance.
• Model gating: Multi-stage automated evaluation excludes problems solvable by open-weight models from hardest splits.
• Access restriction: Direct reviewer exposure limited to two experts per submission; withdrawn or declined problems immediately deleted.
• Embargo: Full dataset will be public in late 2026 (prior to NeurIPS final acceptance), with controlled access to model evaluations in the interim to prevent data leakage.

The methodological stringency is intended to guarantee reliable, future-proof benchmarking for the next generation of mathematical LLMs.

---

In summary: Soohak defines a new standard for evaluating LLMs on research-level mathematical tasks. Its rigorous, expert-driven design and introduction of the refusal dimension provide a benchmark that simultaneously exposes reasoning and metacognitive deficits in current models, highlighting the substantial progress that remains before LLMs can function as reliable collaborators in advanced mathematics (Son et al., 9 May 2026).