MathDuels: Dual-Role LLM Math Benchmark

• MathDuels is a self-play benchmark that measures math reasoning by having LLMs both generate and solve problems, thereby capturing dual capabilities.
• The benchmark employs a three-stage pipeline—including meta-prompting, problem generation, and difficulty amplification—to dynamically evolve problem difficulty.
• A combined Rasch model estimates solver abilities and problem difficulties, offering a nuanced ranking beyond traditional solve-only accuracy metrics.

MathDuels is a self-play benchmark for mathematical reasoning in LLMs that evaluates each model in two roles at once: as a solver of math problems authored by other models, and as an author of new math problems that other models must solve. It was introduced in response to the saturation of static mathematical benchmarks such as GSM8K and MATH, where frontier models attain near-ceiling scores and solve-only evaluation increasingly fails to separate capabilities. Its central design replaces a fixed test set with an arena in which participants compete through authored problems, so benchmark difficulty can co-evolve with model strength rather than remain fixed. Problems are generated through a three-stage pipeline, filtered by an independent verifier that excludes ill-posed questions, and scored with a Rasch model that jointly estimates solver abilities and problem difficulties; author quality is derived from the difficulties of each model’s authored problems (Xu et al., 23 Apr 2026).

1. Motivation and conceptual framing

MathDuels is framed as an analogue of historical mathematical duels: instead of answering a static set of questions, participants challenge one another. The motivating claim is that a strong model may not only be a strong solver, but also a strong problem poser that can construct hard, valid problems exposing weaknesses in other systems. This directly targets a blind spot of solve-only evaluation, namely that important abilities such as constructing hard, valid problems are ignored (Xu et al., 23 Apr 2026).

The benchmark’s motivation is explicit. Static benchmarks are saturating faster than new problems can be produced; even live or competition benchmarks are becoming less durable; and evaluation that measures only solving conflates distinct aspects of mathematical competence. MathDuels therefore measures mathematical competence in dual roles and aims to capture a more frontier-relevant notion of capability. A plausible implication is that the benchmark treats adversarial problem construction as a first-class object of evaluation rather than as auxiliary data generation.

The benchmark is also designed to avoid a fixed ceiling. As newer models enter the arena, they author new problems rather than merely attempt an inherited corpus. This suggests a dynamic evaluation regime in which the hardest items can be endogenously generated by the strongest participants, rather than supplied solely by human curators (Xu et al., 23 Apr 2026).

2. Dual-role protocol and arena structure

Each model participates symmetrically in both roles. In the author role, it generates problems; in the solver role, it attempts problems authored by every other model. If there are $N$ models and each authors $K$ problems, then the arena has $NK$ problems total, and every non-author model attempts every problem. Correctness is judged automatically using symbolic equivalence checking, and the observed data are binary outcomes

$$y_{mp}\in\{0,1\},\qquad y_{mp}=1\;\Longleftrightarrow\;\text{model }m\text{ solves problem }p\text{ correctly}.$$

These outcomes form the solve matrix used for ranking (Xu et al., 23 Apr 2026).

The benchmark reported in the paper evaluates 19 frontier models from nine providers, including variants at different reasoning-effort levels. Each model authors 30 problems distributed across six broad math domains: Analysis, Algebra, Geometry / Topology, Discrete Mathematics, Probability / Statistics, and Applied / Computational Mathematics. The arena therefore produced 570 total problems. After verification, 11 were excluded and 559 valid problems remained, yielding 10,062 binary solve observations (Xu et al., 23 Apr 2026).

This protocol makes the separation between authoring and solving empirically visible. The paper explicitly notes that a model can be a great solver but only a mediocre author, or a weaker solver but a surprisingly strong author. That separation is central to MathDuels, and the leaderboard is constructed to preserve it rather than collapse everything into raw accuracy.

3. Problem generation pipeline and independent verification

MathDuels uses a three-stage pipeline intended to make problems hard while keeping them well-posed and verifiable. In Stage 1, meta-prompting, the model is asked to write a prompt for authoring a problem in a given math domain. In Stage 2, problem generation, that meta-prompt is fed back to the same model, which produces a problem statement $q_k$ and a gold answer $g_k$, expressed as a single scalar mathematical expression. In Stage 3, difficulty amplification, the authoring model may perform one or more hardening rounds to produce a variant requiring deeper reasoning (Xu et al., 23 Apr 2026).

The paper reports an ablation over 432 model-pair observations showing that each stage tends to raise difficulty, with amplification providing the largest increase. The measured error rates were 22 / 432, 5.1% without code execution for 1-step direct generation; 41 / 432, 9.5% for 2-step meta-prompting; 108 / 432, 25.0% for 3-step meta-prompting + amplification; and 84 / 432, 21.8% for 3-step with code execution (Xu et al., 23 Apr 2026). This establishes that the generation pipeline is not merely stylistic scaffolding; it materially changes the difficulty profile of the authored corpus.

Synthetic mathematics benchmarks are vulnerable to ambiguous, ill-posed, internally inconsistent, or incorrectly answered questions. MathDuels addresses this with an independent verifier that is triggered when at least one solver gets the problem wrong, that is, when there is disagreement. The verification step is

$$(\,v,\, a^*\,) \;=\; Verify\!\left(q_p,\; g_p,\; \mathcal{A}_p\right), \qquad v\in\{0,1\},\quad a^*\in\mathcal{A}_p,$$

where $\mathcal{A}_p = \{(a_i, r_i)\}$ is the set of unique solver answers and reasoning traces. The verifier checks whether the problem has a unique, well-defined solution and selects the correct answer among candidate solver answers. If $v=0$, the problem is dropped from scoring; if $v=1$, the verifier-selected answer $K$0 replaces the author’s original gold answer if needed (Xu et al., 23 Apr 2026).

The paper gives robustness evidence for this step. Using GPT-5.4-high versus Gemini-3.1-Pro-high as verifier backbones on 414 problems, the same exclusion decision occurred in 97.5% of cases and the same selected answer in 99.4% (Xu et al., 23 Apr 2026). A common misconception is therefore that MathDuels simply trusts model-authored gold answers; in fact, invalid problems are explicitly filtered, and gold answers can be overridden by verifier-selected answers.

4. Rasch estimation, ratings, and leaderboard construction

MathDuels uses a Rasch item response model to jointly estimate solver ability $K$1 and problem difficulty $K$2 on a shared logit scale: $K$3 The interpretation is direct: if $K$4, the model is likely to solve the problem, whereas if $K$5, the problem is likely to defeat the model. The paper states that this is mathematically identical to a Bradley–Terry model over solver–problem pairs (Xu et al., 23 Apr 2026).

Joint estimation is performed by maximizing

$K$6

where $K$7 is the set of observed solver–problem pairs, excluding invalid problems and self-authored attempts, and $K$8 is a mild $K$9 regularizer on problem difficulties to prevent divergence when a problem is solved by all or none of the solvers. Because the Rasch likelihood is invariant to adding a constant to all parameters, the scale is anchored by fixing a reference model’s solver rating (Xu et al., 23 Apr 2026).

The latent logit scores are converted to Elo-like ratings, and author quality is not directly fit as a parameter. Instead, it is derived from the mean rating-scaled difficulty of the problems authored by a model: $NK$0 The paper adds two safeguards: invalid problems are removed from all likelihood computations, and if the author’s own gold answer is wrong, that problem’s difficulty is capped at $NK$1 when contributing to author rating, so authors do not benefit from hard but wrong problems. Final rating combines solving and authoring equally: $NK$2 Confidence intervals are reported as stratified bootstrap 95% CIs with $NK$3 resamples, stratified by author (Xu et al., 23 Apr 2026).

This makes the leaderboard structurally different from a benchmark based on average accuracy. The paper emphasizes that ranks are built from the joint solve-author structure, not from raw solve rates alone. A plausible implication is that the benchmark measures both the ability to withstand adversarial pressure and the ability to generate it.

5. Empirical findings and capability structure

The paper’s headline result is that authoring and solving capabilities are partially decoupled. GPT-5.4-high is reported as the best solver, whereas Gemini-3.1-Pro-high is the best overall model because it is the strongest author. Gemini-3.1-Pro-high’s authored problems have the lowest average solve rate of any participant, 62.9% (Xu et al., 23 Apr 2026). This means that a solve-only benchmark would miss a distinction that becomes visible once authoring quality is explicitly measured.

Another concrete example is Grok-4.20-high, which has the largest gap between solver and author ratings. The authors interpret such cases as evidence that the ability to solve hard mathematics and the ability to pose hard mathematics are related but not identical capabilities. This supports the benchmark’s central claim that dual-role evaluation reveals capability separations invisible in single-role benchmarks (Xu et al., 23 Apr 2026).

The reported leaderboard at $NK$4 places Gemini-3.1-Pro-high first with composite 1919, GPT-5.4-high second with 1870, GPT-5.2-high third with 1737, Claude-Opus-4.6-high fourth with 1675, and Gemini-3-Flash-high fifth with 1673 (Xu et al., 23 Apr 2026). The paper also notes that the top models have relatively tight rank ranges, whereas mid-tier models have wider overlaps, so confidence intervals remain important for interpretation.

A second major finding concerns non-saturation. The paper defines the prior top-3 solvers excluding the two newest entrants as GPT-5.2-high, Claude-Opus-4.6-high, and Qwen-3.5-397B-A17B. Among the 275 problems authored by the two newest and strongest entrants, 27.6% defeated at least one prior top-3 solver. For the remaining 17 participants, only 8.0% of problems defeated a prior top-3 solver (Xu et al., 23 Apr 2026). This is the benchmark’s strongest empirical evidence that stronger newcomers do not merely score higher on an inherited test; they actively raise the benchmark difficulty by authoring problems that break the old leaders.

At the same time, 39% of valid problems were solved by all non-author solvers (Xu et al., 23 Apr 2026). This constrains interpretation. MathDuels does not eliminate easy items; rather, it relies on the remaining tail to provide discrimination, with that tail increasingly shaped by the best authors.

6. Broader duel paradigms, limitations, and interpretation

MathDuels belongs to a wider family of duel-style formalisms. “The Token Games” also draws inspiration from 16th-century Italian mathematical duels, but implements them through Programming Puzzles, pairwise proposer–solver rounds, and Elo aggregation rather than domain-labeled scalar-answer mathematics (Henniger et al., 19 Feb 2026). “Dueling Algorithms” studies a different abstraction, recasting optimization problems as symmetric constant-sum games in which two players compete to outperform one another rather than minimize expected cost in the usual one-player sense (Immorlica et al., 2011). A plausible implication is that MathDuels inherits the duel idea from this broader tradition while specializing it to automated mathematical reasoning evaluation.

The benchmark’s limitations are explicit. Confidence intervals still matter, especially for mid-tier ranks with overlapping rank ranges. The benchmark relies on final-answer checking rather than proof-based verification. Some problems are still solved by everyone, so more scale may be needed for finer separation. The generated problem distribution depends on the model’s authoring style and the domain taxonomy. Verification depends on the quality of the verifier and tooling, since it uses a strong model plus symbolic checking (Xu et al., 23 Apr 2026).

These limitations clarify several misconceptions. MathDuels is not a pure solve-only benchmark, since problem construction counts as much as solving. It is not equivalent to raw accuracy ranking, since ratings come from a latent ability–difficulty model. It is not a proof benchmark, since verification is based on final-answer checking. And it is not a fixed static dataset, since its intended distinguishing feature is that benchmark difficulty co-evolves with participant strength rather than saturating at a fixed ceiling (Xu et al., 23 Apr 2026).

Within that design space, MathDuels’ distinctive claim is narrow but technically consequential: mathematical competence is evaluated in dual roles, under adversarial self-play, with invalid synthetic problems filtered out and performance summarized through joint latent estimation. The benchmark therefore treats authored difficulty as empirical evidence of capability, not merely as a means of generating more test questions.