Unreasonable Math Problems: Benchmarks & Analysis

• Unreasonable Math Problems are defined as math queries with logical inconsistencies, missing variables, or invalid assumptions that render them unsolvable or incoherent.
• Benchmark construction involves systematically perturbing standard problems and using metrics like accuracy and precision to evaluate AI detection and performance.
• Research shows that state-of-the-art models struggle with UMP detection, highlighting the need for advanced prompt engineering and hybrid mitigation strategies.

Unreasonable Math Problems (UMP) are a central topic in contemporary mathematical AI evaluation, benchmarking, and theory. The term serves as an umbrella for problems whose solution is impeded by ill-posedness, logical, or semantic flaws—ranging from subtle inconsistencies in grade-school questions to structurally adversarial hard-perturbations of research-level mathematics, and even includes problems whose very definition or recurrence in advanced domains challenges prior expectations of regularity or tractability. This article synthesizes definitional foundations, benchmark construction, key empirical findings, relevant theoretical paradigms, and implications for both model design and mathematical logic.

1. Definitional Frameworks for Unreasonability

An Unreasonable Math Problem (UMP) is formulated as a mathematical question where the premises render the query unsatisfiable, undefined, or logically or physically incoherent. The formal diagnosis relies on a spectrum of criteria (Ma et al., 2024):

• Logical Contradiction: There exist premises $p_i, p_j$ such that $p_i \wedge p_j \models \bot$.
• Insufficient Information: Required variables in the query $H$ are absent from the set of premises $P$.
• Incorrect Mathematical Assumptions: The problem invokes operations or objects that are mathematically ill-defined for the given parameterization (e.g., division by zero, non-integer counting of discrete entities).
• Nonsensical Physical Units: Stated quantities have impossible or mismatched physical dimensions.
• Illogical Scenario: The question is inconsistent with basic world or domain knowledge.

The composite criterion is:

$$\mathrm{Unreasonable}(Q) \iff \Bigl(\bigwedge_{i=1}^n p_i\Bigr) \text{ is unsatisfiable} \;\lor\; \exists\,v\, [v \in \mathrm{Vars}(H) \land v \notin \mathrm{Vars}(P)] \;\lor\; \mathrm{dimError}(P)$$

This formalism underpins automatic and human evaluation for UMP detection on benchmarks.

2. Benchmark Construction and Taxonomy

The practical evaluation of UMP phenomena depends on robust benchmark design. The Unreasonable Math Problems (UMP) benchmark (Ma et al., 2024) and MATH-P-Hard (Huang et al., 10 Feb 2025) are two central resources.

Benchmark construction proceeds through these steps:

• Selection of Source Corpus: Start from a curated base of "reasonable" original problems (e.g., GSM8K or MATH Level-5).
• Systematic Generation of Unreasonable Variants:
  • Rewrite original problems via LLM or expert to introduce unreasonability in controlled fashion, targeting categories such as undefined variables, illogical scenarios, incorrect assumptions, misinterpreted units, and inconsistent conditions.
  • For hard-perturbation, minimally edit problems to alter the semantic structure, destroying the applicability of the original reasoning route.
• Filtering and Annotation:
  • Employ embedding-based cosine similarity to ensure minimal lexical drift and enforce subtlety.
  • Annotate each instance by type of unreasonability and include brief explanations.
  • Institute human review to ensure nuanced errors and prevent explicit cues (e.g., "negative people").
• Scale and Composition:
  • Example: UMP benchmark with 150 each of reasonable and unreasonable problems, MATH-P-Hard with 279 hard-perturbed items.

Benchmark categories are summarized as follows:

Category	Example Instance	Generation Mode
Undefined Variable	Solve for $x$, but $x$ never introduced	Directed rewrite
Illogical Scenario	"How many kittens does a hamster have?"	LLM rewrite / human edit
Hard Perturbation	Core conditions altered, solution changes	PhD expert construction
Incorrect Assumption	Division by zero, fractional counting	LLM rewrite
Nonsensical Units	Distance stated as "square meters"	LLM rewrite

This taxonomy creates a substrate for fine-grained error analysis and benchmarking.

3. Empirical Model Performance and Detection Methodologies

Evaluation of LLMs on UMPs involves measuring both solution accuracy and detection of unreasonability. The UMP (Ma et al., 2024) and MATH-Perturb (Huang et al., 10 Feb 2025) benchmarks reveal several key insights:

• Metrics:
  • $\mathrm{Acc}_U$: Proportion of unreasonable problems correctly flagged.
  • $\mathrm{Acc}_R$: Proportion of reasonable problems not incorrectly flagged.
  • Precision, recall, and F1 combine detection of unreasonability and non-over-flagging.
• Prompting Strategies:
  • Basic Chain-of-Thought (CoT): General improvement for arithmetic but does not elicit premise-checking.
  • Direct Plausibility Query: Induces some increase in $\mathrm{Acc}_U$ but can bias toward "reasonable."
  • Critical Calculation and Conclusion (CCC): Forces explicit premise validation; CCC prompts raise $p_i \wedge p_j \models \bot$0 to $p_i \wedge p_j \models \bot$1 for GPT-4 and $p_i \wedge p_j \models \bot$2 for Qwen-Max.
  • Fine-tuning and Multi-task Loss: Proposals suggest a “reasonability-detection head” for separate signal, but may slightly reduce raw mathematical performance.

Model	CoT Acc.	CCC Acc.	Drop: Simple $p_i \wedge p_j \models \bot$3 Hard (\%) (Huang et al., 10 Feb 2025)
GPT-4	74.6	94.6	27.6
Qwen-Max	70.6	94.0	14.33
o1-mini	—	—	15.78
Gemini-2.0-flash	—	—	14.33
Claude-3.5-Sonnet	—	—	25.81

The critical empirical observation is that state-of-the-art models, even under CCC prompting, sometimes fail to distinguish subtle unreasonable cues, and they may hallucinate solutions or copy reasoning templates from similar but inapplicable problems.

4. Failure Modes and the Challenge of Robust Reasoning

Analyzing LLM failure modes on UMPs and hard-perturbed problems exposes foundational limitations:

• Blind Memorization: High-end models often replay solution strategies from near-neighbor original problems without verifying that preconditions still apply (Huang et al., 10 Feb 2025). This differs from simple mode collapse; the reasoning logic is valid for the original but invalid for the perturbation.
• Overthinking and Non-convergence: Some models, especially when prompted to be "critical," produce verbose but unproductive self-correcting loops in unreasonable cases (Ma et al., 2024).
• Substantive Error Chains: As in Riemann-Bench (Garre et al., 8 Apr 2026), frontier models adopt inappropriate theoretical frameworks, invent spurious intermediate claims, and build solutions that are structurally sound but mathematically empty.

Failure analysis from Riemann-Bench indicates current models lack the ability to flexibly adjust solution trajectories in the face of unfamiliar or ill-defined problem structures, leading to sub-10% pass@1 rates on authentic research problems despite perfect Olympiad performance.

5. Theoretical Perspectives: Unreasonability and Mathematical Logic

Unreasonable math problems are also prominent in theoretical computer science and discrete mathematics, both as objects of study and as phenomena challenging classical regularity. Two key paradigms are relevant:

• Unreasonable Effectiveness of Quasi-polynomials: In parametric counting, "unreasonable" instances emerge where quasi-polynomial structure is preserved despite parameter-dependent constraint vectors, generalizing beyond classical Presburger definability (Woods, 2013). Woods formulates the grand conjecture that parametric Presburger families, even with polynomially varying coefficients in atomic constraints, exhibit eventual quasi-polynomial behavior, confirmed for major classes by Chen–Li–Sam and Calegari–Walker.
• Partial One-sided Decision Procedures in Computability: Newberry's construction (Newberry, 2017) defines URM programs that "know" about divergence on a sparse subset of inputs, providing one-sided "unreasonability detectors" without violating uncomputability theorems. This yields analogies to the liar paradox and informs generalizations of incompleteness arguments beyond Peano Arithmetic.

These lines of research indicate that "unreasonability" is both an adversarial diagnostic and a fundamental phenomenon in logic and combinatorics.

6. Strategies for Mitigating UMP Failure and Future Directions

Research converges on a set of recommendations to address UMPs in mathematical AI:

• Prompt Engineering: Always include explicit premise-checking or critical-reasoning stages in the prompt. CCC-style prompting is effective in surfacing latent detection capabilities (Ma et al., 2024).
• Hybrid Pipeline Architectures: Combine lightweight binary reasonability detectors with full CoT solvers. If a detector flags a problem as unreasonable, refrain from solution or escalate for review.
• Adversarial and Hard-Perturbation Training: Expose models during fine-tuning to both the original and hard-perturbed instances, preventing overfit to surface cues (Huang et al., 10 Feb 2025).
• Research-level Evaluation: Employ private, programmatically verified, research-derived benchmarks such as Riemann-Bench to assure contamination resistance and to measure authentic progress on intractable or ill-posed problem classes (Garre et al., 8 Apr 2026).
• Human-in-the-Loop Brainstorming: Maintain iterative, reflective dialogue between model and expert, alternating between idea-generation and error detection, as demonstrated in collaborative protocols (Gu, 2023).

A plausible implication is that reliable deployment of LLMs in mathematical reasoning will require systematic integration of both critical self-evaluation and adversarial robustness at multiple abstraction levels.

7. Impact and Theoretical Significance

UMP research reveals that the gap between near-perfect performance on structured benchmarks and robust, general-purpose mathematical reasoning is substantial. Subtle unreasonability either defeats contemporary LLMs (who generate solutions for unsolvable or nonsensical questions) or induces brittle pattern-matching that collapses under even minor hard perturbation. The emergence of hidden quasi-polynomial structure in "unreasonable" combinatorial families (Woods, 2013) and the existence of partial divergence detectors in computability (Newberry, 2017) underscore the depth and pervasiveness of this phenomenon.

This suggests that the long-term advancement of AI as mathematical collaborator depends not merely on scaling up pattern recognition, but on fundamentally new architectures and training curricula designed to internalize unreasonability detection, structural validation, and flexible reasoning over adversarially reframed problems. The continuing study of unreasonable math problems thus offers both a stress-test for current models and a path toward more principled, intellectually robust mathematical reasoning systems.