Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game

Abstract: While LLMs have achieved notable success on formal mathematics benchmarks such as MiniF2F, it remains unclear whether these results stem from genuine logical reasoning or semantic pattern matching against pre-training data. This paper identifies Architectural Reasoning: the ability to synthesize formal proofs using exclusively local axioms and definitions within an alien math domain, as the necessary ability for future automated theorem discovery AI. We use the Obfuscated Natural Number Game, a benchmark to evaluate Architectural Reasoning. By renaming identifiers in the Natural Number Game in Lean 4, we created a zero-knowledge, closed environment. We evaluate state-of-the-art models, finding a universal latency tax where obfuscation increases inference time. The results also reveal a divergence in robustness: while general models (Claude-Sonnet-4.5, GPT-4o) suffer performance degradation, reasoning models (DeepSeek-R1, GPT-5, DeepSeek-Prover-V2) maintain the same accuracy despite the absence of semantic cues. These findings provide a quantitative metric for assessing the true capacity for mathematical reasoning.

• The paper introduces Architectural Reasoning by using the Obfuscated Natural Number Game to strip semantic cues and test pure logical proof synthesis.
• It demonstrates that reasoning-focused models maintain high accuracy under obfuscation, while general-purpose models experience significant performance drops.
• The study reveals a universal latency tax, underscoring the computational demands inherent to purely logic-driven theorem proving.

Architectural Reasoning Evaluation for LLM Provers via the Obfuscated Natural Number Game

Motivation and Problem Statement

The paper "Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game" (2605.00677) addresses a central question in the AI4Math domain: whether state-of-the-art LLM-based theorem provers genuinely reason through mathematical structures or simply engage in semantic pattern matching facilitated by pretraining on large mathematical corpora. Conventional benchmarks such as MiniF2F and PutnamBench disproportionately reward library retrieval and semantic cue exploitation, inadequately quantifying logical synthesis in unfamiliar axiomatic contexts. The proposed framework introduces Architectural Reasoning as the ability to construct proofs using solely local axioms and definitions, strictly without semantic or library-based assistance—identified as an essential prerequisite for robust automated theorem discovery.

Benchmark Design: Obfuscation for Zero-Knowledge Environments

The core methodology is realized through the Obfuscated Natural Number Game (O-NNG). Identifier names within the Lean 4-based Natural Number Game are randomized using controlled character-level noise, systematically erasing semantic information. The process deconstructs standard naming conventions, leaving only the logical structure of Peano arithmetic accessible to models and abolishing shortcuts from memorized semantic associations. The obfuscated benchmark is parameterized by a noise level ($\lambda$), progressively transitioning from original identifiers ($\lambda = 0$) to complete obfuscation ($\lambda = 1.0$). To maintain syntactic validity, transformations leverage the nlpaug RandomCharAug pipeline with calibrated insertion and deletion probabilities. The dataset comprises 68 formal problems across eight mathematical modules, and all prompts include modular axioms, target theorems, and permissible tactics (excluding high-level automation).

Experimental Pipeline

Each model is interrogated with a chain-of-thought prompt mandating production of both a natural language proof plan and a formal Lean 4 proof. Outputs undergo automated validation via the Lean 4 compiler (v4.27.0). For robustness, every theorem is attempted five times at each noise level, and two primary metrics are reported: Correct Rate (%) and Average Time (s). The selection includes general models (GPT-4o, Claude-Sonnet-4.5) and reasoning-focused architectures (DeepSeek-R1, GPT-5, DeepSeek-Prover-V2), all accessed via API rather than local compute clusters.

Results: Latency Tax and Accuracy Divergence

All models incurred a statistically significant inference time increase on the O-NNG compared to NNG, regardless of performance baseline—a "universal latency tax." This supports the hypothesis that architectural reasoning necessitates more intensive logical context reconstruction in semantically alien environments. With respect to accuracy, a pronounced bifurcation emerges:

• Reasoning models (DeepSeek-R1, GPT-5, DeepSeek-Prover-V2): Maintain near-identical correct rates across all obfuscation levels. Their performance robustness demonstrates independence from semantic cues and validates Architectural Reasoning as a functional capability.
• General-purpose models (GPT-4o, Claude-Sonnet-4.5): Suffer statistically significant correct rate degradation when identifiers are obfuscated, highlighting reliance on semantic pattern matching rather than logic-driven proof synthesis.

These results are supported by rigorous ANOVA significance testing.

Discussion: Limitations, Heuristic Mapping, and Uncanny Valley Effects

The benchmark’s logical structure remains isomorphic to Peano arithmetic, which frontier models might heuristically identify and exploit, although no explicit mapping was observed. Further, a non-linear performance dip was occasionally observed at intermediate obfuscation levels, suggesting a "Reasoning Uncanny Valley" where residual semantic cues hinder structural reasoning. This invites future exploration into whether maximal obfuscation consistently stabilizes performance and if iterative, interactive feedback environments (e.g., Lean compiler responses) can reduce latency.

The framework's domain specificity (natural numbers, Lean 4) and restriction to high-end models limit generalizability. Extensions to more abstract domains or mid-sized, open-source models are warranted. Evaluating model aptitude for creative domain mapping or construction of genuinely novel axiomatic systems would provide a firmer test of Architectural Reasoning.

Implications for AI4Math and LLM Prover Design

The findings advance Architectural Reasoning benchmarks as requirements for evaluating mathematical discovery agents under zero-knowledge constraints. This paradigm compels the design and deployment of reasoning-centric architectures, as general-purpose models are evaluatively insufficient. Moreover, the universal latency accentuates the computational overhead intrinsic to pure logical reasoning, signifying that the reliable automation of theorem synthesis will entail nontrivial resource expenditure and inference time.

O-NNG demonstrates that performance in conventional formal mathematics tasks does not reliably indicate deep logical reasoning capabilities; instead, robustness in zero-knowledge environments should emerge as a gating criterion for future AI4Math development.

Conclusion

The paper rigorously defines and operationalizes Architectural Reasoning, presenting the Obfuscated Natural Number Game as a robust benchmark for evaluating the capacity of LLMs to synthesize proofs in semantically obfuscated, library-free environments. Reasoning-focused models exhibit resilience to semantic perturbation and preserve proof accuracy, while general-purpose models falter, evidencing a reliance on lexical associations. The universal latency tax underscores the computational demands of architectural reasoning in alien domains. The framework establishes a quantitative metric for model evaluation beyond pattern matching, with direct relevance for the construction of domain-agnostic automated reasoning systems. Future research should extend to broader axiomatic domains and interactive feedback modalities to further delineate the limits and operational cost of Architectural Reasoning in LLMs.