Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving (2512.10739v1)

Abstract: LLMs have achieved significant progress in solving complex reasoning tasks by Reinforcement Learning with Verifiable Rewards (RLVR). This advancement is also inseparable from the oversight automated by reliable verifiers. However, current outcome-based verifiers (OVs) are unable to inspect the unreliable intermediate steps in the long reasoning chains of thought (CoTs). Meanwhile, current process-based verifiers (PVs) have difficulties in reliably detecting errors in the complex long CoTs, limited by the scarcity of high-quality annotations due to the prohibitive costs of human annotations. Therefore, we propose the \textbf{O}utcome-based \textbf{P}rocess \textbf{V}erifier (OPV), which verifies the rationale process of summarized outcomes from long CoTs to achieve both accurate and efficient verification and enable large-scale annotation. To empower the proposed verifier, we adopt an iterative active learning framework with expert annotations to progressively improve the verification capability of OPV with fewer annotation costs. Specifically, in each iteration, the most uncertain cases of the current best OPV are annotated and then subsequently used to train a new OPV through Rejection Fine-Tuning (RFT) and RLVR for the next round. Extensive experiments demonstrate OPV's superior performance and broad applicability. It achieves new state-of-the-art results on our held-out \textsc{\thisbench}, outperforming much larger open-source models such as Qwen3-Max-Preview with an F1 score of 83.1 compared to 76.3. Furthermore, OPV effectively detects false positives within synthetic dataset, closely align with expert assessment. When collaborating with policy models, OPV consistently yields performance gains, e.g., raising the accuracy of DeepSeek-R1-Distill-Qwen-32B from 55.2\% to 73.3\% on AIME2025 as the compute budget scales.

• The paper introduces a novel hierarchical multi-agent system that integrates agentic reasoning, lemma summarization, and logical verification to enable deep Olympiad-level problem solving.
• It leverages Bayesian reinforcement learning for process verification, optimizing multi-round deduction and stabilizing high-confidence reasoning through hierarchical policy improvements.
• The method achieves state-of-the-art performance on competitive benchmarks by managing up to 512K context tokens and mitigating error propagation through systematic lemma memory.

Intern-S1-MO: Hierarchical Long-horizon Reasoning for Olympiad-Level Mathematics

Motivation: Context Scalability in Mathematical Reasoning

Large Reasoning Models (LRMs) have advanced the solution frontier for mathematical problems using CoT and RLVR protocols. Despite progress on problems like AIME, context length scaling remains a bottleneck for ultra-hard tasks such as those encountered at the International Mathematical Olympiad (IMO). Existing SOTA LRMs achieve up to 128K context tokens, yet human expert thinking and model token consumption grow exponentially with problem difficulty, rapidly outpacing available model capacity. Figure 1 illustrates the exponential growth of required tokens and demonstrates that Intern-S1-MO enables the use of up to 512K tokens per problem, supporting much deeper reasoning and achieving SOTA on challenging benchmarks.

Figure 1: As problem difficulty increases, both average human thinking time and model token consumption grow exponentially; Intern-S1-MO uses up to 512K tokens per problem and obtains SOTA results on mathematical benchmarks.

Hierarchical Multi-Agent Reasoning Framework

Intern-S1-MO is constructed as an LRM-based multi-agent system with three key agents: reasoner, summarizer, and verifier. The workflow is inherently multi-round and hierarchical. At each reasoning round, the reasoner produces candidate solution traces; the summarizer extracts and condenses exploratory reasoning into formal lemmas, which are passed to the verifier for logical validation. Validated lemmas are stored in a persistent lemma library, which is reincorporated in subsequent rounds as context, enabling lemma-rich exploration unconstrained by prompt size. A final modification loop refines the solution in response to verifier feedback until either successful validation or a rollout limit is reached.

Figure 2: Intern-S1-MO’s agentic framework employs recursive rounds of reasoning, lemma summarization, and verification to accumulate rigorously validated sub-results in persistent memory.

Lemma memory management markedly differs from prior tree-of-thought and parallel search approaches: Intern-S1-MO systematically condenses knowledge into validated sub-results that enable deeper deduction across rounds. This architecture directly addresses reasoning under strict context constraints and mitigates error propagation by explicit theorem verification.

RL Optimization: OREAL-H for Hierarchical Agentic Learning

To drive agentic improvement and learning, OREAL-H extends the Outcome Reward RL paradigm to hierarchical MDPs with multi-agent structure. The process verifier (PV) assigns natural language feedback and a continuous reward signal through multi-shot confidence aggregation. RL optimization proceeds via behavioral cloning for cold-start initialization, then switches to RL with conjugate Bayesian reward modeling for process verification outputs. The sparse/intermediate reward structure is implemented with temporal difference estimation and a lemma dependency graph, which propagates proof success downstream to intermediate lemmas and quantifies their contribution to solution synthesis.

Figure 3: A lemma dependency graph aggregates and quantifies lemma contributions across multiple rollouts, driving value estimation and structured RL advantage computation.

OREAL-H leverages Bayesian modeling for process verifier feedback, smoothing noisy step-level evaluations and denoising stochastic artifacts. The reward formulation computes the dominance probability over a “completely invalid” baseline, strongly suppressing false positives while maintaining gradient signal for high-confidence progress. This method supports stable and scalable agent training under imperfect process-level supervision.

Empirical Results and Performance

Intern-S1-MO sets new state-of-the-art marks across several competitive mathematics benchmarks:

• HMMT2025: 95% pass@1
• AIME2025: 96.6% pass@1
• CNMO2025: 232.4/260 points
• IMO2025 non-geometry: 26/35 points, matching silver medalist level
• CMO2025 (real test participation, human grading): 102/126 points, exceeding gold medal threshold

Performance gains are consistent across both calculation-centric and proof-oriented benchmark tasks. On IMO-level problems, Intern-S1-MO accounts for persistent partial progress, successfully constructs novel auxiliary lemmas, and surpasses inference methods restricted by one-pass context. The ablation paper confirms every system component (multi-round reasoning, lemma verification, process validation, RL) yields non-redundant improvements, and the architecture is parameter-efficient.

Theoretical and Practical Implications

Intern-S1-MO demonstrates that systematic hierarchical reasoning, lemma-based memory, and process-aware RL collectively unlock deep solution capabilities for tasks beyond traditional LRM context bounds. The persistent lemma library and multi-agent architecture are theoretically significant: they enable SRLMs to simulate scratchpad-like state, facilitate iterative decomposition, and approach human-like problem decomposition patterns. Practically, these mechanisms allow LRMs to address a class of problems previously inaccessible to single-round or prompt-engineered methods, with robust verification mitigating error propagation.

The RL aspects, particularly Bayesian reward modeling and hierarchical TD advantage formulation, offer a template for future agentic optimization in language reasoning agents and mark a substantial advance beyond binary outcome-based RLVR training.

Future Directions

This work points towards scalable, agentic LRM systems capable of long-horizon deductive synthesis at or beyond human expert standards. Key open areas include further scaling of persistent memory architectures, automated lemma extraction verification, transferability to other domains (e.g., formal theorem proving, research mathematics), and integration with multimodal or tool-augmented frameworks.

Improving idiosyncratic insight detection (spark-of-insight problems, creative construction) remains a frontier, potentially addressable by policy architecture enhancements and meta-learning protocols. The paradigm demonstrated by Intern-S1-MO is likely to inform next-generation AI capable of rigorous cognitive tasks across both mathematical and broader scientific domains.

Conclusion

Intern-S1-MO advances the practical and theoretical frontier of long-horizon mathematical reasoning by integrating hierarchical multi-round agentic protocols, lemma memory management, robust verification, and process-aware RL training. These mechanisms collectively enable solutions to ultra-hard competition problems beyond the reach of existing context-constrained LRMs, demonstrated by gold/silver medal-level performance on real Olympiad datasets. The methodological innovations described herein will shape future research on scalable reasoning agents for complex intellectual domains.