Reasoning Large Language Models

• RLMs are neural language models that extend traditional LLMs by generating explicit multi-step reasoning traces using chains, trees, or graphs.
• They integrate reinforcement learning and structured inference to optimize planning, credit assignment, and performance in complex cognitive tasks.
• Their modular architectures and adaptive search strategies, including Monte Carlo Tree Search, balance exploration and exploitation to enhance reasoning accuracy.

A Reasoning LLM (RLM) is a neural LLM architected, trained, or supervised to generate not only answers, but also explicit multi-step reasoning traces—chains of thought (CoT), trees, or graphs—thereby extending the standard LLM’s sequence modeling with system-level search, planning, and credit assignment for advanced cognitive tasks such as mathematics, scientific problem solving, code synthesis, and diagnosis. RLMs integrate reinforcement learning, structured inference, and reasoning-centric training pipelines to approximate human-like deductive and inductive abilities. This article surveys the foundational principles, algorithmic techniques, empirical results, and system-level architectures underpinning state-of-the-art RLMs, with a focus on their unique reasoning characteristics and challenges.

1. Conceptual Foundations and Principles

RLMs extend the classic autoregressive LLM by introducing explicit “thought” tokens and multi-phase reasoning structures. The central paradigm is chain-of-thought (CoT) prompting, which elicits multi-step rationales rather than direct answers. Tree-of-thought (ToT) and Graph-of-thought (GoT) architectures generalize CoT, enabling parallel exploration, backtracking, and aggregation of alternative reasoning paths, often governed by search algorithms such as Monte Carlo Tree Search or beam search (Besta et al., 20 Jan 2025).

A canonical RLM reasoning loop is structured as follows:

• State: a partial reasoning chain or graph
• Action: extension (e.g., next CoT step, branching node)
• Operators: {Generate, Refine, Backtrack, Prune, Aggregate, Evaluate, Update}
• Search strategy: selection and expansion policies balance exploration (novel ideas) and exploitation (known good patterns)
• Credit assignment: value models or process reward models (PRMs) estimate the merit of intermediate or final reasoning states

The reasoning process is commonly formalized as an episodic Markov decision process (MDP). The final answer is accompanied by an explicit rationale, supporting human- or verifier-based judgment and structured supervision (Xu et al., 16 Jan 2025, Besta et al., 20 Jan 2025).

2. Reinforcement Learning and Supervisory Techniques

Reinforcement learning (RL) is central in the transition from LLMs to RLMs. Policy-gradient methods, especially Proximal Policy Optimization (PPO) and Group-Relative Policy Optimization (GRPO), have become standard, often replacing value networks with group-level normalization for stability (Xu et al., 16 Jan 2025, Le et al., 3 Apr 2025, Tian et al., 6 Aug 2025). RLMs are trained under either:

• Outcome-Based Supervision (OBS): reward only for final answer correctness
• Process-Based Supervision (PBS): additional reward/penalty signals for each reasoning step, requiring labeled rationales or rewards from process reward models (PRMs) (Liu et al., 2 Oct 2025)

Hybrid schemes such as reverse curriculum RL (R³) “slide” the starting point of RL episodes backward through correct rationales, approximating process-level feedback with outcome-only reward data (Xi et al., 8 Feb 2024). Memory-augmented methods (e.g., Memory-R⁺) use episodic stores of past successes and failures, retrieving nearest-neighbor reasoning traces to form intrinsic rewards, balancing imitation and exploration for small models (Le et al., 3 Apr 2025).

The integration of PRMs enables search-guided inference, step-level refinement, and robust evaluation metrics, but semi-automated data pipelines often provide the dense annotations needed for step-level supervision (Liu et al., 2 Oct 2025).

3. Reasoning Structures and Search Schemes

RLMs are architected with diverse reasoning structures:

• Chain-of-thought (linear): each inference step builds on the previous, producing a single trace per query
• Tree-of-thought (branching): exploration of multiple candidate traces, backtracking as needed, often evaluated with value functions or PRMs
• Graph-of-thought (DAG): allows convergence (merging) and divergence (splitting) of thought chains; enables richer cross-step dependencies and hypothesis testing (Besta et al., 20 Jan 2025, Xiong et al., 20 May 2025)

The search strategy may be fixed (greedy, beam search) or adaptive (MCTS with neural value models, best-of-N with reward filtering), trading off latency, memory, and accuracy (Besta et al., 20 Jan 2025, Xu et al., 16 Jan 2025). Temporal or budget constraints at test time (e.g., token budget, dynamic suppression) further balance cost-efficiency and reasoning depth (Zheng, 29 Sep 2025).

4. Empirical Performance, Robustness, and Limitations

RLMs exhibit strong gains on arithmetic, symbolic, and scientific benchmarks compared to pure LLM baselines:

• SFT-only models leveraging curated CoT datasets achieve 50–73% accuracy on AIME24 (math Olympiad); RLVR-trained distillations hit 79.7–79.8%, matching proprietary DeepSeek-R1 (Zhang et al., 1 May 2025).
• For tiny LLMs (≤1B), intrinsic motivation from memory-augmentation yields +2–7 points over vanilla RL and doubles sample efficiency, especially in the low-data regime (Le et al., 3 Apr 2025).
• Reverse curriculum RL (R³) offers +4 points over outcome-supervised RL in reasoning and program-based CoT tasks, matching larger or closed-source models at 7B (Xi et al., 8 Feb 2024).

However, RLM robustness remains incomplete:

• Fine-tuned LLMs degrade sharply in non-ideal settings: summary inference under aggregation, adversarial distractors, and irrelevant contextual noise—all show ∼5–10 point drops or more, even after RL-based remediation (Tian et al., 6 Aug 2025).
• Prompt dependency is pronounced. Extensive few-shot templates or verbose self-reflective demonstrations can reduce both the density of reasoning graphs and accuracy, while zero-shot or minimal exemplars encourage more exploratory, branched reasoning and higher performance (Xiong et al., 20 May 2025, Raganato et al., 1 May 2025).
• “Overthinking” (unnecessary reflection) inflates computation without benefiting accuracy; adaptive suppression (ARS) can cut token count by up to 53% while preserving or boosting accuracy (Zheng, 29 Sep 2025).

Multilingual capabilities are brittle. Cross-lingual collapse and language mixing are widespread in multilingual RLMs, especially under RLVR, with reasoning traces reverting to dominant pre-training languages, collapsing low-resource CoTs unless heavily penalized, at significant accuracy cost (Park et al., 6 Jun 2025, Wang et al., 20 May 2025). Understanding failures, especially the inability to internally translate low-resource prompts to the dominant reasoning language, are a primary cause of performance gaps; selective translation strategies based on interpretable detection can bridge these gaps efficiently (Kang et al., 31 Oct 2025).

5. System Architectures and Modular Blueprints

Recent RLM designs are systematically modularized, facilitating extensibility:

• Reasoning structure module: defines CoT, ToT, GoT, or nested architectures
• Operator set: discrete generators for expansion, refinement, backtracking, aggregation, external tool invocation, and retrieval
• Inference pipeline: orchestrates operator application under policy/value model guidance, supports test-time scaling, dynamic token budgets, and early-exit mechanisms
• Data pipeline: supports supervised (SFT, PBS), RL (PPO, GRPO, DPO), and synthetic data generation using replay buffers and search-based rollouts (Besta et al., 20 Jan 2025)
• Training pipeline: enables two-phase learning (SFT then RL), with either in-model or external reward/value heads

Systems such as x1 implement these abstractions as swappable Python APIs, supporting MCTS, beam search, trace annotation, and incremental operator design. Notably, rSIM introduces a Stackelberg multi-agent setup—external planner selects step-wise strategies (e.g., self-reflection, decomposition), steering an LLM “follower,” enabling zero-to-RLM uplift for small models, with continual planner learning and plug-in reuse (Chen et al., 9 Dec 2025).

The value function, critical for search and credit assignment, is most reliably implemented as a dedicated external model, trained with outcome- and process-based supervision.

6. Open Challenges, Controversies, and Future Directions

Despite substantial progress, open questions persist:

• Credit assignment and reward hacking: Assigning precise value to early rationale steps is unresolved; policies can exploit reward models via verbosity or circular logic (Liu et al., 2 Oct 2025).
• Data and annotation quality: Reliance on LLM-generated CoTs and automated annotation incurs noise and risks distribution shift; decontamination and explicit benchmarking are weakly enforced (Zhang et al., 1 May 2025).
• Generality and OOD transfer: Test-time scaling and reasoning structure transfer across domains and languages remains inconsistent; explicit separate modules, hybrid symbolic integration, or meta-learning are underexplored (Raganato et al., 1 May 2025, Lin et al., 29 Oct 2025).
• Compute and efficiency: Long token chains and tree search introduce 10–100× inference cost; efficient early-exit, suppression, and selective reasoning are active areas (Zheng, 29 Sep 2025).
• Multimodal and multilingual reasoning: Current RLMs struggle in low-resource scripts, multimodal inputs, and non-STEM domains, with systematic collapse toward English reasoning and instability under even simple input variation (Park et al., 6 Jun 2025, Wang et al., 20 May 2025).
• Safety and alignment: Lengthy, system-2-like reasoning traces are vulnerable to adversarial or unsafe logic; process-level alignment remains an open problem (Zhang et al., 1 May 2025).

Emerging directions include hybrid symbolic–neural modules for logic, automated process-level reward discovery, meta-search for discoverable strategy sets, continual curriculum-based training, and scaling context beyond million-token windows for document-level reasoning and planning.

7. Practical System Design and Engineering Insights

Best practices include:

• Data curation: Difficulty-aware, decontaminated chain-of-thought datasets with rigorous verification yield stronger generalization, even at small scales (Zhang et al., 1 May 2025).
• Stepwise supervision: Process-based supervision and reverse curricula stabilize RL and mitigate sparse-reward pathologies, especially in small models (Le et al., 3 Apr 2025, Xi et al., 8 Feb 2024).
• Modular implementation: Separating structure, search, operator, and reward/value models allows rapid experimentation and extension to new domains (Besta et al., 20 Jan 2025).
• Test-time optimization: Adaptive suppression, budget scaling, and dynamic reasoning depth trade accuracy and cost flexibly, suiting deployment constraints (Zheng, 29 Sep 2025).
• Prompt engineering: Minimal or zero-shot setups encourage exploration, while verbose exemplars or rationale-before-answer formats can degrade performance (Xiong et al., 20 May 2025, Raganato et al., 1 May 2025).
• Multilingual processing: Forcing reasoning in high-resource scripts or selective translation on understanding failures provides the best trade-off among accuracy, efficiency, and language fidelity (Wang et al., 20 May 2025, Kang et al., 31 Oct 2025).

The RLM field continues to evolve toward modular, efficient, robust, and interpretable reasoning systems, blending neural generation, explicit search, and modular control for broad, deep, and explainable AI problem-solving (Besta et al., 20 Jan 2025).