Self-Improvement for Multi-Step Reasoning

Updated 14 December 2025

Self-improvement for multi-step reasoning is a framework that enables LLMs to iteratively refine their solution processes using internal feedback and self-correction.
This paradigm integrates in-context self-reflection, trajectory optimization, and reinforcement learning to systematically address errors and enhance reasoning robustness.
Empirical studies show that these techniques yield measurable gains in accuracy, reduced redundancy, and improved scalability for complex multi-step tasks.

Self-improvement for multi-step reasoning refers to a comprehensive class of methodologies that enable LLMs and related agents to iteratively refine, correct, or optimize their reasoning chains—either at inference time or during (self-)supervised or reinforcement learning—by leveraging various forms of feedback, evaluation, or search over intermediate steps. This paradigm encompasses techniques operating at the prompt, trajectory, and parameter levels, spanning in-context self-reflection, external search and optimization, and RL-based policy improvement. Systematic self-improvement is foundational to closing the gap between static chain-of-thought (CoT) prompting and robust, generalizable multi-step reasoning, and has been instantiated in a diverse array of algorithmic frameworks in LLM and agent research.

1. Conceptual Foundations and Definitions

Self-improvement in the context of multi-step reasoning is defined as the process by which an LLM or agent autonomously iterates on its own solutions, using feedback—either from explicit evaluators, internal consistency checks, or external environment signals—to update, re-sample, or revise intermediate reasoning steps and ultimately enhance both correctness and efficiency. Rather than generating a single, immutable reasoning chain, self-improving systems enact closed feedback loops where output is re-evaluated or re-entered, enabling error correction and trajectory optimization (Plaat et al., 16 Jul 2024, Wei et al., 20 Feb 2025). This can manifest at three principal loci:

In-Context Self-Reflection: The model inspects and critiques its own reasoning steps at inference, triggering step repair or regeneration if low confidence or inconsistency is detected.
Optimization of Solution Trajectories: External or model-internal mechanisms (e.g., beam search, tree search, evolutionary operators) explore alternative step sequences, leveraging evaluative heuristics or process reward models to select and refine better chains.
Policy/Parameter Update via Reinforcement or Imitation Learning: Self-generated or externally labeled feedback is used to directly update model parameters, shaping the policy toward higher-quality multi-step reasoning over time.

Self-improvement is unified by these feedback-driven, iterative dynamics, regardless of the specific underlying algorithm, be it prompt-based, search-based, or RL-based.

2. Taxonomy of Approaches

A wide range of self-improvement frameworks has been developed and are classified according to the locus of control, type of feedback, and whether improvement occurs at inference or through parameter updates (Plaat et al., 16 Jul 2024, Wei et al., 20 Feb 2025). Broadly, the field distinguishes:

Approach Family	Principal Mechanism	Example Instantiations
Self-Reflection/Step Critique	In-context dissassembly & regeneration	MAPS, Self-Notes, Self-Evaluation Guided BS
External Optimization Loops	Search over trajectories, voting, repair	Stepwise Informativeness Search, SE-Agent
RL-based Parameter Update	Step/outcome reward, policy gradient	AutoPRM, OREO, ExIt, ReST meets ReAct
Process Supervision/Verification	Step-wise annotations/verifier signals	PRM+PPO, Tool-augmented verification
Self-Synthesizing Data	Model generates and trains on its own outputs	ReGenesis, SRLM

Each category leverages distinct algorithms but shares the self-improvement loop: evaluation of prior outputs → revised sampling/generation/updating → performance gain.

3. Stepwise Evaluation, Critique, and Repair

A central thread in self-improvement for multi-step reasoning is explicit stepwise evaluation of intermediate steps, as opposed to treating only the final answer as the locus of feedback. This paradigm is embodied in several prominent methods:

Stepwise Informativeness Search (Wang et al., 21 Feb 2025): Inference-time beam search integrates two selection heuristics—grounding-guided (leveraging attention to underutilized prior steps) and novelty-guided (rewarding steps with novel content compared to previous conclusions). Self-grounding prompts reinforce explicit referencing of premises, reducing redundancy and error propagation. Empirical validation shows ∼5% absolute accuracy gain and a 30–40% drop in rationale redundancy versus vanilla CoT.
MAPS (Multi-Layered Self-Reflection with Auto-Prompting) (Loureiro et al., 30 Jun 2025): Iteratively generates a base CoT rationale, detects errors (logical, arithmetic, misreading), synthesizes a tailored meta-prompt reflecting on the diagnosis, and regroups with targeted re-answering. Multi-layered reflection—typically up to three—yields up to +13% accuracy over single-pass CoT on GSM8K, and cost-performance can be tuned by limiting reflection depth.
Self-Notes and Interleaved Explicit Reasoning (Lanchantin et al., 2023): Models can interleave “Self-Notes” directly into the input context, acting as explicit, persistent memory units that reduce reasoning distance and anchor subsequent steps. Outperforms traditional scratchpad and CoT on several multi-hop reasoning and algorithmic tasks.
Self-Evaluating LLMs (Stepwise Confidence Estimation) (Mavi et al., 10 Nov 2025): Models estimate confidence for each reasoning step, with stepwise aggregation outperforming holistic scoring (up to +15% AUC-ROC) in failure detection and enabling targeted correction loops that locally patch problematic reasoning links.

The shared insight is that localized, step-level critiques—rather than end-to-end single-shot evaluation—enable both finer-grained diagnosticity and more targeted correction operations, significantly improving both accuracy and robustness.

4. Trajectory Optimization via External Search and Evolution

Self-improvement is also realized via explicit optimization of solution trajectories, often using search, selection, and recombination techniques:

Stepwise Informativeness Search (Wang et al., 21 Feb 2025) and Self-Evaluation Guided Beam Search (Xie et al., 2023) guide multi-beam search by custom scoring functions evaluating informativeness and/or self-evaluated correctness of partial chains.
SE-Agent (Self-Evolution Trajectory Optimization) (Lin et al., 4 Aug 2025): Introduces revision (self-reflective modification of trajectories), recombination (cross-trajectory inspiration via crossover, transfer, or restructuring), and refinement (fitness- and diversity-based selection), iteratively evolving a pool of solution paths for multi-step, tool-using agents. Yields 30–110% relative accuracy gain over baseline MCTS/code agents on SWE-bench.
Exploratory Iteration (ExIt) (Jiang et al., 4 Sep 2025): RL-based self-improvement policies grow a task space by prioritizing the most informative partial solution trajectories as new starting points, enabling inference-time improvement beyond the average training iteration depth.

These methods are unified by the use of search or population-based approaches that optimize solution objects (reasoning chains, agent trajectories) rather than single-step outputs, leveraging both reward signals and structural diversity for robust self-improvement.

5. RL-Based and Data-Driven Self-Improvement Frameworks

Self-improving multi-step reasoners are often cast as Markov Decision Processes with sparse reward, and advanced reinforcement learning algorithms are employed to assign credit/blame across steps and optimize policies:

AutoPRM (Chen et al., 18 Feb 2024): Models jointly decompose problems into controllable subquestion sequences and use a stepwise verifier as a reward signal for reinforcement learning. Context-guided decoding ensures coherence across decompositions, yielding significant improvements in mathematical and commonsense reasoning.
OREO (Offline Reasoning Optimization) (Wang et al., 20 Dec 2024): Models are trained via entropy-regularized, soft Bellman consistency at each step, outperforming DPO, SFT, and baseline rejection models (e.g., +3–5% on GSM8K/MATH; up to +17.9% with value-guided beam search).
ReST meets ReAct (Aksitov et al., 2023): Agentic LLM interaction with external tools is optimized by a ReST-style “grow and improve” strategy, using LLM-based AI feedback and iterative fine-tuning, producing compact student models approaching large teacher model performance in compositional multi-hop QA.
SRLM (Wang et al., 20 May 2025): Iterative self-training using a small set of human/meta-reasoner-generated “reasoning catalyst” demonstrations amplifies meta-reasoning skills, with best-of-N sampling further unlocking deeper solution chains.
ReGenesis (Peng et al., 3 Oct 2024): Models self-synthesize reasoning paths by scaffolded, three-stage abstract-to-concrete generation—adapting abstract reasoning guidelines to task-specific outlines to full paths—enhancing both in-domain and OOD performance (+6.1% OOD vs. –4.6% for single-distribution self-training approaches).

Data-driven self-synthesis and RL loops converge on a strategy of iterated improvement where each round of generation, reflection, and distillation incrementally raises both correctness and generalization capacity, especially in settings lacking dense human step-level supervision.

6. Feedback, Process and Outcome Rewards, and Verification

Integration of feedback is the pivotal mechanism by which self-improvement is realized (Wei et al., 20 Feb 2025):

Process-Reward Models: Annotate or infer step-level rewards and use policy gradients (notably PPO) to train the model to optimize R_process. This fine granularity enables highly guided trajectory refinement but incurs significant annotation and training cost.
Outcome-Level Rewards: Rely on final answer correctness only, using discriminative or generative verifiers to select among candidate solutions. These approaches are annotation-efficient and synergize with methods like self-consistency sampling.
Training-Free / Tool-Augmented Methods: Include self-evaluation (prompted True/False queries per step), logit-confidence heuristics, and external tool checking (e.g., code execution, theorem proving). No parameter updates are required, and these methods function on frozen models, trading higher inference cost for zero extra training.

Table: Feedback Modalities and Representative Algorithms

Feedback Modality	Example Algorithms	Main Trade-off
Step-wise (process)	PRM+PPO, AutoPRM, MAPS, OREO	High annotation, high granularity
Outcome-level	Outcome-RL, Self-Consistency, Verifier filtering	Low annotation, limited guidance
Training-free	Self-Notes, Self-Polish, Self-Eval BS, Map-SR	No training, high inference cost

Selecting among feedback modalities involves careful consideration of task complexity, annotation budget, and the need for intermediate correction.

7. Empirical Gains, Limitations, and Future Directions

Across arithmetic, logical, and multi-turn reasoning benchmarks, self-improvement methods yield substantial accuracy and robustness gains. Notably:

Stepwise informativeness search with self-grounding achieves up to 5% accuracy gain and reduces reasoning length and redundancy by 30–40% (Wang et al., 21 Feb 2025).
Early-stopping self-consistency sharply reduces CoT sampling cost with negligible accuracy change (Li et al., 19 Jan 2024).
Multi-layer MAPS reflection delivers up to 13% accuracy increase over vanilla CoT on GSM8K (Loureiro et al., 30 Jun 2025).
RL/MCTS/evolutionary and trajectory-based approaches (SE-Agent, ExIt) consistently outperform static or non-iterative optimization baselines by 2–30% or more (Lin et al., 4 Aug 2025, Jiang et al., 4 Sep 2025).
Process-reward and verifier models attain ≥88% on GSM8K and ≥35% on MATH, outperforming SFT and outcome-only reward setups (Wei et al., 20 Feb 2025).

Limitations center on annotation and compute cost (especially for process-level methods), potential for reward gaming or distribution drift, scalability to truly open-ended or multi-modal domains, and the challenge of diagnosing and correcting deeply embedded errors in very long reasoning chains. For practical deployment, balancing inference cost (e.g., beam or sample count, reflection depth), robustness, and generalizability remains a key engineering and research challenge.

Open problems include mechanistically faithful self-reflection, scalable in-context RL, adaptive stopping and difficulty estimation, efficient cross-domain transfer, and minimizing hallucination or reward misalignment in self-improvement loops.

In summary, self-improvement for multi-step reasoning unifies a spectrum of algorithms that iteratively optimize multi-step solution paths by integrating explicit feedback, dynamic search, and learning from self-generated or critique-driven supervision. The core pillars are step-level evaluation/repair, trajectory optimization, RL-based learning, and judicious use of feedback modalities—all driving toward more robust, efficient, and generalizable multi-step reasoners, as exemplified by recent advances and surveyed frameworks (Plaat et al., 16 Jul 2024, Wei et al., 20 Feb 2025, Wang et al., 21 Feb 2025, Loureiro et al., 30 Jun 2025, Lin et al., 4 Aug 2025, Peng et al., 3 Oct 2024).