ReaLM: Enhanced Autonomous Reasoning for SLMs

• ReaLM is a reinforcement learning framework that enhances small language models' reasoning, autonomy, and generalization by integrating multi-route process verification, asymptotic induction, and expert-guided chain-of-thought distillation.
• It employs multi-route process verification to learn from both positive and negative reasoning paths, thereby mitigating biases and brittle inference in multi-step tasks.
• The framework phases out external chains-of-thought via asymptotic induction, enabling models to become fully autonomous and perform well across both in-domain and out-of-domain tasks.

ReaLM (Reflection-Enhanced Autonomous Reasoning with Small LLMs) is a reinforcement learning framework designed to maximize the reasoning capabilities, autonomy, and domain generalization of Small LLMs (SLMs, typically ≤7B parameters) for complex multi-step tasks. The framework addresses issues inherent in prior work—including reasoning brittleness due to biased supervision, excessive reliance on inference-time external guidance, and poor out-of-distribution generalization driven by teacher-dataset biases—by introducing three integrated methodological advances: Multi-Route Process Verification (MRPV), Enabling Autonomy via Asymptotic Induction (EAAI), and guided chain-of-thought distillation of expert rules (Xu et al., 17 Aug 2025).

1. Motivation and Architectural Overview

ReaLM targets the triad of reasoning quality, autonomy, and generalization in SLMs, addressing limitations in co-training and distillation approaches which either filter out negative reasoning paths (limiting robustness), require continual access to external chains-of-thought (CoTs) at inference (limiting autonomy), or encode teacher-specific biases during instruction-based data curation. The core pipeline unifies three components:

• MRPV: Learning from both positive and negative CoTs by contrastive process verification.
• EAAI: Scheduling the fading of external CoTs via cosine-decay sampling, so SLMs become fully autonomous at inference time.
• Expert-guided CoT distillation: Integration of domain-specific rules and annotated rationales into the SLM parameterization for vertical and cross-domain generalization.

ReaLM supports purely RL-based (ReaLM-Zero) and hybrid SFT+RL (ReaLM-R1) recipes.

2. Multi-Route Process Verification (MRPV)

MRPV explicitly incorporates both correct and erroneous reasoning paths (positive and negative CoTs) into the training context. For each reasoning task (e.g., multi-step math or semantic matching), a teacher LLM (e.g., GPT-4-turbo, T=1.5) generates N diverse CoTs without filtering out failure cases. From these, k CoTs are selected per EAAI (see §3) and prepended to the target question. The SLM evaluates the utility of each reference CoT and produces its own reasoning chain and answer.

The reward function is two-stage:

• Stage 1: Answer-based reward, $r_{\mathrm{base}} = 1$ if both answer and format match the ground truth, $0.1$ for format-only, $0$ otherwise.
• Stage 2: Process reward, $$r_{\mathrm{gain}} = \mathrm{scale}_{\mathrm{gain}}\frac{1}{k}\sum_{i=1}^k s_i$$ if $r_{\mathrm{base}}=1$, zero otherwise, where $s_i$ is the SLM’s classification of each reference CoT as leading to the correct answer.

This contrastive approach enables the SLM to learn decisive patterns from inaccurate reasoning trajectories as well as correct ones, robustifying error signals and preventing brittle reasoning. The expected reward strictly dominates that of models which only reference teacher-generated CoTs, as shown by a decomposition $$\mathbb{E}_{\mathrm{CoT}}[R(C_{\mathrm{out}})] = \mathbb{E}[R(\sum_{i=1}^k w_i\,C_i)] + \Delta_{\mathrm{gen}}$$, with all $w_i \ge 0$ (Xu et al., 17 Aug 2025).

3. Enabling Autonomy via Asymptotic Induction (EAAI)

EAAI ensures that SLMs eventually operate without support from externally supplied CoTs. At each training step $t$ (for $t=0\,..\,T$), each candidate CoT is sampled into the prompt with probability $0.1$0, so that as $0.1$1, $0.1$2. This results in a seamless decay from multi-CoT, teacher-driven contexts to fully autonomous SLM reasoning at inference.

As CoTs are phased out, the MRPV process reward is discontinued, and only the answer-based reward $0.1$3 remains, stabilizing the RL reward landscape. This design prevents reward sparsity in late training and aligns the training/inference mismatch, yielding self-sufficient SLMs.

4. Guided Chain-of-Thought Distillation

For vertical (domain-specific) reasoning tasks, such as industrial ad search relevance, ReaLM introduces an expert-annotated dataset $0.1$4, where “comment” encodes domain rationale. Teacher LLMs are prompted to generate CoTs augmented by these comments (with probability $0.1$5), incorporating explicit rule pointers and business logic.

SLMs are then supervised to generate these rule-aware CoTs using cross-entropy minimization

$0.1$6

where $0.1$7 is the reference CoT. This distillation objective is linearly combined with the RL objective: $0.1$8 allowing ReaLM to encode both high-level domain logic and RL-optimized reasoning capabilities.

5. Reinforcement Learning Framework and Training Protocol

ReaLM employs a KL-regularized policy gradient (based on GRPO), where policy $0.1$9 is updated to maximize MRPV reward while remaining close to initialization $0$0. The full loss is: $0$1 In ReaLM-R1, an initial SFT is performed on rule-aware CoTs (Eq. 5 above), followed by RL fine-tuning with fading CoT context.

Training uses teacher LLMs (GPT-4-turbo, $0$2 CoTs/sample), batch size 160, KL coefficient $0$3, actor learning rate $0$4, on 8 A100 GPUs. Metrics include accuracy and F1 (multi-class, vertical task), with up to 1k maximum output tokens.

6. Empirical Performance and Ablations

ReaLM demonstrates marked improvements across both general and vertical reasoning tasks. For DeepSeek-R1-7B:

• On GSM8K (math, in-domain): RL baseline (no CoTs) 84.2%; ReaLM (with MRPV+EAAI, no CoTs) 87.1%.
• On GSM8KPlus (math, out-of-domain): RL baseline 73.5%; ReaLM 75.8%.
• With teacher CoTs at inference, ReaLM-w/o EAAI outperforms CoT-Synthesizer by +4.2% accuracy on average.

On a private ad search relevance task with expert rationale, ReaLM-R1 yields highest accuracy (82.61%) and F1 (82.45%), outperforming strong CoT distillation and incremental symbolic knowledge baselines.

Ablation studies confirm that:

• MRPV (vs. route-only reward) provides +0.6% (ID) / +1.7% (OOD) gains.
• EAAI contributes a distinct +1.8% (ID) / +0.5% (OOD) increase.
• The full pipeline (MRPV + EAAI) exceeds RL-only by 2.9–2.3% on GSM8K, and error analysis reveals a lower fraction of errors matching teacher LLM mistakes.

7. Broader Implications, Limitations, and Extensions

ReaLM demonstrates that contrastive process verification (learning from negative as well as positive reasoning), autonomous curriculum induction, and explicit expert-guided distillation are all substantial levers for SLM reasoning robustness and transfer. The methodology reduces the model’s dependence on inference-time augmentation (e.g., externally supplied CoTs) without sacrificing performance, while increasing generalization across both academic and proprietary vertical tasks.

Limitations include dependency on the quality/diversity of sampled teacher CoTs (garbage-in/garbage-out) and the need for domain-expert rationale in vertical tasks. Future work could couple ReaLM with open-ended reasoning benchmarks and automated generation of counterfactual or adversarial rationales, and extend the approach to larger model classes.

---

Key Reference:

• “ReaLM: Reflection-Enhanced Autonomous Reasoning with Small LLMs” (Xu et al., 17 Aug 2025)