ReflectEvo Framework

• ReflectEvo is a self-improving reflection learning framework that uses an iterative generate–reflect–learn pipeline to enhance small language models’ reasoning.
• It leverages a large-scale auto-curated reflection dataset with diverse prompt templates and applies both supervised fine-tuning and direct preference optimization.
• Empirical results show significant accuracy gains on multiple reasoning benchmarks, demonstrating continual autonomous improvement without external supervision.

ReflectEvo is a self-improving reflection learning framework designed to enhance the meta introspection and reasoning ability of small LLMs (SLMs) based on iterative self-reflection and self-correction cycles. The approach eliminates reliance on large teacher models or fine-grained human annotations by leveraging a generate–reflect–learn pipeline, a large-scale auto-curated reflection dataset, and two distinct optimization paradigms—supervised fine-tuning (SFT) and Direct Preference Optimization (DPO). Empirical results demonstrate substantial improvements in SLMs' reasoning benchmarks, rivaling or surpassing open-source large model baselines, with evidence for continual, autonomous reasoning enhancement without external supervision (Li et al., 22 May 2025).

1. Iterative Generate–Reflect–Learn Pipeline

The ReflectEvo pipeline operationalizes an iterative self-reflection loop. For each example consisting of a query ($q$) and ground-truth answer ($a^*$), a generator ($G$) and a reflector ($R$) share model parameters $\theta$ initialized from a pretrained SLM. At iteration $t$:

• $G(a|q;\theta^{(t)})$ generates an initial solution $a$.
• A binary verifier $f(a,q,a^*) \rightarrow \{$correct, incorrect$\}$ assesses correctness.
• If $f$ is incorrect:
  • $R$ generates a self-reflection $r = R(r|q,a,f;\theta^{(t)})$.
  • $R$ generates a corrected answer $\bar{a} = R(\bar{a}|q,a,f,r;\theta^{(t)})$.

This process yields tuples $(q,a,f,r,\bar{a})$. After data curation, the model updates $\theta$ by training on these reflection-augmented instances. The cycle is repeated for $T$ iterations. The parameter update, ReflectEvoUpdate, is realized by SFT or DPO.

Pseudocode (excerpt):

for t = 0...T-1: for q in Q: a ~ G(a|q;θ^(t)) f = verify(a,q,a*) if f == incorrect: for m in 1...M variants: r_m ~ R(r|q,a,f;θ^(t)) ā_m ~ R(ā|q,a,f,r_m;θ^(t)) D.append((q,a,f,r_m,ā_m)) θ^(t+1) = ReflectEvoUpdate(θ^(t), D)

This mechanism introduces an autonomous self-evolving process where each iteration leverages local "gradients" in the form of failure-focused textual feedback (Li et al., 22 May 2025).

2. Construction of the ReflectEvo-460k Dataset

ReflectEvo-460k is a reflection dataset comprising approximately 461,799 tuples generated via automated reflection over 17 benchmarks spanning 10 reasoning and QA tasks. The benchmarks include logical reasoning, mathematics, coding, contextual and context-free QA, reading comprehension, commonsense, social, causal, and physics reasoning.

Instruction Pool

Reflection prompts are factorized across three stages:

1. Verification: With or without stepwise trace.
2. Error Localization/Diagnosis: Targeting math, logic, rationale, inconsistency, misinterpretation, format, and factual errors (eight options).
3. Correction Planning: High-level or low-level strategies.

Combining these yields 32 distinct reflection prompt templates. This structured prompt diversity is critical for stimulating varied and informative self-critique dynamics.

Generator and Reflector

• $G$ and $R$ are instantiated from the same SLM checkpoint (e.g., Llama-3-8B, Mistral-7B, Gemma-2-9B).
• $G$ follows the ReAct scheme, producing chain-of-thought solutions.
• For each $(q,a,f=\text{incorrect})$, $R$ samples $k=2$ reflection/correction pairs per prompt (via rejection sampling), ensuring diversity.

Data Curation

Three primary sets are derived:

• Positive Set $D^+$: Only $(q,a,f,r,\bar{a})$ with $\bar{a}=a^*$ retained.
• Preference Set $D^{\text{pref}}$: For each $q$, GPT-4o selects a preferred reflection-correction pair, providing a ranked (preferred/rejected) annotation.
• Pairwise Set $D^{\pm}$: For each $q$, one "positive" and one "negative" ($\bar{a}\neq a^*$) correction are randomly sampled.

Summary Table: ReflectEvo-460k Dataset

Attribute	Value	Details
Total Samples	461,799	17 benchmarks, 10 reasoning paradigms
Avg. Reflection Length	~250 tokens
Reflection Prompts	32	2 (stage 1) × 8 (stage 2) × 2 (stage 3)

This comprehensive corpus underpins subsequent reflection training and ablation analyses (Li et al., 22 May 2025).

3. Reflection Learning: SFT and DPO Variants

ReflectEvo supports two principal optimization strategies for reflection learning, applied over $D^+$, $D^{\pm}$, and $D^{\text{pref}}$.

3.1 Supervised Fine-Tuning (SFT)

• One-Stage (joint):

$$\mathcal{L}_1(\theta) = - \mathbb{E}_{(q,a,f,r,\bar{a}) \sim D^+ }\left[ \log R((r, \bar{a}) | q,a,f;\theta) \right]$$

• Two-Stage (decoupled):

$$\mathcal{L}_{2.1}(\theta) = - \mathbb{E}_{(q,a,f,r)\sim D^+} [\log R(r|q,a,f;\theta)]$$

$$\mathcal{L}_{2.2}(\theta) = - \mathbb{E}_{(q,a,f,r,\bar{a})\sim D^+} [\log R(\bar{a}|q,a,f,r;\theta)]$$

3.2 Direct Preference Optimization (DPO)

• On $D^\pm$:

$$\mathcal{L}_3(\theta) = - \mathbb{E}_{(x, r^+, r^-) \sim D^\pm} [ \log \sigma(r_\theta(x, r^+) - r_\theta(x, r^-)) ]$$

• On $D^{\text{pref}}$:

$$\mathcal{L}_4(\theta) = - \mathbb{E}_{(x, r^{\text{cho}}, r^{\text{rej}} ) \sim D^{\text{pref}} } [ \log \sigma(r_\theta(x, r^{\text{cho}}) - r_\theta(x, r^{\text{rej}})) ]$$

with

$$r_\theta(x, r) = \beta \log \left( \frac{ \pi_\theta(r|x) }{ \pi_{\text{ref}}(r|x) } \right)$$

where $\pi_{\text{ref}}$ is the pre-update policy and $\beta>0$ a scaling constant.

DPO systematically ranks and prefers higher-quality reflections, allowing for fine-grained reflection improvement in the absence of gold-standard critique.

4. Empirical Results and Performance Gains

ReflectEvo demonstrates significant accuracy improvements across multiple SLMs and benchmarks:

Model	Big-Bench Prompt-Only	Big-Bench ReflectEvo SFT	Absolute Gain
Llama-3-8B	52.4%	71.2%	+18.8%
Mistral-7B	43.8%	71.1%	+27.3%

Additional gains:

• LogiQA: 30% $\rightarrow$ 50%
• MATH: 15% $\rightarrow$ 25%
• MBPP: 44% $\rightarrow$ 63%

Multiturn (up to 6) reflection cycles can further elevate Big-Bench performance above 80%. Comparative studies indicate that ReflectEvo outperforms methods such as STaR, Re-ReST, and RISE in accuracy deltas between initial and corrected answers (Li et al., 22 May 2025).

5. Analyses of Reflection Data and Error Correction

Error-Type Distribution

• Logic/reasoning errors: 88.4% prevalence in reflections.
• Instruction violations: 47.9%.
• Math calculation errors: ~20.8% on MATH tasks.

Reflection–Thought Correlation

Pearson correlation between the reflection and the subsequent corrected chain-of-thought is a strong predictor of accuracy improvement on tasks with complex reasoning demands (e.g., StrategyQA, Social IQA). For computation-intensive tasks (MATH, MBPP), correlations are weaker, highlighting differential reflection efficacy across domains.

Ablation Comparisons

• GPT-4o-generated reflections outperform SLM self-reflections; however, self-generated reflections still yield +9–19% improvements.
• Ablative results demonstrate the core signal is retained even without external oracles.

Qualitative Function

Reflections serve to:

• Localize reasoning faults
• Articulate specific correction plans
• Bias training towards effective error-correcting behavior

This evidences a form of meta-introspection resembling a textual analog of supervised gradient correction without human or model-based critique.

6. Autonomous Iteration and Continual Self-Improvement

ReflectEvo's reflection loop constitutes a self-bootstrap process. Each reflection identifies local modes of failure and specifies corrections, acting as an iterative feedback signal. Training on $D^+$ encourages the model to prefer corrective behaviors; DPO further biases towards superior reflections.

The sole reliance on binary correctness feedback ($f$) eradicates dependence on expert-labeled critique or large teacher models. Iterating this loop allows SLMs to seed increasingly accurate reasoning chains, forming a foundation for continual autonomous improvement and meta-introspective capability.

A human learning analogy: the SLM repeatedly attempts a problem, scrutinizes its own erroneous logic, identifies flaws, plans remediation, and retrains—progressively improving via self-directed error analysis and correction (Li et al., 22 May 2025).