Reflection-aware AEPO

• Reflection-aware AEPO is a reinforcement learning framework that integrates explicit self-reflection steps and adaptive entropy control to curb echoing errors in LLMs and multimodal models.
• It employs information bottleneck regulation and stage-calibrated entropy shaping to ensure diverse, corrigible, and robust reasoning outcomes during multi-stage inference.
• Empirical results demonstrate significant gains in answer accuracy and reflection quality, validating its effectiveness across tasks like dialogue generation and multimodal reasoning.

Reflection-aware Adaptive Entropy Policy Optimization (AEPO) refers to a class of reinforcement learning algorithms for LLMs and multimodal LLMs that incorporate explicit self-reflection steps during multi-stage inference, with intrinsic mechanisms to control information flow and encourage cognitively meaningful exploratory reasoning. These methods target the systemic shortcomings of standard chain-of-thought (CoT) prompting—most notably, the phenomenon of "Echo Reflection," where model self-reflection reiterates, rather than substantively refines, earlier erroneous reasoning. AEPO schemes quantitatively regulate reflection at policy and reward levels, using information bottleneck techniques and stage-wise entropy shaping, to produce robust, corrigible, and insightful inferences (He et al., 9 Nov 2025). In recent years, this paradigm has been extended and specialized for various modalities and interaction settings, including end-to-end spoken dialogue and empathy-aware conversational AI (Jia et al., 26 Jan 2026), as well as for multimodal reasoning via reflection-calibrated policy optimization (Wan et al., 2 Jun 2025).

1. The Echo Reflection Problem and Motivating Principles

LLMs, and multimodal analogues, have achieved high performance on complex reasoning tasks using multi-stage inference: Thinking → Draft → Reflection → Answer. However, detailed analyses reveal that, especially outside strict mathematical or symbolic domains, the Reflection stage often "echoes" or recapitulates errors from earlier reasoning rather than effecting genuine correction or insight (He et al., 9 Nov 2025). This is attributed to (1) uncontrollable information flow from intermediate answers, which entrenches spurious content, and (2) premature entropy collapse in the reflection policy, leading to inadequate exploration of new reasoning paths. The core guiding principles of reflection-aware AEPO are thus:

• Information Bottleneck (IB) Regulation: Constrain and filter the transfer of cognitive information between inference stages to prevent error propagation.
• Adaptive Policy Entropy Control: Maintain non-collapsing, stage-calibrated entropy to incentivize diverse, corrigible reflective outputs during self-reflection, without sacrificing answer accuracy.

2. Mathematical Formulation of Reflection-aware AEPO

The canonical AEPO framework is structured around the following notation and loss components (He et al., 9 Nov 2025):

Let $Q$ be the question, $T$ the initial thinking output, $R$ the Reflection-stage output, and $L$ the ground-truth answer. The "reflection-aware information filtration" loss is given by:

$\mathcal{L}_{IB} = I(X;R) - \beta\,I(R;L)$

where $X = [Q,T]$, $I(\cdot\,;\,\cdot)$ denotes mutual information, and $\beta$ balances bottleneck tightness against answer relevance.

Because direct MI computation is impractical in LLMs, proxy metrics are introduced:

• Average Reflection Entropy $\bar{\mathcal{H}}_R$: Lower values signal higher dependence on $T$, implying unwanted echoing.
• Contribution Indicator $T$0: Quantifies the causal impact of reflection on the correctness of the final answer.

The surrogate reflection bottleneck loss becomes: $T$1

Adaptive-entropy optimization (AEO) augments this with stage-dependent intrinsic rewards:

$T$2

with $T$3 as empirically optimal entropy. The entropy reward is gated by final answer correctness:

$T$4

The full per-trajectory reflection-aware AEPO loss used with PPO-style policy gradients is:

$T$5

3. Alternating and Interleaved Reflective Inference

Reflection-aware AEPO algorithms are closely connected to alternating-inference architectures, such as ReEmpathy for empathetic spoken dialogue (Jia et al., 26 Jan 2026). These systems interleave explicit self-reflection steps with response generation in a multi-round loop, formalized as:

$T$6

where $T$7 are spoken/textual response chunks and $T$8 are free-form internal reflection traces. Alternation is maintained under end-to-end learning objectives with both likelihood-based and continuous empathy score regressions.

Such mechanisms ensure that every candidate response chunk is explicitly conditioned on the model's most recent internal self-assessment, enabling fine-grained self-correction and affective alignment (Jia et al., 26 Jan 2026). A plausible implication is that this approach generalizes reflection-aware AEPO's error-containment and corrigibility goals beyond classical QA to high-dimensional, open-ended dialogues where "correctness" is non-binary and nuanced.

4. Policy Optimization and Reward Design

AEPO and related frameworks (e.g., Group Relative Policy Optimization (GRPO) in multimodal RL (Wan et al., 2 Jun 2025)) employ policy gradient methods with reflection-calibrated reward schemes. Key reward components include:

• Task Reward: Evaluates answer formatting and task-denoted accuracy, ensuring that base reasoning remains anchored to objective criteria.
• Reflection Reward: Measures error correction, brevity, and formatting in the reflection. For instance,
  • $T$9: Reflection preserves ($R$0) or corrects a wrong answer ($R$1), or, if erroneous, penalizes by $R$2.
  • $R$3: Smoothly rewards brevity, peaking at a specified reflection length target.
• Group-normalized Advantage Estimation: In GRPO, per-group intra-batch normalization of advantage signals replaces explicit value functions, providing stability and encouraging exploratory diversity among sampled reflection trajectories (Wan et al., 2 Jun 2025).

For end-to-end dialogue models, the reward landscape is augmented by natural-language diagnostic traces and continuous evaluation scores (e.g., $R$4 for empathy metrics) computed from learned evaluation heads (Jia et al., 26 Jan 2026).

5. Empirical Performance and Ablation Evidence

Reflection-aware AEPO has demonstrated consistent improvement over standard RL baselines:

Benchmark	Base LLM / MLLM	+AEPO/SRPO Performance	SOTA Baseline
MedQA / MedMcQA	LLaMA-8B	68.5%	58.7%
	Qwen2.5-7B	62.5%	57.0%
MathVista (SRPO-7B)	Qwen-VL-7B	75.8%	68.2%
MathVision (SRPO-7B)	Qwen-VL-7B	32.9%	25.1%

Ablation studies indicate that:

• Removing reflection reward in RL nearly eliminates accuracy gains (e.g., from 75.8% to 70.3% on MathVista (Wan et al., 2 Jun 2025)).
• Disabling entropy shaping or IB constraints results in entropy collapse and echo reflection phenomena.
• Cold-start SFT with reflection-rich supervision accelerates RL convergence and overall reflection quality.
• Continuous empathy scoring and descriptive evaluation traces in spoken dialogue are essential for fine-grained adjustment of response-reflection alternation (Jia et al., 26 Jan 2026, He et al., 9 Nov 2025).

6. Theoretical Insights and Broader Implications

Information bottleneck principles in reflection-aware AEPO ensure that the reflection output $R$5 encodes only the minimal sufficient statistic of $R$6 necessary for correct answer prediction $R$7, thereby directly suppressing the propagation of spurious intermediate errors (He et al., 9 Nov 2025). Adaptive entropy shaping provably maintains an optimal balance between exploration and exploitation across multi-stage reasoning, avoiding both entropy collapse (premature convergence to a single trajectory) and entropy explosion (loss of focus).

A plausible implication is that reflection-aware AEPO provides a principled, extensible template for complex, corrigible, and trustworthy AI reasoning. The same entropy- and bottleneck-inspired criteria readily adapt to multimodal and dialogic generative pipelines, as validated in empathy-aware SLMs (Jia et al., 26 Jan 2026) and visio-linguistic CoT models (Wan et al., 2 Jun 2025).

7. Specialized Architectures and Practical Implementation

Recent advances have layered reflection-aware AEPO into hierarchical or modularized stacks:

• ReEmpathy SLM (Spoken Dialogue): GLM-4-Voice backbone with chunked decoding alternates between speech response generation and internal reflection; training is supervised by both "silver" reflection traces and joint empathy regression (Jia et al., 26 Jan 2026).
• EmpathyEval (Evaluation Module): Qwen3-Omni-based, multi-head architecture outputs both paralinguistic emotion labels and granular empathy score regressions.
• SRPO (Multimodal RL): Supervised cold-start with reflection-guided targets, followed by RL with GRPO and composite task-plus-reflection rewards. Visual encoders are typically frozen; only policy heads and reflection modules are fine-tuned (Wan et al., 2 Jun 2025).

Implementation pipelines universally employ multi-stage inference with alternating or staged rollouts and group-based advantage normalization, instantiated in high-throughput RLHF or PPO frameworks.

---

Reflection-aware AEPO constitutes a rigorous framework for mitigating echo reflection and promoting meaningful self-reflection in both unimodal and multimodal LLM deployments. By combinatorially regulating information flow and stage entropy, it delivers robust improvements in answer accuracy, reflection quality, and behavioral corrigibility across a range of complex reasoning domains (Jia et al., 26 Jan 2026, He et al., 9 Nov 2025, Wan et al., 2 Jun 2025).