---
title: Reinforcement Learning from Reflective Feedback
url: https://www.emergentmind.com/topics/reinforcement-learning-from-reflective-feedback-rlrf
type: topic
---

# Reinforcement Learning from Reflective Feedback

Reinforcement Learning from Reflective Feedback (RLRF) is a class of techniques that integrate reflection—either from models, external evaluators, or self-critique—into the reinforcement learning (RL) loop for sequential decision-making agents. RLRF extends or supersedes traditional reward-based RL and reinforcement learning from human feedback (RLHF) by leveraging free-form, aspect-based, or contextually grounded feedback to guide optimization, facilitate exploration, and improve model sample efficiency. Methods in this paradigm have been applied to large language models (LLMs), code generation, robot control, recommendation systems, and vector graphics tasks across diverse architectures and problem settings.

## 1. Theoretical Foundations and Definitions

RLRF formalizes agent improvement by coupling trial-and-error RL with explicit or internally generated reflective feedback. At each episode or trial, an agent—often a language model or other sequential predictor—obtains feedback that may be:

- **Scalar**: Classic RL signals such as reward or binary correctness
- **Linguistic**: Free-form critiques, suggestions, or explanations, either from a model's own self-reflection or from an external oracle (such as another LLM)
- **Aspect-based**: Multi-dimensional rubric covering factuality, logical correctness, metacognition, and other properties

Distinguishing characteristics of RLRF include:
- Separation of policy improvement into weight-based learning and context-based, prompt-level in-context learning (e.g., Reflexion [2303.11366])
- Use of reflective trajectories, refined via evaluators or feedback models, to shape future queries and outputs
- Algorithmic mechanisms for credit assignment to reflection tokens, as in the token-level policy gradients of "Reflect, Retry, Reward" [2505.24726]
- Feedback potentially delivered after unsuccessful attempts, triggering a revise-and-retry cycle

Mathematically, several RLRF instantiations remain within the RL Markov Decision Process formalism but augment the reward function or learning context:
\[
r(x, y) = r_{\textrm{env}}(x, y) + r_{\textrm{reflect}}(x, y)
\]
or, for token-level credit assignment,
\[
A_t(\tau)=r\quad\text{if } t \text{ is a reflection token},\quad 0 \text{ otherwise}
\]
where $r$ is derived from verifiable success/failure or from fine-grained feedback [2403.14238, 2505.24726].

## 2. Key Algorithms and Architectural Variants

RLRF comprises several architectural and algorithmic paradigms:

### a. Reflexion and In-Context Reflective Loops
The Reflexion framework [2303.11366] eschews weight updates, instead reinforcing agents by maintaining an episodic buffer of verbal reflections. These are concatenated to the agent's prompt at each episode, biasing decisions via in-context information:
- The agent executes a trial, receives scalar/environmental reward, and produces a reflective text
- The memory buffer $M_t$ accumulates the most recent $\Omega$ reflections, forming $a_t \sim \pi(a \mid s_t, M_t)$ for subsequent decisions

### b. RL Fine-Tuning from Fine-Grained Feedback
“Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine‐Grained Self‐Reflection” [2403.14238] alternates between exploration (self-reflection and response refinement) and policy update (via Direct Preference Optimization, DPO):
1. Sample $n$ initial candidate responses per prompt.
2. Score via both preference-based reward and a feedback model, covering aspects such as factuality, logicality, insight, and more.
3. Select the best candidate, induce a self-reflection, and generate $m$ refined responses.
4. Pool initial and refined responses, select positive/negative pairs, and fine-tune with DPO loss.

### c. Token-Level Reflection Credit Assignment
"Reflect, Retry, Reward" [2505.24726] introduces a two-stage process:
- A failed attempt yields a reflection $z$.
- The retry is conditioned on $z$.
- If the retry succeeds, a reward is assigned to the reflection tokens alone via GRPO, maximizing the effectiveness of reflective self-improvement.

### d. Applied RLRF in Robotics, Recommendation, and SVG Generation
- **Robotic Manipulation (Lafite-RL [2311.02379])**: An RL agent receives environment rewards and additional LLM-based scaffolding rewards as $r_t = r_{\rm env} + r_{\rm llm}$. Natural language feedback is translated to $\{+1, 0, -1\}$, directly augmenting sample efficiency in sparse reward tasks.
- **Session-Based Recommendation (Re2LLM [2403.16427])**: Self-reflection extracts error-correcting hints, which are then retrieved by an RL agent (trained via PPO) to steer future recommendations.
- **Vector Graphics (SVG) Generation [2505.20793]**: Offline rewards are computed by comparing rendered outputs to targets (via L2, similarity, and code-length measures), and policy is trained via GRPO to maximize rendering fidelity and efficiency.

### e. Reflection-Aware Online RL
REA-RL [2505.19862] integrates a separate reflection model to detect and truncate “overthinking” in LLM outputs, optimizes a composite reward (accuracy, conciseness, reflection frequency), and adapts reflection dynamically across problem difficulty.

## 3. Feedback Modalities, Collection, and Integration

RLRF methods leverage a wide spectrum of feedback sources:

| Feedback Type        | Source                         | Integration Modality         |
|----------------------|-------------------------------|-----------------------------|
| Scalar (0/1, real)   | Environment/success/failure    | Direct reward augmentation   |
| Natural language     | Self, LLM oracle, heuristics   | Prompt concatenation         |
| Aspect-based         | Feedback model (LLM)           | Multi-aspect scoring         |

Practical integration strategies include:
- Batching reflections into prompt buffers (Reflexion)
- Aspect selection and scoring via LLMs, then presenting top aspects to influence further generation [2403.14238]
- Prompt-based LLM queries to evaluate actions, providing scaffolding rewards via keyword detection [2311.02379]
- Reward mapping from natural-language judgments (e.g. "Good move"/"Bad move") directly to numerical signals

## 4. Empirical Results and Applications

RLRF frameworks yield substantial empirical gains across tasks:

- **Robotics (Lafite-RL [2311.02379])**: Success rates on RLBench tasks increased from 41.6% to 70.3% (“Push Buttons”) and from 26.1% to 68.7% (“Umbrella”) by supplementing sparse rewards with LLM feedback.
- **LLM Alignment and Reasoning (RLRF [2403.14238])**: Gains of $+8$ pp in factuality (FactScore) and $+6$ pp on GSM8K math reasoning benchmarks relative to reward-only RLHF baselines.
- **Reflect–Retry–Reward [2505.24726]**: Smaller RLRR-trained models outperform 10x larger vanilla models; e.g., Qwen2.5-1.5B on “Countdown” achieved $6.0\%\to34.9\%$ first-try accuracy.
- **Recommendation (Re2LLM [2403.16427])**: In few-shot Movie session tasks, NDCG@10 improved by $+20.1\%$ over baselines; similar improvements on Games.
- **Vector Graphics Generation [2505.20793]**: Test MSE improved from 23.31 to 4.79 and SSIM from $62.28\%\to88.76\%$ after RLRF, with enhanced structural fidelity and code efficiency.
- **Reflection-Aware RL (REA-RL [2505.19862])**: Reasoning LLMs reduce inference cost by ~35% (token count) at constant accuracy, adaptively trimming reflection on easy tasks and retaining it on hard ones.

## 5. Comparative Analysis and Limitations

RLRF introduces new axes beyond PPO-based RLHF and supervised fine-tuning:

| Method           | Weight Updates | Episodic Reflection | Fine-Grained Feedback | External Reward | Sample Efficiency |
|------------------|---------------|---------------------|----------------------|-----------------|------------------|
| RLRF/Reflexion   | No            | Yes                 | Yes                  | Yes             | High             |
| RLHF/PPO         | Yes           | No                  | No                   | Yes             | Moderate         |
| Self-Refine      | No            | No                  | Yes                  | No              | Low              |

Key advantages include:
- Improved sample efficiency via contextual or token-level feedback assignment
- Capability to address “overthinking”, verbosity, or reward hacking (e.g. code/length hacks in SVG generation [2505.20793])
- Flexibility to work with diverse feedback, including multi-aspect rubrics, binary validators, or natural language critique

Known limitations:
- Subjectivity and potential noise in reflective feedback (especially aspect-based scoring)
- High computational cost for some paradigms, e.g. repeated generations for self-reflection/exploration [2403.14238]
- Dependence on reflection model or LLM quality for correct error attribution [2505.19862]
- Cost and latency for repetitive LLM-based feedback during training [2311.02379, 2403.16427]
- Scalability to very large models in online RL remains an open question [2505.19862]

## 6. Future Directions and Open Challenges

Open research problems include:
- Automating and objectifying feedback collection (robust, low-cost generation of high-quality reflective signals)
- Joint or continual learning of policy and reflection models to enable richer, adaptive feedback flow [2505.19862]
- Multi-hint or multi-round reflection/retrieval to address complex errors, as suggested for session-based recommendation [2403.16427]
- Extending RLRF beyond tasks with explicit or easily-verifiable correctness criteria (e.g., dialog, open-ended generation)
- Exploring formal convergence and theoretical understanding of text-based pseudo-gradients and in-context policy improvement [2303.11366]
- Richer forms of memory and value reflection for deeper and more persistent improvement [2303.11366]

RLRF represents a growing design space for robust, efficient, and contextually informed reinforcement learning, blending classical RL with the flexible, introspective capacities of modern large-scale models.

Source: https://www.emergentmind.com/topics/reinforcement-learning-from-reflective-feedback-rlrf