--- title: 'RL²F: Reinforcement Learning with Language Feedback' url: https://www.emergentmind.com/topics/reinforcement-learning-with-language-feedback-rl-2-f type: topic --- # RL²F: Reinforcement Learning with Language Feedback Reinforcement Learning with Language Feedback (RL²F) is a paradigm in which reinforcement learning agents leverage natural language as an explicit, actionable learning signal. Departing from classical RL formulations that rely exclusively on scalar, manually engineered rewards, RL²F integrates textual feedback—generated by humans or large language models (LLMs)—to condition policy improvement, shape exploration, impose constraints, deliver dense gradient signals, or facilitate credit assignment. Techniques range from preference modeling and constraint enforcement to span-level update propagation, spanning domains as varied as LLM post-training, robotics, continuous control, embodied agents, and autonomous reasoning. RL²F unifies and generalizes RL from Human Feedback (RLHF), advanced preference learning, and recent efforts on fine-grained, interpretable policy optimization. ## 1. Problem Setting and Core Principles RL²F generalizes the conventional Markov Decision Process (MDP) by augmenting each transition or trajectory with associated natural language feedback. The agent observes a state $s_t$, executes action $a_t$, receives environment reward $r_t$, and additionally receives or generates language feedback $l_t$ or $c_t$ that encodes evaluative, corrective, or prescriptive information about the behavior or outcome. Formally, the interaction proceeds under an MDP or POMDP tuple extended with a language feedback channel (e.g., $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \mathcal{L})$). RL²F systems support multiple feedback modalities: - **Preference or ranking signals**: Pairwise or multi-way comparisons over rollouts/trajectories, often producing a reward model via inverse RL or Bradley–Terry-style cross-entropy [2504.15472]. - **Free-form critiques or structured corrections**: Dense corrections at the span, token, or trajectory level [2505.22338, 2409.00162]. - **Task constraints and prohibitions**: Language-discriminators detecting violations of specified behaviors, steering the policy within a constrained MDP [2005.01618]. - **Exploratory scaffolding and refinement**: Language-derived suggestions or aggregated critiques directly injected as off-policy scaffolds or prompts [2603.04597, 2510.16552]. - **Semantic alignment**: Embedding environment states and goals in a language manifold, with reward defined by semantic similarity [2508.05977]. The agent's policy $\pi_\theta$ is thus conditioned or shaped, directly or indirectly, by language. Feedback can be supplied online during RL, retroactively from a dataset, or synthesized/abstracted through self-reflection, group dynamics, or historical summary [2510.16552, 2603.04597]. ## 2. Representative Algorithms and Feedback Integration RL²F encompasses diverse algorithms that adapt classical RL, preference-based methods, and modern LLM fine-tuning techniques to harness natural language feedback: ### 2.1 Policy Optimization with Preference and Critique Modeling RL²F generalizes RLHF by training a reward model $R_\phi$ from preference or ranking data (human or LLM-generated), often using binary cross-entropy objectives over pairwise differences. The policy is then optimized, typically with PPO or its variants, to maximize expected return under $R_\phi$: $$ J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^T \gamma^t R_\phi(s_t, a_t)\right] $$ [2310.06147, 2504.15472]. Feedback modeling can be extended to sequence-to-sequence reward models, enabling the policy to receive dense, token- or span-level feedback by comparing generated sequences to language-corrected versions, alleviating sparse credit assignment and reward gaming [2409.00162, 2505.22338]. ### 2.2 Group-Level and Off-Policy Refinement Frameworks like GOLF leverage group-level language feedback, aggregating external critiques and intra-group alternative attempts for each problem instance [2603.04597]. Aggregated negative feedback is used to condition the policy to generate high-quality refinements, which are then injected as off-policy scaffolds into RL optimization. This mitigates the vanishing gradient problem in sparse-reward regions and yields substantial gains in sample efficiency and exploration. ### 2.3 Constraint-Augmented RL via Language Discriminators Language can function as a constraint signal: natural-language feedback is distilled into a learned discriminator that predicts violation probabilities, which are penalized through a Lagrangian in the RL objective [2005.01618]: $$ \max_{\pi_\theta} J_R(\pi_\theta) \quad \text{s.t.} \quad J_{C_\phi}(\pi_\theta) \leq \alpha $$ The penalty is updated on a slower timescale, allowing soft, learned constraints that adapt to user preferences. ### 2.4 Token- and Span-Level Update Propagation Text2Grad demonstrates RL²F in the context of policy optimization from span-annotated language critiques [2505.22338]. Free-form text feedback is aligned to output spans; these alignments are converted to token-level pseudo-rewards and propagated through a natural language gradient: $$ \nabla_\mathrm{NL}(f \to y) = \sum_t \delta_t \nabla_\theta \log \pi_\theta(y_t|x, y_{