Verbal Reinforcement Learning (vRL)

• Verbal Reinforcement Learning (vRL) is an emerging paradigm that utilizes natural language feedback to guide policy improvements and enhance interpretability.
• vRL incorporates prompt-based methods, plug-and-play architectures, and memory-augmented models to significantly boost sample efficiency and behavioral alignment.
• Empirical results demonstrate that vRL improves task success rates and robustness in applications like robotics, program synthesis, dialogue systems, and multi-agent coordination.

Verbal Reinforcement Learning (vRL) is an emerging paradigm in which language-based, human- or model-generated feedback—delivered in natural language rather than (or in addition to) scalar rewards—directs the policy improvement of autonomous agents. Unlike traditional reinforcement learning (RL), which updates agent parameters via numerical reward signals and gradient-based optimization, vRL exploits the interpretability, richness, and flexibility of natural-language “verbal” feedback for both online adaptation and offline training. Recent work demonstrates that vRL confers substantial gains in sample efficiency, transparency, behavioral alignment, and robustness across domains spanning sequential decision-making, program synthesis, dialogue systems, robotics, customer service, and multi-agent coordination.

1. Foundations and Paradigm Definition

Verbal reinforcement learning generalizes the RL paradigm by extending the reward channel to include natural-language feedback, which can originate from external evaluators (humans, automated critics, or LLMs) or internal self-reflection by the agent itself. This verbal feedback, or “reflection,” functions as a semantic policy gradient—guiding future actions through lesson-like textual advice, concrete mistake explanations, or structured critiques (Shinn et al., 2023).

Formally, in a generic partially observable Markov decision process, the agent’s observation and action history is augmented by an episodic memory buffer of self-reflections. At each trial, the agent generates a trajectory $\tau$ via its policy, receives scalar and/or linguistic feedback, and stores a reflection $sr$ in memory (Shinn et al., 2023). The agent’s future actions are conditioned not only on recent observations, but also on these linguistic “lessons learned.”

Rather than relying solely on parameter update mechanisms, vRL may—depending on the formulation—eschew gradient updates entirely (prompt-based vRL), or use verbal feedback as an additional learning signal to shape a differentiable reward landscape for policy optimization (Duan et al., 5 Feb 2026, Hao et al., 8 Sep 2025).

2. Canonical Architectures and Algorithms

2.1 Reflexion and Prompt-based vRL

The Reflexion framework exemplifies weight-free, prompt-based vRL. Agents modularly combine three components: (i) an Actor (LLM) that generates the next action conditioned on task description and a sliding window of self-reflections, (ii) an Evaluator which produces task-level feedback, and (iii) a Self-Reflection model that distills episode outcomes into a textual “reflection” stored in memory (Shinn et al., 2023). The full policy is thus defined as

$$a_{t:0:T} \sim \pi_\theta\bigl(\,\cdot\mid \text{prompt},\,\mathrm{mem}_t\bigr)$$

Crucially, the “semantic gradient” is delivered via the prompt context and not through explicit parameter updates. The agent iteratively improves by leveraging its own self-generated corpus of past mistakes and corrected strategies, encoded as free-form text.

2.2 Plug-and-Play and Memory-Augmented vRL

Plug-and-play vRL architectures—such as MemOrb—interpose a verbal-reinforcement memory layer around otherwise frozen LLMs. Episodic strategy reflections are distilled after each episode and indexed in an external memory bank using neural embeddings. At inference, top-$k$ reflections relevant to the current context are retrieved and concatenated into the prompt, steering the agent with prior policy notes but without any online fine-tuning (Huang et al., 23 Sep 2025). This enables continual improvement and long-term consistency in dialogue and customer service scenarios.

2.3 RL with Trainable Policies and Verbal Rewards

Other frameworks integrate verbal feedback as a differentiable reward for gradient-based policy optimization. In reflective verbal reward design for alignment, LLM-based reward models are conditioned on user-reflective dialogue histories, allowing the agent to optimize for person-specific behavioral alignment via in-context verbal reward probabilities (Blair et al., 21 Jun 2025). The overall objective becomes

$$J(\phi) = \mathbb{E}_{\tau \sim \pi_\phi}[V_\theta(C, \tau)]$$

where $V_\theta(C, \tau)$ is the (personalized) verbal reward probability produced by the LLM.

Multi-turn structures like MulFeRL unify cross-turn verbal critique and preference ranking with within-batch policy advantages, yielding complementary learning signals and improved sample efficiency on hard reasoning benchmarks (Li et al., 30 Jan 2026).

2.4 Multi-Agent and Communication-Centric vRL

In multi-agent settings, vRL enables human-interpretable, text-driven coordination and communication. Frameworks such as Verco and OptAgent treat each agent’s message or debate interaction as a verbal action, optimizing communication structures and debate quality directly for improved collaborative performance and solution accuracy (Li et al., 2024, Bi et al., 20 Oct 2025).

3. Empirical Results and Ablative Analyses

Verbal RL yields notable improvements over purely scalar reward or SFT-based baselines across a range of tasks:

• Sequential decision-making (AlfWorld, HotPotQA): Reflexion agents attain up to +22% absolute task success rate, e.g., HumanEval Pass@1 accuracy 91% vs. 80% for GPT-4 (Shinn et al., 2023).
• Programming/code synthesis: Incorporating verbal unit-test feedback or self-reflection enables agents to outperform SOTA GPT-4 on HumanEval and LeetCode HardGym (Shinn et al., 2023).
• Dialogue and open-domain interaction: RAPO shows +12% gain in mood improvement and goal attainment over scalar-only RL by fusing dense verbal hindsight critiques about emotional resonance (Ye et al., 16 Mar 2026).
• Customer service and reliability: MemOrb boosts multi-trial Pass$^3$ consistency by up to +63 pp compared to frozen LLM agents, demonstrating conversion of verbal reflection into robust long-term memory (Huang et al., 23 Sep 2025).
• Math and logic reasoning: Multi-turn, cross-turn feedback models (MulFeRL) reach pass@1 rates up to 68% (Qwen3-4B-Inst) vs. 50–63% for supervised or scalar RL (Li et al., 30 Jan 2026); ALIVE improves generalization and self-correction (+12 pts MMLU-Pro) by integrating instructive verbal evaluation into policy gradients (Duan et al., 5 Feb 2026).
• Interpretability and explainability: Dual-model reflection (DARS) provides transparent policy refinement, outperforming preference optimization while surfacing explicit critique text (Li et al., 26 Feb 2025).

Ablations reveal that omitting self-reflection, feedback-guided regeneration, or verbal reward signals usually degrades performance by 2–10 points, confirming the direct contribution of the verbal channel.

4. Formalizations, Learning Signals, and Processing

vRL systems may employ a variety of feedback types:

• Scalar/binary feedback: Environmental success/failure signals, often weak and sparse.
• Heuristic/domain signals: Rule-based indicators (e.g. hallucination flags, repeated actions).
• LLM-based judgment: Automated, internally simulated evaluation, including unit tests, self-assessment, or chain-of-thought scoring.
• Human or model-generated critiques: Free-form explanations, error localizations, repair plans.

Processing pipelines typically route the full trajectory and feedback to a reflection module, which verbalizes actionable lessons. These reflections are embedded, stored, and surfaced in prompts, or directly converted into reward values for policy gradients (e.g., via likelihood ratios, DPO, preference modeling) (Shinn et al., 2023, Li et al., 30 Jan 2026, Blair et al., 21 Jun 2025).

5. Applications, Sample Efficiency, and Limits

Verbal RL has demonstrated value in:

• Simulated and physical robotics: Closed-loop, vision-language critic feedback for symbolic plan refinement outperforms scalar-score-only baselines in real-world mobile manipulation (Plotnikov et al., 23 Mar 2026), while OLAF achieves +20% absolute policy success improvement via user verbal corrections (Liu et al., 2023).
• Multi-turn text-to-SQL and code completion: Verbal group self-evaluation (PaVeRL-SQL, RepoGenReflex) provides lightweight, training-free policy improvement, yielding up to +7.4% execution accuracy over SOTA zero-shot LLMs (Hao et al., 8 Sep 2025, Wang et al., 2024).
• Multi-agent coordination: Optimizing communication via verbal actions improves sample efficiency and final episodic return compared to numerical embedding-based communication (Li et al., 2024, Bi et al., 20 Oct 2025).
• Agent alignment and customization: Constrained preference optimization over synthetic verbal feedback minimizes overgeneralization while retaining adherence to nuanced instructions (Stephan et al., 2024).

Sample complexity is reduced in vRL settings due to the high density of information encoded in verbal feedback. Studies report rapid convergence within a few episodes or samples, minimal need for fine-tuning, and increased robustness to distributional shift (Shinn et al., 2023, Huang et al., 23 Sep 2025, Blair et al., 21 Jun 2025).

Key limitations include dependency on LLM self-evaluation capabilities, prompt/context window constraints, and potential local minima when the quality of verbal feedback is insufficient or misleading. In open-ended settings, parsing free-form feedback into effective, context-sensitive updates remains an ongoing challenge.

6. Theoretical Perspectives and Future Research

Conceptual analyses interpret vRL as a form of semantic or interpretable policy iteration, with the “gradient” encoded in natural-language rather than in the model’s parameter space (Shinn et al., 2023). Episodic textual memory allows for improved credit assignment and transparent debugging over traditional black-box RL. In multi-agent domains, vRL frameworks (e.g., OptAgent) formalize graph-structured, communication-centric MDPs that optimize both correctness and quality of agent interactions (Bi et al., 20 Oct 2025).

Future research directions envisioned in recent literature include:

• End-to-end learning of reflection generation and retrieval.
• Memory- and context-aware meta-learning for cross-task transfer.
• Integration of vRL into hierarchical and hybrid symbolic-sub-symbolic planning.
• Robustness to adversarial or ambiguous verbal feedback (e.g., uncertainty-aware critics, human-in-the-loop correction).
• Increased agent autonomy in reflective reward modeling and value inference.

Empirical advances, theoretical grounding, and the synthesis of differentiable and prompt-based learning loops are expected to further generalize vRL across domains and deployment settings.