RLFF-ESC: Future-Oriented Emotional Support

• RLFF-ESC is a reinforcement learning framework for emotional support conversations that directly optimizes long-term outcomes.
• It integrates multi-agent dialogue simulation, a learned future-oriented reward model, and GRPO for policy optimization.
• The framework enforces explicit reasoning and structured responses to provide sustained, effective multi-turn emotional support.

Searching arXiv for the RLFF-ESC paper and closely related emotional-support conversation work. RLFF-ESC, short for Reinforcement Learning from Future-oriented Feedback for Emotional Support Conversations, is an end-to-end reinforcement learning framework for LLM-based emotional support conversation systems. It is designed for open-ended, long-term emotional support conversations rather than single-turn supportive generation, and it replaces predefined strategy selection with direct policy optimization under a future-oriented reward that estimates the long-term emotional impact of a response (Yang et al., 18 Aug 2025). The framework combines LLM-based multi-agent dialogue simulation, a learned future-oriented reward model, and GRPO-based policy optimization, and it further imposes an explicit reasoning format in which the model produces a reasoning trace and a final supportive reply (Yang et al., 18 Aug 2025).

1. Problem formulation and conceptual scope

Emotional Support Conversation (ESC) systems are intended to alleviate users’ emotional distress and provide sustained, systematic support for emotional well-being over a multi-turn dialogue. Earlier ESC work, including ESConv, framed the task around a “supporter” helping a “seeker” with strategies such as asking questions, reflecting feelings, and offering suggestions. RLFF-ESC is motivated by the claim that most existing LLM-based ESC systems remain constrained by three limitations: reliance on predefined strategy sets, optimization for immediate turn-level response quality rather than long-term outcome, and underuse of LLM reasoning about emotions, causes, and downstream consequences (Yang et al., 18 Aug 2025).

The framework therefore shifts the unit of optimization from a locally well-formed reply to a reply viewed as a policy action whose value depends on how the subsequent dialogue unfolds. Formally, given a dialogue context $x = c_{1:t-1}$ and a candidate system response $y = u_t^{sys}$, RLFF-ESC seeks a policy

$$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$

where $R(x,y)$ is a future-oriented reward estimating the long-term emotional impact of $y$ (Yang et al., 18 Aug 2025).

This formulation is explicitly tied to open-ended support rather than fixed strategy prediction. In the paper’s framing, effective support must address immediate distress while also promoting long-term emotional well-being. A supportive utterance is therefore evaluated not only by whether it sounds empathic in isolation, but by whether it helps the user’s emotional problem improve or resolve after several additional turns.

2. Multi-agent future-oriented feedback

RLFF-ESC operationalizes long-horizon evaluation through a three-agent simulation mechanism. The system agent $M_{sys}$ is the ESC model being trained. The user simulator $U$ is an LLM prompted to role-play a distressed user given an emotion type, problem type, and situation description. The critic agent $M_{crt}$ acts as an evaluator and outputs a qualitative judgment of the user’s state on a seven-level scale: Same, Slightly Better, Moderately Better, Significantly Better, Slightly Worse, Moderately Worse, and Significantly Worse. These levels are mapped to numeric rewards from $-1.0$ to $+1.0$ (Yang et al., 18 Aug 2025).

For each real dialogue context $y = u_t^{sys}$0, the system samples $y = u_t^{sys}$1 candidate responses,

$y = u_t^{sys}$2

and for each such response simulates a future trajectory by alternating user and system turns. The user’s first simulated reply is

$y = u_t^{sys}$3

after which the system and user continue for up to $y = u_t^{sys}$4 additional turns. At each step the critic evaluates the current conversation:

$y = u_t^{sys}$5

Simulation terminates either when the critic’s reward exceeds a threshold, meaning the user’s issue is considered solved, or when the turn limit is reached (Yang et al., 18 Aug 2025).

The initial response is then assigned a future-oriented reward that balances final emotional state and efficiency. The paper prints

$y = u_t^{sys}$6

and also notes a minor typesetting glitch; the stated intended idea is to average the critic score and an efficiency term $y = u_t^{sys}$7, where AvgT is the number of turns until termination. The resulting reward dataset is

$y = u_t^{sys}$8

This design makes the reward trajectory-level rather than turn-level. In the paper’s description, it captures goal completion, emotional improvement, efficiency, and, implicitly, coherence and strategy quality, because poor responses tend to induce worse simulated futures (Yang et al., 18 Aug 2025).

3. Reward modeling and policy optimization

The pipeline has three stages: multi-agent dialogue simulation, future-oriented reward model training, and reinforcement learning with Group Relative Policy Optimization (GRPO). The reward model is trained from $y = u_t^{sys}$9 by converting scalar simulated rewards into binary labels. Each example contains a conversation context $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$0, a system response $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$1, and a scalar reward $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$2. A prompt template constructs an input sequence $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$3 from the scenario description, the dialogue context, and the system response. The label is

• $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$4 if $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$5,
• $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$6 otherwise.

The model is an LLM-based classifier with frozen LLaMA-3.2-1B as base and a linear layer on top:

$$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$7

The paper states that sigmoid is applied to obtain $$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$8, and training uses binary cross-entropy:

$$\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],$$9

Although the logits are two-dimensional, the paper notes that the implementation likely uses softmax or a 2-dim sigmoid vector (Yang et al., 18 Aug 2025).

The ESC policy model is instantiated with two backbone instruction-tuned LLMs: Qwen2.5-7B-Instruct-1M and LLaMA-3.1-8B-Instruct. RLFF-ESC optimizes outputs of the form

$R(x,y)$0

Its reinforcement signal has two components. The thinking format reward is

$R(x,y)$1

The future-oriented reward $R(x,y)$2 is produced by the trained reward classifier from context and response. The combined RL reward is

$R(x,y)$3

with $R(x,y)$4 in experiments (Yang et al., 18 Aug 2025).

GRPO replaces an explicit critic network with within-group relative comparisons. For each context, the method samples a group of candidate outputs $R(x,y)$5, computes rewards $R(x,y)$6, and forms normalized advantages

$R(x,y)$7

The paper’s GRPO objective includes a PPO-style clipped ratio term and a KL penalty to keep the updated policy near a reference model. During RL, the reward model is applied only to the final supportive response inside <response>, ignoring the > segment.

4. Explicit reasoning protocol

A distinctive feature of RLFF-ESC is its explicit two-part generation format. The <think> ... segment contains the model’s reasoning process, including understanding the user’s emotional state, identifying distress cues and underlying causes, and planning possible support strategies. The <response> ... </response> segment contains the message sent to the user, such as empathic reflection, questions, suggestions, or reframes (Yang et al., 18 Aug 2025).

The RL prompt is correspondingly explicit: the model is instructed to act as a therapist in a counseling conversation, to help the patient reduce emotional distress, and to produce reasoning and response enclosed within the specified tags. The thinking format reward enforces structural compliance, while the future-oriented reward makes that structure consequential for optimization because better reasoning should induce better downstream support trajectories.

The paper characterizes this mechanism as a way to force systematic analysis of emotional context before replying. It is described as analogous to a Chain-of-Thought style process, but targeted specifically at emotional support. A case study includes a breakup scenario in which the reasoning identifies rejection, confusion, and hurt, and the response validates the experience and invites further sharing. In the paper’s interpretation, RL trains the model to reproduce this kind of structured, context-sensitive supportive behavior (Yang et al., 18 Aug 2025).

5. Datasets, evaluation protocol, and empirical results

RLFF-ESC is evaluated on two public ESC datasets. ESConv contains 1,300 crowd-sourced dialogues annotated with 8 emotional support strategies, user problem types, emotion types, and situation descriptions. ExTES contains 11,177 ChatGPT-generated dialogues, human-verified, and annotated with 16 emotional support strategies. The official HuggingFace split is used for ESConv, while ExTES is split 8:1:1 into train, development, and test (Yang et al., 18 Aug 2025).

Automatic evaluation follows two metrics. Success Rate (SR) is the proportion of dialogues in which the goal is successfully achieved within a maximum number of turns. Operationally, GPT-4o acts as both user simulator and critic; if the critic reward exceeds $R(x,y)$8, the goal is considered completed. Average Turn (AT) is the average number of turns required to reach goal completion, with lower AT indicating greater efficiency. Human evaluation is performed on 100 ESConv dialogues by 3 annotators with psychology background, comparing RLFF-ESC against baselines on Fluency, Empathy, Identification, Suggestion, and Overall emotional support quality (Yang et al., 18 Aug 2025).

The training protocol uses GRPO with LoRA on the policy models, reward weights $R(x,y)$9 and $y$0, a maximum simulated dialogue length of 8 turns, and 4 candidate generations per context. Typical hyperparameters include RL learning rate $y$1, batch size 4, 2 epochs, and LoRA rank 8. Training is conducted on 4× NVIDIA A100 with DeepSpeed ZeRo-3.

The main automatic results are as follows.

Setting	Best baseline	RLFF-ESC
ESConv, LLaMA-3.1-8B	ICL-AIF: SR 23.4%, AT 7.18	SR 35.5%, AT 6.83
ESConv, Qwen2.5-7B	ICL-AIF: SR 28.5%, AT 7.43	SR 41.5%, AT 7.18
ExTES, LLaMA-3.1-8B	ICL-AIF: SR 28.4%, AT 7.48	SR 30.2%, AT 7.30
ExTES, Qwen2.5-7B	ICL-AIF: SR 30.3%, AT 7.34	SR 32.5%, AT 7.29

On ESConv with the Qwen2.5-7B backbone, RLFF-ESC also exceeds several much larger zero-shot LLMs in SR: 41.5% for RLFF-ESC versus 30.8% for GPT-4o, 23.9% for LLaMA-3.1-405B-Instruct, and 27.7% for Qwen2.5-72B-Instruct (Yang et al., 18 Aug 2025).

Human evaluation shows majority wins for RLFF-ESC across all measured dimensions against all baselines. Against Standard Prompt, it records 96% win / 4% lose on Fluency, 92% / 8% on Empathy, and 92% / 8% on Overall. Against the stronger baseline ICL-AIF, it records 66% / 34% on Fluency, 78% / 22% on Empathy, 76% / 24% on Suggestion, and 74% / 26% on Overall (Yang et al., 18 Aug 2025).

Ablation results identify the reward model as central. On ESConv with Qwen2.5-7B, Vanilla reaches SR 26.5% and AT 7.58; DPO reaches 31.5% and 7.26; GRPO_Random reaches 18.5% and 7.66; GRPO_Ranking reaches 26.9% and 7.56; and GRPO_Classification, the full RLFF-ESC setup, reaches 41.5% and 7.18. The paper further reports that RLFF-ESC outperforms ICL-AIF in most emotion and problem categories, especially anxiety, depression, ongoing depression, and academic pressure (Yang et al., 18 Aug 2025).

6. Relation to adjacent paradigms, limitations, and future directions

RLFF-ESC is conceptually adjacent to both RLHF and RLAIF, but it differs in the construction of the feedback signal and in the training objective. Like RLHF, it trains a reward model and fine-tunes a policy. Like RLAIF, it uses LLMs as critics rather than relying exclusively on human preference labels. Its specific novelty is the use of future-oriented feedback derived from simulated multi-turn dialogue trajectories, followed by GRPO rather than PPO (Yang et al., 18 Aug 2025).

The paper also positions RLFF-ESC against prior ESC methods such as PPDP and DialogueXpert. PPDP applies reinforcement learning to an external planner for strategy prediction, while DialogueXpert uses a frozen LLM as action prior together with an emotion tracker for strategy-level RL. RLFF-ESC differs by directly optimizing the LLM itself, eliminating reliance on a separate planner, and grounding the reward in multi-turn simulated outcomes rather than local or strategy-level signals.

Several limitations are explicitly acknowledged. First, both user simulation and critic evaluation depend on LLMs such as GPT-4o and Qwen2.5-72B, which introduces possible evaluation biases or misalignment with real human users. Second, the study does not directly measure users’ emotional intensity changes through clinical or longitudinal real-user protocols. Third, the full pipeline—multi-agent simulation, reward modeling, and GRPO—remains computationally expensive, even though LoRA and a smaller reward model partially mitigate that cost (Yang et al., 18 Aug 2025).

The authors identify four major future directions. One is comprehensive user studies with real users and tracking of emotional intensity and well-being over time. A second is improved user simulators and critics. A third is personalization, including user personas, preferences, and histories. A fourth concerns safety and ethics: ESC systems are not therapists, so deployment should include clear boundaries, safe handling of severe distress, and appropriate escalation such as recommending professional help. A further direction is integration of structured psychological knowledge, including cognitive-behavioral techniques and coping strategies (Yang et al., 18 Aug 2025).

In this sense, RLFF-ESC marks a shift in ESC research from turn-level supportive style toward outcome-oriented policy learning. Its central contribution is not merely stronger response generation, but a redefinition of supportive dialogue optimization around simulated future emotional trajectories, goal completion, and conversation efficiency.