---
title: 'RLFF-ESC: Future-Oriented Emotional Support'
url: https://www.emergentmind.com/topics/rlff-esc
type: topic
---
# RLFF-ESC: Future-Oriented Emotional Support
Searching arXiv for the RLFF-ESC paper and closely related emotional-support conversation work.
RLFF-ESC, short for **Reinforcement Learning from Future-oriented Feedback for Emotional Support Conversations**, is an end-to-end reinforcement learning framework for large language model-based emotional support conversation systems. It is designed for **open-ended, long-term emotional support conversations** rather than single-turn supportive generation, and it replaces predefined strategy selection with direct policy optimization under a **future-oriented reward** that estimates the long-term emotional impact of a response [2508.12935]. The framework combines LLM-based multi-agent dialogue simulation, a learned future-oriented reward model, and GRPO-based policy optimization, and it further imposes an explicit reasoning format in which the model produces a reasoning trace and a final supportive reply [2508.12935].
## 1. Problem formulation and conceptual scope
Emotional Support Conversation (ESC) systems are intended to alleviate users’ emotional distress and provide **sustained, systematic support** for emotional well-being over a multi-turn dialogue. Earlier ESC work, including ESConv, framed the task around a “supporter” helping a “seeker” with strategies such as asking questions, reflecting feelings, and offering suggestions. RLFF-ESC is motivated by the claim that most existing LLM-based ESC systems remain constrained by three limitations: reliance on predefined strategy sets, optimization for immediate turn-level response quality rather than long-term outcome, and underuse of LLM reasoning about emotions, causes, and downstream consequences [2508.12935].
The framework therefore shifts the unit of optimization from a locally well-formed reply to a reply viewed as a policy action whose value depends on how the subsequent dialogue unfolds. Formally, given a dialogue context $x = c_{1:t-1}$ and a candidate system response $y = u_t^{sys}$, RLFF-ESC seeks a policy
$$
\pi^* = \arg\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot|x)}[R(x, y)],
$$
where $R(x,y)$ is a **future-oriented reward** estimating the long-term emotional impact of $y$ [2508.12935].
This formulation is explicitly tied to open-ended support rather than fixed strategy prediction. In the paper’s framing, effective support must address immediate distress while also promoting long-term emotional well-being. A supportive utterance is therefore evaluated not only by whether it sounds empathic in isolation, but by whether it helps the user’s emotional problem improve or resolve after several additional turns.
## 2. Multi-agent future-oriented feedback
RLFF-ESC operationalizes long-horizon evaluation through a three-agent simulation mechanism. The **system agent** $M_{sys}$ is the ESC model being trained. The **user simulator** $U$ is an LLM prompted to role-play a distressed user given an emotion type, problem type, and situation description. The **critic agent** $M_{crt}$ acts as an evaluator and outputs a qualitative judgment of the user’s state on a seven-level scale: Same, Slightly Better, Moderately Better, Significantly Better, Slightly Worse, Moderately Worse, and Significantly Worse. These levels are mapped to numeric rewards from $-1.0$ to $+1.0$ [2508.12935].
For each real dialogue context $c_{1:t-1}$, the system samples $m$ candidate responses,
$$
u_{t,j}^{sys} \sim P_{M_{sys}(\cdot \mid p_{sys}, c_{1:t-1})}, \quad j = 1,\dots,m,
$$
and for each such response simulates a future trajectory by alternating user and system turns. The user’s first simulated reply is
$$
u_{t,j}^{usr} \leftarrow P_U(p_{usr}, [c_{1:t-1}, u_{t,j}^{sys}]),
$$
after which the system and user continue for up to $T$ additional turns. At each step the critic evaluates the current conversation:
$$
r_{t+k,j} \leftarrow P_{M_{crt}(p_{crt}, [c_{1:t-1}, C_{t,j}^{f}])}.
$$
Simulation terminates either when the critic’s reward exceeds a threshold, meaning the user’s issue is considered solved, or when the turn limit is reached [2508.12935].
The initial response is then assigned a **future-oriented reward** that balances final emotional state and efficiency. The paper prints
$$
r_{t,j}^{sys} = \frac{r_{t+k,j} + \frac{1}{\text{AvgT}}{2},
$$
and also notes a minor typesetting glitch; the stated intended idea is to average the critic score and an efficiency term $1/\text{AvgT}$, where AvgT is the number of turns until termination. The resulting reward dataset is
$$
\mathcal{D}_r = \{(c_{1:t-1}, u_{t,j}^{sys}, r_{t,j}^{sys})\}_{t,j}.
$$
This design makes the reward trajectory-level rather than turn-level. In the paper’s description, it captures **goal completion**, **emotional improvement**, **efficiency**, and, implicitly, **coherence and strategy quality**, because poor responses tend to induce worse simulated futures [2508.12935].
## 3. Reward modeling and policy optimization
The pipeline has three stages: multi-agent dialogue simulation, future-oriented reward model training, and reinforcement learning with **Group Relative Policy Optimization (GRPO)**. The reward model is trained from $\mathcal{D}_r$ by converting scalar simulated rewards into binary labels. Each example contains a conversation context $c_i$, a system response $u_i^{sys}$, and a scalar reward $r_i^{sys}$. A prompt template constructs an input sequence $s_i$ from the scenario description, the dialogue context, and the system response. The label is
- $y_i = 1$ if $r_i^{sys} > \delta$,
- $y_i = 0$ otherwise.
The model is an LLM-based classifier with frozen **LLaMA-3.2-1B** as base and a linear layer on top:
$$
\mathbf{z}_i = \text{Linear}(\text{Pool}(\text{LLaMA}(s_i))) \in \mathbb{R}^2.
$$
The paper states that sigmoid is applied to obtain $\hat{y}_i = \sigma(\mathbf{z}_i)$, and training uses binary cross-entropy:
$$
\mathcal{L} = -\frac{1}{N} \sum_{i=1}^N \big[y_i \log(\hat{y}_i) + (1 - y_i)\log(1 - \hat{y}_i)\big].
$$
Although the logits are two-dimensional, the paper notes that the implementation likely uses softmax or a 2-dim sigmoid vector [2508.12935].
The ESC policy model is instantiated with two backbone instruction-tuned LLMs: **Qwen2.5-7B-Instruct-1M** and **LLaMA-3.1-8B-Instruct**. RLFF-ESC optimizes outputs of the form
$$
\texttt{} \; \text{reasoning tokens} \; \texttt{} \; \texttt{} \; \text{supportive reply tokens} \; \texttt{}.
$$
Its reinforcement signal has two components. The **thinking format reward** is
$$
R_{\text{thk-fmt}} =
\begin{cases}
1, & \text{if output adheres to the required format;}\\
0, & \text{otherwise.}
\end{cases}
$$
The **future-oriented reward** $R_{\text{fut}}$ is produced by the trained reward classifier from context and response. The combined RL reward is
$$
R_{\text{rlff}} = R_{\text{fut}} + \alpha R_{\text{thk-fmt}},
$$
with $\alpha = 0.5$ in experiments [2508.12935].
GRPO replaces an explicit critic network with within-group relative comparisons. For each context, the method samples a group of candidate outputs $\{o_i\}_{i=1}^G$, computes rewards $r_i = R_{\text{rlff}}(c_{1:t-1}, o_i)$, and forms normalized advantages
$$
A_i = \frac{r_i - \text{mean}(r_1,\dots,r_G)}{\text{std}(r_1,\dots,r_G)}.
$$
The paper’s GRPO objective includes a PPO-style clipped ratio term and a KL penalty to keep the updated policy near a reference model. During RL, the reward model is applied only to the final supportive response inside ``, ignoring the `` segment.
## 4. Explicit reasoning protocol
A distinctive feature of RLFF-ESC is its explicit two-part generation format. The ` ... ` segment contains the model’s reasoning process, including understanding the user’s emotional state, identifying distress cues and underlying causes, and planning possible support strategies. The ` ... ` segment contains the message sent to the user, such as empathic reflection, questions, suggestions, or reframes [2508.12935].
The RL prompt is correspondingly explicit: the model is instructed to act as a therapist in a counseling conversation, to help the patient reduce emotional distress, and to produce reasoning and response enclosed within the specified tags. The thinking format reward enforces structural compliance, while the future-oriented reward makes that structure consequential for optimization because better reasoning should induce better downstream support trajectories.
The paper characterizes this mechanism as a way to force systematic analysis of emotional context before replying. It is described as analogous to a Chain-of-Thought style process, but targeted specifically at emotional support. A case study includes a breakup scenario in which the reasoning identifies rejection, confusion, and hurt, and the response validates the experience and invites further sharing. In the paper’s interpretation, RL trains the model to reproduce this kind of structured, context-sensitive supportive behavior [2508.12935].
## 5. Datasets, evaluation protocol, and empirical results
RLFF-ESC is evaluated on two public ESC datasets. **ESConv** contains 1,300 crowd-sourced dialogues annotated with 8 emotional support strategies, user problem types, emotion types, and situation descriptions. **ExTES** contains 11,177 ChatGPT-generated dialogues, human-verified, and annotated with 16 emotional support strategies. The official HuggingFace split is used for ESConv, while ExTES is split 8:1:1 into train, development, and test [2508.12935].
Automatic evaluation follows two metrics. **Success Rate (SR)** is the proportion of dialogues in which the goal is successfully achieved within a maximum number of turns. Operationally, GPT-4o acts as both user simulator and critic; if the critic reward exceeds $0.5$, the goal is considered completed. **Average Turn (AT)** is the average number of turns required to reach goal completion, with lower AT indicating greater efficiency. Human evaluation is performed on 100 ESConv dialogues by 3 annotators with psychology background, comparing RLFF-ESC against baselines on Fluency, Empathy, Identification, Suggestion, and Overall emotional support quality [2508.12935].
The training protocol uses GRPO with LoRA on the policy models, reward weights $R_{\text{fut}} = 1.0$ and $R_{\text{thk-fmt}} = 0.5$, a maximum simulated dialogue length of 8 turns, and 4 candidate generations per context. Typical hyperparameters include RL learning rate $1\text{e-}6$, batch size 4, 2 epochs, and LoRA rank 8. Training is conducted on 4× NVIDIA A100 with DeepSpeed ZeRo-3.
The main automatic results are as follows.
| Setting | Best baseline | RLFF-ESC |
|---|---|---|
| ESConv, LLaMA-3.1-8B | ICL-AIF: SR 23.4%, AT 7.18 | SR 35.5%, AT 6.83 |
| ESConv, Qwen2.5-7B | ICL-AIF: SR 28.5%, AT 7.43 | SR 41.5%, AT 7.18 |
| ExTES, LLaMA-3.1-8B | ICL-AIF: SR 28.4%, AT 7.48 | SR 30.2%, AT 7.30 |
| ExTES, Qwen2.5-7B | ICL-AIF: SR 30.3%, AT 7.34 | SR 32.5%, AT 7.29 |
On ESConv with the Qwen2.5-7B backbone, RLFF-ESC also exceeds several much larger zero-shot LLMs in SR: **41.5%** for RLFF-ESC versus **30.8%** for GPT-4o, **23.9%** for LLaMA-3.1-405B-Instruct, and **27.7%** for Qwen2.5-72B-Instruct [2508.12935].
Human evaluation shows majority wins for RLFF-ESC across all measured dimensions against all baselines. Against Standard Prompt, it records 96% win / 4% lose on Fluency, 92% / 8% on Empathy, and 92% / 8% on Overall. Against the stronger baseline ICL-AIF, it records 66% / 34% on Fluency, 78% / 22% on Empathy, 76% / 24% on Suggestion, and 74% / 26% on Overall [2508.12935].
Ablation results identify the reward model as central. On ESConv with Qwen2.5-7B, **Vanilla** reaches SR 26.5% and AT 7.58; **DPO** reaches 31.5% and 7.26; **GRPO_Random** reaches 18.5% and 7.66; **GRPO_Ranking** reaches 26.9% and 7.56; and **GRPO_Classification**, the full RLFF-ESC setup, reaches **41.5%** and **7.18**. The paper further reports that RLFF-ESC outperforms ICL-AIF in most emotion and problem categories, especially anxiety, depression, ongoing depression, and academic pressure [2508.12935].
## 6. Relation to adjacent paradigms, limitations, and future directions
RLFF-ESC is conceptually adjacent to both **RLHF** and **RLAIF**, but it differs in the construction of the feedback signal and in the training objective. Like RLHF, it trains a reward model and fine-tunes a policy. Like RLAIF, it uses LLMs as critics rather than relying exclusively on human preference labels. Its specific novelty is the use of **future-oriented feedback** derived from simulated multi-turn dialogue trajectories, followed by GRPO rather than PPO [2508.12935].
The paper also positions RLFF-ESC against prior ESC methods such as PPDP and DialogueXpert. PPDP applies reinforcement learning to an **external planner** for strategy prediction, while DialogueXpert uses a frozen LLM as action prior together with an emotion tracker for strategy-level RL. RLFF-ESC differs by directly optimizing the LLM itself, eliminating reliance on a separate planner, and grounding the reward in multi-turn simulated outcomes rather than local or strategy-level signals.
Several limitations are explicitly acknowledged. First, both user simulation and critic evaluation depend on LLMs such as GPT-4o and Qwen2.5-72B, which introduces possible evaluation biases or misalignment with real human users. Second, the study does not directly measure users’ emotional intensity changes through clinical or longitudinal real-user protocols. Third, the full pipeline—multi-agent simulation, reward modeling, and GRPO—remains computationally expensive, even though LoRA and a smaller reward model partially mitigate that cost [2508.12935].
The authors identify four major future directions. One is **comprehensive user studies** with real users and tracking of emotional intensity and well-being over time. A second is improved user simulators and critics. A third is **personalization**, including user personas, preferences, and histories. A fourth concerns **safety and ethics**: ESC systems are not therapists, so deployment should include clear boundaries, safe handling of severe distress, and appropriate escalation such as recommending professional help. A further direction is integration of structured psychological knowledge, including cognitive-behavioral techniques and coping strategies [2508.12935].
In this sense, RLFF-ESC marks a shift in ESC research from turn-level supportive style toward outcome-oriented policy learning. Its central contribution is not merely stronger response generation, but a redefinition of supportive dialogue optimization around simulated future emotional trajectories, goal completion, and conversation efficiency.
Source: https://www.emergentmind.com/topics/rlff-esc