Effectively leveraging production user signals for RL training

Investigate methods to effectively leverage production user behavior signals, modeled as binary classifiers predicting events such as conversation continuation and emoji reactions (e.g., p(continue), p(love), p(thumb up), p(thumb down), p(feedback) given system prompt, character instructions, conversation history, and the current response), as differentiable rewards for reinforcement learning training of large language models deployed for social chat applications.

Background

The paper trains a set of user signal models as binary classifiers to predict whether specific user behaviors occur after a model response (e.g., continuing the conversation, reacting with emojis, or providing feedback). These models use the same conversational context as the preference models and are initialized from LLaMA 3.1 checkpoints.

Although many user signals are explored, the authors ultimately found only the p(continue) and p(thumb up) models consistently reliable for rejection sampling data selection. They caution that user signals collected from organic interactions are inherently noisy and biased, and they observed failure modes when optimizing directly against such signals, motivating the open problem of how to use these signals effectively in RL training.

References

We note that effectively leveraging user signals for RL training remains an open research question, and further investigation is encouraged to unlock their full potential.

— CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production  (2603.01973 - Nie et al., 2 Mar 2026) in Subsection “User Signal Models” within Section “Reward Models”

Several directions remain open. First, the current implementation relies on an offline diagnostic pass, and more efficient reachability estimation may further reduce post-training cost. Second, our experiments focus on a fixed semantic index with short four-token identifiers; extending the analysis to longer or hierarchical identifiers may reveal different hint-budget trade-offs. Finally, combining reachability-aware hinting with richer recommendation rewards, such as calibrated ranking rewards or user-feedback rewards, is a promising path toward more robust generative recommendation post-training.

— HCGRec: Hint-Conditioned Generative Recommendation with Semantic IDs  (2608.11980 - Zhang et al., 12 Aug 2026) in Conclusion