Know You Before You Speak: User-State Modeling for LLM Personalization in Multi-Turn Conversation

Abstract: Personalized dialogue requires more than recalling explicit user histories: systems also need to infer hidden user states that evolve through interaction and shape appropriate response strategies. Existing memory- and profile-based methods primarily reuse observable user information, offering limited support for modeling user-state dynamics or selecting actions based on how they shape future user states. We propose PUMA (Prospective User-state Modeling for Action selection), a framework grounded in the Free Energy Principle (FEP) that formulates personalization as decision-making under partial observability, centered on an explicit user state model that captures latent user states and their action-conditioned dynamics. At each turn, PUMA maintains a belief over the user's hidden state, refines the user state model for observation generation and action-conditioned state transition, and selects dialogue actions by minimizing expected free energy, balancing epistemic and pragmatic objectives under a unified criterion. This formulation shifts personalization from passive memory retrieval to model-based decision-making over user evolution. We instantiate PUMA on healthcare-oriented counseling and motivational interviewing benchmarks with latent state annotations for rigorous evaluation. Experiments show that PUMA improves long-horizon dialogue outcomes while maintaining strong response quality, and a cross-dataset study demonstrates more reliable user-state estimation and next-state prediction.

• The paper introduces PUMA, an active-inference framework that combines belief updating, action-conditioned world modeling, semantic memory, and expected-free-energy planning for personalized dialogue.
• The paper reports higher counseling performance than strong LLM baselines, including a 1.62 versus 1.07 Stage Lift, 75.9% versus 52.0% preparation rate, and 0.717 next-state prediction accuracy with Qwen3-8B.
• The paper shows that belief updating, world-model input, and EFE-based action selection each improve results, while noting that dynamic gains rely on an author-built simulator and externally defined user-state labels.

Motivation and problem statement

Personalized dialogue systems have largely been built around explicit user records: persona descriptions, preference profiles, retrieved memories, and long interaction histories. The authors of PUMA argue that this framing is insufficient for long-horizon personalization because the factors that determine an appropriate response are often latent rather than stated. The same utterance — for example, a client reporting two weeks on new medication — may call for entirely different counselor behavior depending on whether the underlying state involves anxiety, confidence, or quiet reconsideration of discontinuation. Effective personalization therefore requires inferring hidden user states and reasoning about how system actions shape their evolution, not merely retrieving what the user has said.

The paper positions existing methods along a spectrum: memory-augmented and profile-based systems operate over observable text and handle stable facts well but do not model latent dynamics; latent-variable dialogue models introduce hidden representations that are typically static or descriptive, without action-conditioned transitions. Under either approach, personalization reduces to retrieval or conditioning rather than sequential decision-making under partial observability. PUMA (Prospective User-state Modeling for Action selection) addresses this gap by grounding personalized dialogue in the Free Energy Principle (FEP) and active inference (2605.24647).

Framework

PUMA treats the user as a partially observable dynamical system. The framework separates three components:

• User state: a latent variable $s_t$ capturing the user's decision-relevant internal condition at turn $t$, maintained as a variational belief $q_\phi(s_t)$.
• World model: a factorization $$p_\theta(o_t, s_t \mid s_{t-1}, a_{t-1}) = p_\theta(s_t \mid s_{t-1}, a_{t-1})\, p_\theta(o_t \mid s_t)$$, where the transition model captures action-conditioned state evolution and the observation model captures how states surface in language.
• Semantic memory: a two-level store (short-term session context, long-term cross-session preferences) that supplies content-level personalization independently of state tracking.

Inference and control follow the two standard FEP objectives. Belief updating minimizes variational free energy, balancing KL consistency with the predictive prior against likelihood fit to the observed utterance. World-model learning optimizes the same objective with respect to $\theta$, yielding a trajectory-level loss combining observation likelihood and action-conditioned transition prediction. Action selection minimizes expected free energy (EFE), decomposed into an epistemic term (expected posterior entropy over the next state, favoring uncertainty-reducing actions) and a pragmatic term (negative log-preference of predicted outcomes, favoring goal-directed actions). The selected action conditions response generation together with the belief and retrieved memory.

Two implementation choices deserve emphasis. First, PUMA is realized as a set of LLM-based inference modules with structured JSON interfaces rather than trained neural components; the "world model" is maintained as textual rule sets updated incrementally via add/modify/drop operations. Second, the user-state space is instantiated with gold-labeled annotations from the evaluation datasets, making the internal representation directly evaluable. Both choices trade fidelity of the probabilistic formalism for practicality, and the paper does not claim end-to-end differentiable inference.

Experimental results

Evaluation uses CAMI, a counseling and motivational interviewing benchmark with per-turn Transtheoretical Model stage annotations (precontemplation, contemplation, preparation), and a profile-grounded client simulator, DynPatient, built for dynamic policy comparison. The simulator itself is validated against prior simulators, achieving the highest consistency (4.39 vs. 3.68) and realism (3.76 vs. 3.00) scores among compared systems while maintaining low action-distribution divergence.

Dynamic counseling effectiveness shows substantial gains. With Qwen3-8B, PUMA reaches Stage Lift 1.62 versus 1.07 for the strongest baseline (DIIR), preparation rate 75.9% versus 52.0%, and trigger coverage 62.4% versus 34.0%, while completing sessions in fewer turns (12.2 vs. 13.1). Results with Llama-3.1-8B are similar or stronger (Lift 1.76, Prep 83.0%). These are large margins relative to baselines, though all dynamic results depend on the simulator environment discussed below.

Counselor-side quality is evaluated with MITI Global Scores judged by an LLM. PUMA achieves MITI-Avg 4.37 (Qwen3-8B) and 4.27 (Llama-3.1-8B), exceeding all automated baselines by more than a point and notably scoring above the human-reference responses (3.85). The claim that an automated method surpasses human counselors on judged MI quality should be read cautiously given LLM-judge evaluation, but it indicates the framework does not sacrifice conversational quality for outcome optimization.

Ablations confirm each component contributes: removing belief updating, world-model input, or EFE-based selection each costs roughly 0.17–0.21 Lift, and removing both world model and planning costs 0.28. An interesting inversion appears in trigger coverage: removing world-model input raises TrigCov slightly (63.8% vs. 62.4%), which the authors attribute to the planner becoming biased toward exploratory trigger discovery rather than transition-aware intervention — evidence that EFE's pragmatic term genuinely redirects exploration toward state progression.

State estimation supports the mechanism directly. On gold-annotated turns, PUMA achieves Curr-Acc 0.689 and Next-Acc 0.717, versus 0.667/0.231 for a long-prompt baseline — the next-state prediction gap being the most striking result, indicating that the transition model, not context length, drives prospective accuracy. Next-state tracking improves from 0.613 in early turns to 0.757 after turn 10, consistent with belief accumulation.

Cross-dataset generalization

On AnnoMI, a real-counseling dataset with utterance-level client-talk annotations replacing TTM stages, PUMA with Qwen3-8B attains Curr-Acc 0.639 and Next-Acc 0.532, outperforming a Qwen3-32B long-prompt baseline (0.602/0.430). This suggests the belief-and-transition machinery transfers across state annotation schemes and scales favorably relative to backbone size, although only the state-modeling component — not full dynamic control — is tested here.

Limitations and open questions

Several constraints qualify the results. All dynamic effectiveness findings rest on DynPatient, an LLM-driven simulator whose readiness dynamics, trigger bonuses, and calibrated thresholds are constructed by the authors; the paper explicitly concedes that simulator-based results should not be interpreted as clinical effectiveness evidence. Counselor-side quality relies on LLM judges, so the human-comparison result inherits judge bias risk. The user-state space is externally annotated rather than learned, leaving open how states would be discovered in domains without established taxonomies such as the Transtheoretical Model or MISC action vocabulary. Finally, the FEP objectives are approximated through prompt-engineered modules with fixed hyperparameters ($\lambda_e = 0.4$, $\lambda_p = 0.6$, soft-fusion $\beta = 0.35$); whether learned, differentiable instantiations of the same objective would outperform these approximations remains untested.

Conclusion

PUMA reframes LLM personalization as model-based decision-making over a partially observed user, unifying belief updating, world-model refinement, and expected-free-energy action selection within one criterion. Across simulated motivational interviewing benchmarks it delivers consistent improvements in state tracking, next-state prediction, and long-horizon counseling outcomes over strong LLM baselines, with ablations attributing gains to each component. The principal caveat is dependence on simulator-based evaluation and externally defined state spaces; validating the framework against real users and learning state representations without gold annotations remain open problems.