Persona-Query Benchmark: Dynamic Personalization

• Persona-Query Benchmark is a framework that tests LLMs on dynamic personalization by inferring evolving user states from long multi-session dialogues.
• It challenges models to select responses based on latent dynamic attributes rather than static details, using distractors that mimic outdated user information.
• Empirical results show top models achieve around 52% accuracy, emphasizing the difficulty of long-context, in-situ personalization.

PERSONAMEM is a benchmark for evaluating whether LLM chat assistants can infer, update, and apply an evolving user profile from long multi-session interaction histories, then select a personalized response to a new first-person user query that matches the user’s current state rather than stale or irrelevant past information (Jiang et al., 19 Apr 2025). It targets dynamic personalization rather than static persona recall: a model must read up to 60 sessions and up to 1M tokens of prior dialogue, infer latent dynamic attributes from the history, and choose the single response that aligns with the user’s present preferences, traits, and recent life events. In reported experiments, current frontier models achieved only around 50% overall accuracy, with top models peaking near 52%, indicating that long-horizon user awareness remains a material bottleneck for contemporary LLM systems (Jiang et al., 19 Apr 2025).

1. Conceptual basis and formal task definition

PERSONAMEM formalizes personalization as reasoning over an evolving latent profile state. Let $S_t$ denote the user’s latent profile state at time $t$, comprising static demographic attributes—name, age, gender identity, racial identity, occupation—and dynamic attributes such as traits, preferences, and events. Static attributes are given to models, whereas dynamic attributes must be inferred from the interaction history $H_t = (h_1, h_2, \dots, h_t)$, where each $h_t$ is a multi-turn user–model session (Jiang et al., 19 Apr 2025).

Profile evolution is modeled as

$S_{t+1} = f(S_t, h_t),$

where $f$ composes profile updates induced by the current session, including newly expressed preferences, life events, and reasons for changes. At evaluation time, the user issues an in-situ query $q_t$ from the first-person perspective, and the model must generate or select a response $r_t$ conditioned on the full prior context and basic demographics. The benchmark defines a unique ground-truth target $g_t$ among four candidates, where the correct choice is the one consistent with the current $S_t$ and $t$0 rather than outdated profile states (Jiang et al., 19 Apr 2025).

The principal metric is accuracy:

$t$1

PERSONAMEM also reports query-type accuracies $t$2 over seven skill categories and distance-sensitive performance conditioned on how many sessions have elapsed since the relevant preference was last mentioned. This distance-based slicing is central because the benchmark explicitly probes whether relevant user facts are preserved, displaced, or overwritten over long conversational timelines (Jiang et al., 19 Apr 2025).

For open-weight models, PERSONAMEM also supports generative scoring. Each candidate response is scored by length-normalized joint token probability under the provided history and query, and the candidate with highest score is selected. This design lets the benchmark compare chat-completion APIs and open-weight autoregressive models under a common multiple-choice target while accommodating different model interfaces (Jiang et al., 19 Apr 2025).

2. Benchmark composition and construction pipeline

PERSONAMEM contains 20 curated personas and over 180 simulated interaction histories, with multiple valid interleavings per persona. Histories span 10, 20, or 60 sessions, corresponding approximately to 32k, 128k, and 1M tokens of context; each session contains 15–30 turns. The benchmark covers 15 real-world personalization tasks, including therapy, legal advice, books, music, movies, food, family, dating, health, finance, travel planning, online shopping, studying tips, and home decoration. Across these histories, it provides about 6k in-situ query–response pairs (Jiang et al., 19 Apr 2025).

The benchmark is organized around seven targeted skills.

Skill type	Evaluation target	Failure mode stressed
Recall user-shared facts	Recover previously shared facts	Omission
Suggest new ideas	Propose novel but aligned content	Genericity
Acknowledge latest preferences	Reflect most recent preference state	Staleness
Track full preference evolution	Narrate how preferences changed	Timeline collapse
Revisit reasons behind updates	Use causes of preference changes	Causal forgetting
Provide preference-aligned recommendations	Tailor concrete suggestions	Boilerplate
Generalize to new scenarios	Transfer reasons across domains	Weak abstraction

Persona construction begins by sampling base personas from PersonaHub, then augmenting them with demographics and a general personal timeline covering education, career, and life events. For each of 15 topics, the pipeline generates time-stamped personal histories, initial preferences, possible updates, and explicit reasons for those updates, while ensuring that initial preferences are not duplicated across topics. Each timeline segment is then expanded into a multi-turn user–LLM conversation (Jiang et al., 19 Apr 2025).

The simulation pipeline includes two specific quality-control stages. First, an internal event citation step forces the generator, GPT-4o, to identify the relevant event before writing each turn. Second, a self-reflection pass reviews sessions to fill in missing events so that preference-update sequences remain coherent. Session histories are then topologically sorted by end timestamps, interleaved across topics while preserving within-topic causality, and supplemented with occasional short non-personal interactions. This produces multiple valid long-context histories without regenerating the entire timeline from scratch (Jiang et al., 19 Apr 2025).

Ground-truth responses are persona- and event-grounded under the current $t$3, while distractors are designed to be plausible but outdated or irrelevant. Questions answerable without context are excluded. Human validation on 90 randomly sampled entries, annotated by three evaluators, reported proportions marked True of 97.8% for appropriateness with $t$4, 95.6% for relevance with $t$5, 97.8% for correctness with $t$6, and 90.0% for best-response with $t$7 (Jiang et al., 19 Apr 2025).

3. Evaluation protocol, settings, and memory augmentation

In the main evaluation setting, models receive the full conversation history $t$8 up to time $t$9 and the user’s basic demographics, but not any structured summary of dynamic attributes. They must infer all dynamic characteristics from the dialogue itself. The core task is four-way multiple-choice selection: one option is correct under the current profile state, and the other three encode stale or irrelevant facts and preferences (Jiang et al., 19 Apr 2025).

PERSONAMEM defines two scoring regimes. In the discriminative regime, used for the headline results, the model sees all four options in randomized order and selects one option with a brief justification. In the generative regime, used for open-weight models, each option is scored separately via joint token probabilities with length normalization. Evaluations are conducted at 128k and 1M token windows, with some memory-module experiments run at 32k for efficiency (Jiang et al., 19 Apr 2025).

The benchmark also evaluates two memory-augmented variants. In the RAG configuration, the system retrieves the top-5 relevant messages per question using dense BGE-M3 embeddings and prepends those snippets to the prompt. In the Mem0 configuration, the system builds an LLM-generated fact database over turns and retrieves the top-5 relevant facts at inference time. The recommended reporting protocol includes overall accuracy, per-query-type accuracy, distance-binned accuracy by session position, and with/without-memory deltas. The benchmark explicitly advises against leaking dynamic user histories as structured summaries, because the task is intended to stress inference from the conversation timeline itself (Jiang et al., 19 Apr 2025).

The model suite used in the primary study includes GPT-4.5, GPT-4.1, o1, o3-mini, o4-mini, GPT-4o, GPT-4o-mini, Gemini-2.0-Flash, Gemini-2.0-Flash-Lite, Gemini-1.5-Flash, DeepSeek-R1-671B, Llama-4-Maverick, Llama-3.1-405B, Claude-3.7-Sonnet, and Claude-3.5-Haiku. This breadth is important because PERSONAMEM is intended not as a single-model stress test but as a comparative framework for long-context personalized response selection (Jiang et al., 19 Apr 2025).

4. Empirical results and characteristic error patterns

The benchmark’s central empirical finding is that contemporary LLMs remain weak at dynamic personalization over long horizons. Frontier models such as GPT-4.5, GPT-4.1, and Gemini-1.5 score around 52% at best, many models cluster around 50%, and Llama-4-Maverick is reported at approximately 43%. Reasoning-labeled models, including o1, o3-mini, o4-mini, and DeepSeek-R1, do not show a clear advantage on these tasks, suggesting that the limiting factor is not generic abstract reasoning but memory, retrieval, and temporal profile tracking (Jiang et al., 19 Apr 2025).

Performance varies strongly by query type. Models are comparatively stronger on recalling user-shared facts, tracking preference evolution, and revisiting reasons behind updates, with reported ranges around 60–70%. They are notably weaker on incorporating the latest preference into responses, suggesting new ideas, providing preference-aligned recommendations, and generalizing to new scenarios. The lowest performance is observed on cross-scenario transfer, where a model must use the reason for a preference change in one domain to personalize advice in a different domain (Jiang et al., 19 Apr 2025).

Long-context position effects are pronounced. Accuracy is higher when the relevant information appears near the beginning or end of the timeline and drops when it is buried in the middle, exhibiting the “lost in the middle” effect. The benchmark therefore operationalizes long-context personalization not merely as storage capacity but as a retrieval-and-selection problem over temporally dispersed evidence (Jiang et al., 19 Apr 2025).

The reported error analysis identifies several recurrent failure modes. Models often miss dynamic changes and continue to recommend according to stale preferences, such as continuing Italian-food suggestions after a switch to Mediterranean due to gluten issues. They also produce generic, non-personal answers even when specific user preferences are present in the history. Another recurrent issue is misreading first-person persona-query framing, leading to impersonal or irrelevant advice. Relevant mid-history events are often underused, while more salient but outdated facts are over-weighted. Finally, models are particularly weak at cross-scenario transfer, such as applying a preference for routine-friendly yoga schedules to a new meal-planning context (Jiang et al., 19 Apr 2025).

Memory augmentation improves performance but does not remove the core difficulty. At 32k context, both RAG and Mem0 increase accuracy, with RAG generally outperforming Mem0 while also being more efficient. The largest gains are reported for fact recall and generalization to new scenarios, while gains are smaller for revisiting reasons behind updates. This suggests that lightweight retrieval can alleviate some memory bottlenecks, but robust dynamic personalization still requires more than extracting isolated facts (Jiang et al., 19 Apr 2025).

5. Position within the personalization benchmark landscape

PERSONAMEM occupies a specific niche within the expanding literature on persona- and personalization-oriented evaluation. Relative to Persona-Chat-style work, it emphasizes multi-session, temporally grounded evolution of user states rather than single-session role-playing or static persona conditioning (Jiang et al., 19 Apr 2025). Relative to long-context memory benchmarks, it tests up-to-date personalization under distractors that encode stale but plausible profile states, making temporal correctness, not just retrieval, the decisive criterion (Jiang et al., 19 Apr 2025).

Several adjacent benchmarks clarify this positioning. BehaviorChain evaluates continuous persona-based behavior simulation over 1,001 personas and 15,846 behaviors, focusing on iterative next-behavior inference rather than personalized response selection from first-person user queries (Li et al., 20 Feb 2025). PERSONA is a pluralistic alignment testbed built from 1,586 synthetic personas and 317,200 preference pairs, targeting value diversity and role-playing fidelity rather than long-horizon profile evolution (Castricato et al., 2024). PersonaGym evaluates persona agents across 200 personas and 10,000 questions using a decision-theoretic PersonaScore over five task types, stressing free-form persona adherence across environments rather than temporally updated user-state tracking (Samuel et al., 2024).

Other contemporaneous benchmarks differentiate personalization axes more directly. PersonaConvBench studies personalized reasoning and generation in multi-user Reddit conversations across 10 domains and three tasks—sentence classification, impact regression, and user-centric text generation—using real conversational trajectories rather than simulated long-session assistant histories (Li et al., 20 May 2025). PersonaFeedback decouples persona inference from personalization by supplying explicit personas in 8,298 human-annotated binary-choice cases, thereby isolating response tailoring under controlled persona information (Tao et al., 15 Jun 2025). PersoBench focuses on zero-shot persona-aware response generation with automatic metrics for fluency, diversity, coherence, and personalization, but does not center dynamic profile evolution over long multi-session timelines (Afzoon et al., 2024).

This comparison indicates that PERSONAMEM is defined less by persona realism alone than by the conjunction of four properties: temporally evolving user states, first-person in-situ persona-queries, long-context histories reaching 1M tokens, and distractor design that explicitly tests whether the model can reject stale profile information. A plausible implication is that PERSONAMEM functions as a bridge benchmark between static persona-conditioned generation and future systems that require continuously updated user models.

6. Limitations, ethical considerations, and research significance

The benchmark’s authors identify several limitations. The personas and histories are synthetic, so although they are curated and human-validated, they may not fully represent real-world diversity, behavioral variability, or preference dynamics. Because the benchmark is generated from seed personas and LLM-based simulation, it may encode cultural or demographic biases. Results obtained in this synthetic setting may not transfer directly to production environments with noisier histories, cross-modal signals, or more ambiguous user behavior (Jiang et al., 19 Apr 2025).

The benchmark also foregrounds privacy concerns. Although PERSONAMEM itself uses synthetic data, it models a regime in which assistants retain and act on extensive personal histories. The paper therefore points to data minimization, consent, retention policies, and redaction of sensitive attributes as central concerns for real deployments. This is not incidental: the benchmark’s value partly lies in exposing how much a model can infer from seemingly ordinary conversational residue (Jiang et al., 19 Apr 2025).

Research-wise, PERSONAMEM’s main contribution is diagnostic. It shows that models can often enumerate or recall past preferences yet still fail to respond in a way that reflects the user’s current state. It also shows that reasoning-branded models are not automatically better at personalization and that simple retrieval augmentation helps but does not solve dynamic user modeling. This suggests that progress will likely require tighter integration of long-context retrieval, temporally aware profile tracking, and causal modeling of why preferences change, rather than merely scaling generic instruction-following or chain-of-thought capabilities (Jiang et al., 19 Apr 2025).

In that sense, PERSONAMEM reframes personalized assistance as a sequential inference problem over evolving latent states. Rather than asking whether a model “knows” a user in the abstract, it asks whether the model knows which version of the user is current now.