LLM-Simulated Users

• LLM-Simulated Users are algorithmic agents generated through fine-tuning and prompt engineering that replicate human behavior across diverse domains.
• They overcome limitations of real user data collection by providing scalable, controllable, and cost-efficient simulation for evaluation and experimental design.
• Key methodologies include persona and profile construction, modular memory and action components, and adaptive conditioning to ensure dynamic, context-aware responses.

LLM-simulated users are algorithmic agents, instantiated by prompting or fine-tuning LLMs, to imitate the behaviors, preferences, goals, and decision patterns of human end-users across diverse applications such as recommender systems, task-oriented dialogue, usability and product testing, education, and agentic benchmarking. These agents are surfacing as a scalable and versatile alternative or complement to real-user data collection, enabling systematic, controllable, and low-cost evaluation, data augmentation, and experimental design.

1. Conceptual Frameworks and Motivations

LLM-simulated users address fundamental limitations of conventional user modeling: the expense, latency, and bias of real human data collection, and the lack of flexibility and generalization in rule-based or simplistic supervised simulators. By leveraging LLMs' capacity for in-context reasoning, dynamic response construction, and adaptation to varied tasks, these agents can mimic both population-level statistics and individual traits, and can be tightly conditioned on context, profiles, or domain knowledge.

Distinct instantiation paradigms exist. In dialogue and recommendation, LLM-simulated users are constructed through enhanced persona prompts, role-based system messages, or by fine-tuning on real user logs with task-aligned objectives (Sekulić et al., 2024, Kim et al., 2024). In agentic simulations, personas may be generated to maximize heterogeneity or diversity, including serial and parallel LLM prompting filtered via clustering, or guided by demographic coverage (Ataei et al., 2024, Ahmad et al., 18 Feb 2025). In interactive environments—such as education (Zhang et al., 2024), multi-agent spatial simulations (Almutairi et al., 9 Oct 2025), and legal proceedings (Zhang et al., 24 Aug 2025)—LLMs are equipped with symbolic states, roles, memories (short/long-term), or explicit strategy modules to produce context-consistent and role-appropriate actions.

2. Methodologies for Building and Training LLM-Simulated Users

2.1 Persona and Profile Construction

LLM-simulated users are synthesized via structured prompts or templates encoding demographics, preferences, goals, and behavioral features. Example methodologies include:

• Parallel and serial generation: Sampling multiple agent profiles in parallel and subsequently filtering for diversity via text embeddings + K-Means, or sequential generation with prior-agent context to force heterogeneity (Ataei et al., 2024).
• Information-rich user profiles: JSON-style records encode attributes such as age, gender, occupation, Big Five personality scores, interests, communication style, primary/secondary goals, and behavioral traits, which are either hand-crafted, LLM-generated, or mixed from real datasets (Ahmad et al., 18 Feb 2025, Wang et al., 2023).

2.2 Memory, Reasoning, and Action Modules

Human-like behaviors are approximated by dividing an LLM agent's architecture into modular components:

• Profile module: Static self-description.
• Memory module: Storage and retrieval of sensory input, short-term clustered experiences, and long-term “insights,” with mechanisms for summarization, forgetting (e.g., via recency-weighted functions), and memory-based reflection (Wang et al., 2023).
• Action module: Decision prompts incorporating profile, relevant memories, and environment observations; explicit function-action spaces in clinical and embodied agents (Voigt et al., 19 Aug 2025, Philipov et al., 2024).

2.3 Prompt Engineering and Fine-Tuning

Both prompt-only and parameter-efficient fine-tuning strategies are employed:

• Prompt templates: Context-rich instructions, including complete user/item histories, schemas, or dialogue goals, and role-anchored system messages dictate LLM behavior (Subbaraman et al., 27 Nov 2025, Sekulić et al., 2024, Kim et al., 2024).
• Adaptive conditioning: Prompts may encode the entire decision/interaction history, recent state summaries, and context—identity, goals, conversation, or action space function signatures—to ensure stateful, history-aware responses (Ebrat et al., 2024, Voigt et al., 19 Aug 2025).
• Fine-tuning: E.g., DAUS, a domain-aware user simulator for task-oriented dialogue, uses LoRA adapters on Llama-2 and is optimized by cross-entropy on gold utterance sequences, outperforming rule-based and 2-shot LLM baselines in completion and hallucination rates (Sekulić et al., 2024).

2.4 Ensemble and Logic/Statistical Models

Hybrid techniques have been proposed to address preference model opacity and simulation reliability:

• Explicit logic/statistical ensemble: Synthesizing user engagement via a majority-vote scheme combining logical keyword-matching, semantic-similarity mapping, and statistical user embeddings, as in Zhang et al. (Zhang et al., 2024).
• Plug-in architectures: Modular plugin managers orchestrate multiple response-generation and memory-update plugins, supporting controlled, scalable, and human-in-the-loop simulation (Zhu et al., 2024).

3. Application Domains and Architectures

LLM-simulated users are deployed in a broad spectrum of domains, usually with application-specific adaptations:

Domain	Simulation Features and Mechanisms
Recommender Systems	Profile-based simulation, ensemble engagement models, RL augmentation (Subbaraman et al., 27 Nov 2025, Zhang et al., 2024, Ebrat et al., 2024)
Dialogue/TOD Systems	Goal- and history-conditioned autoregressive LLM generation, fine-tuning (Sekulić et al., 2024, Ahmad et al., 18 Feb 2025)
Usability Testing	Massively parallel persona instantiation, System 1/System 2 dual loops, replay and interview interfaces (Lu et al., 13 Apr 2025)
Product/Needs Elicitation	Agent diversity maximization, context-aware interviews, latent needs extraction (Ataei et al., 2024)
Educational Simulation	Multi-agent collaborative classroom models, meta-planners for procedural control (Zhang et al., 2024, Yue et al., 2024)
Legal/Professional Simulation	Multi-role agent scaffolding, memory/planning/reflection, staged trial procedures (Zhang et al., 24 Aug 2025)
Embodied/Task Environments	Dialogue-act prediction/fine-tuning, function-call action spaces, continuous observation/action cycles (Philipov et al., 2024, Voigt et al., 19 Aug 2025)
Clinical/Therapeutic Assessment	Psychological personas, symptom traits, turn-linked self-report questionnaires (Wang et al., 2024)
Research Ideation, Teams	Expert persona graphs, role-diverse critique, mind-mapping, team simulation in 2D spatial environments (Liu et al., 2024, Almutairi et al., 9 Oct 2025)

4. Evaluation Protocols, Metrics, and Empirical Results

Evaluation of LLM-simulated users is multi-faceted, spanning success on downstream tasks, fidelity to human distributions, diversity, bias, and explainability.

• Cold-start recall in recommendation: LLM-based user augmentation in two-tower systems yields significant relative gains in cold Recall@50 (Beauty: +10 % over best feature, Sports: +13 %), with statistical significance (p < 1e–5), validated against random/user-feature selection and ablations (Subbaraman et al., 27 Nov 2025).
• Dialogue goal fulfillment and hallucinations: DAUS achieves entity P/R/F1 ≈ 0.91, doubles success over comparable LLMs, and cuts hallucination and looping rates at least in half (Sekulić et al., 2024).
• Dynamic adaption and explainability: Simulated user summaries and explanations (Lusifer) enable agents to adapt over time, with RMSE ≈ 1.31 vs. 0.94 for static CF, while supporting transparent reasoning traceability (Ebrat et al., 2024).
• Team and professional simulation validity: Virtual trial agents (SimCourt) and team simulators (VirT-Lab) match or exceed human expert role ratings on procedural, legal, or coordination metrics, with reliability benchmarks through human studies and ablation on memory/planning modules (Zhang et al., 24 Aug 2025, Almutairi et al., 9 Oct 2025).
• Diversity and bias: Entropy, KL divergence, and skewness of sampled synthetic profiles are used to measure population-level variance; controlled sampling (statistical prompts, rejection sampling) is required to avoid LLM-induced demographic or attribute bias (Ahmad et al., 18 Feb 2025, Ataei et al., 2024).
• Survey simulation validity: LLM-Mirror yields Jensen–Shannon divergence and Wasserstein distances to human response distributions of 0.067–0.105 vs. 0.267–0.407 for baselines, with individual-level percent agreement ≈ 69–71 % (Kim et al., 2024).
• Systematic differences from real users: Large-scale studies reveal consistent discrepancies in planning style, engagement, politeness, and hallucination prevalence between LLM-simulated and human users, with simulated users often over-polite, more willing to adopt suggestions, and less likely to trigger factual errors in agents (Wang et al., 22 Sep 2025, Seshadri et al., 23 Jan 2026).

5. Limitations, Calibration, and Fairness Concerns

Systematic analyses reveal key weaknesses of LLM-simulated users as proxies for real end-users:

• Calibration error and fairness: Across τ-Bench tasks, simulated users overestimate agent success on easy tasks and underestimate on hard ones ("Expected Calibration Error" up to 15.1 %, dialect/group disparities up to 20.3 %), differentially disadvantaging real-world populations (e.g., AAVE, Indian English speakers) (Seshadri et al., 23 Jan 2026).
• Artifacts and behavioral mismatches: Simulated users produce more verbose, question-rich, and polite outputs, surface different dialogue error types, and can skew agent tuning toward unnatural interaction patterns (Wang et al., 22 Sep 2025, Seshadri et al., 23 Jan 2026).
• Hallucination awareness: LLM-simulated users less frequently surface agent hallucinations compared to humans, limiting their utility for real-world failure analysis (Wang et al., 22 Sep 2025).
• Persona/role drift: In long or multi-role simulations, simulated users may lose consistency with initial profiles without repeated persona anchoring and memory mechanisms (Wang et al., 2024, Zhang et al., 24 Aug 2025).
• Data leakage and over-specificity: Without careful plugin design or prompt filtering, simulated users leak knowledge of system-internal data, reducing realism (Zhu et al., 2024).

Best practices include multi-model evaluation, direct calibration against human sessions, or hybrid simulation/human-in-the-loop protocols to maintain robustness and validity (Seshadri et al., 23 Jan 2026).

6. Practical Guidelines and Design Patterns

For reproducibility, reliability, and interpretability in LLM-simulated user studies, effective practices include:

• Conditioning on specific prior information: For survey/testbed fidelity, always inject respondent attributes and relevant history into prompts or as persona-paragraph summaries (Kim et al., 2024, Zhu et al., 2024).
• Policy learning for data efficiency: Reinforcement learning-driven user selection (via policy gradients or contextual bandits) can optimize which histories/providers to sample for maximal downstream task gain (Subbaraman et al., 27 Nov 2025).
• Memory, planning, and reflection: Multi-level agent memory modules and explicit strategy or meta-planning facilitate temporally extended, context- and task-consistent user simulation in complex roles (Zhang et al., 24 Aug 2025, Zhang et al., 2024).
• Controllable parameterization: Use plug-in managers, modular intent parsers, and personality/policy sampling controls to scale, personalize, and test edge-cases (Zhu et al., 2024, Ataei et al., 2024, Liu et al., 2024).
• Diversity and bias monitoring: Explicit entropy/KL-based filtering and re-sampling maintain target population structure, avoid LLM-sample collapse, and allow for the study of demographic or behavioral minority groups (Ahmad et al., 18 Feb 2025, Ataei et al., 2024).

7. Emerging Directions and Open Problems

Current research identifies multiple frontiers and unresolved challenges:

• Human-likeness gap: Addressing behavioral artifacts, persona drift, over-politeness, and lack of negative or neutral feedback to match true user diversity and engagement patterns (Wang et al., 22 Sep 2025, Seshadri et al., 23 Jan 2026).
• Cross-domain and multi-modal generalization: Extending user simulation architectures to new languages, modalities (vision, audio), and procedural/factual domains (Sekulić et al., 2024, Zhang et al., 24 Aug 2025).
• Automated needs elicitation and product design: Scaling up latent need discovery via context-diverse persona generation and action/observation reflection, with demonstrated ability to outperform conventional lead-user interviews on latent innovation yield (Ataei et al., 2024).
• Interactive simulation at scale: Integrating utterance/action-level feedback, dynamic state updates, and on-the-fly agent interviews to support iterative usability and experimental design pipelines (Lu et al., 13 Apr 2025, Ebrat et al., 2024).
• Ethics and over-reliance: Avoiding confirmation and authority bias, providing transparency on persona source and provenance, and warning against using LLM agents as direct substitutes for at-risk populations or in sensitive applications (Liu et al., 2024, Voigt et al., 19 Aug 2025, Wang et al., 2024).

LLM-simulated users represent a rapidly advancing paradigm for scalable, diverse, and domain-transferable user modeling. Their continued development and deployment, however, require systematic calibration, fairness-aware design, and human-grounded validation to ensure their outputs remain credible proxies for nuanced, real-world user populations.