- The paper introduces a comprehensive framework that measures the sim2real gap using clustering of multi-dimensional behavioral signatures.
- It applies divergence metrics like JS divergence to reveal substantial differences between simulated and real user behaviors across coding and writing tasks.
- It demonstrates that combining behaviorally complementary simulators can significantly improve simulation fidelity compared to single models.
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
Introduction
Scalable interactive training and evaluation of AI assistants increasingly relies on user simulators, particularly those built from LLMs. However, while recent approaches optimize such simulators to mimic human responses at the utterance level, the critical question of whether these simulators accurately capture the broad, heterogeneous distribution of real user behaviors remains unaddressed. This paper presents a comprehensive methodology for quantifying the distributional gap between real and simulated user behaviors and delivers a systematic analysis across a significant set of LLM-based simulators. The work further proposes and validates interventions for reducing this sim2real gap.
Methodology
Behavioral Representation and Quantization
The core methodology extracts behavioral signatures from both real and simulated user-assistant dialogues by prompting a capable LLM to generate structured behavioral descriptions along six dimensions: requests, responses, context, communication style, DAMSL dialog acts, and SGD dialog acts. By embedding these multi-facet descriptions into a low-dimensional semantic space and subsequently clustering (via k-means), the method forms discrete behavioral distributions for both the real (P) and simulated (Q) populations.
This quantization is essential: direct computation of divergences on high-dimensional embeddings is unreliable, whereas clustering-based distributions enable robust calculation of divergence metrics such as forward/backward KL and Jensen-Shannon (JS) divergences. The approach is validated rigorously through ablations with alternative embedding models and clustering algorithms and through a human "odd-one-out" study demonstrating high alignment between cluster assignments and human similarity judgments.
Experimental Setup
Twenty-four user simulators, spanning 7 closed-source LLMs, 15 open-source LLMs, and 2 trained simulators, are evaluated on coding and writing tasks using samples from WildChat, a large dataset of real user-assistant interactions. The simulated interactions are generated so as to match real user goals extracted from WildChat. Divergence metrics between real and simulator-behavioral distributions are then computed for both tasks.
Main Results
Distributional Gaps
Substantial and persistent distributional gaps are observed across all simulators, with JS divergence from real user distributions often an order of magnitude above the irreducible lower bound established via held-out real user samples. The gap is less pronounced for coding than for writing, but remains significant across tasks. Notably, closed-source models such as GPT-5.4 and Gemini 3.1 Pro perform best, but top-tier open-source models (e.g., gemma-4-31B-it, gpt-oss-120b) close much of the gap and outperform several closed alternatives.
Trained simulators, specifically humanlm-opinion and UserLM-8b, approach the fidelity of leading closed-source LLMs, with the caveat that one (UserLM-8b) is trained on WildChat and thus not fully out-of-domain. Numerical accuracy of the distributional measures is underlined by a classifier's ability to distinguish real from simulated behaviors with ≥90% accuracy for all evaluated simulators.
Analysis by Behavior Facet
Behavioral gaps are non-uniform across facets:
- Requests and Context: Simulators most closely match real distributions.
- Communication Style, DAMSL, and SGD Dialog Acts: Largest divergences, with simulators failing to emulate diversity in expression, dialog act variability, and utterance function. Especially, simulators frequently hallucinate hyper-positive, overly verbose, or inappropriately social behaviors not present in real interactions.
Fine-tuned simulators outperform general-purpose LLMs on these challenging facets, highlighting the importance of targeted behavioral supervision for bridging the sim2real gap.
Model Scale, Family, and Data
Scale alone does not uniformly reduce the distributional gap. For some model families (Gemma, GPT-OSS), larger parameter counts yield improved fidelity; for others (Qwen3.5, Llama), the relationship is non-monotonic, implicating both family design and training data curation as critical.
Simulators Combination
Combining behaviorally complementary simulators further reduces the gap. Randomly selecting between two simulators with divergent behavioral distributions produces mixtures that, in forward- and backward-KL as well as JS, are empirically closer to real user distributions than either constituent alone. However, combining pairs that are already behaviorally similar yields no such benefit.
Interpretability
TF-IDF analysis of clusters uncovers explicit patterns of captured, missed, and hallucinated behaviors. For example, iterative problem-solving and error handling are well-captured, authoritative and terse styles are underrepresented, while hyper-enthusiasm and excessive gratitude are overproduced (hallucinated), especially by certain LLM simulators.
Limitations
The reference distribution leverages English interactions from WildChat, thereby primarily reflecting the IT/developer demographic and not generalizing to broader user populations. The analysis is limited to coding and writing; user behavior in other task families is unexplored. Moreover, differences in assistant models between real and simulated dialogues may induce some artifact in measured divergences. Finally, the framework does not distinguish between undesirable and stress-test divergences—simulator behaviors absent from real user data are not necessarily problematic for certain training objectives.
Practical and Theoretical Implications
The demonstrated gap indicates that user simulators, in their current form, are insufficient proxies for real user populations in training and evaluation pipelines. This can produce systematic biases and poor robustness in downstream AI systems, particularly for stylistic and dialog act diversity. The findings stress the necessity of not only scaling and diversifying training but also incorporating explicit behavioral supervision, possibly via reinforcement learning on behavioral objectives.
The observation that mixtures of complementary simulators enhance fidelity supports further research into meta-simulation strategies, including learned routing—analogous to mixture-of-experts architectures—where simulators are selected or blended dynamically according to task or behavioral context.
The methodology for distributional comparison can be extended to more tasks, more varied datasets, and finer-grained demographic axes, supporting continuous monitoring and correction of user simulator fidelity.
Future Directions
Key directions include: expanding behavioral evaluation to non-English and underrepresented demographics; extending analysis to conversational domains beyond coding and writing; developing targeted data augmentation or adversarial training for hard-to-capture behaviors; learning behavioral router functions over simulator pools; and integrating distributional gap metrics directly into simulator training objectives.
Conclusion
This work provides a robust, interpretable framework for quantifying and analyzing the sim2real gap in user simulators for dialog-based AI. It establishes that current LLM-based simulators systematically fail to match the distributional breadth of real user behaviors and that behavioral complementarity, not merely model scale or parameter count, is essential for progress. These insights are critical for both simulation-based AI training and trustworthy evaluation, motivating new techniques for constructing high-fidelity user simulators that generalize to real-world, diverse populations.
Reference: "Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors".
