Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

Abstract: Evaluating LLMs as multi-turn conversational partners requires probing capabilities that single-turn benchmarks miss: persona consistency, evolving intent tracking, emotional dynamics, and goal completion. We introduce EYT-Bench, a human-centered benchmark built around a three-party decoupled design: a persona-grounded user simulator, a target model that separates intent perception from response generation, and an independent third-party LLM judge with optional multi-judge ensembling. Personas are sampled from public human-curated corpora, Nemotron-Personas-USA and PersonaMem-v2, rather than synthesized, reducing LLM-induced persona bias. EYT-Bench also introduces two trajectory-level metrics: embedding-based intent drift and final-intent completion rate (FICR), inspired by tau-bench. In a 17-target x 200-dialogue evaluation, EYT-Bench reveals four findings: (i) state-of-the-art closed- and open-source models are statistically close on subjective dimensions (empathy / persona / anthropomorphism vary within <= 0.3), but differ by up to 9x on objective intent tracking; (ii) reasoning ("thinking on") sharply improves objective tracking on long-context personas (+0.47-0.50 latent-intent accuracy on Gemma-4) while leaving subjective scores nearly unchanged; (iii) persona format dominates trajectory spread, with FICR saturating above 0.95 on Nemotron-USA but spreading from 0.53 to 0.88 on PersonaMem-v2; and (iv) the warm-up effect is robust on 16/17 models (one outlier, GPT-5.5, reverses the effect), with stable rankings across alpha in [0.05, 0.15]. A cross-judge ablation using deepseek-v4-pro confirms that target rankings and final-intent satisfaction are preserved across judges.

• The paper introduces EYT-Bench, which decouples user simulation, target modeling, and judging to address biases in multi-turn dialogue evaluations.
• It employs two human-curated persona pools and ChatSEEDs to ensure reproducible, context-rich dialogue generation and reliable metric tracking, including FICR.
• Experiments reveal that trajectory-level metrics and reasoning switches are key to differentiating model capabilities, paving the way for more robust evaluations.

EYT-Bench: A Human-Centered Evaluation Benchmark for Multi-Turn Dialogue

Motivation and Benchmark Design

The evaluation of LLMs in multi-turn dialogue has become essential as conversational assistants are increasingly expected to demonstrate persona consistency, nuanced intent tracking, emotional responsiveness, and trajectory-level goal completion across extended interactions. Previous benchmarks typically exhibit systematic limitations: synthetic persona bias, entangled evaluation components (leading to self-preference bias), and reliance on turn-level accuracy metrics that fail to capture user goal attainment.

EYT-Bench addresses these gaps by leveraging two human-curated persona pools and implementing a rigorously decoupled evaluation protocol involving independent user simulation, target modeling, and third-party LLM-based judging.

Figure 1: EYT-Bench framework: Decouples user simulation, target modeling, and judging to mitigate self-preference and enhance metric robustness.

Persona Pool Construction and ChatSEED Specification

EYT-Bench employs two complementary persona pools: Nemotron-Personas-USA, featuring structured demographic attributes, and PersonaMem-v2, composed of paragraph-form profiles distilled from real user–assistant interactions. PersonaMem-v2 introduces high context diversity and conversational realism, enabling evaluation of models' reasoning capabilities over long interaction horizons.

ChatSEEDs define the deterministic initial state and the goal for each dialogue episode, explicitly binding persona, topic, initial emotion, and various intent labels. This conditioning ensures that generated dialogues are reproducible and that intent tracking metrics are well-grounded.

Figure 2: ChatSEED example, encapsulating persona, topic, initial emotion, and explicit/latent/final intent to seed deterministic, goalful dialogue.

Evaluation Metrics and Decoupled Pipeline

EYT-Bench's evaluation pipeline is strictly decoupled: the user simulator emits persona-conditioned utterances and structured annotations (explicit intent, latent intent, emotion, intent progress), the target model operates with distinct perception and generation prompts (eliminating rubric leakage), and the judge scores responses on a multi-dimensional Likert rubric.

Objective metrics include per-turn intent and emotion accuracy, embedding-based intent drift, and the judge-adjudicated final-intent completion rate (FICR), which measures whether the assistant meaningfully fulfills the user's conversational goal.

Trajectory-level metrics are central: they quantify whether the model's responses guide the interaction towards goal completion, with intent drift and FICR capturing longitudinal performance.

Figure 3: Judge rubric decomposes Empathy, Persona Alignment, and Anthropomorphic Interaction into granular sub-indicators for nuanced subjective assessment.

Figure 4: Statistical distributions of persona, topic, and emotion across Nemotron and PersonaMem pools, highlighting diversity and context complexity.

Subjective and Objective Findings

Across 17 models and 200 dialogues, EYT-Bench uncovers several critical findings:

• Closed- and open-source models exhibit minimal spread on subjective metrics, yet differ by up to an order of magnitude ($9\times$) in objective latent intent accuracy.
• Enabling reasoning (thinking on) induces a phase transition in objective intent tracking on PersonaMem-v2, improving latent intent accuracy by $+0.47$--$0.50$ and significantly reducing intent drift. The effect is negligible on structured Nemotron personas.
• Persona format is the dominant factor in goal completion metrics: FICR saturates on Nemotron-USA ($\geq 0.95$ for most models) and demonstrates discriminative spread ($0.53 \to 0.88$) on PersonaMem-v2.
• A robust warm-up effect is evident: early-turn Empathy and Persona Alignment scores are lower, and aggregate rankings are preserved across warm-up weight $\alpha$ values. GPT-5.5 is the unique outlier with reversed turn quality.
• Cross-judge ablation confirms stability: replacing the Gemini-3.1-Pro-Thinking judge with deepseek-v4-pro yields consistent rankings and nearly identical FICR satisfaction signals, validating metric robustness to judge-model calibration.

Figure 5: Judge prompts for Empathy, Persona Alignment, Anthropomorphic Interaction; each expands to five binary indicators, ensuring rubric interpretability and reproducibility.

User Simulator and System Prompt Design

The LLM-based user simulator is conditioned on ChatSEED fields and outputs both utterances and structured annotations, including a self-reported progress label. Early termination is triggered when final intent is judged achieved for two consecutive turns.

Figure 6: User-simulator prompt template, integrating persona, topic, emotion, and final intent for controlled yet contextually rich simulation.

The dialogue model under evaluation receives a system prompt that enforces empathetic, concise, persona-matched responses, maximizing alignment with human expectations.

Figure 7: System prompt template guiding the assistant toward empathetic and persona-consistent dialogue generation.

Perception and Generation Decoupling

The evaluation loop invokes perception and response-generation stages separately in the target model. This architectural separation prevents rubric contamination, ensuring that intent prediction and response quality remain independently assessable.

Figure 8: Perception-stage prediction prompt; intent and emotion JSON emission is isolated from generative response prompt to support unbiased label evaluation.

Implications and Future Directions

EYT-Bench demonstrates that, at the frontier, subjective Likert metrics offer little differentiation between advanced closed- and open-source models. Trajectory-level objective metrics—especially on long-context, paragraph-form personas—emerge as the key levers for discriminating model capability. The reasoning "on/off" switch exhibits qualitative changes in intent tracking, underscoring the need for evaluation settings that stress long-horizon comprehension and reasoning.

Practically, EYT-Bench exposes the necessity for rigorous decoupling in simulation, modeling, and judging to avoid confounded and self-reinforcing evaluations. The cross-judge stability of FICR suggests it is a viable metric for standardized, cross-benchmark comparison.

From a theoretical perspective, persona format and dialogue context length are critical experimental variables; benchmark design must explicitly account for these factors. The methodology sets a precedent for evaluation infrastructure extensible to languages, domains, and richer metrics.

Conclusion

EYT-Bench constitutes a highly configurable, robust benchmark for multi-turn dialogue evaluation, foregrounding objective metrics and mitigating LLM-as-judge biases via three-way decoupling. It evidences that subjective metrics are saturated among advanced models, while trajectory-based intent tracking and persona format are the principal discriminators. The framework's extensibility and principled metrics make it an essential tool for advancing human-centered dialog model evaluation (2607.10428).