PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators

Abstract: Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is particularly challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically grounded diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, resolve emotions too quickly, and follow a uniform negative-to-positive trajectory. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our benchmark is strongly aligned with expert judgments. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible benchmark to guide future simulator design and evaluation.

• The paper presents a systematic evaluation framework that quantifies key divergences between simulated and real depression patient interactions using clinical benchmarks.
• It measures narrative-emotion processes, lexical diversity, verbosity, and depression markers, revealing accelerated emotional progression and uniform emotional trajectories in simulators.
• Expert validation shows strong alignment with PSI-Bench rankings, highlighting that framework design, rather than LLM scale, is pivotal for simulator fidelity.

PSI-Bench: A Clinically Grounded, Interpretable Benchmark for Evaluating Depression Patient Simulators

Introduction and Motivation

The emergence of LLMs as patient simulators has provided scalable tools for clinical education and counselor training, particularly in complex and sensitive domains such as mental health. However, rigorous, interpretable evaluation of these simulators' fidelity to real-world patient behaviors—especially for depression, where patient language, emotion, and variability are significant—remains a substantial challenge. Most existing evaluation methods rely on generic LLM Likert ratings that lack both clinical grounding and distributional interpretability.

PSI-Bench addresses these gaps by introducing a systematic, multidimensional evaluation framework rooted in established psychological and psycholinguistic evidence. Through PSI-Bench, key divergences between simulators and real patients can be objectively quantified across clinically meaningful axes, providing robust diagnostics for simulator improvement and reliable benchmarks for research advancement.

Figure 1: PSI-Bench generates simulated datasets conditioned on real patient profiles and evaluates real and synthetic conversations across five clinically grounded dimensions, enabling interpretable, population-level simulator diagnostics.

Evaluation Framework Design

PSI-Bench operationalizes simulator assessment along five dimensions, each substantiated by established research in depression and psychotherapy:

• Narrative-Emotion Processes (NEP) Markers: Assesses progression through clinical change stages (Problem, Transition, Change, Filler) based on narrative patterns, reflecting therapeutic dynamics.
• Emotional Expression: Tracks temporal dynamics of positive/negative emotions in patient messages, grounded in Beck's cognitive theory.
• Lexical Diversity: Measures intra- and inter-patient lexical repetition (MTLD), sensitive to the LLMs' propensity for high-variance output.
• Response Length: Quantifies verbosity against human baselines, relevant due to the known terseness of patients with depression.
• Linguistic Markers of Depression: Analyzes presence and density of absolute, depressive, and non-fluency markers using curated lexicons.

Each dimension is scored via similarity/divergence to matched distributions from real patient data, with metrics normalized to a $0$–$100$ alignment scale.

Empirical Findings: Simulator vs. Real Patients

Applying PSI-Bench revealed several consistent and substantive divergences between current depression simulators and real patient communication:

• Accelerated Emotional Progression: Simulators transition rapidly (by turn 3) from problem states to emotional "resolution," deviating from the protracted, often unresolved patterns found in real therapy sessions.

Figure 2: Simulators exhibit accelerated progression through NEP markers, with much earlier transitions from "problem" to "change" categories relative to real patients.

• Uniform Emotional Trajectories: Synthetic patients display a stereotyped shift from fear/sadness to positive emotions within a few conversational turns, whereas real patients show gradual and diffuse emotional trends with persistent neutrality and hesitancy.
• Enhanced Lexical Diversity with Low Variability: Nearly all simulators generate highly diverse and uniform language, in contrast to real patients' more limited, variable, and repetitive lexical patterns.

  Figure 3: Simulators produce higher and more consistent lexical diversity (MTLD) than the wide-ranging and generally lower diversity seen in real patient conversations.

• Excessive Verbosity: Patient messages from LLM simulators are substantially longer (often 3–10x) compared to human responses, reducing authenticity and potentially undermining training efficacy.
• Depression Marker Dispersion: Simulators tend to distribute depression-related linguistic markers thinly but frequently within verbose utterances, resulting in higher prevalence but lower token-normalized density relative to real interactions.

  Figure 4: Simulators have more messages containing depressive markers, but with lower marker density per token compared with real patient messages.

These empirical divergences highlight key behavioral and stylistic mismatches, with implications for instructional realism and potential bias in counseling education.

Simulator and Model Analysis

A comprehensive comparison across two simulation frameworks (PATIENT-$\Psi$ and Roleplay-doh) and seven LLMs revealed several nontrivial patterns:

• Framework Dominance: The design of the simulation framework, not the scale or family of LLM, is the primary determinant of simulator fidelity. PATIENT-$\Psi$ consistently outperforms Roleplay-doh across LLMs, suggesting that cognitive modeling and principled grounding are more impactful than raw generative capacity.
• Model Scale Paradox: Larger LLMs do not guarantee improved simulation; smaller models often yield higher-fidelity output. This counters the prevailing assumption that scaling inherently enhances human-likeness, and indicates that linguistic sophistication can in fact reduce the naturalness of depressive patient simulations.
• Backbone Effect: Llama-based models yield the most consistent high-ranking results, outperforming both Qwen and GPT-backbones in most framework pairings.

Human Study and Benchmark Alignment

To validate PSI-Bench, a blinded study was conducted with 20 mental health professionals, who performed both message-level annotation and direct pairwise comparison of synthetic vs. human conversations.

• The PSI-Bench ranking of simulators demonstrated near-perfect agreement with expert aggregate preference ($\kappa = 0.82$).
• Annotators overwhelmingly identified verbosity, excessive organization, articulate self-reflection, and "solution-orientation" as cues of inauthenticity, aligning with quantitative benchmarks.
• More authentic conversations, as judged by experts, were typically shorter, more hesitant, syntactically simpler, and exhibited spontaneous disfluency—attributes captured in top PSI-Bench scores.

These findings confirm both the construct validity and expert-relevance of the PSI-Bench evaluation methodology.

Implications and Future Directions

PSI-Bench introduces a principled, interpretable standard for evaluating the fidelity of depression patient simulators. Practically, this benchmark can guide the retraining, refinement, and selection of simulators for educational and research use. Theoretically, the observed divergences signal critical areas where current LLMs—regardless of size—diverge from lived patient experience, with implications for both simulator improvement and broader studies of language-based clinical AI.

Future extensions may include:

• Explicit modeling or data-driven augmentation for disfluency, hesitancy, and incomplete narrative arc.
• Fine-tuning or reinforcement strategies that penalize verbosity and reward conversational pausing or non-resolution.
• Application of PSI-Bench dimensions to other domains (e.g., anxiety simulations, psychosis) or modalities (audio, video dialogue).

Conclusion

PSI-Bench provides an interpretable, clinically grounded framework for depression patient simulator evaluation, revealing that current LLM-based simulators are overly verbose, prematurely emotionally resolved, and insufficiently diverse relative to real patients. Framework design, rather than LLM capability, is the most influential factor in simulation fidelity. Robust expert alignment establishes PSI-Bench as a reliable tool for future research, development, and deployment of clinical simulators.

Reference:

"PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators" (2604.25840)