PSI-Bench: Evaluation of Depression Simulators

• PSI-Bench is an automatic evaluation framework for depression patient simulators that measures behavioral fidelity at turn, dialogue, and population levels.
• It leverages clinically grounded dimensions—narrative-emotion processes, emotion expression, lexical diversity, response length, and linguistic markers—to compare real and simulated conversations.
• The methodology reveals simulation shortcomings such as excessive verbosity, overly structured emotional transitions, and reduced behavioral variability compared to genuine patient interactions.

PSI-Bench, short for Patient-Simulator-Bench, is an automatic evaluation framework for depression patient simulators: large-language-model systems that play the role of a depressed patient in therapist–patient dialogue for mental-health training. Its central aim is to evaluate whether simulated patients behave like real depressed patients, rather than merely producing responses that appear plausible to an LLM judge. To do so, PSI-Bench compares real and simulated conversations along clinically grounded, interpretable dimensions at turn-level, dialogue-level, and population-level resolution, and converts the resulting discrepancies into normalized similarity scores on a $0$–$100$ scale (Hoang et al., 28 Apr 2026).

1. Motivation and scope

PSI-Bench was introduced in response to a specific methodological gap in the evaluation of mental-health patient simulators. Depression simulators are increasingly used because they offer scalable and repeatable exposure to complex, sensitive interactions, but the fidelity of the simulated patient matters: unrealistic behavior can distort expectations about how depressed patients disclose, emote, resist support, or improve over the course of a dialogue (Hoang et al., 28 Apr 2026).

The framework is motivated by two difficulties that are especially acute for depression simulation. First, safety constraints on LLMs can inhibit faithful imitation of depressive speech and distress patterns. Second, depression is highly heterogeneous, so a useful simulator must capture variation across patients rather than reproduce a single stereotyped interaction style. PSI-Bench therefore treats realism as a matter of distributional alignment with real patient behavior, not as a single scalar plausibility judgment (Hoang et al., 28 Apr 2026).

A further motivation is the paper’s critique of prevailing evaluation practice. Prior simulator assessments often rely on LLM judges asked to rate realism or profile adherence with generic, clinically under-specified prompts. PSI-Bench is designed to avoid using an LLM as an end-to-end realism oracle. Instead, it decomposes simulator behavior into explicit diagnostics such as narrative progression, emotional trajectory, lexical diversity, verbosity, and depression-marker prevalence and rate. This makes failure modes inspectable rather than opaque (Hoang et al., 28 Apr 2026).

The framework is an intrinsic fidelity benchmark. It is not framed as diagnosis, treatment assessment, or safety certification. Its target is the behavioral realism of simulated patients in supportive, counseling, and psychotherapy-adjacent dialogues conditioned on structured patient profiles (Hoang et al., 28 Apr 2026).

2. Clinical grounding and evaluated dimensions

PSI-Bench is described as “clinically grounded” because its dimensions are drawn from psychological and psycholinguistic research on depression and psychotherapy process. It is “interpretable” because the output is not a monolithic realism score but a set of dimension-specific alignment measures (Hoang et al., 28 Apr 2026).

The five core dimensions are Narrative-Emotion Processes (NEP) markers, emotion expression, lexical diversity, response length, and linguistic markers of depression. These span the three analysis levels emphasized by the framework.

Dimension	Primary level(s)	Operationalization
Narrative-Emotion Processes	Dialogue, population	Turn-index label distributions compared with JSD
Emotion expression	Dialogue, population	Plutchik-plus-neutral distributions compared with JSD
Lexical diversity	Dialogue, population	Conversation-level MTLD distribution compared with Wasserstein distance
Response length	Turn	Words per message and words per sentence
Linguistic markers of depression	Turn, population	Marker prevalence per message and rate per 1,000 tokens

For NEP, each patient message is classified as Problem, Transition, Change, or Filler. The category definitions are explicitly clinical in orientation. Problem denotes confusion, helplessness, complaint, being stuck, or lack of insight. Transition denotes beginning reflection, perspective-taking, or consideration of alternatives. Change denotes acceptance, insight, reframing, planning, hope, or empowerment. Filler denotes procedural or neutral content without therapeutic substance (Hoang et al., 28 Apr 2026).

For emotion expression, each patient turn is assigned a dominant emotion from Plutchik’s eight primary emotions—trust, fear, anger, disgust, joy, anticipation, sadness, and surprise—plus neutral. This is intended to capture not merely whether a simulator sounds “sad,” but whether its emotional trajectory over turns resembles real depressed-patient dialogue (Hoang et al., 28 Apr 2026).

For linguistic markers of depression, PSI-Bench tracks three lexical classes: absolutist words such as “always” or “never,” depressive words drawn from a depression-domain lexicon, and non-fluencies such as “uh,” “um,” “hmm,” “you know,” and “I mean.” These are measured both as marker prevalence—the proportion of messages containing at least one marker—and as marker rate, the number of occurrences per 1,000 tokens (Hoang et al., 28 Apr 2026).

Response length and lexical diversity serve as clinically motivated discourse-level diagnostics. The benchmark is explicitly designed to detect whether simulators are too verbose, too lexically rich, or too homogeneous across profiles, all of which are treated as fidelity failures rather than stylistic quirks (Hoang et al., 28 Apr 2026).

3. Measurement framework and scoring methodology

PSI-Bench evaluates real and simulated corpora independently, computes discrepancies between them, and converts those discrepancies into similarity scores on a $0$–$100$ scale. The general rule stated in the paper is that distance or divergence measures $d$ are converted to similarity scores as $100-d$, with higher values indicating higher alignment (Hoang et al., 28 Apr 2026).

For dialogue-level progression, both NEP and emotion are modeled as turn-indexed population distributions across the first $T=16$ turns. If $\psi_i$ denotes the simulator distribution at turn $i$ and $r_i$ the real-patient distribution, the average divergence is

$100$0

where $100$1 is the symmetric Jensen–Shannon divergence (Hoang et al., 28 Apr 2026).

For response length, PSI-Bench compares simulator and human averages for words per message and words per sentence. The manuscript’s printed equation is described as garbled, but the intended form is a symmetric log-ratio similarity,

$100$2

with the final length score given by averaging the message-level and sentence-level similarities (Hoang et al., 28 Apr 2026).

For lexical diversity, the unit of analysis is the whole conversation. All patient messages in a conversation are concatenated and scored with MTLD (Measure of Textual Lexical Diversity). Only conversations with at least 100 patient tokens are included. The simulator and real distributions of conversation-level MTLD are then compared using the Wasserstein distance, which is converted to similarity through min–max normalization (Hoang et al., 28 Apr 2026).

For depression markers, PSI-Bench explicitly combines two statistics that can diverge in practice: prevalence across messages and density per 1,000 tokens. The benchmark uses absolute difference for prevalence and relative percentage difference for the token-normalized rate; the final distance is the average of the two before conversion to similarity (Hoang et al., 28 Apr 2026).

A notable methodological nuance is that PSI-Bench avoids LLMs as end-to-end judges, but it still uses gpt-oss-120b as an automatic labeler for NEP and dominant emotion. The paper treats this as a component in a structured pipeline rather than as a holistic realism evaluator, and later validates its outputs against expert judgments (Hoang et al., 28 Apr 2026).

4. Data sources, simulator frameworks, and evaluation protocol

The initial instantiation of PSI-Bench uses the Eeyore dataset, which pairs real patient conversations with structured patient profiles. The real dialogue corpus is drawn from three public sources: AnnoMI for motivational interviewing, HOPE for counseling, and ESC for emotional-support conversations. In the appendix, the paper reports 167 AnnoMI transcripts, 308 HOPE transcripts, and 923 ESC transcripts, for a total of 1398 conversations after merging consecutive messages from the same speaker so that turn-taking is comparable to synthetic dialogues (Hoang et al., 28 Apr 2026).

The patient profiles used for conditioning contain clinically structured information such as symptom severities, cognitive distortions, resistance toward support, depression severity, suicidal ideation severity, and counseling history. PSI-Bench evaluates fidelity to real patient behavior under this conditioning, not diagnosis accuracy or symptom extraction (Hoang et al., 28 Apr 2026).

For each real profile, the evaluation protocol generates a corresponding synthetic conversation using the same profile, with the simulator producing patient turns and a fixed therapist model—gpt-oss-20b—producing therapist turns. Generation is capped at 16 turns or the number of patient turns in the real conversation, whichever is smaller. This standardization is important because the framework’s NEP and emotion progression analyses are turn-indexed up to $100$3 (Hoang et al., 28 Apr 2026).

The benchmark evaluates two simulator frameworks. The first is PATIENT-$100$4, described as a CBT-based framework using structured patient cognitive models. The second is Roleplay-doh, described as a human–LLM collaborative framework in which expert-defined qualitative principles govern the roleplayed patient, with a principle-adherence refinement pipeline. Across these frameworks, PSI-Bench evaluates seven LLM backbones: GPT-4.1 mini, gpt-oss-20b, gpt-oss-120b, Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Qwen3-30B-A3B-Instruct-2507, and Qwen2.5-72B-Instruct, yielding 14 framework–LLM configurations (Hoang et al., 28 Apr 2026).

The human-validation study involves 20 mental health experts, recruited via Prolific and divided into four groups of five, with job roles including psychologist, clinical psychologist, counselor, occupational therapist, and social worker. The experts perform two tasks: message-level classification of NEP and emotion, and pairwise realism comparison between synthetic dialogue snippets conditioned on a patient profile and shown alongside a real reference conversation (Hoang et al., 28 Apr 2026).

5. Empirical findings and benchmark results

PSI-Bench reports systematic divergences between real and simulated depression-patient dialogue. The most prominent finding is verbosity. Human patients average 18.24 words per message and 9.16 words per sentence, whereas simulator outputs range from 63.91 to 318.90 words per message and 15.30 to 27.94 words per sentence. This yields low response-length similarity scores, typically in the 20s–40s (Hoang et al., 28 Apr 2026).

A second major result is excess lexical richness combined with reduced variability. Human conversations have mean MTLD $100$5 with standard deviation $100$6 and corpus-level “All” $100$7. Simulator means are often substantially higher and their standard deviations markedly lower; reported examples include PS-qwen2.5-72b: mean $100$8, std $100$9, PS-gpt-4.1-mini: mean $0$0, std $0$1, and RD-gpt-oss-120b: mean $0$2, std $0$3. The paper interprets the narrow synthetic distributions as evidence of reduced behavioral variability across simulated personas (Hoang et al., 28 Apr 2026).

The progression metrics show premature emotional and narrative resolution. In NEP space, all populations begin mostly in Problem, but real patients remain in that state much longer: the paper reports that over 40\% are still in Problem throughout the first 16 turns. Simulators, by contrast, move away from Problem by about turn 3. Emotionally, simulated patients follow a relatively uniform negative-to-positive trajectory: fear and sadness early on, then a rapid rise of trust, joy, and anticipation by about turn 5. Real patients are more mixed, more neutral, and less uniformly improving (Hoang et al., 28 Apr 2026).

PSI-Bench also finds that simulators lack neutral and filler behavior. Real conversations often contain filler and neutral content, and about 25–50\% of profiles do not show explicit emotion at a given turn. Simulators almost always produce content that maps to a clear NEP stage and a clear emotion. This contributes to an impression of over-organization and excessive communicative efficiency (Hoang et al., 28 Apr 2026).

The marker analysis reveals an instructive inversion between density and prevalence. Human combined marker statistics are 23.60 occurrences per 1,000 tokens and 34.47\% message prevalence. Example simulator outputs include PS-llama3.1-8b: rate $0$4, prevalence $0$5 and RD-gpt-oss-120b: rate $0$6, prevalence $0$7. This indicates that synthetic patients mention depression-related markers in many more messages, but because the messages are much longer, the marker density is lower (Hoang et al., 28 Apr 2026).

The reported overall ranking is led by PATIENT-$0$8 variants, and the paper highlights that simulation framework has a larger impact on fidelity than model scale.

Rank	System	Overall score
1	PATIENT-$0$9 with Llama-3.1-8B-Instruct	62.54
2	PATIENT-$100$0 with Qwen3-30B-A3B-Instruct-2507	61.47
3	PATIENT-$100$1 with Qwen2.5-72B-Instruct	59.42
4	Roleplay-doh with Llama-3.3-70B-Instruct	56.61
5	PATIENT-$100$2 with Llama-3.3-70B-Instruct	56.29
6	Roleplay-doh with Llama-3.1-8B-Instruct	53.66
7	Roleplay-doh with Qwen2.5-72B-Instruct	52.68
8	PATIENT-$100$3 with GPT-4.1 mini	50.90
9	PATIENT-$100$4 with gpt-oss-20b	50.37
10	Roleplay-doh with Qwen3-30B-A3B-Instruct-2507	47.73
11	PATIENT-$100$5 with gpt-oss-120b	47.33
12	Roleplay-doh with GPT-4.1 mini	45.90
13	Roleplay-doh with gpt-oss-20b	38.43
14	Roleplay-doh with gpt-oss-120b	33.98

Dimension-wise best or near-best results are also reported: best NEP and best Emotion are both achieved by PS-llama3.1-8b-i, with 83.96 and 71.26 respectively; best Lexical Diversity is RD-llama3.1-8b-i with 73.43; best Length and best Markers are PS-qwen3-30b-a3b-i with 43.89 and 67.97 (Hoang et al., 28 Apr 2026). The paper interprets the overall pattern as suggesting that larger, stronger LLMs may produce overly coherent, polished, or therapist-like language, which can reduce patient fidelity.

6. Validation, interpretation, and limitations

A central claim of PSI-Bench is that its automatic diagnostics align with expert judgment. In the human study, agreement between the benchmark pipeline and expert-majority labels is reported as 91.67\% with Cohen’s $100$6 for pairwise comparison, 80.06\% with $100$7 for NEP, and 86.83\% with $100$8 for emotion. The corresponding human–human Fleiss’ $100$9 values are 0.4328, 0.4997, and 0.5712. Using the interpretation reported in the paper, human–human agreement is moderate, whereas PSI-Bench agreement with expert majorities is substantial to almost perfect (Hoang et al., 28 Apr 2026).

The qualitative comments from experts reinforce the quantitative results. Less realistic simulators were described as overly structured, polished, too articulate too early, too self-reflective, and too solution-oriented, while more realistic conversations were described as shorter, less certain, more fragmented, and more likely to exhibit hesitation or self-correction. This strongly parallels the benchmark’s findings on verbosity, non-fluencies, and premature transition into therapeutic “change” (Hoang et al., 28 Apr 2026).

Several limitations are explicit or readily stated in the paper. PSI-Bench is designed specifically for depression patient simulators and does not directly transfer to other disorders without redesign. It measures five clinically motivated dimensions, but not all clinically relevant behavior. It still depends on gpt-oss-120b for NEP and emotion labeling, even though it avoids LLMs as black-box judges. Its progression analyses are capped at the first 16 turns, and its MTLD analysis excludes conversations with fewer than 100 patient tokens. In addition, the real reference data comes from a mixture of motivational interviewing, counseling, and support-chat corpora rather than a single homogeneous psychotherapy corpus (Hoang et al., 28 Apr 2026).

Within those boundaries, PSI-Bench functions as a methodological shift in simulator evaluation. It replaces opaque realism scoring with clinically motivated, distributional comparison against real patient dialogue, and it makes visible a recurring set of failure modes in current systems: overly long responses, excess lexical diversity, reduced population variability, premature movement from problem to change, and a uniform negative-to-positive emotional trajectory. A plausible implication is that future progress in patient simulation will depend at least as much on framework design and behavioral control as on raw LLM scale (Hoang et al., 28 Apr 2026).