---
title: 'PSI-Bench: Evaluation of Depression Simulators'
url: https://www.emergentmind.com/topics/psi-bench
type: topic
---

# PSI-Bench: Evaluation of Depression Simulators

PSI-Bench, short for **Patient-Simulator-Bench**, is an automatic evaluation framework for **depression patient simulators**: large-language-model systems that play the role of a depressed patient in therapist–patient dialogue for mental-health training. Its central aim is to evaluate whether simulated patients behave like **real depressed patients**, rather than merely producing responses that appear plausible to an LLM judge. To do so, PSI-Bench compares real and simulated conversations along **clinically grounded, interpretable dimensions** at **turn-level**, **dialogue-level**, and **population-level** resolution, and converts the resulting discrepancies into normalized similarity scores on a \(0\)–\(100\) scale [2604.25840].

## 1. Motivation and scope

PSI-Bench was introduced in response to a specific methodological gap in the evaluation of mental-health patient simulators. Depression simulators are increasingly used because they offer scalable and repeatable exposure to complex, sensitive interactions, but the fidelity of the simulated patient matters: unrealistic behavior can distort expectations about how depressed patients disclose, emote, resist support, or improve over the course of a dialogue [2604.25840].

The framework is motivated by two difficulties that are especially acute for depression simulation. First, **safety constraints on LLMs** can inhibit faithful imitation of depressive speech and distress patterns. Second, **depression is highly heterogeneous**, so a useful simulator must capture variation across patients rather than reproduce a single stereotyped interaction style. PSI-Bench therefore treats realism as a matter of **distributional alignment with real patient behavior**, not as a single scalar plausibility judgment [2604.25840].

A further motivation is the paper’s critique of prevailing evaluation practice. Prior simulator assessments often rely on **LLM judges** asked to rate realism or profile adherence with generic, clinically under-specified prompts. PSI-Bench is designed to avoid using an LLM as an end-to-end realism oracle. Instead, it decomposes simulator behavior into explicit diagnostics such as **narrative progression**, **emotional trajectory**, **lexical diversity**, **verbosity**, and **depression-marker prevalence and rate**. This makes failure modes inspectable rather than opaque [2604.25840].

The framework is an **intrinsic fidelity benchmark**. It is not framed as diagnosis, treatment assessment, or safety certification. Its target is the behavioral realism of simulated patients in supportive, counseling, and psychotherapy-adjacent dialogues conditioned on structured patient profiles [2604.25840].

## 2. Clinical grounding and evaluated dimensions

PSI-Bench is described as “clinically grounded” because its dimensions are drawn from psychological and psycholinguistic research on depression and psychotherapy process. It is “interpretable” because the output is not a monolithic realism score but a set of dimension-specific alignment measures [2604.25840].

The five core dimensions are **Narrative-Emotion Processes (NEP) markers**, **emotion expression**, **lexical diversity**, **response length**, and **linguistic markers of depression**. These span the three analysis levels emphasized by the framework.

| Dimension | Primary level(s) | Operationalization |
|---|---|---|
| Narrative-Emotion Processes | Dialogue, population | Turn-index label distributions compared with JSD |
| Emotion expression | Dialogue, population | Plutchik-plus-neutral distributions compared with JSD |
| Lexical diversity | Dialogue, population | Conversation-level MTLD distribution compared with Wasserstein distance |
| Response length | Turn | Words per message and words per sentence |
| Linguistic markers of depression | Turn, population | Marker prevalence per message and rate per 1,000 tokens |

For **NEP**, each patient message is classified as **Problem**, **Transition**, **Change**, or **Filler**. The category definitions are explicitly clinical in orientation. *Problem* denotes confusion, helplessness, complaint, being stuck, or lack of insight. *Transition* denotes beginning reflection, perspective-taking, or consideration of alternatives. *Change* denotes acceptance, insight, reframing, planning, hope, or empowerment. *Filler* denotes procedural or neutral content without therapeutic substance [2604.25840].

For **emotion expression**, each patient turn is assigned a dominant emotion from **Plutchik’s eight primary emotions**—trust, fear, anger, disgust, joy, anticipation, sadness, and surprise—plus **neutral**. This is intended to capture not merely whether a simulator sounds “sad,” but whether its emotional trajectory over turns resembles real depressed-patient dialogue [2604.25840].

For **linguistic markers of depression**, PSI-Bench tracks three lexical classes: **absolutist words** such as “always” or “never,” **depressive words** drawn from a depression-domain lexicon, and **non-fluencies** such as “uh,” “um,” “hmm,” “you know,” and “I mean.” These are measured both as **marker prevalence**—the proportion of messages containing at least one marker—and as **marker rate**, the number of occurrences per 1,000 tokens [2604.25840].

**Response length** and **lexical diversity** serve as clinically motivated discourse-level diagnostics. The benchmark is explicitly designed to detect whether simulators are too verbose, too lexically rich, or too homogeneous across profiles, all of which are treated as fidelity failures rather than stylistic quirks [2604.25840].

## 3. Measurement framework and scoring methodology

PSI-Bench evaluates real and simulated corpora independently, computes discrepancies between them, and converts those discrepancies into similarity scores on a \(0\)–\(100\) scale. The general rule stated in the paper is that distance or divergence measures \(d\) are converted to similarity scores as \(100-d\), with higher values indicating higher alignment [2604.25840].

For **dialogue-level progression**, both NEP and emotion are modeled as turn-indexed population distributions across the first \(T=16\) turns. If \(\psi_i\) denotes the simulator distribution at turn \(i\) and \(r_i\) the real-patient distribution, the average divergence is

$$
d = \frac{1}{T} \sum_{i=1}^{T} \mathrm{JSD}(\psi_i \,\|\, r_i),
$$

where \(\mathrm{JSD}\) is the symmetric Jensen–Shannon divergence [2604.25840].

For **response length**, PSI-Bench compares simulator and human averages for **words per message** and **words per sentence**. The manuscript’s printed equation is described as garbled, but the intended form is a symmetric log-ratio similarity,

$$
s(c_{\psi}) = 100 \cdot \exp\left(- \left| \log \frac{c_{\psi}}{c_{r}} \right| \right),
$$

with the final length score given by averaging the message-level and sentence-level similarities [2604.25840].

For **lexical diversity**, the unit of analysis is the whole conversation. All patient messages in a conversation are concatenated and scored with **MTLD** (Measure of Textual Lexical Diversity). Only conversations with at least **100 patient tokens** are included. The simulator and real distributions of conversation-level MTLD are then compared using the **Wasserstein distance**, which is converted to similarity through min–max normalization [2604.25840].

For **depression markers**, PSI-Bench explicitly combines two statistics that can diverge in practice: prevalence across messages and density per 1,000 tokens. The benchmark uses **absolute difference** for prevalence and **relative percentage difference** for the token-normalized rate; the final distance is the average of the two before conversion to similarity [2604.25840].

A notable methodological nuance is that PSI-Bench avoids LLMs as **end-to-end judges**, but it still uses **gpt-oss-120b** as an automatic labeler for NEP and dominant emotion. The paper treats this as a component in a structured pipeline rather than as a holistic realism evaluator, and later validates its outputs against expert judgments [2604.25840].

## 4. Data sources, simulator frameworks, and evaluation protocol

The initial instantiation of PSI-Bench uses the **Eeyore** dataset, which pairs real patient conversations with structured patient profiles. The real dialogue corpus is drawn from three public sources: **AnnoMI** for motivational interviewing, **HOPE** for counseling, and **ESC** for emotional-support conversations. In the appendix, the paper reports **167** AnnoMI transcripts, **308** HOPE transcripts, and **923** ESC transcripts, for a total of **1398** conversations after merging consecutive messages from the same speaker so that turn-taking is comparable to synthetic dialogues [2604.25840].

The patient profiles used for conditioning contain clinically structured information such as **symptom severities**, **cognitive distortions**, **resistance toward support**, **depression severity**, **suicidal ideation severity**, and **counseling history**. PSI-Bench evaluates fidelity to real patient behavior under this conditioning, not diagnosis accuracy or symptom extraction [2604.25840].

For each real profile, the evaluation protocol generates a corresponding synthetic conversation using the same profile, with the simulator producing patient turns and a fixed therapist model—**gpt-oss-20b**—producing therapist turns. Generation is capped at **16 turns** or the number of patient turns in the real conversation, whichever is smaller. This standardization is important because the framework’s NEP and emotion progression analyses are turn-indexed up to \(T=16\) [2604.25840].

The benchmark evaluates two simulator frameworks. The first is **PATIENT-\(\Psi\)**, described as a CBT-based framework using structured patient cognitive models. The second is **Roleplay-doh**, described as a human–LLM collaborative framework in which expert-defined qualitative principles govern the roleplayed patient, with a principle-adherence refinement pipeline. Across these frameworks, PSI-Bench evaluates seven LLM backbones: **GPT-4.1 mini**, **gpt-oss-20b**, **gpt-oss-120b**, **Llama-3.1-8B-Instruct**, **Llama-3.3-70B-Instruct**, **Qwen3-30B-A3B-Instruct-2507**, and **Qwen2.5-72B-Instruct**, yielding **14 framework–LLM configurations** [2604.25840].

The human-validation study involves **20 mental health experts**, recruited via Prolific and divided into four groups of five, with job roles including psychologist, clinical psychologist, counselor, occupational therapist, and social worker. The experts perform two tasks: **message-level classification** of NEP and emotion, and **pairwise realism comparison** between synthetic dialogue snippets conditioned on a patient profile and shown alongside a real reference conversation [2604.25840].

## 5. Empirical findings and benchmark results

PSI-Bench reports systematic divergences between real and simulated depression-patient dialogue. The most prominent finding is **verbosity**. Human patients average **18.24 words per message** and **9.16 words per sentence**, whereas simulator outputs range from **63.91 to 318.90 words per message** and **15.30 to 27.94 words per sentence**. This yields low response-length similarity scores, typically in the 20s–40s [2604.25840].

A second major result is **excess lexical richness combined with reduced variability**. Human conversations have mean **MTLD \(66.3\)** with **standard deviation \(23.8\)** and corpus-level **“All” \(57.5\)**. Simulator means are often substantially higher and their standard deviations markedly lower; reported examples include **PS-qwen2.5-72b: mean \(87.1\), std \(9.48\)**, **PS-gpt-4.1-mini: mean \(100.4\), std \(8.11\)**, and **RD-gpt-oss-120b: mean \(118.1\), std \(9.03\)**. The paper interprets the narrow synthetic distributions as evidence of reduced behavioral variability across simulated personas [2604.25840].

The progression metrics show **premature emotional and narrative resolution**. In NEP space, all populations begin mostly in **Problem**, but real patients remain in that state much longer: the paper reports that **over 40\%** are still in Problem throughout the first 16 turns. Simulators, by contrast, move away from Problem by about **turn 3**. Emotionally, simulated patients follow a relatively uniform **negative-to-positive trajectory**: fear and sadness early on, then a rapid rise of trust, joy, and anticipation by about **turn 5**. Real patients are more mixed, more neutral, and less uniformly improving [2604.25840].

PSI-Bench also finds that simulators lack **neutral and filler behavior**. Real conversations often contain filler and neutral content, and about **25–50\%** of profiles do not show explicit emotion at a given turn. Simulators almost always produce content that maps to a clear NEP stage and a clear emotion. This contributes to an impression of over-organization and excessive communicative efficiency [2604.25840].

The **marker analysis** reveals an instructive inversion between density and prevalence. Human combined marker statistics are **23.60 occurrences per 1,000 tokens** and **34.47\%** message prevalence. Example simulator outputs include **PS-llama3.1-8b: rate \(16.99\), prevalence \(92.35\%\)** and **RD-gpt-oss-120b: rate \(6.56\), prevalence \(90.01\%\)**. This indicates that synthetic patients mention depression-related markers in many more messages, but because the messages are much longer, the marker density is lower [2604.25840].

The reported overall ranking is led by PATIENT-\(\Psi\) variants, and the paper highlights that **simulation framework has a larger impact on fidelity than model scale**.

| Rank | System | Overall score |
|---|---|---:|
| 1 | PATIENT-\(\Psi\) with Llama-3.1-8B-Instruct | 62.54 |
| 2 | PATIENT-\(\Psi\) with Qwen3-30B-A3B-Instruct-2507 | 61.47 |
| 3 | PATIENT-\(\Psi\) with Qwen2.5-72B-Instruct | 59.42 |
| 4 | Roleplay-doh with Llama-3.3-70B-Instruct | 56.61 |
| 5 | PATIENT-\(\Psi\) with Llama-3.3-70B-Instruct | 56.29 |
| 6 | Roleplay-doh with Llama-3.1-8B-Instruct | 53.66 |
| 7 | Roleplay-doh with Qwen2.5-72B-Instruct | 52.68 |
| 8 | PATIENT-\(\Psi\) with GPT-4.1 mini | 50.90 |
| 9 | PATIENT-\(\Psi\) with gpt-oss-20b | 50.37 |
| 10 | Roleplay-doh with Qwen3-30B-A3B-Instruct-2507 | 47.73 |
| 11 | PATIENT-\(\Psi\) with gpt-oss-120b | 47.33 |
| 12 | Roleplay-doh with GPT-4.1 mini | 45.90 |
| 13 | Roleplay-doh with gpt-oss-20b | 38.43 |
| 14 | Roleplay-doh with gpt-oss-120b | 33.98 |

Dimension-wise best or near-best results are also reported: **best NEP** and **best Emotion** are both achieved by **PS-llama3.1-8b-i**, with **83.96** and **71.26** respectively; **best Lexical Diversity** is **RD-llama3.1-8b-i** with **73.43**; **best Length** and **best Markers** are **PS-qwen3-30b-a3b-i** with **43.89** and **67.97** [2604.25840]. The paper interprets the overall pattern as suggesting that larger, stronger LLMs may produce overly coherent, polished, or therapist-like language, which can reduce patient fidelity.

## 6. Validation, interpretation, and limitations

A central claim of PSI-Bench is that its automatic diagnostics align with expert judgment. In the human study, agreement between the benchmark pipeline and expert-majority labels is reported as **91.67\%** with **Cohen’s \(\kappa = 0.8210\)** for pairwise comparison, **80.06\%** with **\(\kappa = 0.6958\)** for NEP, and **86.83\%** with **\(\kappa = 0.7520\)** for emotion. The corresponding human–human **Fleiss’ \(\kappa\)** values are **0.4328**, **0.4997**, and **0.5712**. Using the interpretation reported in the paper, human–human agreement is **moderate**, whereas PSI-Bench agreement with expert majorities is **substantial to almost perfect** [2604.25840].

The qualitative comments from experts reinforce the quantitative results. Less realistic simulators were described as **overly structured**, **polished**, **too articulate too early**, **too self-reflective**, and **too solution-oriented**, while more realistic conversations were described as **shorter**, **less certain**, **more fragmented**, and more likely to exhibit **hesitation** or **self-correction**. This strongly parallels the benchmark’s findings on verbosity, non-fluencies, and premature transition into therapeutic “change” [2604.25840].

Several limitations are explicit or readily stated in the paper. PSI-Bench is designed specifically for **depression patient simulators** and does not directly transfer to other disorders without redesign. It measures five clinically motivated dimensions, but not all clinically relevant behavior. It still depends on **gpt-oss-120b** for NEP and emotion labeling, even though it avoids LLMs as black-box judges. Its progression analyses are capped at the first **16 turns**, and its MTLD analysis excludes conversations with fewer than **100 patient tokens**. In addition, the real reference data comes from a mixture of **motivational interviewing**, **counseling**, and **support-chat** corpora rather than a single homogeneous psychotherapy corpus [2604.25840].

Within those boundaries, PSI-Bench functions as a methodological shift in simulator evaluation. It replaces opaque realism scoring with clinically motivated, distributional comparison against real patient dialogue, and it makes visible a recurring set of failure modes in current systems: **overly long responses**, **excess lexical diversity**, **reduced population variability**, **premature movement from problem to change**, and **a uniform negative-to-positive emotional trajectory**. A plausible implication is that future progress in patient simulation will depend at least as much on **framework design** and **behavioral control** as on raw LLM scale [2604.25840].

Source: https://www.emergentmind.com/topics/psi-bench