---
title: 'PsychiatryBench: Benchmarking LLMs in Psychiatry'
url: https://www.emergentmind.com/topics/psychiatrybench
type: topic
---

# PsychiatryBench: Benchmarking LLMs in Psychiatry

PsychiatryBench denotes a psychiatry-focused benchmarking framework for evaluating large language models on clinically meaningful psychiatric tasks. In the explicit title usage, it refers to "Psychiatry-Bench: A Multi-Task Benchmark for LLMs in Psychiatry," a benchmark grounded exclusively in authoritative, expert-validated psychiatric textbooks and casebooks and comprising eleven question-answering tasks totaling over 5,300 expert-annotated items [2509.09711]. In the surrounding literature, the term also functions more broadly as a label for psychiatry-oriented evaluation designs, including real-EMR clinical benchmarks, DSM-grounded diagnostic decision suites, end-to-end virtual encounters, and ethics-focused psychiatric evaluation modules [2503.01903][2602.03340][2602.12871][2607.08257]. This dual usage is important: PsychiatryBench is both a specific benchmark and, plausibly, a shorthand for an emerging benchmark ecosystem in computational psychiatry.

## 1. Definition and historical positioning

PsychiatryBench emerged in response to a recurrent limitation in mental-health NLP evaluation: existing resources had focused on small clinical interview corpora, social-media-derived datasets, or synthetic dialogues, all of which constrain clinical validity and underrepresent the complexity of psychiatric reasoning [2509.09711]. The benchmark’s stated aim is to probe sophisticated reasoning rather than narrow screening or sentiment detection, covering diagnostic reasoning, treatment selection, longitudinal follow-up, management planning, clinical approach, sequential case analysis, and exam-style psychiatric assessment [2509.09711].

The benchmark sits within a rapidly diversifying research lineage. "PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice" evaluates five psychiatry-specific tasks on 300 de-identified real inpatient cases from three psychiatric centers in China [2503.01903]. "MentalDx Bench," introduced alongside MentalSeek-Dx, targets disorder-level psychiatric diagnosis from 712 de-identified electronic health records under ICD-11 guidelines [2602.03340]. "MentalBench" uses a psychiatrist-built and validated DSM-5 knowledge graph to generate 24,750 synthetic clinical cases varying in completeness and diagnostic complexity [2602.12871]. "MentalHospital" shifts the unit of evaluation from static questions to full psychiatric encounters, instantiating the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning workflow with 1,193 de-identified psychiatric EHR cases and specialist-aligned evaluators [2607.08257].

This literature suggests that PsychiatryBench is best understood not as a single design choice but as a family of benchmark philosophies. Some benchmarks maximize source authority through textbooks and casebooks; some maximize ecological validity through de-identified hospital records; some maximize diagnostic interpretability through DSM-5 or ICD-11 logic; and some maximize process fidelity through interactive simulation.

## 2. Internal structure of the benchmark

In its explicit benchmark form, PsychiatryBench comprises eleven distinct tasks spanning the psychiatric workflow [2509.09711]. The benchmark standardizes inputs through fields such as History, Mental State Examination, Physical Examination, and Question, while adapting the output format to the clinical objective of each task [2509.09711].

The task taxonomy is as follows:

- **Diagnosis**: infer the most likely diagnosis or construct a differential diagnosis from a detailed case vignette.
- **Treatment**: recommend an individualized, evidence-based treatment plan consistent with guidelines.
- **Treatment Follow-Up**: interpret evolving symptoms or side effects and adjust treatment longitudinally.
- **Classification (multi-label)**: assign disorder category labels and specific disorder labels to case narratives.
- **Management Plan**: generate holistic psychiatric management plans including diagnostics, safety planning, psychosocial interventions, pharmacologic steps, coordination, and follow-up.
- **Clinical Approach**: outline the stepwise clinical approach, including history-taking priorities, differential structure, investigations, and initial management.
- **Mental QA**: define psychiatric terms and concepts with clinical precision.
- **Sequential Question Answering**: perform multi-turn case reasoning across ordered questions on differential diagnosis, supporting evidence, etiological factors, treatment options, prognosis, and DSM-5 class mapping.
- **MCQs**: answer single-best-answer psychiatric questions.
- **EMI**: solve extended matching item clusters with shared option pools.
- **Exam Simulations**: complete grouped exam-style multiple-choice sets [2509.09711].

The benchmark reports 5,308 total items and task-specific counts including 467 diagnosis items, 258 treatment items, 27 treatment follow-up items, 418 classification items, 337 management plan items, 56 clinical approach items, 326 mental QA items, 32 sequential QA vignettes, 1,473 MCQs, 277 EMI clusters, and 600 exam items [2509.09711]. An "EMI Separated" format with 1,037 single items is provided as an evaluation convenience rather than a distinct task [2509.09711].

A central design choice is heterogeneity of reasoning form. Some tasks are free-text and open-ended; some are objective selection tasks; some are inherently longitudinal. This is significant because psychiatric competence is not reducible to single-turn diagnosis. PsychiatryBench explicitly includes treatment adaptation, prognosis, management planning, and sequential case analysis, thereby operationalizing psychiatric reasoning as a workflow rather than a label-assignment problem [2509.09711].

## 3. Source curation, annotation, and evaluation formalism

PsychiatryBench is grounded in expert-authored educational sources rather than social media, patient forums, or model-generated synthetic cases. Its materials were manually curated from authoritative psychiatric textbooks and casebooks, including DSM-5 Clinical Cases, DSM-5-TR Self-Exam Questions, Stahl’s Essential Psychopharmacology and associated case studies, Case Files Psychiatry, 100 Cases in Psychiatry, Core Clinical Cases in Psychiatry, Clinical Cases in Psychiatry, Geriatric Psychiatry, MRCPsych revision MCQs and EMIs, and MCQs in Psychology [2509.09711]. Subject matter experts manually extracted high-yield vignettes and exam items, standardized style and metadata, and filtered ambiguity, redundancy, and overly simple prompts [2509.09711].

For the multi-label classification task, disorder category and specific-disorder lists were derived from DSM-5-TR and ICD-11 and validated by a licensed psychiatrist [2509.09711]. Where label reconciliation was needed, Gemini 2.5 Pro was used to map source names to standardized lists, followed by human verification [2509.09711]. The paper does not report inter-rater agreement statistics such as Cohen’s $\kappa$ for the broader curation pipeline [2509.09711].

Evaluation combines conventional task metrics with an LLM-as-judge similarity framework. For MCQ-type tasks, the benchmark uses accuracy and balanced accuracy:
$$
BA = \frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right), \qquad
Accuracy = \frac{TP+TN}{TP+TN+FP+FN}.
$$

For multi-label classification, it uses subset accuracy and weighted-average $F_1$:
$$
\text{Subset Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{I}(\hat{Y}_i = Y_i),
$$
$$
F_{1,l} = 2 \times \frac{P_l \times R_l}{P_l + R_l}, \qquad
\text{F1-weighted} = \sum_{l \in \mathcal{L}} w_l \cdot F_{1,l}.
$$

For EMI clusters, it uses a partial scoring system:
$$
EMI\_Score_i = \frac{1}{S_i}\sum_{j=1}^{S_i} c_{ij}, \qquad
EMI\_Accuracy = \frac{1}{N}\sum_{i=1}^N EMI\_Score_i.
$$
It also defines consistency, agreement, and divergence measures across full-cluster and separated-item formats [2509.09711].

The judge model selected for free-text evaluation is LLaMA 3.3 70B, chosen after comparison with LearnLM, LLaMA-3 70B, and GPT-4o mini [2509.09711]. Two rubric families were tested: a point-based rubric generated by GPT-4.5 and a logic-driven, example-based prompt generated by Gemini. The example-based, logic-driven prompt was adopted for final use because it was more robust across tasks and models [2509.09711]. Free-text task outputs are scored on a 0–100 scale and aggregated by arithmetic mean [2509.09711].

The benchmark is intended for non-commercial research and academic use under fair-use principles, with no patient data and an emphasis on transformative evaluation rather than source substitution [2509.09711].

## 4. Reported model behavior and empirical findings

PsychiatryBench evaluates a broad set of frontier and medical LLMs, including Gemini 2.5 Pro Preview, Gemini 2.5 Flash Preview, Gemini 2.0 Flash, Claude 3.5 Sonnet, DeepSeek R1, DeepSeek Chat, LLaMA 3.3 70B, QWQ-32, Qwen 3 32B, OpenBioLLM, Palmyra-Med-70B, MedGemma, and other medical-domain models [2509.09711]. The reported results show that top models can reach strong similarity scores on several single-turn tasks, but they remain inconsistent on longitudinal and fine-grained psychiatric reasoning [2509.09711].

On **Diagnosis**, the top score is 86.0% for Gemini 2.5 Pro, with QWQ-32 at 85.9% [2509.09711]. On **Treatment**, Gemini 2.5 Flash (Thinking) reaches 82.9%, followed by LLaMA 3.3 at 82.3% and QWQ-32 at 82.1% [2509.09711]. On **Treatment Follow-Up**, Gemini 2.5 Pro reaches 86.5%, QWQ-32 83.7%, and Gemini 2.5 Flash (Thinking) 82.6%, but the total spread across models is large, from 66.5% to 86.5%, which the benchmark highlights as evidence that longitudinal psychiatric management remains difficult [2509.09711].

The **Classification** task exposes a sharper drop in fine-grained psychiatric specificity. For disorder categories, DeepSeek R1 attains weighted $F_1 = 0.72$ and subset accuracy $= 0.54$; for specific disorders, the same model attains weighted $F_1 = 0.50$ and subset accuracy $= 0.44$ [2509.09711]. This difference between category-level and disorder-level performance is consistent with other psychiatry benchmarks, notably MentalDx Bench, which reports strong coarse diagnostic categorization but systematic failure at disorder-level diagnosis across many models [2602.03340].

On **Management Plan**, DeepSeek R1 scores 84.4%, Gemini 2.5 Pro 84.1%, and DeepSeek Chat 83.5% [2509.09711]. On **Sequential QA**, Gemini 2.5 Pro reaches 96.9%, with other models ranging from 81.4% to 90.0% [2509.09711]. On **MCQs**, Gemini 2.5 Pro obtains 81.7% accuracy and 79.3% balanced accuracy under one prompt, and 82.8% accuracy and 74.4% balanced accuracy under another [2509.09711]. On **EMI** full clusters, Gemini 2.5 Pro reaches 87.6–87.9%, while DeepSeek R1 and Gemini 2.5 Flash (Thinking) also exceed 85% [2509.09711]. On **Exam Simulations**, Gemini 2.5 Pro achieves 80.4% on one prompt, while Flash (Thinking) is reported as robust on another [2509.09711].

The headline interpretation is not simple superiority. The paper explicitly reports substantial gaps in clinical consistency and safety, particularly in multi-turn follow-up and management tasks [2509.09711]. Prompt reformulation also matters: in Clinical Approach, one prompt version yields 86.7–90.1%, while another yields 82.8–87.5%, with Gemini 2.5 Pro showing the smallest decline at 1.7% [2509.09711]. This indicates that even high-performing models can be brittle under changes in instructional framing.

## 5. PsychiatryBench as an ecosystem of complementary benchmarks

The broader literature shows that no single psychiatry benchmark covers all relevant axes of evaluation. The current landscape is therefore modular.

| Resource | Primary emphasis | Salient design |
|---|---|---|
| Psychiatry-Bench | Multi-task psychiatry QA | Textbook- and casebook-grounded; 11 tasks [2509.09711] |
| PsychBench | LLM-assisted psychiatric clinical practice | 300 de-identified inpatient EMRs; 5 clinical tasks [2503.01903] |
| MentalDx Bench | Disorder-level psychiatric diagnosis | 712 EHRs; 76 disorders; ICD-11 labels [2602.03340] |
| MentalBench | DSM-5 diagnostic decision-making | MentalKG; 24,750 synthetic cases [2602.12871] |
| MentalHospital | End-to-end psychiatric encounters | SOAP workflow; 1,193 EHR cases; MentalEval [2607.08257] |
| EthicsMH | Ethical reasoning in mental health | 125 scenarios; stakeholder-rich schema [2509.11648] |
| PsychEthicsBench | Jurisdiction-aware ethics | Australian principles; MCQ and OEQ tasks [2601.03578] |
| RSPC | Relational context in mental-health NLP | 1,799 psychiatrist-annotated Reddit posts [2606.27247] |
| PsychBench (epidemiological) | Population fidelity of simulated patients | 28,800 profiles; intersectional audit [2604.17359] |
| PSI-Bench | Depression patient simulator evaluation | Turn-, dialogue-, and population-level diagnostics [2604.25840] |
| LingxiDiagBench | Dynamic psychiatric consultation | 16,000 EMR-aligned synthetic dialogues [2602.09379] |

These resources occupy distinct niches. MentalHospital evaluates full encounters rather than answers to static prompts and reports that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck [2607.08257]. MentalBench isolates DSM-5 rule application and shows that models struggle to calibrate diagnostic commitment when disorders overlap [2602.12871]. MentalDx Bench emphasizes real-world EHR diagnosis and identifies a "paradigm misalignment" between category recognition and disorder-level diagnostic reasoning [2602.03340]. PsychBench evaluates authentic psychiatric clinical practice from real inpatient EMRs and finds that current LLMs are not yet adequate as stand-alone decision-making tools, though they can materially assist junior psychiatrists [2503.01903].

Two ethics-oriented resources are especially relevant to extensions of PsychiatryBench. EthicsMH provides 125 scenarios across confidentiality, autonomy versus beneficence, and bias in AI, with structured fields such as Options, Expected Reasoning, Model Behavior, Real World Impact, and Viewpoints; the paper explicitly proposes it as an ethics module within PsychiatryBench [2509.11648]. PsychEthicsBench, grounded in Australian psychology and psychiatry guidance, shows that refusal rates are poor indicators of ethical behavior and that some domain-specific fine-tuning can degrade ethical robustness [2601.03578].

A plausible implication is that PsychiatryBench is evolving toward a modular architecture: textbook-grounded reasoning, EHR-grounded diagnosis, interactive interview simulation, ethical alignment, population-fidelity auditing, and socially contextual psychiatric inference are increasingly being benchmarked as separable but complementary competencies.

## 6. Conceptual issues, misconceptions, and future directions

A common misconception is that psychiatric benchmarking is equivalent to exam-style multiple-choice performance. PsychiatryBench itself contradicts this by placing diagnosis, treatment, follow-up, management planning, clinical approach, sequential reasoning, EMI clusters, and free-text definitions in the same benchmark [2509.09711]. The surrounding literature strengthens the point: MentalHospital shows that strong performance on isolated psychiatric tasks does not imply competence in complete encounters, and that models lag clinicians substantially on objective psychiatric competence [2607.08257].

A second misconception is that safety can be approximated by refusal. PsychEthicsBench explicitly finds that refusal rates are poor indicators of ethical behavior in mental health and that clinically inadequate refusals can be perceived as unempathetic and discourage help-seeking [2601.03578]. A third misconception is that clinically plausible simulated patients are sufficient for psychiatric evaluation. The epidemiological PsychBench finds a coherence–fidelity dissociation: models produce clinically plausible individuals while misrepresenting real populations, with variance compression from 14 percent to 62 percent and substantial demographic calibration biases [2604.17359]. A fourth misconception is that patient simulation realism can be read off a single holistic judge score. PSI-Bench argues instead for interpretable turn-, dialogue-, and population-level diagnostics, showing that current depression simulators are overly long, less variable, and too uniformly negative-to-positive in emotional trajectory [2604.25840].

The limitations of PsychiatryBench as a specific benchmark are also clear. Its cases are textbook- and casebook-derived rather than drawn from noisy clinical EHRs or spontaneous patient dialogue, its disorder distribution reflects teaching goals rather than epidemiologic prevalence, and its large-scale free-text evaluation depends on LLM-as-judge scoring rather than psychiatrist panel adjudication [2509.09711]. The paper does not specify a public repository URL or formal leaderboard, and train/dev/test splits are not defined because the benchmark is primarily used as a curated test suite [2509.09711].

Future directions are already articulated across the benchmark literature. PsychiatryBench proposes expanded coverage in child and adolescent, perinatal, forensic, and culturally diverse psychiatry, multimodal inputs, automated extraction with human-in-the-loop validation, enhanced safety metrics, calibration analysis, and broader model comparison [2509.09711]. EthicsMH proposes community and expert contributions to scale scenario diversity and cultural representation [2509.11648]. MentalHospital plans richer mental status exams, multimodal simulation, crisis and adversarial safety scenarios, fairness audits, and longitudinal follow-ups [2607.08257]. RSPC argues for sequential inputs, dyadic conversation context, longitudinal trajectories, and multilingual, cross-platform expansion for relationally contextual mental-health NLP [2606.27247].

Taken together, these directions suggest that the long-term significance of PsychiatryBench lies not only in any single leaderboard score, but in the formalization of psychiatric evaluation as a multi-axis problem: diagnostic rigor, treatment adequacy, longitudinal coherence, ethical alignment, interpersonal context, simulator realism, and epidemiological fidelity must all be measured if LLMs are to be assessed credibly in psychiatry.

Source: https://www.emergentmind.com/topics/psychiatrybench