PsychiatryBench: Benchmarking LLMs in Psychiatry

• PsychiatryBench is a psychiatry-focused multi-task framework that evaluates LLM clinical reasoning using expert-validated, textbook-derived cases.
• It comprises eleven distinct tasks—from diagnosis to treatment follow-up—ensuring comprehensive evaluation across static, longitudinal, and open-ended scenarios.
• Empirical findings indicate strong performance on single-turn tasks while revealing challenges in maintaining safety and consistency in longitudinal psychiatric reasoning.

PsychiatryBench denotes a psychiatry-focused benchmarking framework for evaluating LLMs on clinically meaningful psychiatric tasks. In the explicit title usage, it refers to "Psychiatry-Bench: A Multi-Task Benchmark for LLMs in Psychiatry," a benchmark grounded exclusively in authoritative, expert-validated psychiatric textbooks and casebooks and comprising eleven question-answering tasks totaling over 5,300 expert-annotated items (Fouda et al., 7 Sep 2025). In the surrounding literature, the term also functions more broadly as a label for psychiatry-oriented evaluation designs, including real-EMR clinical benchmarks, DSM-grounded diagnostic decision suites, end-to-end virtual encounters, and ethics-focused psychiatric evaluation modules (Liu et al., 28 Feb 2025, Sun et al., 3 Feb 2026, Song et al., 13 Feb 2026, Yang et al., 9 Jul 2026). This dual usage is important: PsychiatryBench is both a specific benchmark and, plausibly, a shorthand for an emerging benchmark ecosystem in computational psychiatry.

1. Definition and historical positioning

PsychiatryBench emerged in response to a recurrent limitation in mental-health NLP evaluation: existing resources had focused on small clinical interview corpora, social-media-derived datasets, or synthetic dialogues, all of which constrain clinical validity and underrepresent the complexity of psychiatric reasoning (Fouda et al., 7 Sep 2025). The benchmark’s stated aim is to probe sophisticated reasoning rather than narrow screening or sentiment detection, covering diagnostic reasoning, treatment selection, longitudinal follow-up, management planning, clinical approach, sequential case analysis, and exam-style psychiatric assessment (Fouda et al., 7 Sep 2025).

The benchmark sits within a rapidly diversifying research lineage. "PsychBench: A comprehensive and professional benchmark for evaluating the performance of LLM-assisted psychiatric clinical practice" evaluates five psychiatry-specific tasks on 300 de-identified real inpatient cases from three psychiatric centers in China (Liu et al., 28 Feb 2025). "MentalDx Bench," introduced alongside MentalSeek-Dx, targets disorder-level psychiatric diagnosis from 712 de-identified electronic health records under ICD-11 guidelines (Sun et al., 3 Feb 2026). "MentalBench" uses a psychiatrist-built and validated DSM-5 knowledge graph to generate 24,750 synthetic clinical cases varying in completeness and diagnostic complexity (Song et al., 13 Feb 2026). "MentalHospital" shifts the unit of evaluation from static questions to full psychiatric encounters, instantiating the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning workflow with 1,193 de-identified psychiatric EHR cases and specialist-aligned evaluators (Yang et al., 9 Jul 2026).

This literature suggests that PsychiatryBench is best understood not as a single design choice but as a family of benchmark philosophies. Some benchmarks maximize source authority through textbooks and casebooks; some maximize ecological validity through de-identified hospital records; some maximize diagnostic interpretability through DSM-5 or ICD-11 logic; and some maximize process fidelity through interactive simulation.

2. Internal structure of the benchmark

In its explicit benchmark form, PsychiatryBench comprises eleven distinct tasks spanning the psychiatric workflow (Fouda et al., 7 Sep 2025). The benchmark standardizes inputs through fields such as History, Mental State Examination, Physical Examination, and Question, while adapting the output format to the clinical objective of each task (Fouda et al., 7 Sep 2025).

The task taxonomy is as follows:

• Diagnosis: infer the most likely diagnosis or construct a differential diagnosis from a detailed case vignette.
• Treatment: recommend an individualized, evidence-based treatment plan consistent with guidelines.
• Treatment Follow-Up: interpret evolving symptoms or side effects and adjust treatment longitudinally.
• Classification (multi-label): assign disorder category labels and specific disorder labels to case narratives.
• Management Plan: generate holistic psychiatric management plans including diagnostics, safety planning, psychosocial interventions, pharmacologic steps, coordination, and follow-up.
• Clinical Approach: outline the stepwise clinical approach, including history-taking priorities, differential structure, investigations, and initial management.
• Mental QA: define psychiatric terms and concepts with clinical precision.
• Sequential Question Answering: perform multi-turn case reasoning across ordered questions on differential diagnosis, supporting evidence, etiological factors, treatment options, prognosis, and DSM-5 class mapping.
• MCQs: answer single-best-answer psychiatric questions.
• EMI: solve extended matching item clusters with shared option pools.
• Exam Simulations: complete grouped exam-style multiple-choice sets (Fouda et al., 7 Sep 2025).

The benchmark reports 5,308 total items and task-specific counts including 467 diagnosis items, 258 treatment items, 27 treatment follow-up items, 418 classification items, 337 management plan items, 56 clinical approach items, 326 mental QA items, 32 sequential QA vignettes, 1,473 MCQs, 277 EMI clusters, and 600 exam items (Fouda et al., 7 Sep 2025). An "EMI Separated" format with 1,037 single items is provided as an evaluation convenience rather than a distinct task (Fouda et al., 7 Sep 2025).

A central design choice is heterogeneity of reasoning form. Some tasks are free-text and open-ended; some are objective selection tasks; some are inherently longitudinal. This is significant because psychiatric competence is not reducible to single-turn diagnosis. PsychiatryBench explicitly includes treatment adaptation, prognosis, management planning, and sequential case analysis, thereby operationalizing psychiatric reasoning as a workflow rather than a label-assignment problem (Fouda et al., 7 Sep 2025).

3. Source curation, annotation, and evaluation formalism

PsychiatryBench is grounded in expert-authored educational sources rather than social media, patient forums, or model-generated synthetic cases. Its materials were manually curated from authoritative psychiatric textbooks and casebooks, including DSM-5 Clinical Cases, DSM-5-TR Self-Exam Questions, Stahl’s Essential Psychopharmacology and associated case studies, Case Files Psychiatry, 100 Cases in Psychiatry, Core Clinical Cases in Psychiatry, Clinical Cases in Psychiatry, Geriatric Psychiatry, MRCPsych revision MCQs and EMIs, and MCQs in Psychology (Fouda et al., 7 Sep 2025). Subject matter experts manually extracted high-yield vignettes and exam items, standardized style and metadata, and filtered ambiguity, redundancy, and overly simple prompts (Fouda et al., 7 Sep 2025).

For the multi-label classification task, disorder category and specific-disorder lists were derived from DSM-5-TR and ICD-11 and validated by a licensed psychiatrist (Fouda et al., 7 Sep 2025). Where label reconciliation was needed, Gemini 2.5 Pro was used to map source names to standardized lists, followed by human verification (Fouda et al., 7 Sep 2025). The paper does not report inter-rater agreement statistics such as Cohen’s $\kappa$ for the broader curation pipeline (Fouda et al., 7 Sep 2025).

Evaluation combines conventional task metrics with an LLM-as-judge similarity framework. For MCQ-type tasks, the benchmark uses accuracy and balanced accuracy:

$$BA = \frac{1}{2}\left(\frac{TP}{TP+FN}+\frac{TN}{TN+FP}\right), \qquad Accuracy = \frac{TP+TN}{TP+TN+FP+FN}.$$

For multi-label classification, it uses subset accuracy and weighted-average $F_1$:

$$\text{Subset Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{I}(\hat{Y}_i = Y_i),$$

$$F_{1,l} = 2 \times \frac{P_l \times R_l}{P_l + R_l}, \qquad \text{F1-weighted} = \sum_{l \in \mathcal{L}} w_l \cdot F_{1,l}.$$

For EMI clusters, it uses a partial scoring system:

$$EMI\_Score_i = \frac{1}{S_i}\sum_{j=1}^{S_i} c_{ij}, \qquad EMI\_Accuracy = \frac{1}{N}\sum_{i=1}^N EMI\_Score_i.$$

It also defines consistency, agreement, and divergence measures across full-cluster and separated-item formats (Fouda et al., 7 Sep 2025).

The judge model selected for free-text evaluation is LLaMA 3.3 70B, chosen after comparison with LearnLM, LLaMA-3 70B, and GPT-4o mini (Fouda et al., 7 Sep 2025). Two rubric families were tested: a point-based rubric generated by GPT-4.5 and a logic-driven, example-based prompt generated by Gemini. The example-based, logic-driven prompt was adopted for final use because it was more robust across tasks and models (Fouda et al., 7 Sep 2025). Free-text task outputs are scored on a 0–100 scale and aggregated by arithmetic mean (Fouda et al., 7 Sep 2025).

The benchmark is intended for non-commercial research and academic use under fair-use principles, with no patient data and an emphasis on transformative evaluation rather than source substitution (Fouda et al., 7 Sep 2025).

4. Reported model behavior and empirical findings

PsychiatryBench evaluates a broad set of frontier and medical LLMs, including Gemini 2.5 Pro Preview, Gemini 2.5 Flash Preview, Gemini 2.0 Flash, Claude 3.5 Sonnet, DeepSeek R1, DeepSeek Chat, LLaMA 3.3 70B, QWQ-32, Qwen 3 32B, OpenBioLLM, Palmyra-Med-70B, MedGemma, and other medical-domain models (Fouda et al., 7 Sep 2025). The reported results show that top models can reach strong similarity scores on several single-turn tasks, but they remain inconsistent on longitudinal and fine-grained psychiatric reasoning (Fouda et al., 7 Sep 2025).

On Diagnosis, the top score is 86.0% for Gemini 2.5 Pro, with QWQ-32 at 85.9% (Fouda et al., 7 Sep 2025). On Treatment, Gemini 2.5 Flash (Thinking) reaches 82.9%, followed by LLaMA 3.3 at 82.3% and QWQ-32 at 82.1% (Fouda et al., 7 Sep 2025). On Treatment Follow-Up, Gemini 2.5 Pro reaches 86.5%, QWQ-32 83.7%, and Gemini 2.5 Flash (Thinking) 82.6%, but the total spread across models is large, from 66.5% to 86.5%, which the benchmark highlights as evidence that longitudinal psychiatric management remains difficult (Fouda et al., 7 Sep 2025).

The Classification task exposes a sharper drop in fine-grained psychiatric specificity. For disorder categories, DeepSeek R1 attains weighted $F_1 = 0.72$ and subset accuracy $= 0.54$; for specific disorders, the same model attains weighted $F_1 = 0.50$ and subset accuracy $= 0.44$ (Fouda et al., 7 Sep 2025). This difference between category-level and disorder-level performance is consistent with other psychiatry benchmarks, notably MentalDx Bench, which reports strong coarse diagnostic categorization but systematic failure at disorder-level diagnosis across many models (Sun et al., 3 Feb 2026).

On Management Plan, DeepSeek R1 scores 84.4%, Gemini 2.5 Pro 84.1%, and DeepSeek Chat 83.5% (Fouda et al., 7 Sep 2025). On Sequential QA, Gemini 2.5 Pro reaches 96.9%, with other models ranging from 81.4% to 90.0% (Fouda et al., 7 Sep 2025). On MCQs, Gemini 2.5 Pro obtains 81.7% accuracy and 79.3% balanced accuracy under one prompt, and 82.8% accuracy and 74.4% balanced accuracy under another (Fouda et al., 7 Sep 2025). On EMI full clusters, Gemini 2.5 Pro reaches 87.6–87.9%, while DeepSeek R1 and Gemini 2.5 Flash (Thinking) also exceed 85% (Fouda et al., 7 Sep 2025). On Exam Simulations, Gemini 2.5 Pro achieves 80.4% on one prompt, while Flash (Thinking) is reported as robust on another (Fouda et al., 7 Sep 2025).

The headline interpretation is not simple superiority. The paper explicitly reports substantial gaps in clinical consistency and safety, particularly in multi-turn follow-up and management tasks (Fouda et al., 7 Sep 2025). Prompt reformulation also matters: in Clinical Approach, one prompt version yields 86.7–90.1%, while another yields 82.8–87.5%, with Gemini 2.5 Pro showing the smallest decline at 1.7% (Fouda et al., 7 Sep 2025). This indicates that even high-performing models can be brittle under changes in instructional framing.

5. PsychiatryBench as an ecosystem of complementary benchmarks

The broader literature shows that no single psychiatry benchmark covers all relevant axes of evaluation. The current landscape is therefore modular.

Resource	Primary emphasis	Salient design
Psychiatry-Bench	Multi-task psychiatry QA	Textbook- and casebook-grounded; 11 tasks (Fouda et al., 7 Sep 2025)
PsychBench	LLM-assisted psychiatric clinical practice	300 de-identified inpatient EMRs; 5 clinical tasks (Liu et al., 28 Feb 2025)
MentalDx Bench	Disorder-level psychiatric diagnosis	712 EHRs; 76 disorders; ICD-11 labels (Sun et al., 3 Feb 2026)
MentalBench	DSM-5 diagnostic decision-making	MentalKG; 24,750 synthetic cases (Song et al., 13 Feb 2026)
MentalHospital	End-to-end psychiatric encounters	SOAP workflow; 1,193 EHR cases; MentalEval (Yang et al., 9 Jul 2026)
EthicsMH	Ethical reasoning in mental health	125 scenarios; stakeholder-rich schema (Kasu, 15 Sep 2025)
PsychEthicsBench	Jurisdiction-aware ethics	Australian principles; MCQ and OEQ tasks (Shen et al., 7 Jan 2026)
RSPC	Relational context in mental-health NLP	1,799 psychiatrist-annotated Reddit posts (Vangapandu et al., 25 Jun 2026)
PsychBench (epidemiological)	Population fidelity of simulated patients	28,800 profiles; intersectional audit (Keough, 19 Apr 2026)
PSI-Bench	Depression patient simulator evaluation	Turn-, dialogue-, and population-level diagnostics (Hoang et al., 28 Apr 2026)
LingxiDiagBench	Dynamic psychiatric consultation	16,000 EMR-aligned synthetic dialogues (Xu et al., 10 Feb 2026)

These resources occupy distinct niches. MentalHospital evaluates full encounters rather than answers to static prompts and reports that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck (Yang et al., 9 Jul 2026). MentalBench isolates DSM-5 rule application and shows that models struggle to calibrate diagnostic commitment when disorders overlap (Song et al., 13 Feb 2026). MentalDx Bench emphasizes real-world EHR diagnosis and identifies a "paradigm misalignment" between category recognition and disorder-level diagnostic reasoning (Sun et al., 3 Feb 2026). PsychBench evaluates authentic psychiatric clinical practice from real inpatient EMRs and finds that current LLMs are not yet adequate as stand-alone decision-making tools, though they can materially assist junior psychiatrists (Liu et al., 28 Feb 2025).

Two ethics-oriented resources are especially relevant to extensions of PsychiatryBench. EthicsMH provides 125 scenarios across confidentiality, autonomy versus beneficence, and bias in AI, with structured fields such as Options, Expected Reasoning, Model Behavior, Real World Impact, and Viewpoints; the paper explicitly proposes it as an ethics module within PsychiatryBench (Kasu, 15 Sep 2025). PsychEthicsBench, grounded in Australian psychology and psychiatry guidance, shows that refusal rates are poor indicators of ethical behavior and that some domain-specific fine-tuning can degrade ethical robustness (Shen et al., 7 Jan 2026).

A plausible implication is that PsychiatryBench is evolving toward a modular architecture: textbook-grounded reasoning, EHR-grounded diagnosis, interactive interview simulation, ethical alignment, population-fidelity auditing, and socially contextual psychiatric inference are increasingly being benchmarked as separable but complementary competencies.

6. Conceptual issues, misconceptions, and future directions

A common misconception is that psychiatric benchmarking is equivalent to exam-style multiple-choice performance. PsychiatryBench itself contradicts this by placing diagnosis, treatment, follow-up, management planning, clinical approach, sequential reasoning, EMI clusters, and free-text definitions in the same benchmark (Fouda et al., 7 Sep 2025). The surrounding literature strengthens the point: MentalHospital shows that strong performance on isolated psychiatric tasks does not imply competence in complete encounters, and that models lag clinicians substantially on objective psychiatric competence (Yang et al., 9 Jul 2026).

A second misconception is that safety can be approximated by refusal. PsychEthicsBench explicitly finds that refusal rates are poor indicators of ethical behavior in mental health and that clinically inadequate refusals can be perceived as unempathetic and discourage help-seeking (Shen et al., 7 Jan 2026). A third misconception is that clinically plausible simulated patients are sufficient for psychiatric evaluation. The epidemiological PsychBench finds a coherence–fidelity dissociation: models produce clinically plausible individuals while misrepresenting real populations, with variance compression from 14 percent to 62 percent and substantial demographic calibration biases (Keough, 19 Apr 2026). A fourth misconception is that patient simulation realism can be read off a single holistic judge score. PSI-Bench argues instead for interpretable turn-, dialogue-, and population-level diagnostics, showing that current depression simulators are overly long, less variable, and too uniformly negative-to-positive in emotional trajectory (Hoang et al., 28 Apr 2026).

The limitations of PsychiatryBench as a specific benchmark are also clear. Its cases are textbook- and casebook-derived rather than drawn from noisy clinical EHRs or spontaneous patient dialogue, its disorder distribution reflects teaching goals rather than epidemiologic prevalence, and its large-scale free-text evaluation depends on LLM-as-judge scoring rather than psychiatrist panel adjudication (Fouda et al., 7 Sep 2025). The paper does not specify a public repository URL or formal leaderboard, and train/dev/test splits are not defined because the benchmark is primarily used as a curated test suite (Fouda et al., 7 Sep 2025).

Future directions are already articulated across the benchmark literature. PsychiatryBench proposes expanded coverage in child and adolescent, perinatal, forensic, and culturally diverse psychiatry, multimodal inputs, automated extraction with human-in-the-loop validation, enhanced safety metrics, calibration analysis, and broader model comparison (Fouda et al., 7 Sep 2025). EthicsMH proposes community and expert contributions to scale scenario diversity and cultural representation (Kasu, 15 Sep 2025). MentalHospital plans richer mental status exams, multimodal simulation, crisis and adversarial safety scenarios, fairness audits, and longitudinal follow-ups (Yang et al., 9 Jul 2026). RSPC argues for sequential inputs, dyadic conversation context, longitudinal trajectories, and multilingual, cross-platform expansion for relationally contextual mental-health NLP (Vangapandu et al., 25 Jun 2026).

Taken together, these directions suggest that the long-term significance of PsychiatryBench lies not only in any single leaderboard score, but in the formalization of psychiatric evaluation as a multi-axis problem: diagnostic rigor, treatment adequacy, longitudinal coherence, ethical alignment, interpersonal context, simulator realism, and epidemiological fidelity must all be measured if LLMs are to be assessed credibly in psychiatry.