PsyCrisis-Bench: Crisis Evaluation for LLMs

• PsyCrisis-Bench is a family of benchmarks designed to evaluate large language models in psychological crisis settings via hotline transcript analysis, safety alignment in dialogues, and multi-turn caregiving scenarios.
• It employs clinically grounded annotations, reference-free LLM-as-Judge evaluations, and deterministic safety gates to assess binary tasks, longitudinal interactions, and regulatory compliance.
• Quantitative findings show enhanced performance from fine-tuning and few-shot learning, while highlighting persistent challenges such as mood recognition and reliable crisis signal detection.

PsyCrisis-Bench, also written as PsyCrisisBench in part of the 2025 literature, denotes a cluster of benchmark designs for evaluating LLMs in psychological crisis settings rather than a single uniform artifact. Across the cited works, the name is used for three distinct but related regimes: a clinically grounded benchmark built from 540 anonymized hotline transcripts for four binary crisis assessment tasks; a reference-free, LLM-as-Judge benchmark for safety alignment in high-risk Chinese mental-health dialogues; and a purpose-built, multi-turn pre-deployment safety gate for psychological and caregiving AIs that adapts core ideas from InvisibleBench to crisis detection in relationship contexts (Deng et al., 2 Jun 2025, Cai et al., 11 Aug 2025, Madad, 25 Nov 2025).

1. Nomenclature and conceptual scope

The literature uses closely related names for benchmarks that target different operational problems. One line of work evaluates transcript-level classification over real hotline conversations; another evaluates the safety alignment of generated replies when no gold-standard response exists; a third evaluates longitudinal conversational behavior, including crisis routing, compliance, belonging, and memory, in simulated caregiver–AI interactions. This suggests that “PsyCrisis-Bench” functions as a benchmark family label for crisis-oriented evaluation rather than as a single fixed dataset or protocol (Deng et al., 2 Jun 2025, Cai et al., 11 Aug 2025, Madad, 25 Nov 2025).

Variant	Setting	Core target
PsyCrisisBench	540 hotline transcripts	Four binary assessment tasks
PsyCrisis-Bench	608 Chinese utterances	Five safety-alignment dimensions
PsyCrisis-Bench adaptation of InvisibleBench	17 multi-turn scenarios	Five weighted dimensions with autofail

The variants differ along several axes. The hotline benchmark uses full transcripts and standard classification metrics such as F1-score; the reference-free benchmark scores model replies directly against expert-derived criteria; the multi-turn benchmark treats crisis handling as a deployment-readiness problem in which a single failure mode can deterministically zero out a scenario score. The resulting family spans classification, response evaluation, and longitudinal interaction testing.

2. Hotline transcript benchmark for structured crisis assessment

The benchmark introduced in “Evaluating LLMs in Crisis Detection: A Real-World Benchmark from Psychological Support Hotlines” is built from 540 anonymized call transcripts collected from January 1 to December 31, 2023 at the Hangzhou Psychological Assistance Hotline, after exclusions from an original pool of 21,527 calls for calls under 60 seconds, non-help requests, and repeat callers (Deng et al., 2 Jun 2025). Dialogue statistics are substantial: the average turns per call are $238 \pm 165$, the average operator word count is $2{,}708 \pm 2{,}085$, and the average caller word count is $3{,}495 \pm 3{,}461$.

Each transcript receives four independent binary labels assigned by trained hotline operators using structured post-call annotation forms: Mood Status Recognition $\{\text{Depression}, \text{Normal}\}$, Suicidal Ideation Detection $\{\text{Yes}, \text{No}\}$, Suicide Plan Identification $\{\text{Yes}, \text{No}\}$, and Risk Assessment $\{\text{High-risk}, \text{Non-high-risk}\}$. The label distribution is reported as Mood status: Depression 233 and Normal 307; Suicidal ideation: Yes 305 and No 235; Suicide plan: Yes 196 and No 344; Risk level: High-risk 270 and Non-high-risk 270. Inter-annotator agreement is not explicitly reported.

The evaluation protocol covers zero-shot, static few-shot, dynamic few-shot, and fine-tuning. Zero-shot uses only a system prompt plus the full transcript. Static few-shot uses four fixed examples from an independent 2022 hotline dataset. Dynamic few-shot experiments on Qwen-2.5-7B vary the number of random shots across $0,2,4,6,8$, each repeated $10\times$. Fine-tuning is performed on Qwen-2.5-1.5B-Instruct using 520 transcripts from 2022, with 3 epochs, learning rate $= 1\times 10^{-5}$, and $2{,}708 \pm 2{,}085$0 with $2{,}708 \pm 2{,}085$1. The prompt requires strict JSON output; if the JSON format is violated, the response is counted as incorrect (Deng et al., 2 Jun 2025).

Performance is reported with Precision, Recall, and F1-score, and statistical comparisons use Welch’s $2{,}708 \pm 2{,}085$2-tests. The best task-wise F1 values improve from zero-shot to 4-shot as follows: mood status from $2{,}708 \pm 2{,}085$3 to $2{,}708 \pm 2{,}085$4, suicidal ideation from $2{,}708 \pm 2{,}085$5 to $2{,}708 \pm 2{,}085$6, suicide plan from $2{,}708 \pm 2{,}085$7 to $2{,}708 \pm 2{,}085$8, and risk assessment from $2{,}708 \pm 2{,}085$9 to $3{,}495 \pm 3{,}461$0. Average few-shot gains are $3{,}495 \pm 3{,}461$1 for mood, $3{,}495 \pm 3{,}461$2 for ideation, $3{,}495 \pm 3{,}461$3 for plan, and $3{,}495 \pm 3{,}461$4 for risk. The fine-tuned Qwen2.5-1.5B model reaches $3{,}495 \pm 3{,}461$5 on mood, $3{,}495 \pm 3{,}461$6 on ideation, $3{,}495 \pm 3{,}461$7 on plan, and $3{,}495 \pm 3{,}461$8 on risk, exceeding all zero-shot and static few-shot baselines in mood and ideation. The paper characterizes mood recognition as the principal bottleneck, attributing difficulty to lost paralinguistic cues and ambiguity in transcript-only inference (Deng et al., 2 Jun 2025).

3. Reference-free safety alignment benchmark in Chinese mental-health dialogues

The benchmark described in “Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge” addresses a different evaluation problem: open-ended crisis response generation in settings where there is no single correct answer and reference-based metrics such as BLEU or BART-Score are unsuitable (Cai et al., 11 Aug 2025). It is explicitly formulated as a reference-free task: given a user utterance $3{,}495 \pm 3{,}461$9 and a model reply $\{\text{Depression}, \text{Normal}\}$0, the reply is judged directly against expert-derived criteria.

The dataset contains 608 crisis-level user utterances drawn from three publicly available Chinese datasets: PsyQA, SOS-HL-1K, and Emotional First Aid. After deduplication and high-risk filtering, the resulting set is categorized into Suicidal Ideation with 289 samples, Non-Suicidal Self-Injury with 115 samples, and Existential Distress with 204 samples. Construction involves keyword-based high-risk filtering, semantic clustering to remove near-duplicates, manual classification by two trained annotators, and adjudication by a PhD-level reviewer. A random 100-sample check yields 100% consistency after consensus discussion. For safety ratings, six experts rate a subset of 800 model replies on five binary dimensions, producing inter-rater agreement of Cohen’s $\{\text{Depression}, \text{Normal}\}$1, Matthews CC $\{\text{Depression}, \text{Normal}\}$2, and $\{\text{Depression}, \text{Normal}\}$3 (Cai et al., 11 Aug 2025).

The five expert-defined dimensions are Empathy and Relational Stance, Evidence-Based Emotion Regulation Strategies, Exploration of Client Concerns, Risk Assessment and Identification, and Referral to External Resources. The judge outputs a dictionary of binary decisions: $\{\text{Depression}, \text{Normal}\}$4 Scoring is point-wise and additive: $\{\text{Depression}, \text{Normal}\}$5 and

$\{\text{Depression}, \text{Normal}\}$6

The framework embeds expert-written chain-of-thought exemplars in prompts so that the judge mirrors professional reasoning grounded in crisis intervention guidelines (Cai et al., 11 Aug 2025).

Agreement is evaluated on 800 question–answer pairs, corresponding to 3600 total binary judgments across five dimensions. Relative to a general baseline and a rule-based baseline, the reported system-level results for PsyCrisis-Bench are Pearson $\{\text{Depression}, \text{Normal}\}$7, Spearman $\{\text{Depression}, \text{Normal}\}$8, Kendall $\{\text{Depression}, \text{Normal}\}$9, and pairwise agreement $\{\text{Yes}, \text{No}\}$0. Category-level breakdowns are SI $\{\text{Yes}, \text{No}\}$1, NSSI $\{\text{Yes}, \text{No}\}$2, and ED $\{\text{Yes}, \text{No}\}$3 for Pearson, Spearman, and Kendall respectively. In a human preference study on 1,200 explanation pairs, the benchmark’s rationales win 65.3% against the general baseline and 82.3% against the rule baseline, with evaluation criteria of Rationality, Traceability, and Consistency. The paper notes a mild leniency bias in the LLM judge, with model ratings slightly higher than expert ratings (Cai et al., 11 Aug 2025).

4. Multi-turn caregiving and relationship evaluation

A third benchmark bearing the PsyCrisis-Bench name is described as a purpose-built, multi-turn evaluation suite designed as a pre-deployment safety gate for psychological and caregiving AIs. It adapts the core ideas of InvisibleBench to focus on crisis detection within relationship contexts while preserving five evaluation dimensions, a tiered scenario architecture, a scoring rubric with hard autofail gates, and a reproducible code–data release (Madad, 25 Nov 2025).

The benchmark measures model behavior in simulated caregiver–AI conversations spanning 3–20+ turns. It comprises 17 hand-crafted scenarios—5 in Tier 1, 9 in Tier 2, and 3 in Tier 3—with 68 total model–scenario interactions. The five orthogonal dimensions and weights are Safety (20%), Compliance (15%), Trauma-Informed Design (15%), Belonging/Cultural Fitness (34%), and Memory (16%). The weights are renormalized when a scenario omits dimensions; Tier 1, for example, omits Memory Hygiene checks. Safety targets explicit or masked crisis signals such as suicidal ideation and means stockpiling. Compliance enforces medical-boundary regulations such as the Illinois WOPR Act and prohibits diagnosis, treatment plans, or dosing advice. Trauma-Informed Design aligns with seven trauma-informed care principles: safety, trust/transparency, choice/control, collaboration, empowerment/agency, cultural sensitivity, and flexibility. Belonging/Cultural Fitness evaluates cultural competence, relational warmth, and practical support, with four penalty categories: explicit othering $\{\text{Yes}, \text{No}\}$4, implicit stereotyping $\{\text{Yes}, \text{No}\}$5, power-over language $\{\text{Yes}, \text{No}\}$6, and individual blame $\{\text{Yes}, \text{No}\}$7. Memory covers longitudinal consistency and privacy hygiene in multi-session scenarios (Madad, 25 Nov 2025).

Scenario tiers operationalize distinct stressors. Tier 1 covers Foundational Safety in 3–5 turns and focuses on explicit and masked crisis detection, medical-boundary compliance, and trauma-informed response behavior. Tier 2 covers Memory & Attachment in 8–12 turns and probes memory consistency, de-escalation of parasocial dependency, cultural fitness under repeated prompts, and progressive boundary drift. Tier 3 is Multi-Session Longitudinal, spanning 20+ turns across 2–3 sessions, and tests memory hygiene, context re-entry after temporal gaps, trajectory consistency, and attachment engineering over long engagements. Crises include direct self-harm statements, indirect cues, medication misuse, caregiver burnout, grief, and social isolation. Escalation criteria revolve around explicit ideation or plan, masked indications of means or intent, and patterns of increasing distress across turns or sessions (Madad, 25 Nov 2025).

The evaluation uses an LLM-as-Judge framework augmented by deterministic checks. Judges apply 0–3 point rubrics per turn per dimension. For dimensions with hard rules—Safety, Compliance, and Trauma—sampling occurs $\{\text{Yes}, \text{No}\}$8–$\{\text{Yes}, \text{No}\}$9 times at $\{\text{Yes}, \text{No}\}$0, with majority vote for binary decisions and mean aggregation for continuous ones. The normalization and aggregation procedure is: $\{\text{Yes}, \text{No}\}$1

$\{\text{Yes}, \text{No}\}$2

$\{\text{Yes}, \text{No}\}$3

Crisis detection is summarized as

$\{\text{Yes}, \text{No}\}$4

and compliance is quantified as

$\{\text{Yes}, \text{No}\}$5

Autofail conditions set $\{\text{Yes}, \text{No}\}$6 for any single occurrence of a missed explicit crisis, a medical boundary violation, harmful information, or attachment engineering such as permanence promises or exclusivity claims (Madad, 25 Nov 2025).

5. Quantitative findings and comparative performance

The hotline transcript benchmark evaluates 64 LLMs across 15 families, including closed-source series such as GPT, Claude, Gemini, Grok, Doubao, Kimi, Moonshot, Hunyuan, Baichuan, and ERNIE, and open-source series such as Mistral, Llama, DeepSeek, Qwen, QwQ, GLM, InternLM, and Hunyuan-Large. It reports that open-source models perform comparably to closed-source systems on suicidal ideation, suicide plan, and risk assessment, while closed-source systems retain an advantage in mood recognition, where the comparison between Claude-3.7-Sonnet at 0.7088 and QwQ-32B at 0.6786 yields $\{\text{Yes}, \text{No}\}$7 and $\{\text{Yes}, \text{No}\}$8. The benchmark also reports no significant differences between top reasoning and non-reasoning models across the four tasks, though reasoning mode in the Qwen3 hybrid series improves mood recognition in several cases at the cost of 4.5× longer outputs (Deng et al., 2 Jun 2025).

The same benchmark examines scaling and quantization. Within model families, F1 generally increases with parameter count, but saturation appears beyond approximately 10–14B parameters. AWQ quantization reduces peak GPU memory substantially—for example, QwQ-32B from 31.3 GB to 9.5 GB and Qwen3-32B from 31.2 GB to 9.4 GB—while causing only minimal F1 degradation. In some cases, such as InternLM3-8B, quantized models slightly outperform full-precision variants, which the paper attributes to improved instruction adherence (Deng et al., 2 Jun 2025).

The multi-turn PsyCrisis-Bench adaptation evaluates four state-of-the-art models over 17 scenarios and 68 runs. DeepSeek Chat v3 obtains the highest overall score at 75.9%, followed by Gemini 2.5 Flash at 73.6%, GPT-4o Mini at 73.0%, and Claude Sonnet 4.5 at 65.4%. Dimension leaders differ: DeepSeek Chat v3 leads Memory at 92.3% and Belonging at 91.7%; Gemini 2.5 Flash leads Trauma-Informed Design at 85.0%; GPT-4o Mini leads Compliance at 88.2%; and Claude Sonnet 4.5 leads Safety with a crisis detection rate of 44.8%. The crisis detection rate ranges from 11.8% for GPT-4o Mini to 44.8% for Claude Sonnet 4.5, averaging 25.4%. The benchmark states that no single model passes the 70%+ threshold with zero autofails, and that all tested LLMs miss 55–88% of crisis signals, motivating deterministic crisis routing in production systems (Madad, 25 Nov 2025).

The reference-free Chinese safety-alignment benchmark does not report leaderboard-style task F1 because its object of evaluation is the quality of generated replies under expert-defined safety dimensions. Its principal quantitative claim is evaluative validity: compared with a general baseline and a rule-based baseline, the LLM-as-Judge method achieves the highest agreement with expert assessments and produces more interpretable rationales, albeit with a mild leniency bias (Cai et al., 11 Aug 2025).

6. Limitations, misconceptions, and research directions

A common misconception is that PsyCrisis-Bench denotes a single benchmark with a unified dataset and metric suite. The literature instead presents at least three distinct instantiations with different units of analysis: full hotline transcripts, single-turn high-risk user utterances paired with model replies, and scripted multi-turn caregiver–AI scenarios. A plausible implication is that benchmark choice should be driven by the intended deployment role of the model: structured crisis triage, generative response assessment, or longitudinal relationship management (Deng et al., 2 Jun 2025, Cai et al., 11 Aug 2025, Madad, 25 Nov 2025).

Each variant also states explicit limitations. The hotline benchmark identifies mood status recognition as the hardest task because tone and prosody are lost in transcription, and it advises against unsupervised deployment; its recommended pipeline is to fine-tune smaller open-source models on historical hotline data, quantize them for on-premise real-time inference, and integrate them with human-in-the-loop workflows for final judgment and empathetic support (Deng et al., 2 Jun 2025). The reference-free benchmark is single-turn only, Chinese-only, based on six experts, and does not fine-tune the evaluator model; it proposes multi-turn datasets, multilingual extension, broader expert participation, and evaluator fine-tuning as future work (Cai et al., 11 Aug 2025). The multi-turn benchmark emphasizes that it is a deployment-readiness evaluation rather than a clinical claims paper, and it provides open-source artifacts under MIT/CC BY 4.0, including 17 JSON scenario files, judge prompt templates in YAML, scoring configuration, Python evaluation scripts, and results data, together with instructions for adding scenarios, changing regulatory boundaries, plugging in models via standard API, adjusting weights, and replacing WOPR Act rules for other jurisdictions (Madad, 25 Nov 2025).

Taken together, these works define a research space in which crisis-oriented LLM evaluation is moving along three complementary directions: clinically grounded transcript classification, reference-free safety judgment for open-ended responses, and multi-turn deployment gating with deterministic failure conditions. This suggests an increasingly modular evaluation stack in which offline task performance, safety alignment, and longitudinal risk handling are treated as separable but interacting requirements for mental-health and caregiving AI systems.