Verily Mental Health Crisis Dataset v1.0

Updated 16 July 2026

The dataset is a clinician-reviewed benchmark that uses a two-stage approach for binary crisis detection and multi-label crisis-type classification.
It consists of 1,800 synthetic, English single-turn messages evenly split between crisis and non-crisis, annotated by licensed clinicians with near-perfect agreement.
Evaluation using the Verily Behavioral Health Safety Filter demonstrated high accuracy (>0.99) in crisis detection and robust performance in classifying nine crisis categories.

The Verily Mental Health Crisis Dataset v1.0 is a clinician-reviewed benchmark for detecting mental health crises in text-based conversations. It was introduced as the evaluation substrate for the Verily Behavioral Health Safety Filter (VBHSF) and is defined by a deliberately narrow scope: identifying whether a single text message contains any mental health crisis and, if so, assigning one or more crisis categories relevant to psychiatric emergency response and safety triage. The dataset is synthetic, English-only, and single-turn rather than a corpus of real user messages or multi-turn counseling logs, because the authors state that real mental health crisis text is highly sensitive and difficult to access (Nelson et al., 14 Oct 2025).

1. Definition and intended use

The dataset is not framed as a general moderation resource, a generic toxicity benchmark, or a diagnostic corpus. Its intended role is more specific: support behavioral health safety filtering for LLM-enabled systems operating in text-first digital health and conversational AI settings, including LLM safety monitoring, chatbot oversight, clinical triage systems, integrated behavioral health platforms, and large-scale digital health conversational agents. The paper explicitly argues that existing moderation tools are “not fit-for-purpose for detecting psychiatric crises,” and positions the dataset as infrastructure for a specialized safety layer rather than for autonomous clinical decision-making (Nelson et al., 14 Oct 2025).

Property	Value
Corpus type	Simulated English single-turn text messages
Dataset size	1,800 messages
Top-level balance	900 crisis / 900 non-crisis
Primary tasks	Stage 1 binary crisis detection; Stage 2 multi-label crisis category classification
Clinical review	Two licensed clinicians
Reported agreement	Cohen’s $\kappa = 0.99$
Access	Available upon researcher request

This design makes the dataset closer to a mental-health-specific safety benchmark than to broader mental-health NLP resources centered on help-seeking, counseling quality, or disorder prediction. The paper also treats high sensitivity as a core requirement, because in this application domain missed crises are operationally more serious than over-flagging (Nelson et al., 14 Oct 2025).

2. Corpus construction and message composition

The dataset contains 1,800 simulated messages generated with pre-trained LLMs. The authors state that synthesis was used “to mitigate the unavailability of highly sensitive mental health crisis messages.” The top level is exactly balanced: 900 crisis messages and 900 non-crisis messages. All items are single messages, not dialogue histories, and the paper identifies this single-turn design as a limitation (Nelson et al., 14 Oct 2025).

The 900 crisis messages are organized into nine crisis groupings of 100 messages each:

Crisis grouping	Messages
Abuse	100
Neglect	100
Eating-disorder behaviors	100
Psychosis	100
Self-harm	100
Suicide	100
Substance misuse	100
Violence toward others	100
Mixed crisis presentations	100

The paper sometimes describes the system as identifying “eight dimensions” of crisis, but the construction section lists nine crisis message categories, with mixed crisis presentations functioning as an additional grouping. That inconsistency is internal to the paper itself (Nelson et al., 14 Oct 2025).

The 900 non-crisis messages are not generic background chatter. They are drawn from health-related but non-emergency conversational scenarios, including appointment logistics, behavior change, follow-up care, follow-up clarification, health education, health monitoring, insurance and billing, lab and wearable data interpretation, lifestyle modification, locating providers, medical record request, preventative care, referral request, test scheduling, symptom tracking, medication pickup logistics, chronic condition management, and diet and nutrition. This produces what the paper describes as near-domain distractor content, which is important for evaluating systems intended for digital health messaging rather than open-domain social media (Nelson et al., 14 Oct 2025).

The message surface form was also designed to resemble informal texting. Among crisis messages, 91.10% were classified as direct explicit expressions of risk, 8.90% as ambiguous statements of harm, and 8.6% included slang or masked language. Across the full dataset, language mechanics errors occur in 55.90% of messages, textese in 45.80%, and emojis and symbolic markers in 13.50%. The descriptive statistics reported for the full corpus are 96.70 mean characters per message ( $\text{SD }84.50$ ), 18.07 mean words ( $\text{SD }15.14$ ), 1.50 mean sentences ( $\text{SD }0.82$ ), 4,695 unique tokens, 32,527 total tokens, and a type-token ratio of 0.144 (Nelson et al., 14 Oct 2025).

3. Clinical review and label schema

The annotation workflow, as documented, is clinician-centered but relatively compact. The synthetic messages were independently reviewed by two licensed clinicians, identified in the paper as BWN and AR (Benjamin W. Nelson and Alanna Robinson). They annotated all 1,800 messages as crisis or non-crisis, achieved Cohen’s $\kappa = 0.99$ , and resolved disagreements through adjudication and consensus (Nelson et al., 14 Oct 2025).

The label design is hierarchical. Stage 1 is a binary decision: any mental health crisis versus non-crisis. Stage 2 is a multi-label crisis-type classification step evaluated in a one-vs-rest framework. The crisis taxonomy comprises the nine groupings listed above, with mixed crisis presentations allowing composite cases. The paper does not provide a full formal ontology or an exact annotation handbook, but it is explicit that the category system is intended to cover “the most urgent and high-risk presentations” linked to adverse outcomes and, in many cases, mandatory reporting obligations (Nelson et al., 14 Oct 2025).

The stage-2 design is importantly not a severity scale. The paper reports no low/moderate/high crisis gradation, no suicidality severity ladder, and no explicit operational threshold definitions for imminent danger. It also does not give DSM-style operational criteria for categories such as psychosis, and it notes that grave disability and homicidality are not separate labels; the closest reported class for the latter is violence toward others. The taxonomy therefore supports crisis-category recognition rather than formal clinical risk stratification (Nelson et al., 14 Oct 2025).

The paper further indicates that stage-2 labeling is genuinely non-exclusive. Category totals reported in its evaluation tables exceed the 900 crisis-message count, which is consistent with the multi-label setup. This is especially relevant for mixed crisis presentations, where a single message can express more than one urgent crisis type (Nelson et al., 14 Oct 2025).

4. Evaluation protocol and reported performance

The principal model evaluated on the dataset is the Verily Behavioral Health Safety Filter, described as “a transformer-based LLM (GPT architecture) that uses advanced prompt engineering and clinical reasoning to detect the presence of a crisis within a message and classify the specific crisis type.” The evaluation is split into Stage 1 overall crisis detection and Stage 2 category-level classification, with comparison against OpenAI Omni Moderation Latest and NVIDIA NeMo Guardrails (Nelson et al., 14 Oct 2025).

For Stage 1 overall crisis detection on the Verily dataset, the reported VBHSF performance is:

Accuracy: 0.991 (95% CI: 0.986–0.995)
Sensitivity: 0.990 (95% CI: 0.981–0.995)
Specificity: 0.992 (95% CI: 0.984–0.996)
PPV: 0.992 (95% CI: 0.984–0.996)
NPV: 0.990 (95% CI: 0.981–0.995)

The paper states that this corresponded to 7 false positives and 9 false negatives (Nelson et al., 14 Oct 2025).

System	Sensitivity	Specificity
VBHSF	0.990	0.992
OpenAI Omni Moderation Latest	0.419	0.999
NVIDIA NeMo Guardrails	0.759	0.756

The between-system differences were tested statistically. The paper reports Cochran’s $Q(2)=751.12, p<0.001$ for sensitivity and Cochran’s $Q(2)=418.68, p<0.001$ for specificity. Pairwise McNemar tests with Bonferroni correction $\alpha=0.017$ found that VBHSF sensitivity was higher than both OpenAI and NVIDIA ( $p<0.001$ in both cases), and VBHSF specificity was higher than NVIDIA ( $p<0.001$ ) but not significantly different from OpenAI ( $\text{SD }84.50$ 0) (Nelson et al., 14 Oct 2025).

For Stage 2 crisis subtype classification, the paper reports the following macro-averaged performance for VBHSF:

Macro F1-score: 0.939 (95% CI: 0.927–0.951)
Macro sensitivity: 0.957 (95% CI: 0.937–0.978)
Macro PPV: 0.923 (95% CI: 0.894–0.952)

Its narrative summary states that per-category sensitivity ranged from 0.917 to 0.992 and specificity was at or above 0.978 for all crisis types. The paper also notes that among 898 messages flagged as “Crisis”, 88 messages (9.8%) contained at least one category misclassification, yielding 125 individual category errors (Nelson et al., 14 Oct 2025).

The authors additionally evaluate VBHSF on a separate benchmark, the NVIDIA Aegis AI Content Safety Dataset subsetted to 794 mental health-related messages. On that set, VBHSF remained highly sensitive, with sensitivity 0.982, accuracy 0.921, and specificity 0.859. The paper interprets this as evidence of generalization beyond the internal Verily benchmark, though still within text-based safety filtering (Nelson et al., 14 Oct 2025).

A further operational analysis projects PPV under low crisis prevalence. At 2% prevalence, the projected PPV is 0.716 for VBHSF, 0.895 for OpenAI, and 0.060 for NVIDIA. The paper uses this result to emphasize the low-base-rate difficulty of crisis screening: even high-performing systems can create substantial false-alarm burden when deployed autonomously at population scale (Nelson et al., 14 Oct 2025).

5. Relation to adjacent mental-health and crisis datasets

Within the broader literature, the Verily dataset occupies a specific niche. It is much narrower than help-seeking datasets such as M-Help, which is a Reddit-based resource for detecting whether a user is “actively or indirectly seeking assistance related to mental health,” plus disorder and cause annotations. M-Help explicitly overlaps with crisis-like content but is not a crisis-severity or emergency-triage dataset (Sathvik et al., 21 Aug 2025). It is also distinct from MentalChat16K, which is an English benchmark for conversational mental health assistance composed of counseling-style prompt-response pairs and is not organized around crisis detection, suicidality, or emergency escalation (Xu et al., 13 Mar 2025).

Compared with newer crisis-specific benchmarks, Verily also reflects a different design choice. “Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs” introduces a crisis evaluation resource built from 12 public datasets, with a six-category clinically informed crisis taxonomy and a 5-point expert-designed rubric for evaluating response appropriateness; its focus is benchmark evaluation of crisis handling rather than construction of a healthcare-messaging safety filter (Arnaiz-Rodriguez et al., 29 Sep 2025). CRADLE Bench is a clinician-annotated Reddit-based benchmark for multi-faceted crisis and safety risk detection with seven crisis types and explicit ongoing/past temporal labels, together with a weakly supervised training corpus (Byun et al., 27 Oct 2025). CRADLE-Dialogue extends that direction to 600 multi-turn dialogues and turn-level crisis detection, explicitly showing that identifying when risk emerges in conversation is substantially harder than classifying static text (Byun et al., 9 Jun 2026).

Against this background, the Verily dataset can be characterized factually as English-only, single-turn, synthetic, clinician-reviewed, and centered on behavioral health safety filtering in digital health messaging. A 2025 survey of clinical mental health AI datasets does not explicitly mention a dataset called “Verily Mental Health Crisis Dataset”, so the resource is absent from that survey’s catalog and taxonomy (Mandal et al., 13 Aug 2025).

6. Limitations, ethics, and access conditions

The dataset’s main strengths are also the source of its main limitations. Because all messages are simulated, the benchmark avoids direct release of sensitive real-world crisis text, but it also cannot fully capture the distribution of naturally occurring user language. The paper states plainly that “all messages were simulated, not real user messages.” It also acknowledges that the resource is English-only, single-turn only, and may not capture all real-world slang or rapidly evolving coded language. No train/validation/test split is reported for the Verily dataset itself, and the paper does not provide a full prompt-generation pipeline or a complete annotation manual. These are substantial reproducibility constraints (Nelson et al., 14 Oct 2025).

The single-turn design is particularly consequential. The paper notes that “safety guardrails are known to fail more frequently in multi-turn conversations,” so performance on isolated messages may overestimate robustness in deployment settings where risk emerges gradually across dialogue. That caution aligns with later conversational crisis benchmarks showing marked degradation when models must localize risk turn by turn rather than classify a single text in isolation (Byun et al., 9 Jun 2026).

The paper does not present the dataset as a substitute for diagnosis or as a self-sufficient emergency system. It instead describes the filter as most useful as a screening tool with human-in-the-loop oversight, especially where avoiding missed crises, reducing alert fatigue, and pushing personalized crisis resources are critical. This suggests a workflow in which automated crisis detection supports escalation and review rather than replacing clinician judgment (Nelson et al., 14 Oct 2025).

Access is restricted rather than openly released. The paper’s data sharing statement says: “Data from this study are available upon researcher request.” No explicit license is provided for the Verily dataset, and the paper does not release a public benchmark package, model weights, or a complete dataset card. In that respect, the dataset differs from several public arXiv-era crisis benchmarks, even while addressing a similar safety problem (Nelson et al., 14 Oct 2025).