---
title: Verily Mental Health Crisis Dataset v1.0
url: https://www.emergentmind.com/topics/verily-mental-health-crisis-dataset
type: topic
---

# Verily Mental Health Crisis Dataset v1.0

The **Verily Mental Health Crisis Dataset v1.0** is a clinician-reviewed benchmark for detecting mental health crises in text-based conversations. It was introduced as the evaluation substrate for the **Verily Behavioral Health Safety Filter (VBHSF)** and is defined by a deliberately narrow scope: identifying whether a single text message contains **any mental health crisis** and, if so, assigning one or more crisis categories relevant to psychiatric emergency response and safety triage. The dataset is **synthetic**, **English-only**, and **single-turn** rather than a corpus of real user messages or multi-turn counseling logs, because the authors state that real mental health crisis text is highly sensitive and difficult to access [2510.12083].

## 1. Definition and intended use

The dataset is not framed as a general moderation resource, a generic toxicity benchmark, or a diagnostic corpus. Its intended role is more specific: support **behavioral health safety filtering** for LLM-enabled systems operating in **text-first digital health and conversational AI settings**, including **LLM safety monitoring**, **chatbot oversight**, **clinical triage systems**, **integrated behavioral health platforms**, and **large-scale digital health conversational agents**. The paper explicitly argues that existing moderation tools are “not fit-for-purpose for detecting psychiatric crises,” and positions the dataset as infrastructure for a specialized safety layer rather than for autonomous clinical decision-making [2510.12083].

| Property | Value |
|---|---|
| Corpus type | Simulated English single-turn text messages |
| Dataset size | 1,800 messages |
| Top-level balance | 900 crisis / 900 non-crisis |
| Primary tasks | Stage 1 binary crisis detection; Stage 2 multi-label crisis category classification |
| Clinical review | Two licensed clinicians |
| Reported agreement | Cohen’s \(\kappa = 0.99\) |
| Access | Available upon researcher request |

This design makes the dataset closer to a **mental-health-specific safety benchmark** than to broader mental-health NLP resources centered on help-seeking, counseling quality, or disorder prediction. The paper also treats high sensitivity as a core requirement, because in this application domain missed crises are operationally more serious than over-flagging [2510.12083].

## 2. Corpus construction and message composition

The dataset contains **1,800 simulated messages** generated with **pre-trained language models**. The authors state that synthesis was used “to mitigate the unavailability of highly sensitive mental health crisis messages.” The top level is exactly balanced: **900 crisis messages** and **900 non-crisis messages**. All items are **single messages**, not dialogue histories, and the paper identifies this single-turn design as a limitation [2510.12083].

The **900 crisis messages** are organized into **nine crisis groupings of 100 messages each**:

| Crisis grouping | Messages |
|---|---:|
| Abuse | 100 |
| Neglect | 100 |
| Eating-disorder behaviors | 100 |
| Psychosis | 100 |
| Self-harm | 100 |
| Suicide | 100 |
| Substance misuse | 100 |
| Violence toward others | 100 |
| Mixed crisis presentations | 100 |

The paper sometimes describes the system as identifying “eight dimensions” of crisis, but the construction section lists **nine crisis message categories**, with **mixed crisis presentations** functioning as an additional grouping. That inconsistency is internal to the paper itself [2510.12083].

The **900 non-crisis messages** are not generic background chatter. They are drawn from **health-related but non-emergency conversational scenarios**, including **appointment logistics, behavior change, follow-up care, follow-up clarification, health education, health monitoring, insurance and billing, lab and wearable data interpretation, lifestyle modification, locating providers, medical record request, preventative care, referral request, test scheduling, symptom tracking, medication pickup logistics, chronic condition management, and diet and nutrition**. This produces what the paper describes as **near-domain distractor content**, which is important for evaluating systems intended for digital health messaging rather than open-domain social media [2510.12083].

The message surface form was also designed to resemble informal texting. Among crisis messages, **91.10%** were classified as **direct explicit expressions of risk**, **8.90%** as **ambiguous statements of harm**, and **8.6%** included **slang or masked language**. Across the full dataset, **language mechanics errors** occur in **55.90%** of messages, **textese** in **45.80%**, and **emojis and symbolic markers** in **13.50%**. The descriptive statistics reported for the full corpus are **96.70** mean characters per message (\(\text{SD }84.50\)), **18.07** mean words (\(\text{SD }15.14\)), **1.50** mean sentences (\(\text{SD }0.82\)), **4,695 unique tokens**, **32,527 total tokens**, and a **type-token ratio of 0.144** [2510.12083].

## 3. Clinical review and label schema

The annotation workflow, as documented, is clinician-centered but relatively compact. The synthetic messages were independently reviewed by **two licensed clinicians**, identified in the paper as **BWN** and **AR** (**Benjamin W. Nelson** and **Alanna Robinson**). They annotated all **1,800 messages** as **crisis** or **non-crisis**, achieved **Cohen’s \(\kappa = 0.99\)**, and resolved disagreements through **adjudication and consensus** [2510.12083].

The label design is hierarchical. **Stage 1** is a binary decision: **any mental health crisis** versus **non-crisis**. **Stage 2** is a **multi-label crisis-type classification** step evaluated in a **one-vs-rest framework**. The crisis taxonomy comprises the nine groupings listed above, with **mixed crisis presentations** allowing composite cases. The paper does not provide a full formal ontology or an exact annotation handbook, but it is explicit that the category system is intended to cover “the most urgent and high-risk presentations” linked to adverse outcomes and, in many cases, mandatory reporting obligations [2510.12083].

The stage-2 design is importantly **not** a severity scale. The paper reports **no low/moderate/high crisis gradation**, **no suicidality severity ladder**, and **no explicit operational threshold definitions for imminent danger**. It also does not give DSM-style operational criteria for categories such as **psychosis**, and it notes that **grave disability** and **homicidality** are not separate labels; the closest reported class for the latter is **violence toward others**. The taxonomy therefore supports crisis-category recognition rather than formal clinical risk stratification [2510.12083].

The paper further indicates that stage-2 labeling is genuinely non-exclusive. Category totals reported in its evaluation tables exceed the **900** crisis-message count, which is consistent with the multi-label setup. This is especially relevant for **mixed crisis presentations**, where a single message can express more than one urgent crisis type [2510.12083].

## 4. Evaluation protocol and reported performance

The principal model evaluated on the dataset is the **Verily Behavioral Health Safety Filter**, described as “a transformer-based LLM (GPT architecture) that uses advanced prompt engineering and clinical reasoning to detect the presence of a crisis within a message and classify the specific crisis type.” The evaluation is split into **Stage 1 overall crisis detection** and **Stage 2 category-level classification**, with comparison against **OpenAI Omni Moderation Latest** and **NVIDIA NeMo Guardrails** [2510.12083].

For **Stage 1 overall crisis detection** on the Verily dataset, the reported VBHSF performance is:

- **Accuracy**: **0.991** (95% CI: **0.986–0.995**)
- **Sensitivity**: **0.990** (95% CI: **0.981–0.995**)
- **Specificity**: **0.992** (95% CI: **0.984–0.996**)
- **PPV**: **0.992** (95% CI: **0.984–0.996**)
- **NPV**: **0.990** (95% CI: **0.981–0.995**)

The paper states that this corresponded to **7 false positives** and **9 false negatives** [2510.12083].

| System | Sensitivity | Specificity |
|---|---:|---:|
| VBHSF | 0.990 | 0.992 |
| OpenAI Omni Moderation Latest | 0.419 | 0.999 |
| NVIDIA NeMo Guardrails | 0.759 | 0.756 |

The between-system differences were tested statistically. The paper reports **Cochran’s \(Q(2)=751.12, p<0.001\)** for sensitivity and **Cochran’s \(Q(2)=418.68, p<0.001\)** for specificity. Pairwise **McNemar tests** with **Bonferroni correction \(\alpha=0.017\)** found that VBHSF sensitivity was higher than both OpenAI and NVIDIA (**\(p<0.001\)** in both cases), and VBHSF specificity was higher than NVIDIA (**\(p<0.001\)**) but not significantly different from OpenAI (**\(p=0.094\)**) [2510.12083].

For **Stage 2 crisis subtype classification**, the paper reports the following macro-averaged performance for VBHSF:

- **Macro F1-score**: **0.939** (95% CI: **0.927–0.951**)
- **Macro sensitivity**: **0.957** (95% CI: **0.937–0.978**)
- **Macro PPV**: **0.923** (95% CI: **0.894–0.952**)

Its narrative summary states that **per-category sensitivity ranged from 0.917 to 0.992** and **specificity was at or above 0.978 for all crisis types**. The paper also notes that among **898 messages flagged as “Crisis”**, **88 messages (9.8%)** contained at least one category misclassification, yielding **125 individual category errors** [2510.12083].

The authors additionally evaluate VBHSF on a separate benchmark, the **NVIDIA Aegis AI Content Safety Dataset** subsetted to **794 mental health-related messages**. On that set, VBHSF remained highly sensitive, with **sensitivity 0.982**, **accuracy 0.921**, and **specificity 0.859**. The paper interprets this as evidence of generalization beyond the internal Verily benchmark, though still within text-based safety filtering [2510.12083].

A further operational analysis projects **PPV** under low crisis prevalence. At **2% prevalence**, the projected PPV is **0.716** for VBHSF, **0.895** for OpenAI, and **0.060** for NVIDIA. The paper uses this result to emphasize the low-base-rate difficulty of crisis screening: even high-performing systems can create substantial false-alarm burden when deployed autonomously at population scale [2510.12083].

## 5. Relation to adjacent mental-health and crisis datasets

Within the broader literature, the Verily dataset occupies a specific niche. It is much narrower than **help-seeking** datasets such as **M-Help**, which is a Reddit-based resource for detecting whether a user is “actively or indirectly seeking assistance related to mental health,” plus disorder and cause annotations. M-Help explicitly overlaps with crisis-like content but is not a crisis-severity or emergency-triage dataset [2508.15440]. It is also distinct from **MentalChat16K**, which is an English benchmark for **conversational mental health assistance** composed of counseling-style prompt-response pairs and is not organized around crisis detection, suicidality, or emergency escalation [2503.13509].

Compared with newer crisis-specific benchmarks, Verily also reflects a different design choice. **“Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs”** introduces a crisis evaluation resource built from **12 public datasets**, with a **six-category clinically informed crisis taxonomy** and a **5-point expert-designed rubric** for evaluating response appropriateness; its focus is benchmark evaluation of crisis handling rather than construction of a healthcare-messaging safety filter [2509.24857]. **CRADLE Bench** is a clinician-annotated Reddit-based benchmark for **multi-faceted crisis and safety risk detection** with **seven crisis types** and explicit **ongoing/past temporal labels**, together with a weakly supervised training corpus [2510.23845]. **CRADLE-Dialogue** extends that direction to **600 multi-turn dialogues** and **turn-level crisis detection**, explicitly showing that identifying when risk emerges in conversation is substantially harder than classifying static text [2606.10380].

Against this background, the Verily dataset can be characterized factually as **English-only**, **single-turn**, **synthetic**, **clinician-reviewed**, and centered on **behavioral health safety filtering** in digital health messaging. A 2025 survey of clinical mental health AI datasets does **not explicitly mention** a dataset called **“Verily Mental Health Crisis Dataset”**, so the resource is absent from that survey’s catalog and taxonomy [2508.09809].

## 6. Limitations, ethics, and access conditions

The dataset’s main strengths are also the source of its main limitations. Because all messages are **simulated**, the benchmark avoids direct release of sensitive real-world crisis text, but it also cannot fully capture the distribution of naturally occurring user language. The paper states plainly that “all messages were simulated, not real user messages.” It also acknowledges that the resource is **English-only**, **single-turn only**, and may not capture all real-world slang or rapidly evolving coded language. No **train/validation/test split** is reported for the Verily dataset itself, and the paper does not provide a full prompt-generation pipeline or a complete annotation manual. These are substantial reproducibility constraints [2510.12083].

The single-turn design is particularly consequential. The paper notes that “safety guardrails are known to fail more frequently in multi-turn conversations,” so performance on isolated messages may overestimate robustness in deployment settings where risk emerges gradually across dialogue. That caution aligns with later conversational crisis benchmarks showing marked degradation when models must localize risk turn by turn rather than classify a single text in isolation [2606.10380].

The paper does not present the dataset as a substitute for diagnosis or as a self-sufficient emergency system. It instead describes the filter as most useful as a **screening tool** with **human-in-the-loop oversight**, especially where **avoiding missed crises**, **reducing alert fatigue**, and **pushing personalized crisis resources** are critical. This suggests a workflow in which automated crisis detection supports escalation and review rather than replacing clinician judgment [2510.12083].

Access is **restricted rather than openly released**. The paper’s data sharing statement says: **“Data from this study are available upon researcher request.”** No explicit license is provided for the Verily dataset, and the paper does not release a public benchmark package, model weights, or a complete dataset card. In that respect, the dataset differs from several public arXiv-era crisis benchmarks, even while addressing a similar safety problem [2510.12083].

Source: https://www.emergentmind.com/topics/verily-mental-health-crisis-dataset