Verily Behavioral Health Safety Filter

• VBHSF is an AI-based system for detecting mental health crises using a two-stage hierarchical classification that prioritizes sensitivity.
• It employs a binary crisis detector followed by a multi-label crisis-type classifier, leveraging advanced prompt engineering and clinical reasoning.
• Empirical validation shows VBHSF outperforms state-of-the-art guardrails with robust recall and precision across internal and external crisis datasets.

The Verily Behavioral Health Safety Filter (VBHSF) is an AI-based, transformer-driven content moderation system engineered specifically to detect and categorize mental health crises in user-generated, single-turn text-based conversations. VBHSF operationalizes advanced prompt engineering and clinical reasoning to maximize sensitivity and specificity in identifying mental health emergencies, distinguishing it from general-purpose moderation guardrails through robust hierarchical classification and empirical validation across multiple datasets (Nelson et al., 14 Oct 2025).

1. System Architecture and Operational Workflow

VBHSF employs a two-stage hierarchical classification architecture comprising a primary crisis detector and a secondary multi-label crisis-type classifier. Input messages undergo standard normalization, including lowercasing, removal of excessive whitespace, and substitution of common internet-slang (e.g., "u" → "you", "unalive" → "kill myself"). No domain-specific handcrafted features are used beyond tokenization; the backbone is a transformer-based LLM based on the GPT paradigm, fine-tuned with advanced prompt engineering.

• Stage 1: Binary Crisis Detector The model conducts binary classification (Crisis vs. Non-Crisis), generating both a probability score and a binary label. The decision threshold is specifically tuned to optimize recall, prioritizing sensitivity to ensure minimal missed crises.
• Stage 2: Crisis-Type Classifier Invoked only for messages flagged as “Crisis,” this head employs a one-vs-rest multi-label output for eight crisis dimensions: abuse, neglect, eating-disorder behaviors, psychosis, self-harm, suicide, substance misuse, violence toward others, and mixed presentations.
• Prompt Design and Clinical Reasoning Prompts directly instruct the LLM in natural language with detailed descriptions and examples for each crisis category, including explicit (“I want to kill myself”), ambiguous, and masked (“I’m feeling unalive”) expressions. There is no rule-based post-processing; the model’s outputs are directly adopted as the filter’s verdict.

2. Datasets and Labeling Methodology

VBHSF was validated on two rigorously labeled datasets:

• Verily Mental Health Crisis Dataset v1.0 (Internal) This dataset comprises 1,800 simulated single-turn messages (900 crisis, 900 non-crisis), with the crisis subset uniformly distributed across nine clinical categories. The dataset is characterized by high linguistic variability: 56% of messages contain spelling/grammar errors, 46% use textese, and 14% include emojis or symbols. Expressions of crisis span direct risk (91%), ambiguous (9%), and masked/slang (9%), with overlap. Dual clinician annotation yielded outstanding interrater reliability (Cohen's κ = 0.99), and multi-category adjudication for all positive cases.
• NVIDIA Aegis AI Content Safety Dataset 2.0 Subset (External) This corpus includes 794 human-generated messages (397 crisis, 397 non-crisis) drawn from the “Suicide and Self Harm” domain, after removing duplicates and suicide-detection samples. Messages were relabeled by two clinicians, with 6.9% of original labels corrected. Only binary labeling (crisis/non-crisis) was utilized for this dataset.

3. Evaluation Metrics and Formulation

VBHSF’s empirical evaluation aligns with standard metrics:

• Sensitivity (Recall):

$$\mathrm{Sensitivity} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}$$

• Specificity:

$$\mathrm{Specificity} = \frac{\mathrm{TN}}{\mathrm{TN} + \mathrm{FP}}$$

• F1-score:

$$F_1 = 2 \times \frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}} = \frac{2\,\mathrm{TP}}{2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN}}$$

These metrics were reported with 95% confidence intervals, providing a quantitative framework for cross-system efficacy assessment.

4. Empirical Results and Performance Comparison

VBHSF demonstrated high and consistent performance on both internal and external datasets.

Table: Overview of Key Performance Metrics

Dataset	Sensitivity	Specificity	F1-score*	Accuracy
Verily Crisis (n=1,800)	0.990	0.992	0.939	—
NVIDIA Aegis Subset (n=794)	0.982	0.859	—	0.921

*F1-score reported for Verily dataset only

For the Verily Mental Health Crisis Dataset:

• Binary crisis detection: Sensitivity 0.990 (CI: 0.981–0.995), Specificity 0.992 (CI: 0.984–0.996).
• Crisis-type multi-label classification: Macro-averaged F1-score 0.939 (CI: 0.927–0.951), sensitivity ranged 0.917–0.992 per category, and specificity was ≥0.978 for all types.

On the NVIDIA Aegis subset:

• Sensitivity was 0.982 (CI: 0.964–0.991), specificity 0.859 (CI: 0.821–0.889), and accuracy 0.921 (CI: 0.900–0.937).

5. Comparative Analysis: VBHSF Versus State-of-the-Art Guardrails

VBHSF was benchmarked against OpenAI Omni Moderation Latest and NVIDIA NeMo Guardrails:

System	Sensitivity (Verily)	Specificity (Verily)	Sensitivity (NVIDIA)	Specificity (NVIDIA)
VBHSF	0.990	0.992	0.982	0.859
OpenAI Omni Moderation	0.419	0.999	0.882	0.899
NVIDIA NeMo Guardrails	0.759	0.756	0.907	0.886

Statistical analyses (Cochran’s Q, McNemar with Bonferroni correction) demonstrated that VBHSF achieved significantly higher sensitivity than both comparators on all splits (p<0.001), and significantly higher specificity than NVIDIA NeMo on the Verily dataset (p<0.001). Specificity differences with OpenAI were not significant (p=0.094). At the crisis-type level, VBHSF sensitivity ranged from 0.880–0.992 (consistent and high), while OpenAI (0.097–0.916) and NVIDIA NeMo (0.503–1.000) exhibited highly variable results, including sensitivity below 0.10 for certain categories.

6. Design Priorities and Clinical Implications

VBHSF explicitly prioritizes sensitivity to minimize missed mental health crises (“never miss a crisis”), a critical property in healthcare triage and clinical safety applications. Authors report the filter’s high generalizability across simulated and real-world datasets, with negligible loss in sensitivity between internal and external message distributions. The lack of handcrafted feature extraction and absence of post-hoc rule-based refinement emphasize reliance on the transformer’s learned clinical reasoning, bolstered by natural language prompt engineering. These characteristics position VBHSF as a strong candidate for integration within human-in-the-loop triage, digital behavioral health platforms, and large-scale content safety systems, where low false negative rates are paramount (Nelson et al., 14 Oct 2025).

7. Limitations and Future Research Considerations

While VBHSF exhibits robust sensitivity and specificity, specificity is modestly reduced on external data (NVIDIA Aegis subset). The comparator systems, though highly specific, fail to generalize across crisis types, with some category-level sensitivities below clinically acceptable thresholds. A plausible implication is that VBHSF’s performance is largely driven by the joint effect of its high-quality prompt engineering, clinical input, and supervised transformer fine-tuning, whereas traditional guardrails lack such clinical adaptation. Future investigations could target improved specificity in real-world settings and further diversification of message modalities.