Social Media Screener: Mental Health NLP

• Social Media Screener is a tool that uses transformer models tailored for analyzing mental health content in social media posts.
• It employs targeted pretraining techniques and lexicon-guided masking to focus on clinical cues from specialized mental-health domains.
• Benchmark evaluations demonstrate improved detection of depression, stress, and suicidal ideation compared to generic NLP models.

MentalBERT is a family of domain-adaptive pretrained LLMs engineered for natural language understanding and classification tasks in mental healthcare. These transformer architectures employ targeted pretraining regimens on large mental-health–oriented corpora to optimize for the detection and analysis of mental disorders, stress, and suicidal ideation in social media and digital text. MentalBERT and its variants, including Chinese MentalBERT, have demonstrably advanced the state of the art in psychiatric NLP by surpassing generic and biomedical models on benchmark tasks, while also revealing nuanced limitations in narrative comprehension and topic word dependency.

1. Pretraining Strategies and Domain-Specific Corpora

MentalBERT leverages domain-adaptive pretraining schemes that continue training classic transformer architectures (BERT-base, RoBERTa-base) using corpora exclusively sourced from mental-health domains. The canonical English MentalBERT is initialized from BERT-base-uncased and further pretrained on approximately 13.7 million sentences from Reddit subreddits explicitly focused on mental health (e.g., r/depression, r/SuicideWatch, r/Anxiety, r/offmychest, r/bipolar, r/mentalillness, r/mentalhealth), totaling hundreds of millions of tokens (Ji et al., 2021). No user profile or personally identifiable information is harvested; only public text is used.

In the Chinese context, Chinese MentalBERT aggregates over 3.36 million posts from Weibo tree-hole communities, depression super-topics, and curated depression datasets, subject to de-duplication and removal of noisy entries (Zhai et al., 14 Feb 2024). The mental-health language style and vocabulary differ substantially from both clinical documents and general social media, motivating in-domain selection for pretraining data.

A unique aspect of Chinese MentalBERT is lexicon-guided masking. Seeded lexicons constructed from labeled posts guide the MLM masking process, prioritizing salience on depression or suicide-related terms. Masking all lexicon tokens and upweighting their prediction loss biases the model toward clinical cues, distinguishing it from conventional random masking approaches.

2. Model Architectures and Training Protocols

All MentalBERT variants retain the architectural parameters of their base models:

• MentalBERT (English): 12 Transformer encoder layers, hidden size 768, 12 self-attention heads per layer, intermediate size 3072, WordPiece vocabulary (30,522 tokens), ~110M parameters (Ji et al., 2021).
• MentalRoBERTa: initialized from RoBERTa-base, dynamic masking, vocabulary 50,265, ~125M parameters.
• Chinese MentalBERT: based on Chinese-BERT-wwm-ext, 12 layers, hidden size 768, whole-word masking optimized for Chinese segmentation, vocabulary ~21,000 tokens (Zhai et al., 14 Feb 2024).

Pretraining employs classic objectives: Masked Language Modeling (MLM) and (for English) Next Sentence Prediction (NSP):

$$\mathcal{L}_{\text{MLM}} = -\sum_{i\in M} \log P_{\theta}(x_i \mid x_{\setminus M})$$

Additionally, lexicon-guided weighting in Chinese models modifies the MLM loss:

$$\mathcal{L}_{\text{MLM}} = -\sum_{i\in M} w_i\;\log P(x_i \mid x_{\backslash i})$$

where $w_i = \alpha$ if $x_i$ belongs to the lexicon ($\alpha > 1$), otherwise $w_i = 1$.

Hyperparameters closely follow base model defaults: Adam/AdamW optimizer, learning rates ($5\text{–}10^{-5}$ for pretraining, $1\text{–}3\times10^{-5}$ for fine-tuning), dropout 0.1, batch sizes 16–128, linear warmup, and early stopping based on validation loss. Fine-tuning heads are single-layer MLPs with tanh activation over the [CLS] embedding, followed by softmax.

3. Downstream Evaluation and Benchmark Performance

MentalBERT family models are systematically evaluated on binary and multi-class classification tasks spanning depression, stress, anxiety, bipolar, and suicidal ideation, using Reddit, Twitter, SMS, and Weibo text.

English MentalBERT demonstrates consistent improvement in recall and F1 over base BERT/RoBERTa and biomedical/clinical variants (BioBERT, ClinicalBERT) on tasks such as:

• Depression detection: CLPsych15, eRisk18 (Ji et al., 2021)
• Stress classification: Dreaddit
• Suicidal ideation: T-SID, UMD Suicidality

Example: MentalRoBERTa achieves F1 = 93.38 on eRisk18 depression (Recall = 93.38), outperforming generic RoBERTa (F1 = 92.25).

Chinese MentalBERT with lexicon-guided masking achieves macro-F1 gains of +1–2.6 percentage points over general Chinese BERT and up to +7 over non-domain competitors on tasks including suicide risk, emotion classification, and cognitive distortion labeling (Zhai et al., 14 Feb 2024). In fine-grained suicide risk detection (SOS-1K), it yields F1 = 88.39% (binary) and 50.89% (11-class) (Qi et al., 19 Apr 2024).

Multiclass and Real-world Screening: Ajayi et al. train MentalBERT as part of a ten-class social media screening framework; after end-to-end fine-tuning, MentalBERT attains accuracy 0.92 and macro-F1 0.76, outperforming generic BERT (0.87/0.70) and zero-shot LLMs (Ajayi et al., 25 Nov 2025).

Ensembling: Ensemble strategies (simple averaging, Bayesian) combining MentalBERT with generic transformers improve performance further, highlighting MentalBERT's in-domain complementary strength (Tavchioski et al., 2023).

4. Model Calibration, Feature Injection, and Explainability

MentalBERT has been the subject of extensive studies on model calibration and multimodal embedding strategies (Ilias et al., 2023). Injecting external linguistic features (LIWC, NRC, LDA topics, Top2Vec) via Multimodal Adaptation Gate (MAG) boosts detection F1 scores by ~1–2 points. Label smoothing ($\epsilon = 0.001$) reduces calibration errors (ECE, ACE) from $\sim$0.07–0.16 to $\sim$0.03–0.06, vital for operational reliability in high-risk psychiatric screening.

Ajayi et al. introduce a SHAP–LLM hybrid framework for interpretability. SHAP values quantify token contributions to MentalBERT's predictions, which are then summarized by a smaller LLM into actionable rationales for human moderators within dashboard interfaces. This approach ensures human-in-the-loop monitoring and enforceable audit trails, critical for deployment in mental health contexts (Ajayi et al., 25 Nov 2025).

5. Narrative Sensitivity, Limitations, and Robustness Insights

Recent work exposes MentalBERT’s reliance on explicit topic and symptom words for optimum classification accuracy. In expressive narrative analysis (ENS), removing high-frequency mental-health terms reduces MentalBERT’s accuracy by a statistically significant margin ($\sim$0.6 percentage points, $p<0.01$) while general BERT remains robust (Tang et al., 20 Dec 2024).

Disrupting sentence order within posts also degrades MentalBERT’s classification, though the impact ($\sim$1.4 pp) remains less than that of keyword removal. In contrast, general BERT maintains performance, suggesting superior sensitivity to distributed linguistic features.

This limitation constrains MentalBERT's applicability in settings where narratives lack overt psychiatric vocabulary, underscoring the necessity for contrastive adaptation, coherence objectives, or pretraining on label-sparse expressive text.

6. Multilingual and Lexicon-Guided Adaptations

Chinese MentalBERT exemplifies the move toward multilingual, culturally adaptive models. Continued pretraining on Chinese social media, paired with psychological lexicon guidance, yields best-in-class performance on Chinese emotion, stress, and suicide detection tasks (Zhai et al., 14 Feb 2024, Qi et al., 19 Apr 2024).

Lexicon-guided MLM, wherein all psychologically salient tokens are masked and subject to loss up-weighting, focuses capacity on target semantics. Empirical results show such guided models shift predictions toward emotionally relevant tokens (e.g., “悲伤,” “折磨,” “困难”) and outperform random-masking baselines in both quantitative and qualitative measures (all improvements $p<0.05$).

7. Applications, Limitations, and Future Directions

Applications span early screening for depression/suicidal ideation, automated triage support, continuous monitoring, and embedding services for downstream psychiatric chatbot and moderation platforms. Human-in-the-loop dashboard prototypes integrate MentalBERT outputs with explainability analytics, allowing informed expert intervention (Ajayi et al., 25 Nov 2025).

Limitations include:

• English and Chinese language restriction (no current support for other languages)
• Reliance on public, potentially biased corpora
• Lack of clinical diagnostic validity; models are screening tools only
• Explainability, fairness, and sensitivity to topic word presence remain open concerns

Future directions are actively investigated:

• Extension to multilingual and broader psychological domains
• Deeper interpretability via attention visualization, feature attribution
• Training from scratch on enlarged and diversified corpora (e.g., interview transcripts)
• Systematic bias and fairness analysis in mental-health risk prediction
• Integration of multimodal (text+image+emoji) signals for enhanced assessment granularity

Summary Table: MentalBERT Benchmarks (Selected Tasks)

Model	Task(s)	Macro-F1 (%)	Key Adaptation
MentalBERT (English)	Depression, stress	58–94	Reddit mental health posts
MentalRoBERTa	Depression	93	Reddit, dynamic masking
Chinese MentalBERT	Suicide risk, emotions	67–88	Weibo, lexicon-guided masking
SHAP–LLM + MentalBERT	Multiclass screening	76	Explainable human-in-loop

MentalBERT and its derivatives define the contemporary standard for transformer-based psychiatric text analytics, with continuous innovation in model calibration, explainability, and domain adaptation shaping their evolution (Ji et al., 2021, Zhai et al., 14 Feb 2024, Ajayi et al., 25 Nov 2025, Ilias et al., 2023, Tang et al., 20 Dec 2024, Qi et al., 19 Apr 2024, Tavchioski et al., 2023).