MentalBERT: Domain-Adapted Transformer

• MentalBERT is a family of domain-adapted pretrained transformers that extend BERT with mental health–specific corpora to detect conditions like depression and suicidality.
• It employs optimized pretraining strategies, including lexicon-guided masked language modeling, to achieve significant performance gains on both English and Chinese datasets.
• Evaluations reveal robust detection in mental health screening tasks while highlighting challenges with implicit cues and the need for multilingual and fairness enhancements.

MentalBERT denotes a family of domain-adapted pretrained transformer models, initially introduced for automatic mental health assessment from social media text. These models extend the BERT architecture via continued pretraining on mental health–related corpora and domain-specific pretraining strategies, yielding improved detection of depression, stress, suicidality, and related conditions across English and Chinese digital platforms. Distinct variants such as MentalBERT (English), Chinese MentalBERT, and MentalRoBERTa are documented, with multiple independent evaluations confirming performance advantages over general-purpose pretrained LLMs and biomedical or clinical LMs (Ji et al., 2021).

1. Model Architecture and Domain-Adaptive Pretraining

English MentalBERT

English MentalBERT is initiated from BERT-base-uncased: 12 Transformer encoder layers, hidden size 768, 12 self-attention heads, 3072-dimensional intermediate feedforward layer, and vocabulary of 30,522 WordPiece tokens. Total parameters: ~110M. Domain-adaptive pretraining is performed by continuing masked language modeling (MLM) and Next Sentence Prediction (NSP) objectives on 13.6M sentences from target-domain subreddits (e.g., r/depression, r/Anxiety, r/SuicideWatch, r/offmychest, r/bipolar, r/mentalillness) (Ji et al., 2021, Tavchioski et al., 2023, Ajayi et al., 25 Nov 2025). Tokenization follows the original BERT protocol (WordPiece, uncased).

Chinese MentalBERT

The Chinese MentalBERT line is constructed atop Chinese-BERT-wwm-ext, which features 12 layers, 768 hidden size, and Whole-Word Masking (WWM) for Chinese segmentation granularity. Pretraining is conducted on a corpus aggregating >3M Weibo posts focused on depression, suicide risk, and “tree-hole” narratives. Uniquely, psychological lexicons are integrated into the masking process: tokens in key lexica are preferentially masked and their prediction loss is reweighted: $$\mathcal{L}_{\rm MLM} = -\sum_{i\in M} w_i \log p(x_i|x_{\setminus i}), \quad w_i = \alpha \text{ if token } i \in \text{lexicon},\ w_i = 1 \text{ otherwise}$$ Pretraining uses a fixed-length 128-token input, batch size 128, and learning rate $5\times10^{-5}$ (Zhai et al., 14 Feb 2024, Qi et al., 19 Apr 2024).

Hyperparameters

Both English and Chinese variants use standard Adam/AdamW optimizers, typical learning rates ($10^{-5}$ to $10^{-4}$), batch sizes from 16–128 per GPU, and linear warm-up/decay. Dropout=0.1, weight decay=0.01 are standard.

2. Training Objectives and Domain Adaptation

All core MentalBERT variants are pretrained primarily with MLM—masking 15% (English) or 20% (Chinese) of tokens (with lexicon-guided upweighting in Chinese models)—and, on English, with the BERT-style NSP objective. For Chinese, only MLM is adopted in later work. Domain adaptation is realized via “continued pretraining”: the model is initialized from general-domain BERT(-wwm-ext), then trained further on domain-specific corpora, exposing it to colloquial symptom expressions, risk cues, and idiomatic mental health discourse absent from standard Wikipedia/BookCorpus/Baike data.

Lexicon-guided masking and weighted loss (applied in Chinese variants) explicitly bias representation learning toward psychiatric and crisis vocabulary, empirically improving downstream performance in depression, suicidality, and cognitive distortion classification (Zhai et al., 14 Feb 2024, Qi et al., 19 Apr 2024).

3. Downstream Evaluation and Benchmarks

English

Eight classification targets are evaluated in (Ji et al., 2021): depression (eRisk18, CLPsych15, Depression_Reddit), stress (Dreaddit), suicidal ideation (UMD Suicidality, T-SID), multi-label disorder detection (SWMH), and SMS-style stressor detection (SAD). Standard input representation uses the [CLS] token’s final hidden state, projecting to softmax output via a 1-layer MLP with tanh, typically with learning rates $1\times10^{-5}$–$3\times10^{-5}$. Performance metrics: precision, recall, F1-score (macro/micro as appropriate).

Chinese

Task coverage includes six-way emotion classification, COVID-era emotion analysis, cognitive distortion detection (12-label multi-label), and binary/high-low suicide risk (EWECT-usual, EWECT-epidemic, cognitive distortion, SOS-1K). Macro-F1 is the main metric. Model comparison includes domain-general BERT, DKPLM-medical, and word2vec-based models (Zhai et al., 14 Feb 2024, Qi et al., 19 Apr 2024).

Task	BERT-wwm-ext	MentalBERT (random)	MentalBERT (guided)
EWECT-usual	74.85	76.13	76.74
EWECT-epidemic	63.82	66.48	67.77
Cognitive distortion	84.16	84.66	86.77
Suicide risk	84.39	85.71	86.15

On the SOS-1K suicide risk dataset, Chinese MentalBERT achieved F1=88.39% (binary) and F1=50.89% (fine-grained 0–10), outperforming all other contemporary Chinese PLMs (Qi et al., 19 Apr 2024).

4. Comparative Performance and Error Analysis

Across multiple studies, domain-adaptive MentalBERT generally outperforms architectures pretrained on general, biomedical, or clinical corpora for in-domain mental health benchmarks:

• On English Reddit depression detection (multi-class), MentalBERT: F1=0.577 vs BERT/RoBERTa/BERTweet: 0.561–0.563 (Tavchioski et al., 2023).
• On English multiclass mental health/cyberbullying, MentalBERT: Acc=0.92, Macro-F1=0.76 vs BERT-base: 0.87/0.70; RoBERTa: 0.88/0.70 (Ajayi et al., 25 Nov 2025).
• In Chinese emotion/risk tasks, guided-masking MentalBERT outperforms prior SOTA by up to 7pp F1 (cognitive distortion), 3.75pp (suicide risk) (Zhai et al., 14 Feb 2024).

Error/robustness studies report that while MentalBERT excels when classification involves explicit mental health terminology (“topic words”), its performance degrades more than general BERT if these terms are removed, indicating reliance on domain-specific lexical cues rather than distributed semantic understanding (Tang et al., 20 Dec 2024).

5. Model Extensions: Multimodal and Calibration Augmentations

Ilias et al. (Ilias et al., 2023) introduced a Multimodal Adaptation Gate to inject linguistic and psychological features (LIWC, LDA, NRC, Top2Vec vectors) as auxiliary context, further improving F1 by ~1–2 pp across stress/depression detection tasks. Label smoothing ($\epsilon=0.001$) is employed to calibrate model confidences, halving the expected calibration error (ECE). The best F1 scores for binary depression recognition approach 93.1; for stress, 83.4.

Chinese MentalBERT incorporates lexicon-guided masking and weighted loss functions to amplify detection of rarer, semantically subtle psychiatric cues (Zhai et al., 14 Feb 2024, Qi et al., 19 Apr 2024).

6. Explainability, Human-in-the-Loop, and Deployment

Ajayi et al. (Ajayi et al., 25 Nov 2025) present a hybrid SHAP-LLM explainability framework: SHAP values are computed per token to identify crucial input features, which are then interpreted by a local LLM to produce narrative rationales. A “Social Media Screener” dashboard visualizes highlighted cues, model predictions, and LLM explanations, gated by moderator review (confirm, dismiss, recategorize). The intended deployment context is not clinical diagnosis, but human-in-the-loop screening for triage and moderation—explicitly addressing ethical accountability, transparency, and operator skepticism.

7. Limitations, Open Challenges, and Future Directions

• Language Scope: English and Chinese-only; robust domain adaptation for other languages is unaddressed (Ji et al., 2021, Zhai et al., 14 Feb 2024).
• Domain Generalization: Performance remains sensitive to the presence of explicit psychiatric keywords—general BERT architectures may surpass MentalBERT when such cues are absent or in subtle narrative text (Tang et al., 20 Dec 2024).
• Fairness/Bias: No thorough audit of demographic or content bias in pretraining corpora; future work cited for bias/fairness assessment (Ji et al., 2021).
• Calibration and Interpretability: Initial progress with label smoothing and SHAP-LLM pipelines, but post-hoc explainability and causal analysis are underdeveloped (Ajayi et al., 25 Nov 2025, Ilias et al., 2023).
• Augmentation/Scalability: For fine-grained classes (e.g., suicide risk levels 0–10), core limit is data sparsity and label subjectivity; augmentation approaches (e.g., round-trip translation, LLM generation, synonym replacement) yield modest F1 improvements (Qi et al., 19 Apr 2024).

Anticipated research directions include constructing multilingual variants, integrating broader psychological lexica and interview data, developing auxiliary objectives for deeper semantic modeling, and systematic fairness/robustness evaluations (Ji et al., 2021, Zhai et al., 14 Feb 2024, Tang et al., 20 Dec 2024).

---

Key Papers:

• “MentalBERT: Publicly Available Pretrained LLMs for Mental Healthcare” (Ji et al., 2021)
• “Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis” (Zhai et al., 14 Feb 2024)
• “A Machine Learning Approach for Detection of Mental Health Conditions and Cyberbullying from Social Media” (Ajayi et al., 25 Nov 2025)
• “Calibration of Transformer-based Models for Identifying Stress and Depression in Social Media” (Ilias et al., 2023)
• “SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis” (Qi et al., 19 Apr 2024)
• “Detection of depression on social networks using transformers and ensembles” (Tavchioski et al., 2023)
• “Decoding Linguistic Nuances in Mental Health Text Classification Using Expressive Narrative Stories” (Tang et al., 20 Dec 2024)