MentalRoBERTa: Domain-Adapted Mental Health Transformers

• MentalRoBERTa is a suite of RoBERTa-based transformer models pretrained on mental-health discourse, enhancing psychiatric classification tasks.
• The models use dynamic masking and continued pretraining on data from support forums to achieve improved recall and precision over general-domain and biomedical transformers.
• Applications include depression detection, early suicide risk stratification, conversational early warning, and multimodal fusion for comprehensive mental health assessment.

MentalRoBERTa refers to a family of RoBERTa-based transformer models that undergo continued pretraining (or, in some cases, pretraining from scratch) on mental-health–specific discourse—primarily user-generated text from mental-health support forums and subreddits such as r/depression, r/anxiety, r/mentalillness, among others. The resultant models, exemplified by MentalRoBERTa-base, MentalRoBERTa-large, and language-adapted variants, are then fine-tuned or probed on downstream psychiatric classification, risk detection, and mental-health assessment tasks, consistently demonstrating advantages over general-domain transformers and even biomedical/clinical domain-adapted alternatives in recall and precision for key mental-health benchmarks (Ji et al., 2021, Marmol-Romero et al., 24 Sep 2025).

1. Model Architecture and Pretraining Objectives

The prototypical MentalRoBERTa model adopts the RoBERTa-base or RoBERTa-large blueprint: either 12 or 24 Transformer encoder layers, with hidden sizes of 768 or 1024, 12 or 16 self-attention heads per layer, and approximately 125M to 355M parameters respectively. Pretraining is conducted via dynamic masking on unlabeled corpora of mental-health–relevant text, minimizing the standard masked language modeling (MLM) objective:

$$L_{\mathrm{MLM}} = -\frac{1}{|M|} \sum_{i\in M} \log P_\theta(x_i \mid x_{/M})$$

where $M$ is the set of masked positions and $P_\theta$ denotes the output softmax probabilities (Ji et al., 2021, Marmol-Romero et al., 24 Sep 2025).

The tokenization is byte-level BPE, identical to RoBERTa-base, with a vocabulary of 50,265 tokens. No new special tokens or lexical adaptations are introduced unless specified for language-specific variants (see Section 6). Models are typically initialized from the publicly released RoBERTa checkpoints and then “domain-adapted” through further MLM pretraining on a corpus focused on mental-health self-disclosure, including over 13M sentences from seven major mental-health subreddits (Ji et al., 2021).

2. Fine-Tuning Protocols and Downstream Task Configurations

Standard fine-tuning attaches a task-specific classification MLP atop either the pooled [CLS] representation or mean-pooled final-layer token embeddings. For binary and multi-class detection (e.g., depression vs. control, multi-disorder), the training signal is categorical cross-entropy (or its ordinal/flavored variants as in risk-level regression):

$$L_{\mathrm{CE}} = -\frac{1}{N} \sum_{i=1}^N \sum_{c=1}^C y_{i,c} \log \hat{y}_{i,c}$$

Common fine-tuning hyperparameters use learning rates in the range of $1 \times 10^{-5}$ to $5 \times 10^{-5}$, batch sizes of 4–32, AdamW optimizer, and early stopping on validation F1 or macro-F1. For specific early-detection and risk-assessment pipelines, additional mechanisms are attached: e.g., dual-head models combining Consistent Rank Logits (CORAL) with categorical softmax (Yang et al., 23 Oct 2025), temporal embedding of posting intervals, and hierarchical context encoding (Marmol-Romero et al., 24 Sep 2025).

A summary of representative configurations is presented below:

Variant	Layers	Hidden Size	Pretraining Corpus	Task Head
MentalRoBERTa-base	12	768	Reddit (mental health subs)	MLP
MentalRoBERTa-large	24	1024	Reddit (mental health subs)	MLP
belabBERT (Dutch)	12	768	Dutch OSCAR + clinical text	MLP

3. Applications: Classification, Assessment, and Retrieval

Depression Detection and Risk Assessment:

MentalRoBERTa models yield state-of-the-art recall and F1 in binary and multi-class mental-health classification benchmarks (eRisk, CLPsych, UMD, T-SID). For example, on depression detection tasks, MentalRoBERTa achieves F1 up to 93.38 (eRisk T1) and often outperforms BioBERT and ClinicalBERT (Ji et al., 2021).

Early Detection and Conversational Strategies:

In the eRisk@CLEF 2025 task, MentalRoBERTa-large enabled the SINAI-UJA system to rapidly issue early predictions with high recall (1.00) at the cost of modest precision (0.21), evidencing the trade-off between detection speed and false positives (Marmol-Romero et al., 24 Sep 2025). Context encoding included hierarchical flattening of multi-user conversation threads with explicit [MSG], [USER], and role-type markers and additional pruning/post-processing to reduce irrelevant context.

Suicide Risk Stratification:

A hierarchical dual-head model based on MentalRoBERTa with partially frozen layers and temporal embedding successfully modeled ordinal suicide risk levels, integrating cross-entropy, CORAL, and focal loss terms to address class imbalance and ordinal structure (Yang et al., 23 Oct 2025).

Sentence Embedding and Retrieval:

MentalRoBERTa’s sentence-level embeddings are obtained by mean-pooling the output token vectors. While effective for mental-health classification, these embeddings are less suited for semantic search than models like MPNet, which are explicitly fine-tuned for retrieval objectives (Bucur, 2023).

4. Empirical Performance and Benchmarking

Quantitative Results:

MentalRoBERTa matches or exceeds off-the-shelf BERT, RoBERTa, and domain-specialized models on almost all evaluated tasks. Representative benchmark scores for depression detection:

Model	eRisk T1 F1	CLPsych15 F1	Depression_Reddit F1
BERT-base	88.54	62.75	90.90
RoBERTa-base	92.25	66.07	95.11
BioBERT	78.86	65.50	90.98
MentalBERT	86.20	62.63	94.62
MentalRoBERTa	93.38	69.71	94.23

(Ji et al., 2021)

In early-detection settings, MentalRoBERTa Large achieved perfect recall with an F1 of 0.35, issuing alerts at an average of three messages per user thread and maintaining high throughput (0.99 threads/sec) (Marmol-Romero et al., 24 Sep 2025).

For Dutch psychiatric text, belabBERT (MentalRoBERTa-style for Dutch) demonstrated a 6–10% accuracy gain over RobBERT and audio-only baselines in text-based classification of psychotic and depressive patients (Wouts et al., 2021).

5. Design Decisions: Pretraining Corpus and Layer Utilization

Domain-Adaptive Pretraining:

Empirical results demonstrate that continued pretraining on mental-health corpus (user discourse from Reddit) is essential: models pretrained on biomedical (BioBERT), clinical narratives (ClinicalBERT), or even general RoBERTa do not internalize the distinctive emotional, colloquial, and self-reporting linguistic patterns found in mental-health communication (Ji et al., 2021).

Layer Probing and Feature Extraction:

Depression assessment pipelines employing MentalRoBERTa-large find maximal predictive accuracy when using pooled features extracted from “distributed second-half” layers (e.g., 16, 19, 22, 24), rather than the standard practice of using only the last four layers (Matero et al., 2021). The optimal single layer was found to be layer 19; aggregating multiple such layers further improved mean squared error and correlation with depression scores.

6. Language Adaptations and Multimodal Extensions

Non-English Variants:

belabBERT exemplifies adaptation of MentalRoBERTa-style models to other languages. It incorporates a Dutch-specific BPE tokenizer, is pretrained from scratch on 32 GB of Dutch web data (OSCAR), and achieves superior psychiatric classification results relative to the previous best Dutch model, RobBERT (Wouts et al., 2021).

Multimodal Fusion:

Late-fusion approaches combining text-based MentalRoBERTa embeddings with acoustic features yield further gains in psychiatric status classification, with hybrid models outperforming both unimodal counterparts (Wouts et al., 2021).

7. Limitations and Prospective Research Directions

Limitations:

Corpus coverage is limited to Reddit and, in language-specific cases, OSCAR dumps; generalization to platforms such as Twitter and Facebook has not been comprehensively validated. Annotation is frequently weakly supervised, relying on subreddit labeling rather than clinical diagnosis. Distributional shifts, cross-platform discourse variation, and the representation of comorbid or ambiguous cases remain open challenges (Ji et al., 2021).

Future Work:

Ongoing research recommends:

• Incorporating domain-expert manual annotation, especially for edge cases or multi-label data.
• Exploring confidence-based decision thresholds, ERDE cost calibration, and dynamic context pruning to refine early-detection tradeoffs (Marmol-Romero et al., 24 Sep 2025).
• Building longer-context transformers (e.g., “Mental Longformer”) for extended dialogues.
• Expanding to raw-audio or multimodal transformer stacks for richer psychiatric state classification.
• Developing interpretability mechanisms (saliency, probing) to elucidate linguistic indicators of mental health (Wouts et al., 2021).

A plausible implication is that robust real-world deployment requires both further corpus diversification and architecture adaptations capable of handling heterogeneous, temporally extended social-media discourse.