MentaLLaMA-IMHI: Mental Health NLP Framework

• MentaLLaMA-IMHI is a research framework integrating the IMHI dataset and LLaMA2-based models for interpretable mental health analysis on social media.
• It produces natural-language, step-by-step rationales alongside classification outputs, ensuring transparency and diagnostic clarity.
• The system attains near state-of-the-art accuracy across diverse tasks while providing human-aligned explanations and robust multi-task performance.

MentaLLaMA-IMHI is a research framework for interpretable mental health analysis on social media, uniting the IMHI dataset—a large, expert-evaluated multi-task corpus—and the MentaLLaMA family of instruction-tuned LLMs based on LLaMA2. The system targets two core limitations of prior work in mental health NLP: the lack of comprehensive, high-quality labeled data with explanations, and the absence of open-source foundation models tailored for interpretable diagnosis and reasoning. MentaLLaMA models provide natural-language, step-by-step rationales alongside classification outputs, leveraging domain-specific instruction tuning to approach state-of-the-art (SOTA) accuracy while producing high-quality, human-aligned explanations (Yang et al., 2023).

1. IMHI Dataset: Composition and Construction

The Interpretable Mental Health Instruction (IMHI) dataset aggregates 105,000 annotated social media samples spanning eight mental-health tasks and ten publicly available sources. The tasks include depression detection (DR), stress detection (Dreaddit), loneliness identification, multi-class mental health labeling (SWMH), cause and risk factor identification (CAMS, IRF), stressor categorization (SAD), and wellness dimension detection (MultiWD). Each task is derived from Reddit, Twitter, or similar domains, and labeled with fine-grained categories (e.g., {Yes, No}, {Suicide, Anxiety, Bipolar, Depression, None}).

Task	Label Set / Aspects	Train	Val	Test
DR	{Yes, No}	1003	430	405
Dreaddit (Stress)	{Yes, No}	2837	300	414
SWMH	{Suicide, Anxiety, Bipolar, ...}	34,822	8,705	10,882
MultiWD	{Spiritual, Physical, ...}	15,744	1,500	2,441

Seeded by expert-written instructions and 350 “gold” demonstrations formatted as $$[\text{label}]~\text{Reasoning:}~[\text{explanation}]$$, each instance is expanded via a three-part prompt template: task-specific instruction, few-shot gold exemplars sampled per class, and raw user text with its annotation. ChatGPT is then prompted to generate both a label and detailed reasoning. Contradictions between generated and gold labels (occurring in 5–30% of cases) trigger expert re-annotation or explanation revision, followed by automated and manual filtering for correctness and consistency.

2. Evaluation and Quality Control of Explanations

Explanations generated during dataset construction undergo automatic and human evaluation along three axes:

• Correctness: Defined as agreement between ChatGPT's predicted label and the gold annotation. Agreement varies by task, e.g., 68% (T-SID) to 96% (DR).
• Consistency: Assessed by training a MentalBERT classifier $f(r) \rightarrow \ell$ to predict labels from explanations, achieving weighted $F_1 > 93.5\%$ on test data, defined as

$$\mathrm{F}_1 = 2 \frac{\mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

• Quality (BART-score): Uses BART-score to compare generated explanations $r$ to gold references $r^*$:

$$\mathrm{BARTscore}(r, r^*) = \sum_{t=1}^{|r|}\log P_{\mathrm{BART}}(r^*_t | r, r^*_{<t})$$

Zero-shot, few-shot, and with-label prompts yield mean BART-scores of 1.8, 2.4, and 2.8, respectively (on a 0–4 scale).

Human evaluation involves ratings by three psychology PhD annotators on 200 explanations according to Consistency, Reliability, Professionality, and Overall Effectiveness (0–3 scale). Inter-annotator agreement (Krippendorff’s alpha) is ≈0.68, with mean scores: Consistency 2.7, Reliability 2.2, Professionality 2.1, Overall 2.4.

3. MentaLLaMA Model Architecture and Instruction-Tuning

MentaLLaMA models build on Meta’s LLaMA2 architectures, including LLaMA2-7B, LLaMA2-chat-7B, and LLaMA2-chat-13B. Instruction-tuning is conducted as a sequence-to-sequence task:

• Input: $q =$“Post: … Question: …”
• Output: $r = [\text{label}]~\|~\text{Reasoning: …}$

The objective is standard left-to-right log-likelihood maximization:

$$\max_{\phi}\;\sum_{(q,r)\in\mathcal D} \sum_{j=1}^{|r|} \log P_\phi(r_j | q, r_{<j})$$

Training is performed for 10 epochs with AdamW optimizer (weight decay 0.01), peak learning rate $f(r) \rightarrow \ell$0, effective batch size 256, and sequence length 2048 using FlashAttention.

Instructionally formatted training data encompass 72,095 examples, canonicalized as Post, Question, Answer, and Reasoning fields. Baseline models (T5, BART, vanilla LLaMA2) use a “completion” prompt style for comparison.

4. Model Performance: Classification and Explanation Metrics

Classification correctness is assessed by weighted $f(r) \rightarrow \ell$1 on each of the ten IMHI test sets. Discriminative baselines (MentalRoBERTa, MentalBERT) lead in overall accuracy, but instruction-tuned MentaLLaMA-chat-13B closes the gap to SOTA discriminative methods, achieving F1 within 5 points on 7 of 10 tasks.

Model	DR	Dreaddit	SWMH	MultiWD	T-SID
MentalRoBERTa	94.2	81.8	72.2	—	89.0
LLaMA2-13B (zero-shot)	54.1	36.3	40.5	53.7	25.3
ChatGPT (few-shot)	84.2	75.4	60.2	64.9	43.9
MentaLLaMA-chat-13B	85.7	75.8	71.7	75.1	75.3

Explanation quality, as measured by BART-score, finds MentaLLaMA-chat-13B scoring ≈2.9 versus 2.8 for ChatGPT (few-shot) on gold benchmarks.

When four tasks are held out from training (CAMS, Dreaddit, IRF, T-SID), MentaLLaMA-chat-13B outperforms ChatGPT (zero-shot) in weighted $f(r) \rightarrow \ell$2 by 5–45 points, and maintains explanation quality advantages over T5/BART models.

5. Qualitative Analysis: Strengths, Weaknesses, and Example Outputs

MentaLLaMA models generate coherent, stepwise explanations that align with psychological diagnostic criteria. Consistency between explanation and predicted label exceeds 94%. The models generalize robustly to new platforms (e.g., Twitter) and unseen tasks.

Notable weaknesses include lower “professionality” (human rating) scores relative to ChatGPT—lagging by 0.4 points—due to less frequent citation of high-criticality symptoms, as well as occasional hallucinations on rare wellness dimensions.

Example outputs reflect transparency and interpretability:

• DR task:

Input: “I’ve been feeling empty and can’t sleep for nights…” Output: “Yes. Reasoning: The user explicitly reports insomnia and emotional emptiness over multiple nights, consistent with the DSM-5 criterion of depressed mood and sleep disturbance.”

• CAMS (cause detection):

Input: “Ever since they laid me off, I can’t shake the feeling I’m worthless.” Output: “Jobs and career. Reasoning: The stressor arose directly from job loss and feelings of professional inadequacy, pointing to workplace/career cause.”

6. Significance and Availability

MentaLLaMA-IMHI establishes the first open-source instruction-tuned LLMs for interpretable multi-task mental health analysis on social media. By uniting high-quality, multi-domain instruction data and carefully tuned LLMs, the framework addresses both accuracy and interpretability. The IMHI dataset and MentaLLaMA models advance the field toward transparent, evidence-based automated mental health assessment, with evaluation demonstrating that generated explanations approach human quality and models reach near-state-of-the-art prediction accuracy (Yang et al., 2023). A plausible implication is broader applicability to related domains requiring interpretable, instruction-based classification over user-generated content.