---
title: Interpretable Mental Health Instruction Dataset
url: https://www.emergentmind.com/topics/interpretable-mental-health-instruction-imhi-dataset
type: topic
---

# Interpretable Mental Health Instruction Dataset

The Interpretable Mental Health Instruction (IMHI) dataset collectively refers to a family of rigorously curated resources that advance mental health analysis by pairing textual samples with structured instructions, expert-annotated labels, and natural-language decision rationales. IMHI datasets are foundational to the development, instruction tuning, and benchmarking of large language models (LLMs) for explainable, robust, and clinically relevant mental health inference on both social media and real-world clinical data. These datasets are designed to address the limitations of traditional opaque deep learning models by enabling not only high-accuracy multi-task classification but also providing interpretability essential for clinical trust, model validation, and safe deployment [2309.13567][2410.10323][2304.03347][2502.16051][2305.18727][2503.03607][2506.18116].

## 1. Dataset Scope and Motivations

IMHI is engineered for instruction-based learning and explainable benchmarking in mental health NLP. Its core objectives are:

- Enabling multi-task analysis of mental health–related text, spanning binary detection (e.g., depression, suicide risk), multi-class/multi-label syndrome/disorder categorization, cause/factor and cognitive pathway extraction, and multi-turn counseling strategies.
- Pairing every inference with interpretable, human-style natural language explanations, either as model-generated rationales or annotator-extracted rationales, allowing downstream validation, error analysis, and clinical application.
- Providing a transparent testbed for evaluating the reasoning, calibration, and bias of LLMs in diverse and intersectional mental health contexts, with explicit support for multilingual and cross-cultural studies ([2309.13567], [2304.03347], [2503.03607], [2410.10323]).

Interpretability is emphasized to bridge the trust gap in deep learning, to facilitate clinical and research auditing of AI decisions, to support error analysis, and to advance bias mitigation and fairness assessment ([2506.18116]).

## 2. Composition, Task Spectrum, and Formalization

IMHI resources decompose into several major multilingual and task-diverse datasets, the largest totaling over 100,000 instruction–response pairs [2309.13567]. Key instantiations include:

### Data Sources and Tasks

| Dataset/Source                   | Tasks/Labels                                               | Language         | # Samples    |
|----------------------------------|------------------------------------------------------------|------------------|--------------|
| MentaLLaMA-IMHI [2309.13567]     | 8 tasks (binary/multi-class detection, causes, IRF, etc.)  | EN, CH, others   | 105,794      |
| C-IMHI [2410.10323]              | Suicide risk (binary), cognitive distortion (12-way ML), cognitive pathway (4+19 hierarchical ML) | Chinese, EN (CP only) | 9,251        |
| IMHI small [2304.03347]          | 5 task families (binary, multi-class, causal, ERC, CEE)    | English          | 163 expl.    |
| IRF/Reddit [2305.18727]          | Interpersonal risk factors (TBE/PBU), extractive rationale | English          | 3,522        |
| Psy-Insight [2503.03607]         | Counseling dialogues (emotion, strategy, therapy rationale)| Chinese, English | 12,000+      |

#### Formal Annotation Format

A generic IMHI dataset $D$ is a collection of $(I_i, O_i)$ pairs:

$$
D = \{ (I_i, O_i) \}_{i=1}^N
$$

Where each instruction consists of a detailed task description, target text (e.g., social media post or dialogue), and explicit query; each output includes structured labels (task-dependent) and a natural-language explanation. Functional fields (as per C-IMHI):

- $d_i$: task description
- $t_i$: text/post
- $q_i$: instruction prompt
- $c_i \in Y$: label(s) for task $Y$ (class, multi-label, etc.)
- $e_i$: explanation

Representative JSON schema:

```json
{
  "id": ...,
  "task": "...",
  "instruction": "...",
  "text": "...",
  "query": "...",
  "label": ...,
  "explanation": "..."
}
```
([2410.10323], [2309.13567]).

## 3. Annotation Protocols and Quality Assurance

IMHI datasets are engineered for high annotator agreement and reliability, following distinctive protocols for different domains:

- **Expert protocols:** Annotation by trained clinicians, psychologists, or advanced domain PhDs; peer or group review to resolve conflicts ([2502.16051], [2305.18727]).
- **Automated filtering:** For large-scale, LLM-generated explanations, post hoc correctness (label match), and consistency (classifier-based) checks ([2309.13567], [2410.10323]).
- **Human validation:** Random sampling with multi-axis (Consistency, Reliability, Professionality) human scoring on a 0–3 scale. Consistency ≈2.73, reliability ≈2.30, professionality ≈2.03 for C-IMHI; similar metrics for MentaLLaMA and others ([2410.10323], [2309.13567]).
- **Inter-annotator Agreement:** Cohen’s κ, Fleiss’ κ, or Krippendorff’s α applied where appropriate. For IRF: κ(TBE)=0.7883, κ(PBU)=0.8239 ([2305.18727]). For IMHI-small: κ≥0.41 for most human explanation ratings ([2304.03347]).
- **Uncertainty estimation:** For ambiguous questions, preference probabilities via hierarchical Bradley–Terry models, soft label distribution, and confidence intervals ([2502.16051]).

Explanations may be LLM-generated (validated against expert-written few-shot templates and spot-checked), or extractive/highlighted rationales for clinical interpretability ([2410.10323], [2304.03347], [2305.18727]).

## 4. Instruction Structure and Representative Examples

IMHI samples couple detailed instructions, test samples, gold labels, and rationales. Examples:

- **Binary:**
  - “请判断下面的微博是否表示高自杀风险，并解释理由。”
  - Label: “high_risk”
  - Explanation: “‘结束这一切’直接表明自杀意图，且语气坚定，故属于高自杀风险。”
- **Multi-label:**
  - “请识别这条微博中包含的所有认知扭曲类型，并解释原因。”
  - Label: [“过度概括”, “灾难化”]
  - Explanation: “‘每次…都’属于过度概括；‘一辈子都没希望’是灾难化思维。”
- **Cognitive Pathway:**
  - Extract four parent and nineteen child pathway elements hierarchically, with rationale for each mapping.
- **Counseling Dialogues:**
  - Each therapist turn: {(utterance, strategy_label, reasoning)}, each client turn: {(utterance, emotion_label, observation)}, with session-level guidance summaries ([2503.03607]).

Such instructive and explanation-rich formatting enables modeling both human-like reasoning and fine-grained multi-task capabilities.

## 5. Evaluation Protocols and Benchmarks

Standard IMHI evaluation is multi-level:

- **Classification Metrics:** Micro-averaged Precision, Recall, and $F_1$; for multi-class, weighted $F_1$ as
  $$
  \text{Precision}_c = \frac{TP_c}{TP_c + FP_c},
  \quad
  \text{Recall}_c = \frac{TP_c}{TP_c + FN_c},
  \quad
  F1_c = 2 \frac{\text{Prec}_c \cdot \text{Rec}_c}{\text{Prec}_c + \text{Rec}_c}
  $$
  ([2410.10323], [2309.13567]).
- **Explanation Quality:** BART-Score, BLEU, ROUGE-L, BERTScore, and human evaluations (0–3 scales for fluency, completeness, reliability) ([2304.03347], [2309.13567]).
- **Fairness/Bias Auditing:** For intersectional bias, $B_{\text{sent}}$, $B_{\text{dem}}$, $B_{\text{cond}}$ computed by groupwise comparison, and bias amplification factors $F_k$ through multi-hop QA ([2506.18116]).
- **Clinical/ambiguous tasks:** Use of soft label calibration scores (Brier, cross-entropy), ambiguity metrics (Krippendorff’s α; BT probabilities) ([2502.16051]).

IMHI-tuned LLMs (e.g., MentalLLaMA, MentalGLM) are benchmarked against BERT-family PLMs, ChatGPT, GPT-4, and open-source LLMs, showing competitive correctness (F1), high explanation quality, and, in some settings, superior performance to zero/few-shot LLMs ([2309.13567], [2410.10323]).

## 6. Applications and Impact

IMHI frameworks underpin a variety of research and applied efforts:

- **Instruction/LLM Tuning:** MentaLLaMA, MentalGLM, and related open-source LLMs for Chinese and English are instruction-tuned on IMHI, improving explainability and robustness ([2309.13567], [2410.10323]).
- **Clinical and Social Media Use:** Addressing a wide landscape of conditions and explanatory needs—suicide risk, depression, cognitive distortions, counseling dynamics—and supporting both academic and clinical validation ([2305.18727], [2503.03607]).
- **AI Bias Auditing:** Systematic, intersectional audit of LLM outputs for amplification/silencing of marginalized perspectives ([2506.18116]).
- **Evaluation Tooling:** IMHI’s format allows prompt engineering experiments, automatic metric development, and alignment evaluation.
- **Human-in-the-loop Use:** Interpretability and explanation generation facilitate safe integration into triage, early warning, documentation, and research settings ([2410.10323], [2502.16051]).

## 7. Limitations and Prospective Extensions

Known limitations:

- Scarcity of published κ or α metrics for all tasks/languages; several splits rely partially on LLM-simulated expert explanation, which may underperform in "professionality" compared to consistency ([2410.10323], [2309.13567]).
- Current datasets are skewed toward English and Chinese, with limited cross-lingual coverage ([2503.03607]).
- Clinically validated reasoning quality and calibration remain active research areas, particularly for domain transfer and real-world deployment ([2410.10323], [2502.16051]).

Future work includes RLHF-enhanced instruction finetuning, fuller localization and multi-language expansion, longitudinal and platform-diverse data captures, integration of structured clinical instruments, and deployment in real-time human–AI collaboration scenarios ([2410.10323], [2304.03347], [2503.03607], [2506.18116]).

---

The IMHI datasets and their derivatives constitute the principal shared research infrastructure for explainable, instruction-tuned NLP in mental health across clinical and social domains, providing essential resources for advancing model development, reliability, interpretability, and equity in mental health AI [2309.13567][2410.10323][2304.03347][2502.16051][2305.18727][2503.03607][2506.18116].

Source: https://www.emergentmind.com/topics/interpretable-mental-health-instruction-imhi-dataset