A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

Abstract: The growing availability of online support groups has opened up new windows to study mental health through NLP. However, it is hindered by a lack of high-quality, well-validated datasets. Existing studies have a tendency to build task-specific corpora without collecting them into widely available resources, and this makes reproducibility as well as cross-task comparison challenging. In this paper, we present a uniform benchmark set of four Reddit-based datasets for disjoint but complementary tasks: (i) detection of suicidal ideation, (ii) binary general mental disorder detection, (iii) bipolar disorder detection, and (iv) multi-class mental disorder classification. All datasets were established upon diligent linguistic inspection, well-defined annotation guidelines, and human-judgmental verification. Inter-annotator agreement metrics always exceeded the baseline agreement score of 0.8, ensuring the labels' trustworthiness. Previous work's evidence of performance on both transformer and contextualized recurrent models demonstrates that these models receive excellent performances on tasks (F1 ~ 93-99%), further validating the usefulness of the datasets. By combining these resources, we establish a unifying foundation for reproducible mental health NLP studies with the ability to carry out cross-task benchmarking, multi-task learning, and fair model comparison. The presented benchmark suite provides the research community with an easy-to-access and varied resource for advancing computational approaches toward mental health research.

• The paper presents a consolidated benchmark suite of four Reddit-derived datasets addressing various mental health detection tasks with high inter-annotator agreement (κ > 0.8).
• The study employs rigorous linguistic analysis and annotation protocols, achieving F1 scores up to 99.54% across tasks when using state-of-the-art NLP models.
• The consolidated datasets facilitate cross-task benchmarking and multi-task model development, enhancing reproducibility in mental health NLP research.

Benchmarking Mental Health Detection: Consolidation of Reddit-Derived Datasets

Motivation and Context

Mental health disorders — including depression, anxiety, bipolar disorder, and PTSD — are prevalent globally, with significant impact on affected individuals. Automated detection from language has emerged as a promising application of NLP, yet progress is impeded by the lack of standardized datasets. Prior efforts often constructed task-specific corpora with limited accessibility, making reproducibility and fair comparison across studies difficult. This paper introduces a consolidated benchmark suite of four Reddit-derived datasets addressing disjoint but complementary classification tasks: suicidal ideation detection, binary general mental disorder detection, bipolar disorder detection, and multi-class mental disorder classification. Each dataset was constructed via rigorous linguistic analysis, clear annotation guidelines, and human verification, yielding inter-annotator agreement metrics (Cohen's $\kappa > 0.8$) well above accepted thresholds.

Dataset Construction and Annotation Quality

Suicidal Ideation Detection

Posts were collected from r/SuicideWatch as the positive class and a set of other subreddits as the negative class, totaling 37,821 posts. The TextRank algorithm flagged salient phrases (e.g., "no hope", "suicidal thoughts", "just a burden"), which align with established clinical markers. Suicidal posts exhibited shorter lengths, fewer hashtags/URLs, and higher frequency of emotionally intense language.

Human annotation on a random 2% sample yielded $\kappa = 0.897$ and $\kappa = 0.854$, signifying nearly perfect agreement.

Bipolar Disorder Detection

Posts from r/bipolar formed the positive class, with negatives drawn from other illness and control subreddits. Rigorous self-identification and filtering (per Cohan et al.) prevented contamination. Resulting corpus contained 23,000 bipolar and 26,000 non-bipolar posts.

Linguistic analysis demonstrated that bipolar posts were more expressive and internally focused, with higher verb and pronoun usage; non-bipolar posts showed greater external referencing via hashtags/URLs.

Notably, the distribution of sentiment variance was significantly wider in bipolar posts, consistent with theoretical expectations of mood oscillations.

Figure 1: Intra-post sentiment variance distribution for non-bipolar and bipolar posts, highlighting pronounced emotional fluctuation in bipolar content.

Judgmental validation achieved Cohen's $\kappa = 0.83$ and $0.875$, reinforcing annotation reliability and label distinctiveness.

General Mental Disorder Detection (Binary)

Data from mental health and control subreddits were meticulously filtered, resulting in 144,000 posts split into primary (for model benchmarking) and external (for generalization) sets. Linguistically, mental health posts were longer and used more verbs, pronouns, and adjectives; control posts tended toward factual, brief discourse.

Human annotation ($\kappa > 0.94$) confirmed consistent, reproducible label assignment.

Multi-Class Mental Disorder Classification

Reddit posts spanning six disorder-specific and several control subreddits yielded a balanced sample (105,000 posts across seven classes). Stylistic and POS metrics revealed longer posts and richer verb/noun usage in CPTSD and depression classes; control posts had markedly more external linking.

Pairwise Jensen–Shannon divergence mapped lexical distances between all classes, evidencing the separation between disorder-specific and control posts, as well as nuanced proximity within disorder classes (e.g., depression–bipolar).

Figure 2: Pairwise divergence map across classes, quantifying lexical distance using Jensen–Shannon divergence.

Manual annotation of a 1% sample ($\kappa = 0.92$, $0.91$) underscored guideline clarity and label consistency, supporting downstream classifier training.

Performance Benchmarking

Applying state-of-the-art models (RoBERTa, BERT+LSTM, DistilBERT) to each task yielded highly consistent, robust F1 scores:

• Suicidal ideation: F1 up to 93.14%
• Bipolar disorder: F1 ≈ 98%
• General mental disorder (binary): F1 up to 99.54% (hold-out), 95.96% (external)
• Multi-class mental disorder: F1 range 88.03%–99.20% across classes

Transformer-based architectures reliably delivered optimal results across tasks, and contextualized LSTM models were competitive. These numerical outcomes underscore the discriminative quality of the benchmark datasets and the consistency of their signal, supporting reproducible and fair model comparison.

Theoretical Implications and Practical Impact

This benchmark suite provides a foundation for standardized evaluation in mental health NLP, supporting both multi-task learning and methodological comparability. The consolidation of task-specific datasets enables:

• Cross-task benchmarking: Researchers may systematically evaluate algorithms on distinct clinical contexts.
• Multi-task model development: Models can exploit shared linguistic or psychological markers across disorders.
• Data-driven mental health science: The resource facilitates hypothesis testing beyond single-disorder focus, enabling evidence synthesis and generalizability.

On a practical level, the datasets advance the field by supplying rigorously annotated, varied data sources, overcoming the reproducibility bottleneck noted in previous literature. The high annotation reliability and empirical performance render these datasets suitable as gold standards for academic and applied research.

Future Research Directions

Potential avenues for further development include:

• Unified multi-task architectures leveraging the shared benchmark to train models capable of simultaneous disorder detection and classification.
• Robustness evaluation: Standardized benchmarks enable systematic exploration of domain adaptation, label noise, and fairness concerns.
• Interpretability studies: The availability of diverse linguistic markers across tasks encourages new work in explainable mental health NLP.

Conclusion

This benchmark suite consolidates four linguistically and empirically validated Reddit-derived datasets into a coherent resource for mental health detection via NLP. Uniform annotation protocols, careful linguistic analysis, and robust performance metrics substantiate the datasets' quality. By standardizing evaluation across suicidal ideation, bipolar disorder, general mental disorder, and multi-class classification, this suite mitigates previous methodological fragmentation, thereby promoting replicability and multi-task modeling in mental health NLP.