Turkish Suicidal Ideation Corpus

• Turkish Suicidal Ideation Corpus is a Turkish-language dataset built from Ekşi Sözlük posts that focus exclusively on suicide-related discussions.
• It incorporates a multi-stage annotation process combining graduate annotators, domain experts, and language models to resolve label disagreements across four categories.
• Zero-shot cross-lingual evaluations reveal that off-the-shelf models may capture community-specific cues rather than genuine ideation signals, emphasizing the need for transparent annotation practices.

The Turkish Suicidal Ideation Corpus is a Turkish-language dataset for suicidal ideation detection derived from social media posts on Ekşi Sözlük, introduced in "Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation" (Dzafic et al., 19 Jul 2025). It was constructed to address two under-explored problems in mental health NLP: limited language coverage beyond English and unreliable annotation practices. The corpus is paired with a resource-efficient annotation framework involving three human annotators and two LLMs, and it is evaluated in a bidirectional cross-lingual setting against three English suicidal ideation datasets using eight pre-trained sentiment and emotion classifiers. Within that study, the corpus functions both as a new Turkish benchmark and as an instrument for examining label reliability, model consistency, and the limits of zero-shot transfer learning in suicide-related NLP (Dzafic et al., 19 Jul 2025).

1. Data origin and collection boundaries

The corpus was built by scraping Ekşi Sözlük, described as one of the oldest text-only social platforms in Turkey. Data collection targeted six topic threads that were clearly related to suicidal ideation: "hiç yaşamamış olmayı ister miydin" ("Would you prefer never existing?"), "intihar" ("Suicide"), "intihar eden bir insanı anlamak" ("Understanding one who committed suicide"), "intiharı düşünmek" ("Thinking of suicide"), "kendini asmak" ("Hanging yourself"), and "sözlük yazarlarının olası intihar notları" ("Potential suicide notes"). Using a Python BeautifulSoup scraper, every user entry on these threads was harvested over the period May 4, 1999–March 6, 2025. No additional text normalization, including lowercasing or stopword removal, was applied prior to annotation, so annotators saw the material in its raw user-generated form (Dzafic et al., 19 Jul 2025).

Topic	Posts / Authors	Length (mean ± SD)
Would you prefer never existing?	210 / 210	30 ± 41
Suicide	4,525 / 3,498	88 ± 168
Understanding one who committed suicide	1,193 / 1,097	73 ± 134
Thinking of suicide	645 / 608	54 ± 79
Hanging yourself	154 / 153	56 ± 80
Potential suicide notes	1,147 / 1,090	27 ± 101

The source selection has methodological significance because the corpus is not assembled from generic social media traffic but from threads explicitly associated with suicide-related discourse. This suggests that the collection strategy prioritized topical density over broad platform sampling. A plausible implication is that the dataset is particularly suited to studying fine-grained distinctions within suicide-related discussion, rather than first-pass retrieval from heterogeneous social media streams.

2. Label schema and annotation workflow

The annotation design used a four-way label schema guided by two graduate-level annotators and one domain expert. The labels were defined as follows: Positive, explicit first-person suicidal ideation; Mixed, ideation plus discouragement of self-harm; Negative, opposition to suicide or advocacy of help; and Other, neutral or off-topic (Dzafic et al., 19 Jul 2025).

The workflow was explicitly multi-stage. In Stage 1, both graduate annotators independently labeled all 7,874 posts. In Stage 2, labels were binarized into "ideation present" for Positive or Mixed and "ideation absent" for Negative or Other. In Stage 3, if the two humans agreed on the binary presence or absence of ideation but disagreed on the four-way label, the post was submitted to ChatGPT-4o and Gemini 2.5. Where an LLM’s label exactly matched one human’s, that label was adopted. Only 380 posts remained undecided under these rules; for those, one of the human annotations was randomly chosen, which the study describes as introducing controlled noise while preserving the binary ideation decision. In Stage 4, the 1,335 posts on which the two humans disagreed about the presence versus absence of ideation were deferred to the third annotator, a senior domain expert. A majority vote among the expert and one of the original annotators resolved all but 231 posts, and those remaining 231 were labeled by the expert alone (Dzafic et al., 19 Jul 2025).

This architecture is notable because it differentiates between disagreements that are treated as non-sensitive and disagreements that are treated as sensitive. LLMs are used only when the binary ideation decision is already stable, whereas expert adjudication is reserved for presence-versus-absence conflicts. The framework therefore operationalizes a risk-sensitive distinction between label refinement and ideation detection proper.

3. Agreement structure and finalized corpus composition

Before adjudication, the paper recorded 54 % raw agreement between the two primary annotators, denoted as $P_o = 0.54$. Although formal $\kappa$ statistics were not reported, the study presents the standard Cohen’s $\kappa$ framework as:

$$P_e = \sum_\ell (p_{1\ell}\cdot p_{2\ell}), \qquad \kappa = \frac{P_o - P_e}{1 - P_e}.$$

For the four-class setting, $P_e$ is the sum of the products of each label’s marginal probability under each annotator. After the multi-stage adjudication process, all posts received a final consensus label (Dzafic et al., 19 Jul 2025).

Label	Authors / Posts	Length (mean ± SD)
Positive	2,529 / 2,985	55 ± 121
Mixed	2,118 / 2,429	85 ± 156
Negative	1,213 / 1,301	80 ± 129
Other	1,073 / 1,154	52 ± 143

The corpus contains 7,874 posts and 6,078 distinct authors. Only 311 authors contributed more than one post; among those, 301 were consistent in their final labels, which the paper presents as suggesting intra-author stability. The study does not prescribe fixed train, validation, and test splits, instead leaving that choice to downstream users and noting that researchers will likely stratify by label, for example using an 80/10/10 or 70/15/15 split (Dzafic et al., 19 Jul 2025).

A common misconception in suicidal ideation detection is that label quality can be inferred from task framing alone. The Turkish corpus instead foregrounds adjudication mechanics as part of the dataset definition. In that sense, the corpus is not only a collection of texts and labels but also an explicit account of how disagreements were handled and which disagreements were escalated.

4. Cross-lingual evaluation protocol

The accompanying evaluation applied eight off-the-shelf transformers directly to the Turkish corpus and to three benchmark English Reddit datasets: C-SSRS, SDD, and SWMH. No further fine-tuning was performed on the Turkish corpus; all Turkish evaluations were zero-shot. The four Turkish models were two multilingual sentiment classifiers, MULTI-1 and MULTI-2, a DistilBERT trained on translated Twitter-based emotion data, and BERTurk-TREMO, fine-tuned on a native Turkish emotion corpus. The four English models were two suicide-detection classifiers, SENTINET and RoBERTa, and two multi-label emotion detectors, EmoRoBERTa and BERT-Emo (Dzafic et al., 19 Jul 2025).

The Turkish models covered sentiment or emotion label spaces rather than a native suicide-detection objective. MULTI-1 used a five-class multilingual sentiment scheme trained on synthetic multilingual sentiment instances, and MULTI-2 used the same five-class multilingual scheme trained on parallel sentences. DistilBERT was trained on translated Twitter emotion data, while BERTurk-TREMO was fine-tuned on TREMO. The English suicide-detection models were trained on data such as C-SSRS + partial SDD + Twitter suicidal for SENTINET and a small Reddit suicide set for RoBERTa. EmoRoBERTa used GoEmotions (Reddit), and BERT-Emo used GoEmotions + Twitter (Dzafic et al., 19 Jul 2025).

For reliability and consistency analysis, the study used confusion matrices between model predictions and human labels, and Matthews Correlation Coefficient (MCC) for multi-class agreement. Binary evaluation employed Precision, Recall, F$_1$-score, macro-F$_1$, and AUC. This setup is methodologically important because it examines both annotation reliability and model behavior without adapting the models to the Turkish corpus itself. A plausible implication is that the evaluation is designed less as a leaderboard exercise than as an audit of transfer validity.

5. Zero-shot results and what they demonstrate

The study reports binary suicide-detection results for SENTINET and RoBERTa on the three English datasets as follows (Dzafic et al., 19 Jul 2025):

Model	C-SSRS (F$_1$ / AUC)	SDD (F$_1$ / AUC)	SWMH (F$_1$ / AUC)
SENTINET	54.7 % / 57.3 %	96.6 % / 99.5 %	45.3 % / 82.4 %
RoBERTa	16.3 % / 55.8 %	96.9 % / 99.5 %	59.5 % / 81.0 %

The contrast between datasets is central to the paper’s argument. On C-SSRS, identified as a human-annotated gold standard, both models hover around chance with AUC ≈ 0.56, which the paper interprets as failure to detect true ideation signals. On SDD and SWMH, which are described as auto-labeled by subreddit, the same models achieve F$\kappa$0 > 96 %. The study states that this indicates they have in effect learned subreddit-specific cues rather than suicidal ideation per se (Dzafic et al., 19 Jul 2025).

The emotion-detection models behaved similarly problematically. BERT-Emo and EmoRoBERTa returned neutral or superficial emotion tags across all labels, with low inter-model MCC (≈ 0.2 – 0.3) on both Turkish and English data. This finding is relevant beyond model selection: it challenges the assumption that generic emotion classifiers can serve as adequate proxies for suicidal ideation detection in cross-lingual or zero-shot settings.

One controversy directly addressed by these results concerns the status of auto-labeled corpora. High performance on such datasets can be mistaken for robust ideation detection. The reported pattern instead indicates that apparent success may derive from metadata or community markers, not from clinically relevant linguistic signal.

6. Methodological implications and recommended practice

The paper distills several best practices for language-inclusive and reliable suicidal ideation corpus construction. It recommends prioritizing gold-standard, human-validated labels, noting that even small expert adjudication subsets can dramatically raise data reliability. It recommends using LLMs judiciously as tie-breakers in non-critical cases, but never in place of domain experts on the presence versus absence of ideation. It further recommends always reporting inter-annotator statistics such as Cohen’s $\kappa$1, Fleiss’ $\kappa$2, or raw agreement, so that downstream users can assess label trustworthiness (Dzafic et al., 19 Jul 2025).

The study also recommends avoiding uncritical reuse of auto-labeled datasets or off-the-shelf models without auditing their training data, because models will often learn to detect metadata or community markers, such as subreddit names, rather than underlying clinical signals. It additionally recommends transparent documentation of topic selection, scraping periods, annotation guidelines, and decision rules, even if raw data cannot be publicly released (Dzafic et al., 19 Jul 2025).

Taken together, the Turkish Suicidal Ideation Corpus is significant less as an isolated dataset than as a tightly specified annotation and evaluation regime for suicide-related NLP in a non-English setting. The work presents the corpus and its cross-lingual probes as evidence that progress in suicidal ideation detection depends not only on model architecture but also on trustworthy annotation, transparent dataset construction, and explicit testing against distributional shortcuts. This suggests that the corpus is best understood as both a Turkish benchmark and a methodological intervention in how mental health NLP resources are validated.