---
title: Turkish Suicidal Ideation Corpus
url: https://www.emergentmind.com/topics/turkish-suicidal-ideation-corpus
type: topic
---

# Turkish Suicidal Ideation Corpus

The Turkish Suicidal Ideation Corpus is a Turkish-language dataset for suicidal ideation detection derived from social media posts on Ekşi Sözlük, introduced in "Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation" [2507.14693]. It was constructed to address two under-explored problems in mental health NLP: limited language coverage beyond English and unreliable annotation practices. The corpus is paired with a resource-efficient annotation framework involving three human annotators and two large language models, and it is evaluated in a bidirectional cross-lingual setting against three English suicidal ideation datasets using eight pre-trained sentiment and emotion classifiers. Within that study, the corpus functions both as a new Turkish benchmark and as an instrument for examining label reliability, model consistency, and the limits of zero-shot transfer learning in suicide-related NLP [2507.14693].

## 1. Data origin and collection boundaries

The corpus was built by scraping Ekşi Sözlük, described as one of the oldest text-only social platforms in Turkey. Data collection targeted six topic threads that were clearly related to suicidal ideation: "hiç yaşamamış olmayı ister miydin" ("Would you prefer never existing?"), "intihar" ("Suicide"), "intihar eden bir insanı anlamak" ("Understanding one who committed suicide"), "intiharı düşünmek" ("Thinking of suicide"), "kendini asmak" ("Hanging yourself"), and "sözlük yazarlarının olası intihar notları" ("Potential suicide notes"). Using a Python BeautifulSoup scraper, every user entry on these threads was harvested over the period May 4, 1999–March 6, 2025. No additional text normalization, including lowercasing or stopword removal, was applied prior to annotation, so annotators saw the material in its raw user-generated form [2507.14693].

| Topic | Posts / Authors | Length (mean ± SD) |
|---|---:|---:|
| Would you prefer never existing? | 210 / 210 | 30 ± 41 |
| Suicide | 4,525 / 3,498 | 88 ± 168 |
| Understanding one who committed suicide | 1,193 / 1,097 | 73 ± 134 |
| Thinking of suicide | 645 / 608 | 54 ± 79 |
| Hanging yourself | 154 / 153 | 56 ± 80 |
| Potential suicide notes | 1,147 / 1,090 | 27 ± 101 |

The source selection has methodological significance because the corpus is not assembled from generic social media traffic but from threads explicitly associated with suicide-related discourse. This suggests that the collection strategy prioritized topical density over broad platform sampling. A plausible implication is that the dataset is particularly suited to studying fine-grained distinctions within suicide-related discussion, rather than first-pass retrieval from heterogeneous social media streams.

## 2. Label schema and annotation workflow

The annotation design used a four-way label schema guided by two graduate-level annotators and one domain expert. The labels were defined as follows: **Positive**, explicit first-person suicidal ideation; **Mixed**, ideation plus discouragement of self-harm; **Negative**, opposition to suicide or advocacy of help; and **Other**, neutral or off-topic [2507.14693].

The workflow was explicitly multi-stage. In **Stage 1**, both graduate annotators independently labeled all 7,874 posts. In **Stage 2**, labels were binarized into "ideation present" for Positive or Mixed and "ideation absent" for Negative or Other. In **Stage 3**, if the two humans agreed on the binary presence or absence of ideation but disagreed on the four-way label, the post was submitted to ChatGPT-4o and Gemini 2.5. Where an LLM’s label exactly matched one human’s, that label was adopted. Only 380 posts remained undecided under these rules; for those, one of the human annotations was randomly chosen, which the study describes as introducing controlled noise while preserving the binary ideation decision. In **Stage 4**, the 1,335 posts on which the two humans disagreed about the presence versus absence of ideation were deferred to the third annotator, a senior domain expert. A majority vote among the expert and one of the original annotators resolved all but 231 posts, and those remaining 231 were labeled by the expert alone [2507.14693].

This architecture is notable because it differentiates between disagreements that are treated as non-sensitive and disagreements that are treated as sensitive. LLMs are used only when the binary ideation decision is already stable, whereas expert adjudication is reserved for presence-versus-absence conflicts. The framework therefore operationalizes a risk-sensitive distinction between label refinement and ideation detection proper.

## 3. Agreement structure and finalized corpus composition

Before adjudication, the paper recorded **54 % raw agreement** between the two primary annotators, denoted as $P_o = 0.54$. Although formal $\kappa$ statistics were not reported, the study presents the standard Cohen’s $\kappa$ framework as:
$$
P_e = \sum_\ell (p_{1\ell}\cdot p_{2\ell}), \qquad
\kappa = \frac{P_o - P_e}{1 - P_e}.
$$
For the four-class setting, $P_e$ is the sum of the products of each label’s marginal probability under each annotator. After the multi-stage adjudication process, all posts received a final consensus label [2507.14693].

| Label | Authors / Posts | Length (mean ± SD) |
|---|---:|---:|
| Positive | 2,529 / 2,985 | 55 ± 121 |
| Mixed | 2,118 / 2,429 | 85 ± 156 |
| Negative | 1,213 / 1,301 | 80 ± 129 |
| Other | 1,073 / 1,154 | 52 ± 143 |

The corpus contains **7,874 posts** and **6,078 distinct authors**. Only **311 authors** contributed more than one post; among those, **301 were consistent in their final labels**, which the paper presents as suggesting intra-author stability. The study does not prescribe fixed train, validation, and test splits, instead leaving that choice to downstream users and noting that researchers will likely stratify by label, for example using an **80/10/10** or **70/15/15** split [2507.14693].

A common misconception in suicidal ideation detection is that label quality can be inferred from task framing alone. The Turkish corpus instead foregrounds adjudication mechanics as part of the dataset definition. In that sense, the corpus is not only a collection of texts and labels but also an explicit account of how disagreements were handled and which disagreements were escalated.

## 4. Cross-lingual evaluation protocol

The accompanying evaluation applied **eight off-the-shelf transformers** directly to the Turkish corpus and to three benchmark English Reddit datasets: **C-SSRS**, **SDD**, and **SWMH**. No further fine-tuning was performed on the Turkish corpus; all Turkish evaluations were **zero-shot**. The four Turkish models were two multilingual sentiment classifiers, **MULTI-1** and **MULTI-2**, a **DistilBERT** trained on translated Twitter-based emotion data, and **BERTurk-TREMO**, fine-tuned on a native Turkish emotion corpus. The four English models were two suicide-detection classifiers, **SENTINET** and **RoBERTa**, and two multi-label emotion detectors, **EmoRoBERTa** and **BERT-Emo** [2507.14693].

The Turkish models covered sentiment or emotion label spaces rather than a native suicide-detection objective. MULTI-1 used a five-class multilingual sentiment scheme trained on synthetic multilingual sentiment instances, and MULTI-2 used the same five-class multilingual scheme trained on parallel sentences. DistilBERT was trained on translated Twitter emotion data, while BERTurk-TREMO was fine-tuned on TREMO. The English suicide-detection models were trained on data such as **C-SSRS + partial SDD + Twitter suicidal** for SENTINET and a **small Reddit suicide set** for RoBERTa. EmoRoBERTa used **GoEmotions (Reddit)**, and BERT-Emo used **GoEmotions + Twitter** [2507.14693].

For reliability and consistency analysis, the study used confusion matrices between model predictions and human labels, and **Matthews Correlation Coefficient (MCC)** for multi-class agreement. Binary evaluation employed **Precision**, **Recall**, **F$_1$-score**, **macro-F$_1$**, and **AUC**. This setup is methodologically important because it examines both annotation reliability and model behavior without adapting the models to the Turkish corpus itself. A plausible implication is that the evaluation is designed less as a leaderboard exercise than as an audit of transfer validity.

## 5. Zero-shot results and what they demonstrate

The study reports binary suicide-detection results for SENTINET and RoBERTa on the three English datasets as follows [2507.14693]:

| Model | C-SSRS (F$_1$ / AUC) | SDD (F$_1$ / AUC) | SWMH (F$_1$ / AUC) |
|---|---:|---:|---:|
| SENTINET | 54.7 % / 57.3 % | 96.6 % / 99.5 % | 45.3 % / 82.4 % |
| RoBERTa | 16.3 % / 55.8 % | 96.9 % / 99.5 % | 59.5 % / 81.0 % |

The contrast between datasets is central to the paper’s argument. On **C-SSRS**, identified as a **human-annotated gold standard**, both models hover around chance with **AUC ≈ 0.56**, which the paper interprets as failure to detect true ideation signals. On **SDD** and **SWMH**, which are described as **auto-labeled by subreddit**, the same models achieve **F$_1$ > 96 %**. The study states that this indicates they have in effect learned **subreddit-specific cues rather than suicidal ideation per se** [2507.14693].

The emotion-detection models behaved similarly problematically. **BERT-Emo** and **EmoRoBERTa** returned neutral or superficial emotion tags across all labels, with **low inter-model MCC (≈ 0.2 – 0.3)** on both Turkish and English data. This finding is relevant beyond model selection: it challenges the assumption that generic emotion classifiers can serve as adequate proxies for suicidal ideation detection in cross-lingual or zero-shot settings.

One controversy directly addressed by these results concerns the status of auto-labeled corpora. High performance on such datasets can be mistaken for robust ideation detection. The reported pattern instead indicates that apparent success may derive from metadata or community markers, not from clinically relevant linguistic signal.

## 6. Methodological implications and recommended practice

The paper distills several best practices for language-inclusive and reliable suicidal ideation corpus construction. It recommends prioritizing **gold-standard, human-validated labels**, noting that even small **expert adjudication** subsets can dramatically raise data reliability. It recommends using **LLMs judiciously as tie-breakers in non-critical cases**, but never in place of domain experts on the presence versus absence of ideation. It further recommends always reporting inter-annotator statistics such as **Cohen’s $\kappa$**, **Fleiss’ $\kappa$**, or **raw agreement**, so that downstream users can assess label trustworthiness [2507.14693].

The study also recommends avoiding uncritical reuse of auto-labeled datasets or off-the-shelf models without auditing their training data, because models will often learn to detect metadata or community markers, such as subreddit names, rather than underlying clinical signals. It additionally recommends transparent documentation of **topic selection**, **scraping periods**, **annotation guidelines**, and **decision rules**, even if raw data cannot be publicly released [2507.14693].

Taken together, the Turkish Suicidal Ideation Corpus is significant less as an isolated dataset than as a tightly specified annotation and evaluation regime for suicide-related NLP in a non-English setting. The work presents the corpus and its cross-lingual probes as evidence that progress in suicidal ideation detection depends not only on model architecture but also on trustworthy annotation, transparent dataset construction, and explicit testing against distributional shortcuts. This suggests that the corpus is best understood as both a Turkish benchmark and a methodological intervention in how mental health NLP resources are validated.

Source: https://www.emergentmind.com/topics/turkish-suicidal-ideation-corpus