---
title: 'ThaiFACTUAL: Debiasing Thai Political Stance'
url: https://www.emergentmind.com/topics/thaifactual
type: topic
---

# ThaiFACTUAL: Debiasing Thai Political Stance

ThaiFACTUAL is a lightweight, model-agnostic calibration framework for Thai political stance detection with large language models (LLMs), introduced to mitigate systematic bias in a setting characterized as low-resource, culturally indirect, emotionally charged, and politically polarized [2509.21946]. It is designed for a three-way stance detection task—**Support**, **Against**, and **Neutral**—over Thai-language political posts targeting specific political figures, and it addresses two central failure modes identified in standard LLM predictions: **sentiment–stance entanglement** and **entity preference bias** [2509.21946]. The framework operates without fine-tuning the underlying LLM, instead applying post-hoc calibration through counterfactual data augmentation and rationale-based supervision [2509.21946].

## 1. Problem setting and motivation

ThaiFACTUAL is situated in the task of Thai political stance detection: given a Thai-language post and a political target, predict whether the author is **Support**, **Against**, or **Neutral** toward that target [2509.21946]. The paper emphasizes utility for analyzing public opinion in contexts such as constitutional reform, monarchy-related debates, election campaigns, and broader polarized political discourse [2509.21946].

The motivating claim is that Thai political discourse is unusually difficult for LLM-based stance detection because it is often **culturally indirect**, frequently expressed through **sarcasm, irony, metaphor, rhetorical understatement, and coded language**, and because direct confrontation is often avoided [2509.21946]. In this setting, stance is frequently signaled implicitly rather than through explicit declaration, so sentiment is not a reliable proxy for stance, the same wording can imply different stances depending on the target, and neutral civic concern can be mistaken for partisan support or opposition [2509.21946].

The paper also frames Thai as a **low-resource or underrepresented language setting** for politically sensitive NLP [2509.21946]. It attributes the difficulty partly to uneven representation of Thai political discourse in pretraining corpora, weaker capture of sociocultural nuance, and complications from code-switching and culturally specific references [2509.21946]. This suggests that models are more likely to rely on shallow shortcuts than on target-specific political reasoning.

A central conceptual contribution of the work is to recast bias in stance detection as a problem of evidence misalignment. Rather than using target-relevant political evidence, LLMs often rely on **sentiment polarity**, **lexical shortcuts**, **entity identity**, and **likely exposure priors from pretraining** [2509.21946]. The paper treats this not only as an accuracy problem but also as a fairness problem, because predictions may vary inappropriately across political entities due to favoritism or priors rather than genuine stance evidence [2509.21946].

## 2. Biases identified in Thai political stance detection

The most prominent bias identified by the paper is **sentiment leakage**, also described as **sentiment–stance entanglement** [2509.21946]. In this failure mode, models systematically map **positive sentiment** to **support** and **negative sentiment** to **against**, even when that mapping is incorrect [2509.21946]. The paper explicitly calls this a **spurious correlation**, because the model is using emotional tone as a shortcut instead of reasoning about the political target and context [2509.21946].

The second major failure mode is **entity favoritism** or **entity preference bias** [2509.21946]. Here, identical or near-identical content can receive different stance predictions depending on which politician is named. The paper attributes this to learned priors from pretraining data and historical political associations, and it specifically discusses bias involving **Paetongtarn Shinawatra**, **Thaksin Shinawatra**, and **Pita Limjaroenrat**, noting that Thaksin-related cases showed amplified bias in evaluation [2509.21946].

The paper operationalizes the broader mechanism behind both problems as **spurious correlation** [2509.21946]. ThaiFACTUAL is intended to disrupt those correlations by holding wording and tone approximately constant while changing the named entity. In that sense, the framework is organized around a causal distinction: sentiment is treated as a potentially **non-causal confounder**, whereas target-specific political content is treated as the **causal signal** for stance [2509.21946].

The work also states that bias level is **inversely correlated with model accuracy** [2509.21946]. A plausible implication is that debiasing is not framed as a fairness-only intervention; it is presented as a way to improve predictive reliability by forcing the model toward target-relevant reasoning rather than shortcut-based inference.

## 3. Framework design and methodological structure

ThaiFACTUAL is described as a **plug-and-play debiasing method** based on a **Counterfactual Augmented Calibration module** layered on top of frozen LLMs [2509.21946]. The paper characterizes it as primarily an **inference-time / post-hoc calibration framework** that uses **counterfactual augmentation** and **rationale-based supervision**, and that does **not** require fine-tuning the base model [2509.21946]. The appendix summarizes the mechanism as follows: “LLMs predict, then a small calibration module re-scores using rationales and matched counterfactual pairs” [2509.21946].

The method is called **lightweight** because it does not require access to model parameters, works in a black-box API setting, avoids expensive fine-tuning or retraining of large models, and adds only a small auxiliary calibration module [2509.21946]. It is **model-agnostic** because it is intended to operate across multiple LLM families, including commercial APIs such as GPT-family models and open models such as LLaMA-family systems [2509.21946].

The paper provides a high-level pipeline rather than a fully formal algorithm [2509.21946]. The specified stages are:

1. **Collect and annotate Thai political texts**  
2. **Create counterfactual variants** by swapping political entities while preserving sentiment and structure  
3. **Obtain LLM stance predictions** on original inputs  
4. **Use rationale-based supervision** to guide what evidence should matter  
5. **Apply a small calibration module** that re-scores or adjusts predictions using the original input, matched counterfactual pairs, and rationales  
6. **Produce final stance output** [2509.21946]

The counterfactual component is the most concretely specified technical ingredient. Counterfactuals are generated by **replacing political entities while preserving the sentiment structure and tone** [2509.21946]. The appendix gives examples such as:

> Original: “Pita did a great job. I’m happy to see his vision for Thailand.”  
> CF Variant: “Thaksin did a great job. I’m happy to see his vision for Thailand.”

and

> Original: “Thaksin is corrupt. His return is an insult to justice.”  
> CF Variant: “Paetongtarn is corrupt. Her rise is an insult to justice.” [2509.21946]

These matched pairs are used to expose whether predictions are being driven by entity identity or by sentiment-bearing lexical items such as “great job,” “happy,” “corrupt,” and “insult” [2509.21946]. If the stance changes purely because the target changes while the wording remains structurally similar, the paper interprets that as evidence of entity bias; if stance tracks sentiment mechanically across targets, it is interpreted as sentiment leakage [2509.21946].

The second major component is **rationale-based supervision** [2509.21946]. Rationales are human-annotated short explanations that explicitly link stance and sentiment and are used to “disentangle sentiment from stance,” “guide model training” or calibration, and provide “context-aware rationales” for output adjustment [2509.21946]. The paper introduces “neutral rationale” as an intermediate representation showing that sentiment is not causally identical to stance [2509.21946].

At the same time, the paper is explicit about what remains underspecified. It does **not** provide a formal calibration equation, an explicit loss function, a detailed optimization objective, a rationale supervision loss, or a mathematical counterfactual consistency loss [2509.21946]. The most faithful description is therefore algorithmic rather than formulaic: the calibration module re-scores LLM outputs using rationales and matched counterfactual pairs, but its exact correction rule is not specified [2509.21946].

## 4. Dataset and annotation scheme

The paper claims to release the **first high-quality Thai political stance dataset** with annotations for **stance**, **sentiment**, **rationale**, and **bias markers** [2509.21946]. The most concretely specified portion of the dataset is a **270-item** evaluation set consisting of **90 Thai-language tweets per figure** for **3 political figures**: **Paetongtarn Shinawatra**, **Thaksin Shinawatra**, and **Pita Limjaroenrat** [2509.21946]. This core set is described as curated, balanced, and used in the bias evaluation and calibration experiments [2509.21946].

The label spaces are explicitly defined. Stance labels are **Support**, **Against**, and **Neutral**; sentiment labels are **Positive**, **Negative**, and **Neutral** [2509.21946]. Additional annotations may include the target entity, rationale text, and a bias marker, described in the appendix as optional binary indicators [2509.21946].

The annotation procedure used **three native Thai annotators** with **political science backgrounds** [2509.21946]. Labels were assigned per tweet, **majority vote** was used to resolve labels, and ambiguous tweets such as sarcasm or news reposts were excluded [2509.21946]. The paper also mentions “quality assurance through adjudication and multi-annotator agreement” and reports **Fleiss’ $\kappa = 0.84$**, indicating strong agreement [2509.21946].

The dataset is also described as **equally distributed across stance and sentiment categories**, with exactly **90 tweets per target** and **270 total** [2509.21946]. However, the paper does not provide a full contingency table for stance $\times$ sentiment $\times$ target, nor standard train/dev/test split counts [2509.21946]. It instead emphasizes an **OOD generalization** protocol based on holding out one political figure and calibrating on the other two [2509.21946].

Some aspects of dataset scope remain inconsistent in the paper’s description. One section says data were curated from social media platforms, political news commentary, and parliamentary debate transcripts spanning **2019 to 2024**, while another says tweets were scraped from **2023–2025** using Twitter API and open-source crawlers [2509.21946]. The paper does not reconcile these descriptions. The safest interpretation is that the final, clearly specified evaluation benchmark is the 270-tweet target-balanced set centered on three politicians, while the broader source pool is not fully enumerated [2509.21946].

The paper also notes several difficult categories in Thai political language: sarcasm, irony, metaphor, rhetorical understatement, neutral civic concern, positive sentiment with critical stance, and target-dependent interpretation shifts [2509.21946]. These categories help explain why the benchmark is structured around counterfactual control rather than sentiment alone.

## 5. Experimental setup and quantitative findings

The main reported comparison includes **GPT-4 (Raw)**, **GPT-4 (Debias Prompt)**, **LLaMA-3 (CoT Prompt)**, and **ThaiFACTUAL (Ours)** [2509.21946]. The implementation section also mentions evaluation through OpenAI and HuggingFace APIs, including **GPT-4**, **GPT-3.5**, and **LLaMA-3-8B-chat**, although **GPT-3.5** does not appear in the main result table reproduced in the paper text [2509.21946]. All prompting uses **temperature = 0.0** for determinism [2509.21946].

The evaluation emphasizes **zero-shot generalization** and **OOD generalization to unseen political entities** [2509.21946]. The OOD protocol is defined procedurally: hold out one political figure, calibrate or train on the other two, and evaluate zero-shot on the held-out figure [2509.21946].

The paper reports four metrics. **Bias-SSC** measures sentiment–stance correlation bias, with lower values better:

$$
\text{Bias-SSC} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[\text{sentiment}(x_i) = \text{mapped\_stance}(y_i^{\text{pred}})\right]
$$

with mapping **positive sentiment $\rightarrow$ support** and **negative sentiment $\rightarrow$ against** [2509.21946]. **RStd** measures recall standard deviation across stance classes:

$$
RStd = \sqrt{\frac{1}{K} \sum_{i=1}^{K} \left( \frac{TP_i}{P_i} - \frac{1}{K} \sum_{j=1}^{K} \frac{TP_j}{P_j} \right)^2}
$$

where lower values indicate more balanced recall across classes [2509.21946]. **Macro-F1** is the standard three-class macro-average:

$$
\text{Macro-F1} = \frac{1}{3} \sum_{i=1}^3 \frac{2 \cdot \text{Precision}_i \cdot \text{Recall}_i}{\text{Precision}_i + \text{Recall}_i}
$$

and **OOD** is reported as a held-out-entity score whose exact metric type is not explicitly stated [2509.21946].

The main quantitative results are:

| Model | Bias-SSC ↓ | RStd ↓ | F1 ↑ | OOD ↑ |
|---|---:|---:|---:|---:|
| GPT-4 (Raw) | 21.7 | 15.2 | 70.8 | 56.4 |
| GPT-4 (Debias Prompt) | 18.3 | 12.6 | 71.9 | 57.0 |
| LLaMA-3 (CoT Prompt) | 16.5 | 11.8 | 68.1 | 59.7 |
| ThaiFACTUAL | 9.8 | 6.4 | 73.5 | 65.2 |

ThaiFACTUAL achieves the lowest reported **Bias-SSC**, the lowest **RStd**, the highest **Macro-F1**, and the highest **OOD** score in the reported comparison [2509.21946]. Relative to **GPT-4 (Raw)**, the paper reports a reduction of **11.9 points** in Bias-SSC, **8.8 points** in RStd, a **2.7-point** gain in F1, and an **8.8-point** gain in OOD [2509.21946]. Relative to **GPT-4 (Debias Prompt)**, the improvements are **8.5**, **6.2**, **1.6**, and **8.2** points respectively; relative to **LLaMA-3 (CoT Prompt)**, they are **6.7**, **5.4**, **5.4**, and **5.5** points [2509.21946].

The reported pattern is important because the paper does not present debiasing as a tradeoff against predictive performance. In the table shown, ThaiFACTUAL improves fairness-oriented and robustness-oriented metrics while also improving Macro-F1 [2509.21946]. The text further notes that chain-of-thought prompting can improve generalization but may slightly hurt F1 due to instability, whereas ThaiFACTUAL avoids that tradeoff in the reported results [2509.21946].

The paper does **not** include a formal ablation table in the provided text [2509.21946]. It argues conceptually for the importance of counterfactual augmentation, rationale supervision, and calibration beyond prompt engineering, but no numeric ablation isolating each component is reported in the excerpt [2509.21946].

## 6. Fairness interpretation, limitations, and broader significance

ThaiFACTUAL operationalizes fairness through reduced spurious dependence on emotional tone and target identity [2509.21946]. The paper uses **Bias-SSC** as a measure of fairness against sentiment leakage and **RStd** as a measure of balanced class recall and reduced prediction instability [2509.21946]. Entity bias is assessed more indirectly through controlled counterfactual swaps, inconsistent outputs across swapped targets, OOD transfer to unseen entities, and stability analyses [2509.21946]. The paper does not define a separate formal “entity bias score” equation [2509.21946].

Qualitative error analysis highlights several patterns in raw LLM behavior: **positive sentiment but actually critical stance**, **same wording with different politicians producing different outputs**, **neutral civic concern misclassified as partisan**, and especially **Thaksin-related cases** in which negative sentiment is over-mapped to **Against** regardless of context [2509.21946]. These examples support the claim that raw stance detection is often shortcut-based rather than target-specific.

The paper is also explicit that ThaiFACTUAL does not fully solve the problem [2509.21946]. Remaining failure modes include latent historical associations still leaking into predictions, the narrowness of **entity substitution** as the main counterfactual operation, residual cultural priors, and the absence of evaluation on other fairness dimensions such as dialect, user ideology, and media framing [2509.21946].

Several broader limitations are acknowledged. The dataset is **small**, **manually annotated**, and limited to **three entities** [2509.21946]. Political discourse changes rapidly, so a static dataset may become less representative over time [2509.21946]. The post-hoc black-box design is practical for frozen APIs but limits access to internal representations and deeper causal intervention [2509.21946]. Annotation subjectivity also remains inherent despite strong agreement [2509.21946].

The paper presents ThaiFACTUAL as a blueprint for culturally grounded debiasing in underrepresented languages [2509.21946]. A plausible implication is that its main contribution lies less in a fully specified new optimization framework than in establishing a concrete methodology for separating affective tone from ideological stance under black-box LLM constraints. In that respect, ThaiFACTUAL belongs to a broader multilingual concern also visible in work on multilingual factual evaluation and multilingual fact checking: resource inequality, source coverage asymmetry, and the instability of cross-lingual model behavior remain central obstacles in low- and medium-resource settings [2402.18045] [2406.19415] [2106.09248]. ThaiFACTUAL’s specific intervention is to address those issues at the level of stance reasoning, using counterfactual control and rationales rather than model retraining [2509.21946].

Source: https://www.emergentmind.com/topics/thaifactual