ThaiFACTUAL: Debiasing Thai Political Stance

• ThaiFACTUAL is a model-agnostic calibration framework that disentangles sentiment from stance in Thai political discourse using counterfactual augmentation and rationale supervision.
• It addresses key biases by mitigating sentiment–stance entanglement and entity preference bias, thereby enhancing both fairness and predictive accuracy.
• Operating post-hoc on frozen LLMs, ThaiFACTUAL improves bias metrics and macro-F1 scores without the need for costly fine-tuning.

ThaiFACTUAL is a lightweight, model-agnostic calibration framework for Thai political stance detection with LLMs, introduced to mitigate systematic bias in a setting characterized as low-resource, culturally indirect, emotionally charged, and politically polarized (Sermsri et al., 26 Sep 2025). It is designed for a three-way stance detection task—Support, Against, and Neutral—over Thai-language political posts targeting specific political figures, and it addresses two central failure modes identified in standard LLM predictions: sentiment–stance entanglement and entity preference bias (Sermsri et al., 26 Sep 2025). The framework operates without fine-tuning the underlying LLM, instead applying post-hoc calibration through counterfactual data augmentation and rationale-based supervision (Sermsri et al., 26 Sep 2025).

1. Problem setting and motivation

ThaiFACTUAL is situated in the task of Thai political stance detection: given a Thai-language post and a political target, predict whether the author is Support, Against, or Neutral toward that target (Sermsri et al., 26 Sep 2025). The paper emphasizes utility for analyzing public opinion in contexts such as constitutional reform, monarchy-related debates, election campaigns, and broader polarized political discourse (Sermsri et al., 26 Sep 2025).

The motivating claim is that Thai political discourse is unusually difficult for LLM-based stance detection because it is often culturally indirect, frequently expressed through sarcasm, irony, metaphor, rhetorical understatement, and coded language, and because direct confrontation is often avoided (Sermsri et al., 26 Sep 2025). In this setting, stance is frequently signaled implicitly rather than through explicit declaration, so sentiment is not a reliable proxy for stance, the same wording can imply different stances depending on the target, and neutral civic concern can be mistaken for partisan support or opposition (Sermsri et al., 26 Sep 2025).

The paper also frames Thai as a low-resource or underrepresented language setting for politically sensitive NLP (Sermsri et al., 26 Sep 2025). It attributes the difficulty partly to uneven representation of Thai political discourse in pretraining corpora, weaker capture of sociocultural nuance, and complications from code-switching and culturally specific references (Sermsri et al., 26 Sep 2025). This suggests that models are more likely to rely on shallow shortcuts than on target-specific political reasoning.

A central conceptual contribution of the work is to recast bias in stance detection as a problem of evidence misalignment. Rather than using target-relevant political evidence, LLMs often rely on sentiment polarity, lexical shortcuts, entity identity, and likely exposure priors from pretraining (Sermsri et al., 26 Sep 2025). The paper treats this not only as an accuracy problem but also as a fairness problem, because predictions may vary inappropriately across political entities due to favoritism or priors rather than genuine stance evidence (Sermsri et al., 26 Sep 2025).

2. Biases identified in Thai political stance detection

The most prominent bias identified by the paper is sentiment leakage, also described as sentiment–stance entanglement (Sermsri et al., 26 Sep 2025). In this failure mode, models systematically map positive sentiment to support and negative sentiment to against, even when that mapping is incorrect (Sermsri et al., 26 Sep 2025). The paper explicitly calls this a spurious correlation, because the model is using emotional tone as a shortcut instead of reasoning about the political target and context (Sermsri et al., 26 Sep 2025).

The second major failure mode is entity favoritism or entity preference bias (Sermsri et al., 26 Sep 2025). Here, identical or near-identical content can receive different stance predictions depending on which politician is named. The paper attributes this to learned priors from pretraining data and historical political associations, and it specifically discusses bias involving Paetongtarn Shinawatra, Thaksin Shinawatra, and Pita Limjaroenrat, noting that Thaksin-related cases showed amplified bias in evaluation (Sermsri et al., 26 Sep 2025).

The paper operationalizes the broader mechanism behind both problems as spurious correlation (Sermsri et al., 26 Sep 2025). ThaiFACTUAL is intended to disrupt those correlations by holding wording and tone approximately constant while changing the named entity. In that sense, the framework is organized around a causal distinction: sentiment is treated as a potentially non-causal confounder, whereas target-specific political content is treated as the causal signal for stance (Sermsri et al., 26 Sep 2025).

The work also states that bias level is inversely correlated with model accuracy (Sermsri et al., 26 Sep 2025). A plausible implication is that debiasing is not framed as a fairness-only intervention; it is presented as a way to improve predictive reliability by forcing the model toward target-relevant reasoning rather than shortcut-based inference.

3. Framework design and methodological structure

ThaiFACTUAL is described as a plug-and-play debiasing method based on a Counterfactual Augmented Calibration module layered on top of frozen LLMs (Sermsri et al., 26 Sep 2025). The paper characterizes it as primarily an inference-time / post-hoc calibration framework that uses counterfactual augmentation and rationale-based supervision, and that does not require fine-tuning the base model (Sermsri et al., 26 Sep 2025). The appendix summarizes the mechanism as follows: “LLMs predict, then a small calibration module re-scores using rationales and matched counterfactual pairs” (Sermsri et al., 26 Sep 2025).

The method is called lightweight because it does not require access to model parameters, works in a black-box API setting, avoids expensive fine-tuning or retraining of large models, and adds only a small auxiliary calibration module (Sermsri et al., 26 Sep 2025). It is model-agnostic because it is intended to operate across multiple LLM families, including commercial APIs such as GPT-family models and open models such as LLaMA-family systems (Sermsri et al., 26 Sep 2025).

The paper provides a high-level pipeline rather than a fully formal algorithm (Sermsri et al., 26 Sep 2025). The specified stages are:

1. Collect and annotate Thai political texts
2. Create counterfactual variants by swapping political entities while preserving sentiment and structure
3. Obtain LLM stance predictions on original inputs
4. Use rationale-based supervision to guide what evidence should matter
5. Apply a small calibration module that re-scores or adjusts predictions using the original input, matched counterfactual pairs, and rationales
6. Produce final stance output (Sermsri et al., 26 Sep 2025)

The counterfactual component is the most concretely specified technical ingredient. Counterfactuals are generated by replacing political entities while preserving the sentiment structure and tone (Sermsri et al., 26 Sep 2025). The appendix gives examples such as:

Original: “Pita did a great job. I’m happy to see his vision for Thailand.” CF Variant: “Thaksin did a great job. I’m happy to see his vision for Thailand.”

and

Original: “Thaksin is corrupt. His return is an insult to justice.” CF Variant: “Paetongtarn is corrupt. Her rise is an insult to justice.” (Sermsri et al., 26 Sep 2025)

These matched pairs are used to expose whether predictions are being driven by entity identity or by sentiment-bearing lexical items such as “great job,” “happy,” “corrupt,” and “insult” (Sermsri et al., 26 Sep 2025). If the stance changes purely because the target changes while the wording remains structurally similar, the paper interprets that as evidence of entity bias; if stance tracks sentiment mechanically across targets, it is interpreted as sentiment leakage (Sermsri et al., 26 Sep 2025).

The second major component is rationale-based supervision (Sermsri et al., 26 Sep 2025). Rationales are human-annotated short explanations that explicitly link stance and sentiment and are used to “disentangle sentiment from stance,” “guide model training” or calibration, and provide “context-aware rationales” for output adjustment (Sermsri et al., 26 Sep 2025). The paper introduces “neutral rationale” as an intermediate representation showing that sentiment is not causally identical to stance (Sermsri et al., 26 Sep 2025).

At the same time, the paper is explicit about what remains underspecified. It does not provide a formal calibration equation, an explicit loss function, a detailed optimization objective, a rationale supervision loss, or a mathematical counterfactual consistency loss (Sermsri et al., 26 Sep 2025). The most faithful description is therefore algorithmic rather than formulaic: the calibration module re-scores LLM outputs using rationales and matched counterfactual pairs, but its exact correction rule is not specified (Sermsri et al., 26 Sep 2025).

4. Dataset and annotation scheme

The paper claims to release the first high-quality Thai political stance dataset with annotations for stance, sentiment, rationale, and bias markers (Sermsri et al., 26 Sep 2025). The most concretely specified portion of the dataset is a 270-item evaluation set consisting of 90 Thai-language tweets per figure for 3 political figures: Paetongtarn Shinawatra, Thaksin Shinawatra, and Pita Limjaroenrat (Sermsri et al., 26 Sep 2025). This core set is described as curated, balanced, and used in the bias evaluation and calibration experiments (Sermsri et al., 26 Sep 2025).

The label spaces are explicitly defined. Stance labels are Support, Against, and Neutral; sentiment labels are Positive, Negative, and Neutral (Sermsri et al., 26 Sep 2025). Additional annotations may include the target entity, rationale text, and a bias marker, described in the appendix as optional binary indicators (Sermsri et al., 26 Sep 2025).

The annotation procedure used three native Thai annotators with political science backgrounds (Sermsri et al., 26 Sep 2025). Labels were assigned per tweet, majority vote was used to resolve labels, and ambiguous tweets such as sarcasm or news reposts were excluded (Sermsri et al., 26 Sep 2025). The paper also mentions “quality assurance through adjudication and multi-annotator agreement” and reports Fleiss’ $\kappa = 0.84$, indicating strong agreement (Sermsri et al., 26 Sep 2025).

The dataset is also described as equally distributed across stance and sentiment categories, with exactly 90 tweets per target and 270 total (Sermsri et al., 26 Sep 2025). However, the paper does not provide a full contingency table for stance $\times$ sentiment $\times$ target, nor standard train/dev/test split counts (Sermsri et al., 26 Sep 2025). It instead emphasizes an OOD generalization protocol based on holding out one political figure and calibrating on the other two (Sermsri et al., 26 Sep 2025).

Some aspects of dataset scope remain inconsistent in the paper’s description. One section says data were curated from social media platforms, political news commentary, and parliamentary debate transcripts spanning 2019 to 2024, while another says tweets were scraped from 2023–2025 using Twitter API and open-source crawlers (Sermsri et al., 26 Sep 2025). The paper does not reconcile these descriptions. The safest interpretation is that the final, clearly specified evaluation benchmark is the 270-tweet target-balanced set centered on three politicians, while the broader source pool is not fully enumerated (Sermsri et al., 26 Sep 2025).

The paper also notes several difficult categories in Thai political language: sarcasm, irony, metaphor, rhetorical understatement, neutral civic concern, positive sentiment with critical stance, and target-dependent interpretation shifts (Sermsri et al., 26 Sep 2025). These categories help explain why the benchmark is structured around counterfactual control rather than sentiment alone.

5. Experimental setup and quantitative findings

The main reported comparison includes GPT-4 (Raw), GPT-4 (Debias Prompt), LLaMA-3 (CoT Prompt), and ThaiFACTUAL (Ours) (Sermsri et al., 26 Sep 2025). The implementation section also mentions evaluation through OpenAI and HuggingFace APIs, including GPT-4, GPT-3.5, and LLaMA-3-8B-chat, although GPT-3.5 does not appear in the main result table reproduced in the paper text (Sermsri et al., 26 Sep 2025). All prompting uses temperature = 0.0 for determinism (Sermsri et al., 26 Sep 2025).

The evaluation emphasizes zero-shot generalization and OOD generalization to unseen political entities (Sermsri et al., 26 Sep 2025). The OOD protocol is defined procedurally: hold out one political figure, calibrate or train on the other two, and evaluate zero-shot on the held-out figure (Sermsri et al., 26 Sep 2025).

The paper reports four metrics. Bias-SSC measures sentiment–stance correlation bias, with lower values better:

$$\text{Bias-SSC} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left[\text{sentiment}(x_i) = \text{mapped\_stance}(y_i^{\text{pred}})\right]$$

with mapping positive sentiment $\rightarrow$ support and negative sentiment $\rightarrow$ against (Sermsri et al., 26 Sep 2025). RStd measures recall standard deviation across stance classes:

$$RStd = \sqrt{\frac{1}{K} \sum_{i=1}^{K} \left( \frac{TP_i}{P_i} - \frac{1}{K} \sum_{j=1}^{K} \frac{TP_j}{P_j} \right)^2}$$

where lower values indicate more balanced recall across classes (Sermsri et al., 26 Sep 2025). Macro-F1 is the standard three-class macro-average:

$$\text{Macro-F1} = \frac{1}{3} \sum_{i=1}^3 \frac{2 \cdot \text{Precision}_i \cdot \text{Recall}_i}{\text{Precision}_i + \text{Recall}_i}$$

and OOD is reported as a held-out-entity score whose exact metric type is not explicitly stated (Sermsri et al., 26 Sep 2025).

The main quantitative results are:

Model	Bias-SSC ↓	RStd ↓	F1 ↑	OOD ↑
GPT-4 (Raw)	21.7	15.2	70.8	56.4
GPT-4 (Debias Prompt)	18.3	12.6	71.9	57.0
LLaMA-3 (CoT Prompt)	16.5	11.8	68.1	59.7
ThaiFACTUAL	9.8	6.4	73.5	65.2

ThaiFACTUAL achieves the lowest reported Bias-SSC, the lowest RStd, the highest Macro-F1, and the highest OOD score in the reported comparison (Sermsri et al., 26 Sep 2025). Relative to GPT-4 (Raw), the paper reports a reduction of 11.9 points in Bias-SSC, 8.8 points in RStd, a 2.7-point gain in F1, and an 8.8-point gain in OOD (Sermsri et al., 26 Sep 2025). Relative to GPT-4 (Debias Prompt), the improvements are 8.5, 6.2, 1.6, and 8.2 points respectively; relative to LLaMA-3 (CoT Prompt), they are 6.7, 5.4, 5.4, and 5.5 points (Sermsri et al., 26 Sep 2025).

The reported pattern is important because the paper does not present debiasing as a tradeoff against predictive performance. In the table shown, ThaiFACTUAL improves fairness-oriented and robustness-oriented metrics while also improving Macro-F1 (Sermsri et al., 26 Sep 2025). The text further notes that chain-of-thought prompting can improve generalization but may slightly hurt F1 due to instability, whereas ThaiFACTUAL avoids that tradeoff in the reported results (Sermsri et al., 26 Sep 2025).

The paper does not include a formal ablation table in the provided text (Sermsri et al., 26 Sep 2025). It argues conceptually for the importance of counterfactual augmentation, rationale supervision, and calibration beyond prompt engineering, but no numeric ablation isolating each component is reported in the excerpt (Sermsri et al., 26 Sep 2025).

6. Fairness interpretation, limitations, and broader significance

ThaiFACTUAL operationalizes fairness through reduced spurious dependence on emotional tone and target identity (Sermsri et al., 26 Sep 2025). The paper uses Bias-SSC as a measure of fairness against sentiment leakage and RStd as a measure of balanced class recall and reduced prediction instability (Sermsri et al., 26 Sep 2025). Entity bias is assessed more indirectly through controlled counterfactual swaps, inconsistent outputs across swapped targets, OOD transfer to unseen entities, and stability analyses (Sermsri et al., 26 Sep 2025). The paper does not define a separate formal “entity bias score” equation (Sermsri et al., 26 Sep 2025).

Qualitative error analysis highlights several patterns in raw LLM behavior: positive sentiment but actually critical stance, same wording with different politicians producing different outputs, neutral civic concern misclassified as partisan, and especially Thaksin-related cases in which negative sentiment is over-mapped to Against regardless of context (Sermsri et al., 26 Sep 2025). These examples support the claim that raw stance detection is often shortcut-based rather than target-specific.

The paper is also explicit that ThaiFACTUAL does not fully solve the problem (Sermsri et al., 26 Sep 2025). Remaining failure modes include latent historical associations still leaking into predictions, the narrowness of entity substitution as the main counterfactual operation, residual cultural priors, and the absence of evaluation on other fairness dimensions such as dialect, user ideology, and media framing (Sermsri et al., 26 Sep 2025).

Several broader limitations are acknowledged. The dataset is small, manually annotated, and limited to three entities (Sermsri et al., 26 Sep 2025). Political discourse changes rapidly, so a static dataset may become less representative over time (Sermsri et al., 26 Sep 2025). The post-hoc black-box design is practical for frozen APIs but limits access to internal representations and deeper causal intervention (Sermsri et al., 26 Sep 2025). Annotation subjectivity also remains inherent despite strong agreement (Sermsri et al., 26 Sep 2025).

The paper presents ThaiFACTUAL as a blueprint for culturally grounded debiasing in underrepresented languages (Sermsri et al., 26 Sep 2025). A plausible implication is that its main contribution lies less in a fully specified new optimization framework than in establishing a concrete methodology for separating affective tone from ideological stance under black-box LLM constraints. In that respect, ThaiFACTUAL belongs to a broader multilingual concern also visible in work on multilingual factual evaluation and multilingual fact checking: resource inequality, source coverage asymmetry, and the instability of cross-lingual model behavior remain central obstacles in low- and medium-resource settings (Shafayat et al., 2024, 2406.19415, Gupta et al., 2021). ThaiFACTUAL’s specific intervention is to address those issues at the level of stance reasoning, using counterfactual control and rationales rather than model retraining (Sermsri et al., 26 Sep 2025).