---
title: 'WangchanThaiInstruct: Thai Instruction Benchmark'
url: https://www.emergentmind.com/topics/wangchanthaiinstruct
type: topic
---

# WangchanThaiInstruct: Thai Instruction Benchmark

WangchanThaiInstruct is a human-authored Thai instruction-following dataset designed for both evaluation and instruction tuning in settings where cultural specificity, professional domain knowledge, and multitask competence matter simultaneously. It was introduced to address the limits of English-centric and machine-translated benchmarks, which can preserve surface task form while missing Thai idioms, institutions, legal distinctions, and other local constraints. The dataset contains **35,014 total samples**, split into **28,098 training samples** and **6,916 test samples**, spans **Finance**, **Legal**, **Medical**, and **Retail**, covers **seven task types**, and contains **no LLM-generated content** in its data creation process [2508.15239].

## 1. Rationale and problem setting

The central motivation for WangchanThaiInstruct is that Thai instruction-following has been under-measured and under-supervised relative to English. The dataset paper identifies four linked deficiencies in prior practice: **English-centric evaluation**, **translated benchmarks that miss cultural and professional nuance**, **scarcity of native Thai supervised instruction data**, and the lack of **domain-sensitive evaluation** in settings where errors can be harmful or costly, especially legal, financial, medical, and retail/business contexts [2508.15239].

A defining property of the resource is that it is **human-authored in Thai**, rather than translated from English. The paper argues that translated benchmarks often preserve English task structure, rely on generic or culture-neutral prompts, do not reflect Thai law, Thai medicine, Thai finance, or Thai retail realities, and therefore can overestimate model quality for real Thai deployment. WangchanThaiInstruct instead uses Thai web sources, Thai domain experts, culturally tagged examples, and task formulations written directly in Thai by native annotators [2508.15239].

This design also frames a recurrent misconception in multilingual LLM evaluation: fluent Thai output is not equivalent to correct or culturally grounded Thai reasoning. The zero-shot analysis explicitly reports that models can produce fluent Thai while remaining wrong on factual or culturally specific questions, which positions WangchanThaiInstruct as a benchmark for alignment quality rather than surface fluency alone [2508.15239].

## 2. Corpus composition and benchmark structure

WangchanThaiInstruct spans **four professional domains**—**Finance**, **Legal**, **Medical**, and **Retail**—and **seven task types**: **Brainstorming**, **Classification**, **Closed QA**, **Creative Writing**, **Multiple Choice**, **Open QA**, and **Summarization** [2508.15239]. This mixture is methodologically important because it combines objective tasks with open-ended generation, allowing evaluation of correctness, reasoning, fluency, and cohesiveness within one benchmark.

The source collection is large and document-grounded. The paper reports **27,351 documents** collected from **86 Thai websites**, mostly government agencies and publicly listed companies. HTML was cleaned, content-rich documents were retained, and deduplication was performed using **mUSE** with cosine similarity threshold **0.8** [2508.15239]. The dataset also includes substantial topical breadth within domains, with **18 subtopics** in finance, **133 subtopics** in legal, and **28 subtopics** in medical; retail subtopics are also present [2508.15239].

A notable technical feature is the benchmark’s long-context regime. The test set contains contexts ranging from **6 to 26,405 tokens**, making the resource relevant not only for Thai instruction following and domain grounding, but also for long-context retrieval and reasoning behavior [2508.15239]. This long-context property materially increases task difficulty, especially for tasks that require cross-document or document-wide evidence aggregation.

The dataset’s identity in the literature has evolved. A 2024 synthetic-data study treats WangchanThaiInstruct as a benchmark with **6,287 samples total**, split into **Thai culture-specific** and **general** versions, across **Legal**, **Medical**, and **Finance**, while using the same seven task families [2411.15484]. The later WangchanThaiInstruct paper presents the larger **35,014-sample** human-authored release with explicit train and test splits [2508.15239]. This suggests that the benchmark name was already in use before the later expanded dataset formulation.

## 3. Annotation pipeline, expert review, and licensing

The construction pipeline is explicitly multi-stage and combines annotators, domain experts, and AI researchers. In the first stage, annotators had to be **native Thai speakers** and pass an **annotation exam** covering all seven task types. Each annotator received a Thai website document, a task type, and instructions to create both the question or instruction and the gold answer. They were also asked to provide reasoning for answers in most task types, except **creative writing**, **brainstorming**, and **summarization** [2508.15239].

The second stage performed question-level quality control by annotators. A sample of **10% of examples** was re-checked and edited for question correctness, instruction quality, answer correctness, and reasoning correctness [2508.15239]. The third stage introduced domain-specialist verification in **medical**, **legal**, and **finance**. Experts could accept examples, reject them with reasons and references, or send them back for revision; they also added a **cultural tag** indicating whether the sample was Thai-specific. The paper states that this process was repeated until all expert-reviewed data were accepted [2508.15239].

The final stage was formatting control by AI researchers, who enforced structures needed for instruction tuning, including proper multiple-choice formatting, full-sentence closed QA, and inclusion of rationale when needed [2508.15239]. This stage matters because it makes the dataset usable not only as a benchmark but also as supervised fine-tuning data.

To reduce leakage, splitting was performed at the **document level**, so all samples from the same source document remain in the same split [2508.15239]. The release uses a mixed license: **30,000 samples** under **CC BY-NC** and **5,014 samples** under **CC BY-SA 4.0** [2508.15239].

## 4. Evaluation protocol and metric design

The WangchanThaiInstruct paper explicitly rejects exclusive reliance on surface-overlap metrics such as BLEU or ROUGE-L for instruction-following evaluation. Its main protocol adapts an **LLM-as-a-judge** setup inspired by MT-Bench. For tasks with objective answers, the judge evaluates both **correctness** and **reasoning quality**; for creative tasks, it evaluates **fluency** and **cohesiveness** [2508.15239].

For definitive-answer tasks, the judging prompt requires the model to output `Correctness: [[1]]` if correct or `[[0]]` if incorrect, together with `Rating: [[<score>]]` on a **1–10** scale for reasoning quality. The prompt explicitly conditions on the user question, the reference answer, and the assistant answer [2508.15239]. In the appendix, judge models are compared against human rankings on **1,200 sampled examples** using Kendall’s $\tau$. The reported average values are **GPT-4.1: 0.1259**, **Gemini 2.5: 0.1255**, **Claude Sonnet 3.7: 0.0962**, **BLEU: 0.0319**, and **ROUGE-L: 0.0053**; GPT-4.1 is chosen because it has the best overall alignment [2508.15239]. The paper also reports a **Spearman correlation of 0.78** between reasoning rating and accuracy, indicating a strong association between better reasoning and correct answers [2508.15239].

Earlier work used WangchanThaiInstruct differently. The 2024 seed-free synthetic-data paper evaluates on the benchmark using WangchanX-10k-style metrics—**BLEU**, **METEOR**, **ChrF**, **ROUGE-1**, **ROUGE-2**, **ROUGE-L**, **ROUGE-Lsum**, and **BERTScore**—and states that **BERTScore is the most reliable metric** for this benchmark because it measures semantic similarity better than n-gram overlap metrics [2411.15484]. Taken together, these two uses show a transition from metric-based benchmarking toward judge-based evaluation tailored to reasoning-heavy Thai instruction following.

## 5. Zero-shot findings and difficulty profile

The zero-shot study evaluates **Gemini 2.0**, **Qwen2.5-7B**, **Qwen2.5-72B**, **Llama-3.1-8B**, **Llama-3.1-70B**, **Sailor2-8B**, and **Sailor2-20B** under a uniform prompt and zero-shot setup [2508.15239]. A consistent result is that **cultural** samples are harder than **general** samples, particularly in **Multiple Choice**, **Open QA**, and **Legal** tasks. The paper’s interpretation is that Thai fluency by itself does not provide the cultural and professional grounding needed for these items [2508.15239].

The domain-level difficulty ordering is especially clear. **Legal** is the hardest domain, and **no model exceeds 73% accuracy** there. By contrast, **Medical** is the easiest domain, with **Gemini 2.0 reaching 88.71%** [2508.15239]. The paper attributes legal difficulty, at least in part, to the scarcity of legal data in pretraining and to the local specificity of Thai legal systems.

The benchmark also overturns a standard intuition about multiple-choice tasks. Although prior work often treats multiple choice as comparatively easy for LLMs, WangchanThaiInstruct reports lower-than-expected multiple-choice accuracy because these items require correct reasoning rather than superficial answer-pattern selection [2508.15239]. This is one reason the benchmark is harder than translated MCQ sets.

Long-context analysis further sharpens the difficulty profile. The study groups examples into **head** (shortest 20%), **body** (middle 40%), and **tail** (longest 40%), and finds that long context degrades performance especially on **Brainstorming**, **Multiple Choice**, **Open QA**, and **Summarization**; even strong models degrade on long inputs, and **Sailor2-20B** drops substantially on tail-group summarization [2508.15239].

## 6. Instruction tuning and the effect of native Thai supervision

The instruction-tuning study fine-tunes **Llama-3.1-8B**, **Gemma-2-9B**, and **SEA-LIONv2-8B** using **LlamaFactory** with the configuration
$\text{Learning rate} = 2 \times 10^{-4}$, $\text{schedule} = \text{cosine}$, $\text{effective batch size} = 128$, $\text{max token length} = 2048$, $\text{warmup ratio} = 0.1$, and $\text{epochs} = 3$ [2508.15239]. The comparison datasets are translated Thai instruction corpora, specifically **Alpaca-cleaned-52k-th** and **Dolly-15k-th** [2508.15239].

In the full-data comparison, models trained on WangchanThaiInstruct combined with Alpaca or Dolly improve on both **in-domain WangchanThaiInstruct** and **out-of-domain Thai benchmarks**, with the new data outperforming Alpaca- or Dolly-only training in **31 out of 42 cases** [2508.15239]. The paper emphasizes that gains are clear and consistent in-domain, while out-of-domain results are more mixed on NLG metrics such as BLEU and ROUGE-L.

The balanced ablation is more probative because it size-matches the mixtures: **Alpaca 5k + WangchanThaiInstruct 5k**, **Alpaca 10k + WangchanThaiInstruct 10k**, **Alpaca 15k + WangchanThaiInstruct 15k**, and corresponding Dolly mixtures [2508.15239]. Under these matched conditions, adding WangchanThaiInstruct still improves performance substantially. The reported counts are **Llama-3.1-8B: better in 31/42 cases**, **Gemma-2-9B: better in 41/42 cases**, and **SEA-LIONv2-8B: better in 28/42 cases** [2508.15239]. The strongest gains are on in-domain accuracy, reasoning ratings, and fluency, while out-of-domain improvements are more reliable for **NLU** and in-domain instruction following than for **NLG**.

The broader implication drawn in the paper is that **native Thai supervision** contributes something not recoverable from translated supervision alone. Translated datasets may still help, but WangchanThaiInstruct yields clearer advantages in domain understanding, culturally accurate responses, and instruction alignment [2508.15239].

## 7. Use in subsequent Thai LLM research

WangchanThaiInstruct has also entered the Thai LLM ecosystem as a resource for both benchmarking and data construction. In **Typhoon 2**, it is not the sole or central dataset, but it appears in two specific roles. First, it is used as **positive data** for training a Thai high-quality text classifier that filters the continual-pretraining corpus. Second, it contributes **random queries** to the **TyphoonIF** instruction-tuning dataset used in supervised fine-tuning for Typhoon2-Text [2412.13702]. The Typhoon 2 paper is explicit that its instruction-tuned models are produced from a broader multilingual and multi-stage post-training stack, so WangchanThaiInstruct is one ingredient in a larger pipeline rather than the defining supervision source [2412.13702].

The 2024 seed-free synthetic-data study uses WangchanThaiInstruct primarily as an evaluation benchmark for small synthetic Thai SFT datasets. Its best **5,000-instruction** dataset, **F+ C+ D+**, is reported as the **second-best model on BERTScore** on both benchmark splits. On the **Thai Culture Test Set**, it reaches **69.50**, compared with **68.80** for WangchanX, **64.50** for OpenThaiGPT, and **74.10** for Typhoon; on the **General Test Set**, it reaches **73.20**, compared with **72.20**, **67.50**, and **76.50**, respectively [2411.15484]. In that paper, WangchanThaiInstruct functions as a stress test for three dataset properties—**fluency**, **diversity**, and **cultural context**—and the consistent advantage of **F+ C+ D+** supports the view that Thai-native fluency and cultural grounding are materially important for performance on the benchmark [2411.15484].

A parallel Thai instruction-following line in audio language models does not directly use WangchanThaiInstruct. The Thai audio study on Typhoon-Audio instead relies on **SelfInstruct-Th**, **Airoboros-Th**, **Alpaca-Th**, translated Thai prompt-response pairs, and Thai speech-instruction mixtures for speech instruction following [2409.10999]. A plausible implication is that WangchanThaiInstruct currently has its clearest direct role in text-centric evaluation and text instruction tuning, while closely related Thai instruction-following work in speech has, so far, depended on other Thai supervision sources.

Source: https://www.emergentmind.com/topics/wangchanthaiinstruct