WangchanThaiInstruct: Thai Instruction Benchmark

• WangchanThaiInstruct is a Thai instruction-following dataset that captures native cultural, legal, and professional nuances.
• It spans four domains and seven task types, enabling comprehensive evaluation from closed QA to creative writing.
• The dataset is rigorously annotated by native speakers and domain experts, ensuring high-quality, context-rich instruction tuning.

WangchanThaiInstruct is a human-authored Thai instruction-following dataset designed for both evaluation and instruction tuning in settings where cultural specificity, professional domain knowledge, and multitask competence matter simultaneously. It was introduced to address the limits of English-centric and machine-translated benchmarks, which can preserve surface task form while missing Thai idioms, institutions, legal distinctions, and other local constraints. The dataset contains 35,014 total samples, split into 28,098 training samples and 6,916 test samples, spans Finance, Legal, Medical, and Retail, covers seven task types, and contains no LLM-generated content in its data creation process (Limkonchotiwat et al., 21 Aug 2025).

1. Rationale and problem setting

The central motivation for WangchanThaiInstruct is that Thai instruction-following has been under-measured and under-supervised relative to English. The dataset paper identifies four linked deficiencies in prior practice: English-centric evaluation, translated benchmarks that miss cultural and professional nuance, scarcity of native Thai supervised instruction data, and the lack of domain-sensitive evaluation in settings where errors can be harmful or costly, especially legal, financial, medical, and retail/business contexts (Limkonchotiwat et al., 21 Aug 2025).

A defining property of the resource is that it is human-authored in Thai, rather than translated from English. The paper argues that translated benchmarks often preserve English task structure, rely on generic or culture-neutral prompts, do not reflect Thai law, Thai medicine, Thai finance, or Thai retail realities, and therefore can overestimate model quality for real Thai deployment. WangchanThaiInstruct instead uses Thai web sources, Thai domain experts, culturally tagged examples, and task formulations written directly in Thai by native annotators (Limkonchotiwat et al., 21 Aug 2025).

This design also frames a recurrent misconception in multilingual LLM evaluation: fluent Thai output is not equivalent to correct or culturally grounded Thai reasoning. The zero-shot analysis explicitly reports that models can produce fluent Thai while remaining wrong on factual or culturally specific questions, which positions WangchanThaiInstruct as a benchmark for alignment quality rather than surface fluency alone (Limkonchotiwat et al., 21 Aug 2025).

2. Corpus composition and benchmark structure

WangchanThaiInstruct spans four professional domains—Finance, Legal, Medical, and Retail—and seven task types: Brainstorming, Classification, Closed QA, Creative Writing, Multiple Choice, Open QA, and Summarization (Limkonchotiwat et al., 21 Aug 2025). This mixture is methodologically important because it combines objective tasks with open-ended generation, allowing evaluation of correctness, reasoning, fluency, and cohesiveness within one benchmark.

The source collection is large and document-grounded. The paper reports 27,351 documents collected from 86 Thai websites, mostly government agencies and publicly listed companies. HTML was cleaned, content-rich documents were retained, and deduplication was performed using mUSE with cosine similarity threshold 0.8 (Limkonchotiwat et al., 21 Aug 2025). The dataset also includes substantial topical breadth within domains, with 18 subtopics in finance, 133 subtopics in legal, and 28 subtopics in medical; retail subtopics are also present (Limkonchotiwat et al., 21 Aug 2025).

A notable technical feature is the benchmark’s long-context regime. The test set contains contexts ranging from 6 to 26,405 tokens, making the resource relevant not only for Thai instruction following and domain grounding, but also for long-context retrieval and reasoning behavior (Limkonchotiwat et al., 21 Aug 2025). This long-context property materially increases task difficulty, especially for tasks that require cross-document or document-wide evidence aggregation.

The dataset’s identity in the literature has evolved. A 2024 synthetic-data study treats WangchanThaiInstruct as a benchmark with 6,287 samples total, split into Thai culture-specific and general versions, across Legal, Medical, and Finance, while using the same seven task families (Pengpun et al., 2024). The later WangchanThaiInstruct paper presents the larger 35,014-sample human-authored release with explicit train and test splits (Limkonchotiwat et al., 21 Aug 2025). This suggests that the benchmark name was already in use before the later expanded dataset formulation.

3. Annotation pipeline, expert review, and licensing

The construction pipeline is explicitly multi-stage and combines annotators, domain experts, and AI researchers. In the first stage, annotators had to be native Thai speakers and pass an annotation exam covering all seven task types. Each annotator received a Thai website document, a task type, and instructions to create both the question or instruction and the gold answer. They were also asked to provide reasoning for answers in most task types, except creative writing, brainstorming, and summarization (Limkonchotiwat et al., 21 Aug 2025).

The second stage performed question-level quality control by annotators. A sample of 10% of examples was re-checked and edited for question correctness, instruction quality, answer correctness, and reasoning correctness (Limkonchotiwat et al., 21 Aug 2025). The third stage introduced domain-specialist verification in medical, legal, and finance. Experts could accept examples, reject them with reasons and references, or send them back for revision; they also added a cultural tag indicating whether the sample was Thai-specific. The paper states that this process was repeated until all expert-reviewed data were accepted (Limkonchotiwat et al., 21 Aug 2025).

The final stage was formatting control by AI researchers, who enforced structures needed for instruction tuning, including proper multiple-choice formatting, full-sentence closed QA, and inclusion of rationale when needed (Limkonchotiwat et al., 21 Aug 2025). This stage matters because it makes the dataset usable not only as a benchmark but also as supervised fine-tuning data.

To reduce leakage, splitting was performed at the document level, so all samples from the same source document remain in the same split (Limkonchotiwat et al., 21 Aug 2025). The release uses a mixed license: 30,000 samples under CC BY-NC and 5,014 samples under CC BY-SA 4.0 (Limkonchotiwat et al., 21 Aug 2025).

4. Evaluation protocol and metric design

The WangchanThaiInstruct paper explicitly rejects exclusive reliance on surface-overlap metrics such as BLEU or ROUGE-L for instruction-following evaluation. Its main protocol adapts an LLM-as-a-judge setup inspired by MT-Bench. For tasks with objective answers, the judge evaluates both correctness and reasoning quality; for creative tasks, it evaluates fluency and cohesiveness (Limkonchotiwat et al., 21 Aug 2025).

For definitive-answer tasks, the judging prompt requires the model to output Correctness: [[1]] if correct or [[0]] if incorrect, together with Rating: [[<score>]] on a 1–10 scale for reasoning quality. The prompt explicitly conditions on the user question, the reference answer, and the assistant answer (Limkonchotiwat et al., 21 Aug 2025). In the appendix, judge models are compared against human rankings on 1,200 sampled examples using Kendall’s $\tau$. The reported average values are GPT-4.1: 0.1259, Gemini 2.5: 0.1255, Claude Sonnet 3.7: 0.0962, BLEU: 0.0319, and ROUGE-L: 0.0053; GPT-4.1 is chosen because it has the best overall alignment (Limkonchotiwat et al., 21 Aug 2025). The paper also reports a Spearman correlation of 0.78 between reasoning rating and accuracy, indicating a strong association between better reasoning and correct answers (Limkonchotiwat et al., 21 Aug 2025).

Earlier work used WangchanThaiInstruct differently. The 2024 seed-free synthetic-data paper evaluates on the benchmark using WangchanX-10k-style metrics—BLEU, METEOR, ChrF, ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-Lsum, and BERTScore—and states that BERTScore is the most reliable metric for this benchmark because it measures semantic similarity better than n-gram overlap metrics (Pengpun et al., 2024). Taken together, these two uses show a transition from metric-based benchmarking toward judge-based evaluation tailored to reasoning-heavy Thai instruction following.

5. Zero-shot findings and difficulty profile

The zero-shot study evaluates Gemini 2.0, Qwen2.5-7B, Qwen2.5-72B, Llama-3.1-8B, Llama-3.1-70B, Sailor2-8B, and Sailor2-20B under a uniform prompt and zero-shot setup (Limkonchotiwat et al., 21 Aug 2025). A consistent result is that cultural samples are harder than general samples, particularly in Multiple Choice, Open QA, and Legal tasks. The paper’s interpretation is that Thai fluency by itself does not provide the cultural and professional grounding needed for these items (Limkonchotiwat et al., 21 Aug 2025).

The domain-level difficulty ordering is especially clear. Legal is the hardest domain, and no model exceeds 73% accuracy there. By contrast, Medical is the easiest domain, with Gemini 2.0 reaching 88.71% (Limkonchotiwat et al., 21 Aug 2025). The paper attributes legal difficulty, at least in part, to the scarcity of legal data in pretraining and to the local specificity of Thai legal systems.

The benchmark also overturns a standard intuition about multiple-choice tasks. Although prior work often treats multiple choice as comparatively easy for LLMs, WangchanThaiInstruct reports lower-than-expected multiple-choice accuracy because these items require correct reasoning rather than superficial answer-pattern selection (Limkonchotiwat et al., 21 Aug 2025). This is one reason the benchmark is harder than translated MCQ sets.

Long-context analysis further sharpens the difficulty profile. The study groups examples into head (shortest 20%), body (middle 40%), and tail (longest 40%), and finds that long context degrades performance especially on Brainstorming, Multiple Choice, Open QA, and Summarization; even strong models degrade on long inputs, and Sailor2-20B drops substantially on tail-group summarization (Limkonchotiwat et al., 21 Aug 2025).

6. Instruction tuning and the effect of native Thai supervision

The instruction-tuning study fine-tunes Llama-3.1-8B, Gemma-2-9B, and SEA-LIONv2-8B using LlamaFactory with the configuration $\text{Learning rate} = 2 \times 10^{-4}$, $\text{schedule} = \text{cosine}$, $\text{effective batch size} = 128$, $\text{max token length} = 2048$, $\text{warmup ratio} = 0.1$, and $\text{epochs} = 3$ (Limkonchotiwat et al., 21 Aug 2025). The comparison datasets are translated Thai instruction corpora, specifically Alpaca-cleaned-52k-th and Dolly-15k-th (Limkonchotiwat et al., 21 Aug 2025).

In the full-data comparison, models trained on WangchanThaiInstruct combined with Alpaca or Dolly improve on both in-domain WangchanThaiInstruct and out-of-domain Thai benchmarks, with the new data outperforming Alpaca- or Dolly-only training in 31 out of 42 cases (Limkonchotiwat et al., 21 Aug 2025). The paper emphasizes that gains are clear and consistent in-domain, while out-of-domain results are more mixed on NLG metrics such as BLEU and ROUGE-L.

The balanced ablation is more probative because it size-matches the mixtures: Alpaca 5k + WangchanThaiInstruct 5k, Alpaca 10k + WangchanThaiInstruct 10k, Alpaca 15k + WangchanThaiInstruct 15k, and corresponding Dolly mixtures (Limkonchotiwat et al., 21 Aug 2025). Under these matched conditions, adding WangchanThaiInstruct still improves performance substantially. The reported counts are Llama-3.1-8B: better in 31/42 cases, Gemma-2-9B: better in 41/42 cases, and SEA-LIONv2-8B: better in 28/42 cases (Limkonchotiwat et al., 21 Aug 2025). The strongest gains are on in-domain accuracy, reasoning ratings, and fluency, while out-of-domain improvements are more reliable for NLU and in-domain instruction following than for NLG.

The broader implication drawn in the paper is that native Thai supervision contributes something not recoverable from translated supervision alone. Translated datasets may still help, but WangchanThaiInstruct yields clearer advantages in domain understanding, culturally accurate responses, and instruction alignment (Limkonchotiwat et al., 21 Aug 2025).

7. Use in subsequent Thai LLM research

WangchanThaiInstruct has also entered the Thai LLM ecosystem as a resource for both benchmarking and data construction. In Typhoon 2, it is not the sole or central dataset, but it appears in two specific roles. First, it is used as positive data for training a Thai high-quality text classifier that filters the continual-pretraining corpus. Second, it contributes random queries to the TyphoonIF instruction-tuning dataset used in supervised fine-tuning for Typhoon2-Text (Pipatanakul et al., 2024). The Typhoon 2 paper is explicit that its instruction-tuned models are produced from a broader multilingual and multi-stage post-training stack, so WangchanThaiInstruct is one ingredient in a larger pipeline rather than the defining supervision source (Pipatanakul et al., 2024).

The 2024 seed-free synthetic-data study uses WangchanThaiInstruct primarily as an evaluation benchmark for small synthetic Thai SFT datasets. Its best 5,000-instruction dataset, F+ C+ D+, is reported as the second-best model on BERTScore on both benchmark splits. On the Thai Culture Test Set, it reaches 69.50, compared with 68.80 for WangchanX, 64.50 for OpenThaiGPT, and 74.10 for Typhoon; on the General Test Set, it reaches 73.20, compared with 72.20, 67.50, and 76.50, respectively (Pengpun et al., 2024). In that paper, WangchanThaiInstruct functions as a stress test for three dataset properties—fluency, diversity, and cultural context—and the consistent advantage of F+ C+ D+ supports the view that Thai-native fluency and cultural grounding are materially important for performance on the benchmark (Pengpun et al., 2024).

A parallel Thai instruction-following line in audio LLMs does not directly use WangchanThaiInstruct. The Thai audio study on Typhoon-Audio instead relies on SelfInstruct-Th, Airoboros-Th, Alpaca-Th, translated Thai prompt-response pairs, and Thai speech-instruction mixtures for speech instruction following (Manakul et al., 2024). A plausible implication is that WangchanThaiInstruct currently has its clearest direct role in text-centric evaluation and text instruction tuning, while closely related Thai instruction-following work in speech has, so far, depended on other Thai supervision sources.