Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Typhoon: Thai Large Language Models (2312.13951v1)

Published 21 Dec 2023 in cs.CL and cs.AI

Abstract: Typhoon is a series of Thai LLMs developed specifically for the Thai language. This technical report presents challenges and insights in developing Thai LLMs, including data preparation, pretraining, instruction-tuning, and evaluation. As one of the challenges of low-resource languages is the amount of pretraining data, we apply continual training to transfer existing world knowledge from a strong LLM. To evaluate the Thai knowledge encapsulated in each model from the pretraining stage, we develop ThaiExam, a benchmark based on examinations for high-school students and investment professionals in Thailand. In addition, we fine-tune Typhoon to follow Thai instructions, and we evaluate instruction-tuned models on Thai instruction datasets as well as translation, summarization, and question-answering tasks. Experimental results on a suite of Thai benchmarks show that Typhoon outperforms all open-source Thai LLMs, and its performance is on par with GPT-3.5 in Thai while having only 7 billion parameters and being 2.62 times more efficient in tokenizing Thai text.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528, 2023.
  2. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online, July 2020. Association for Computational Linguistics.
  3. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
  4. BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
  5. CrossSum: Beyond English-centric cross-lingual summarization for 1,500+ language pairs. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2541–2564, Toronto, Canada, July 2023. Association for Computational Linguistics.
  6. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  7. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  8. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  9. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.
  10. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672, 2022.
  11. Efficiently adapting pretrained language models to new languages. arXiv preprint arXiv:2311.05741, 2023.
  12. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Singapore, December 2023. Association for Computational Linguistics.
  13. Common Crawl Foundation. Statistics of common crawl monthly archives by commoncrawl. https://commoncrawl.github.io/cc-crawl-statistics/plots/languages, 2023.
  14. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online, August 2021. Association for Computational Linguistics.
  15. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
  16. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
  17. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  18. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
  19. Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models. arXiv preprint arXiv:2309.06085, 2023.
  20. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023.
  21. Holistic evaluation of language models, 2023.
  22. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  23. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  24. LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models, 2023.
  25. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
  26. Wangchanberta: Pretraining transformer-based thai language models. arXiv preprint arXiv:2101.09635, 2021.
  27. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023.
  28. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online, July 2020. Association for Computational Linguistics.
  29. Cedille: A large autoregressive french language model. arXiv preprint arXiv:2202.03371, 2022.
  30. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042, Online, November 2020. Association for Computational Linguistics.
  31. Seallms–large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023.
  32. The National Institute of Educational Testing Service. Basic statistical values of o-net test results. https://www.niets.or.th/th/content/view/11821, 2021.
  33. Consortium of Thai Medical Schools. Scores report of the tpat1 exam for thai medical schools admission. https://www9.si.mahidol.ac.th/cotmes_stat.html, 2023.
  34. Council of University Presidents of Thailand. Basic statistical report tgat/tpat examination. https://www.mytcas.com/stat/, 2023.
  35. OpenAI. GPT-4 technical report, 2023.
  36. OpenThaiGPT. Released openthaigpt 7b 1.0.0-beta. https://openthaigpt.aieat.or.th/, 2023.
  37. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online, July 2020. Association for Computational Linguistics.
  38. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  39. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
  40. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only, 2023.
  41. Sabi\\\backslash\’a: Portuguese large language models. arXiv preprint arXiv:2304.07880, 2023.
  42. WangChanGLM — The Multilingual Instruction-Following Model, April 2023.
  43. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online, November 2020. Association for Computational Linguistics.
  44. Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
  45. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  46. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023.
  47. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022.
  48. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023.
  49. It5: Large-scale text-to-text pretraining for italian language understanding and generation. arXiv preprint arXiv:2203.03759, 2022.
  50. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149, 2023.
  51. AI Singapore. Sea-lion (southeast asian languages in one network): A family of large language models for southeast asia. https://github.com/aisingapore/sealion, 2023.
  52. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  53. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  54. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  55. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  56. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.
  57. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
  58. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  59. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023.
  60. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online, June 2021. Association for Computational Linguistics.
  61. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023.
  62. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. arXiv preprint arXiv:2306.05179, 2023.
  63. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
Citations (14)
View on Semantic Scholar

Summary

Introduction to Typhoon

In the field of LLMs, various models like GPT and Llama have established impressive benchmarks in NLP tasks, heavily relying on English-centric datasets. However, for the Thai language—a low-resource language spoken by over 70 million people—the landscape is sparse, as its representation in global data sets is minimal. Recognizing the linguistic and cultural richness of Thai that existing multilingual and general LLMs might not fully capture, the introduction of Typhoon, a 7-billion parameter Thai LLM, is a significant development. Typhoon aims to encapsulate Thai knowledge better by pretraining on a specifically curated Thai corpus.

Challenges and Data Preparation

Developing Thai-specific LLMs faces unique hurdles, chiefly the scarcity of large-scale Thai data. Typhoon's creators tackled this by sourcing Thai internet data, which constitutes less than half a percent of the Common Crawl data, and addressed quality issues by applying rigid deduplication methods. A meticulous methodology — from increasing dataset size to implementing heuristic filtering — led to a refined 3 TB Thai corpus integrated with English data to preserve English-language skills. They also devised a more efficient tokenizer for Thai text, achieving 2.62 times better efficiency than GPT-4.

Evaluations and Results

The evaluation of Typhoon’s Thai knowledge was meticulously conducted using ThaiExam, an innovative benchmark comprising high-school and professional examinations in Thailand. Furthermore, its capability to follow Thai instructions was rigorously tested against a suite of Thai benchmarks, where Typhoon outperformed existing open-source models and even held ground against proprietary models like GPT-3.5 while demonstrating greater efficiency in tokenizing Thai text.

Instruction Tuning and Future Directions

Due to the lack of Thai instruction-tuning data, an innovative approach was adopted by translating English datasets to Thai or generating instruction sets using the Self-Instruct technique. Through such Supervised Fine-Tuning (SFT), Typhoon was aligned closely with user intent. In zero-shot NLP tasks, Typhoon's performance was superior across a majority of metrics compared to other models. Moving forward, expansions in pretraining data and the application of larger base models could enhance Typhoon's capabilities, which is made possible as Typhoon is publicly available under the Apache-2.0 license, encouraging further research and development.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube