---
title: 'PhoGPT: Vietnamese LLM Benchmark'
url: https://www.emergentmind.com/topics/phogpt
type: topic
---

# PhoGPT: Vietnamese LLM Benchmark

PhoGPT refers to a class of large, open-source Vietnamese generative language models based on the Transformer architecture, designed and released by VinAI Research in 2023. PhoGPT comprises both a base pre-trained monolingual model (PhoGPT-4B) and an instruction-following chat variant (PhoGPT-4B-Chat), establishing itself as a benchmark series for Vietnamese large language modeling. The models are engineered specifically for the Vietnamese language, utilizing a large-scale, diverse dataset and benefiting from state-of-the-art advances in Transformer implementations and efficient training strategies [2311.02945].

## 1. Model Architecture and Tokenization

PhoGPT-4B is a decoder-only transformer that follows the standard transformer decoder stack, augmented with modern optimizations for large-scale training and inference. The key architectural details include:

- **Parameter Count:** 3.7 billion (often rounded to "4B" for family designation).
- **Depth and Width:** 32 transformer layers ($n_\text{layers}=32$), model dimension $d_\text{model}=3072$, multi-head attention with 24 heads.
- **Attention Optimizations:** Incorporates Triton-based flash attention, enabling memory-efficient high-throughput operations and scalability to long contexts, as well as ALiBi (Attention with Linear Biases), which enables robust extrapolation to longer sequences.
- **Context Window:** Maximally 8192 tokens, supporting both extended prompt chaining and long-form document modeling.
- **Tokenizer:** A byte-level BPE tokenizer tailored for Vietnamese with a vocabulary of 20,480 token types. This choice balances coverage of Vietnamese morphological variation, word segmentation, and semantic preservation by reducing over-segmentation common with smaller vocabulary sizes.

The model complies with engineering standards for integration within popular frameworks such as "transformers", "vllm", and "llama.cpp".

## 2. Training Data and Pre-training Regimen

PhoGPT-4B is trained from scratch on an extensive and diverse Vietnamese corpus comprising 102 billion tokens (totalling 482 GB post-cleaning and deduplication):

- **Text Sources Include:**
  - 1GB Vietnamese Wikipedia dump (May 2023 snapshot)
  - 1.5GB medical texts (various public/clinical sources)
  - 3GB literature corpus (public domain books)
  - 12GB legal documents (from Vietnamese law databases)
  - 40GB news corpus variant
  - 88GB OSCAR-2301 Vietnamese data
  - 336GB Vietnamese subset of mC4

Two full epochs are performed over the dataset to ensure sufficient token exposure for rare and complex linguistic phenomena.

**Fine-tuning (PhoGPT-4B-Chat):**
- **Instruction Tuning:** 70,000 human-generated prompt-response pairs spanning poetry, essays, corrections, summarizations, Q&A, etc.
- **Conversational Tuning:** 290,000 Vietnamese dialogue samples from Bactrian-X (67K), ChatAlpaca (20K), ShareGPT (40K, filtered for non-code/mathematical), and UltraChat (230K), all either originally in or translated into Vietnamese.

This multi-stage fine-tuning enables enhanced conversational fluency, instruction-following, and context-appropriate response generation for Vietnamese users.

## 3. Performance Evaluation and Comparative Benchmarks

PhoGPT-4B-Chat is empirically benchmarked against both open-source and commercial models:

- **ViTruthfulQA Evaluation:** On a 199-question Vietnamese question-answering dataset, the model achieves an overall accuracy of 41.7% (83/199 questions correct), with leading results (43.5%, 64/147) on Vietnam-specific categories. Tasks involve fact verification and contextually specialized knowledge.
- **Comparison Set:** GPT-4-0125-preview, GPT-3.5-turbo, Gemini Pro 1.0, and multiple open-source Vietnamese models.
- **Relative Strengths:** PhoGPT-4B-Chat is either competitive with or outperforms commercial models in questions requiring deep understanding of Vietnamese context and culture.

Empirical metrics used focus on accuracy, with task-specific variants as relevant for open-domain QA and cultural knowledge probes.

## 4. Applications and Deployment Potential

PhoGPT is designed for a broad set of Vietnamese NLP tasks:

- **Conversational Agents:** Chatbots for customer service, personal assistants, and task automation.
- **Document Processing:** Summarization, translation, and information retrieval in long and complex Vietnamese texts by leveraging the 8192-token context capability.
- **Domain-Specific Question Answering:** Applications in legal, medical, and news reporting domains based on the model's exposure to specialized corpora.
- **Education and Content Creation:** Supporting instructional tools, educational chatbots, spelling/grammar correction, and creative writing.

The model's open-source nature and modular exportability enable integration within both research and production systems, fostering further fine-tuning and adaptation.

## 5. Technical Contributions and Implementation Strategies

PhoGPT’s architecture and approach reflect several technical advances:

- **Efficient Long-Context Attention:** The combination of flash attention and ALiBi yields practical training and inference at context sizes previously deemed operationally challenging.
- **Custom Language-Specific Preprocessing:** The byte-level BPE tokenizer explicitly addresses Vietnamese diacritics, compound words, and proper segmentation—allowing the vocabulary of 20,480 types to balance granularity and modeling efficiency.
- **Scalability:** Designed for compatibility with distributed and hardware-accelerated training/inference stacks, supporting deployment ranging from consumer hardware (via quantized “llama.cpp” inference) to high-throughput clusters.
- **Fine-tuning at Scale:** The multi-source, large-scale instructional and conversational data makes PhoGPT especially responsive in few-shot and instructional scenarios.

## 6. Availability and Community Impact

PhoGPT-4B and PhoGPT-4B-Chat are distributed under an open-source license:

- **Model Access:** All model weights and inference code are available via the project repository:  
  https://github.com/VinAIResearch/PhoGPT
- **Ecosystem Integration:** Ready compatibility with major LLM deployment frameworks.
- **Research Utility:** As the leading Vietnamese-specific LLM, PhoGPT serves as a baseline for research in low-resource language modeling, cross-lingual transfer, dialogue systems, and instruction tuning in non-English contexts.

The model’s open-source release is expected to stimulate advances in Vietnamese NLP, resource-efficient modeling for low-resource languages, and broader Southeast Asian NLP infrastructure development.

## 7. Context and Significance in the Language Model Landscape

PhoGPT is positioned as the de facto Vietnamese open large language model series, analogous in scope to Llama for English and multi-lingual contexts. Its design principles—training from scratch on a curated, massive, language-specific corpus; large context handling; and modular fine-tuning—reflect broader shifts towards high-performance, culturally-relevant large language models in global NLP research and development.

Source: https://www.emergentmind.com/topics/phogpt