PhoGPT: Generative Pre-training for Vietnamese
This paper presents PhoGPT, a state-of-the-art generative pre-trained transformer model tailored for the Vietnamese language. The authors introduce two main components: PhoGPT-4B, which serves as the base model, and PhoGPT-4B-Chat, a variant fine-tuned for conversational tasks. With 3.7 billion parameters, the models are designed to leverage a comprehensive Vietnamese corpus, aiming to provide advanced LLMing capabilities for Vietnamese NLP tasks.
Model Development
PhoGPT-4B is built upon a Transformer decoder architecture with innovations such as Triton flash attention and ALiBi for effective context length extrapolation. It utilizes a byte-level BPE tokenizer with a vocabulary size of 20,480 tokens and is trained on a robust dataset of 102 billion tokens. This corpus covers diverse text types from Wikipedia to legal and medical texts, ensuring a broad linguistic foundation. The detailed token breakdown in the pre-training corpus signifies a commitment to covering various domains integral to understanding Vietnamese language nuances.
The fine-tuning process to create PhoGPT-4B-Chat involved a dataset of 70,000 instructional prompts with responses and 290,000 conversational exchanges. By drawing on datasets such as Bactrian-X and UltraChat, the model is optimized to handle a variety of conversational interactions in Vietnamese.
Performance Evaluation
PhoGPT-4B-Chat's performance was assessed against both closed-source models like Gemini Pro and GPT-3.5-turbo and open-source models such as Vistral-7B-Chat. Using the ViTruthfulQA dataset, which comprises 199 Vietnamese truthful questions, the model's accuracy demonstrated its competitiveness, achieving notable scores compared to larger 7B-parameter models. Specifically, PhoGPT-4B-Chat showed robust performance on Vietnam-specific questions, highlighting its specialized language capabilities.
Implications and Future Directions
The introduction of PhoGPT models is significant for Vietnamese NLP, offering an open-source alternative that aligns with the linguistic and cultural context of Vietnamese users. The models present potential for various applications, from natural language understanding to conversational AI. However, the authors acknowledge limitations in areas like reasoning and coding, which suggests avenues for future improvement and research.
In essence, PhoGPT opens pathways for further exploration in Vietnamese generative LLMs, encouraging advancements and broader participation in regional AI development. Future work can focus on expanding the model's capabilities and evaluating it across diverse real-world tasks to enhance applicability and reliability.