Sabiá-2: A New Generation of Portuguese Large Language Models (2403.09887v2)

Abstract: We introduce Sabi\'a-2, a family of LLMs trained on Portuguese texts. The models are evaluated on a diverse range of exams, including entry-level tests for Brazilian universities, professional certification exams, and graduate-level exams for various disciplines such as accounting, economics, engineering, law and medicine. Our results reveal that our best model so far, Sabi\'a-2 Medium, matches or surpasses GPT-4's performance in 23 out of 64 exams and outperforms GPT-3.5 in 58 out of 64 exams. Notably, specialization has a significant impact on a model's performance without the need to increase its size, allowing us to offer Sabi\'a-2 Medium at a price per token that is 10 times cheaper than GPT-4. Finally, we identified that math and coding are key abilities that need improvement.

Insights into "Sabiá-2: A New Generation of Portuguese LLMs"

The paper under review introduces Sabiá-2, a family of LLMs trained specifically on Portuguese texts. This research targets a pivotal area in computational linguistics: the adaptation and specialization of LLMs for linguistic and cultural contexts beyond the dominant global lingua franca, English. By focusing on Portuguese, the fifth most spoken language globally, the Sabiá-2 LLMs contribute to the growing recognition of the value in developing monolingual and culturally-tailored LLMs.

Key Findings and Contributions

1. Performance Benchmarks: Sabiá-2 models were evaluated using a comprehensive set of academic and professional exams, including entry-level university tests and professional certification exams in Brazil. The Sabiá-2 Medium model, the centerpiece of this paper, matches or surpasses GPT-4's performance in 23 out of 64 exams and outperforms GPT-3.5 in 58 out of those exams. These robust numerical results signify the proficiency of Sabiá-2 in handling assessments tailored to the Brazilian educational and professional landscape.
2. Cost Efficiency and Specialization: A standout feature of the Sabiá-2 Medium model is its cost-effectiveness. Despite its high-performance metrics, the model offers a pricing structure that is significantly more affordable—up to ten times cheaper than GPT-4 per token. This economic advantage is attributed to specialization strategies that enhance task efficiency without increasing model size.
3. Implications for Domain-Specific Specialization: The research underscores the potential gains of domain-specific specialization. By aligning training data with targeted linguistic and cultural domains, Sabiá-2 exemplifies how smaller, focused models can compete with, and often outperform, larger, more generalized ones in niche areas. This approach is parallel to observed trends in fields such as finance, medicine, and engineering, as mentioned in the paper.
4. Limitations and Future Directions: While Sabiá-2 models excel in many domains, their performance in math and coding reveals areas for further enhancement. The paper identifies these as key areas requiring improvement, aligning with the broader challenges faced by LLMs in handling complex numerical and logical reasoning tasks. This insight foreshadows a trajectory for future research focused on hybrid models that combine domain specialization with improved quantitative and structured problem-solving abilities.

Practical and Theoretical Implications

Practically, the Sabiá-2 models' proficiency on Brazilian benchmarks indicates their immediate applicability in educational platforms and professional certification processes in Portuguese-speaking regions. By lowering costs and maintaining high performance, Sabiá-2 holds promise for democratizing access to advanced AI-driven educational tools.

Theoretically, the paper contributes to discussions on the benefits of monolingual versus multilingual model training. It complements findings from other research advocating for language-specific pretraining, showcasing how monolingual models can capture linguistic intricacies more effectively than their multilingual counterparts.

Conclusion

The paper on Sabiá-2 presents a compelling case for specialized LLM development, emphasizing that language-specific training enriches both linguistic comprehension and cultural contextual understanding. Sabiá-2's success serves as a testament to the growing necessity of diversifying AI research beyond predominant languages, ensuring technologies align with the linguistic breadth and cultural nuances of global users. This specificity not only elevates model performance but also aligns with inclusive AI development principles.

As AI continues to evolve, research like that on Sabiá-2 underscores a shift towards nuanced, localized models that serve distinct communities with precision and affordability. The ongoing conversation in AI fields about monolingual versus multilingual approaches will benefit from such explorations, as they offer empirical evidence of the advantages inherent in targeted specialization.