Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier (2412.04261v1)

Abstract: We introduce the Aya Expanse model family, a new generation of 8B and 32B parameter multilingual LLMs, aiming to address the critical challenge of developing highly performant multilingual models that match or surpass the capabilities of monolingual models. By leveraging several years of research at Cohere For AI and Cohere, including advancements in data arbitrage, multilingual preference training, and model merging, Aya Expanse sets a new state-of-the-art in multilingual performance. Our evaluations on the Arena-Hard-Auto dataset, translated into 23 languages, demonstrate that Aya Expanse 8B and 32B outperform leading open-weight models in their respective parameter classes, including Gemma 2, Qwen 2.5, and Llama 3.1, achieving up to a 76.6% win-rate. Notably, Aya Expanse 32B outperforms Llama 3.1 70B, a model with twice as many parameters, achieving a 54.0% win-rate. In this short technical report, we present extended evaluation results for the Aya Expanse model family and release their open-weights, together with a new multilingual evaluation dataset m-ArenaHard.

• The paper details the Aya Expanse model family's novel use of multilingual data arbitrage, preference optimization, and model merging to enhance language processing.
• It shows that the 32B model outperforms larger models by achieving a 54% win rate on benchmarks like m-ArenaHard and Dolly.
• The study underscores democratizing AI by providing open model weights and evaluation datasets that encourage broader multilingual research.

Evaluation of the Aya Expanse Model Family for Multilingual Language Processing

The Aya Expanse model family represents a significant step forward in the development of multilingual LLMs (MLMs). These models, with 8B and 32B parameters respectively, offer insight into advancing the capabilities of multilingual AI while addressing the performance gaps associated with monolingual counterparts. This analysis focuses on the methodologies employed, the critical results derived, and the implications for future AI developments based on the research presented in the technical report.

Methodological Advances

Aya Expanse is characterized by its employment of innovative methods, including multilingual data arbitrage, preference optimization, and model merging. Central to the model's effectiveness is its approach to synthetic data generation through data arbitrage, which strategically samples from a diverse pool of teacher models. This strategy effectively mitigates the limitations faced when relying on a single-teacher model, thereby enhancing the quality of synthetic multilingual datasets.

The iterative multilingual preference training method is critical for aligning model outputs with human preferences across diverse languages. By generating high-quality multilingual preference data pairs, Aya Expanse successfully overcomes challenges related to multilingual optimization. Furthermore, the model merging approach aims to reduce computational costs while maintaining high performance across languages through cross-lingual transfer and language family diversity optimization.

Strong Numerical Results

The evaluations presented in the report demonstrate the superior performance of Aya Expanse models across several benchmarks. When evaluated on the m-ArenaHard and Dolly datasets, Aya Expanse models showed significant win-rate advantages over competitive models within their parameter class. Particularly noteworthy is the Aya Expanse 32B's ability to outperform Llama 3.1 70B, achieving a 54.0% win rate despite having fewer parameters.

Academic benchmarks also illustrate the Aya Expanse models' advantages. The models achieve high accuracy rates across a range of discriminative tasks, including XCOPA and XStoryCloze. The multilingual Global-MMLU task revealed a notable improvement, with Aya Expanse models surpassing their predecessors, Aya 23, and setting new performance standards.

Implications and Future Developments

The release of the Aya Expanse model family holds several implications for AI's multilingual capabilities. By providing open model weights and evaluation datasets, this work contributes to democratizing access to high-performance multilingual models, encouraging further research and development in this domain. The success of multilingual data arbitrage, as shown in this work, suggests potential applications in other areas of AI, especially where high-quality data is scarce or fragmented.

Moreover, the research underscores the importance of optimizing alignment techniques, such as preference training, for handling dual-language environments efficiently. This indicates a promising trajectory toward more inclusive language technologies in AI applications ranging from machine translation to multilingual conversational agents.

Conclusion

The Aya Expanse model family marks a significant advancement in the field of multilingual language processing, effectively bridging the gap between monolingual and multilingual model performance. By leveraging cutting-edge methodologies in data generation, preference alignment, and model merging, these models achieve outstanding results that not only enhance current technological limits but also set a foundation for subsequent innovation in AI. The paper's outcomes advocate for ongoing research focused on enriching AI's global applicability, ensuring diverse linguistic representations, and facilitating equitable access to technological advancements.