PALO: A Polyglot Large Multimodal Model for 5B People (2402.14818v2)

Abstract: In pursuit of more inclusive Vision-LLMs (VLMs), this study introduces a Large Multilingual Multimodal Model called PALO. PALO offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned LLM, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.

• The paper presents Palo, a multilingual multimodal model that employs a semi-automated LLM-based translation process to adapt vision-language datasets.
• The study scales the model across 1.7B, 7B, and 13B parameters, achieving notable performance improvements, especially in underrepresented languages.
• The approach aligns vision encoders with language models to deliver dynamic, multilingual responses, significantly advancing global AI accessibility.

Palo: Bridging Linguistic Divides in Vision-Language Modeling for Global Accessibility

Introduction to Palo

Recent advancements in Generative Artificial Intelligence have ushered in the era of Large Multimodal Models (LMMs), which have shown promising results in synthesizing textual responses from visual inputs. However, these models have been predominantly centered around English, creating a significant linguistic gap in Vision-LLMs (VLMs) for non-English languages. Addressing this gap, this paper introduces Palo, a Large Multilingual Multimodal Model, designed to offer visual reasoning capabilities across ten major languages covering approximately 65% of the world's population. These languages include English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese.

Palo distinguishes itself by employing a semi-automated translation approach leveraging a fine-tuned LLM for dataset adaptation. This method ensures linguistic fidelity across languages with minimal manual effort, facilitating scalability. Notably, the model has been trained across multiple scales (1.7B, 7B, and 13B parameters), showcasing substantial performance enhancements over strong baselines, especially in underrepresented languages.

Architectural Overview

Palo integrates a vision encoder with a LLM to process both the input image and the user's text query, generating a natural language response. It uses CLIP ViT-L/14 as the vision encoder and employs both a two-layer MLP projector and a Lightweight Downsample Projector (LDP) depending on the variant (1.7B, 7B, and 13B). The LDP, used in the MobilePalo-1.7B variant, utilizes depth-wise separable convolutions offering a compute-efficient solution. This architectural design aligns vision features with the LLM's input embedding space, facilitating dynamic response generation in ten different languages.

Multilingual Dataset and Training

The creation of a comprehensive multilingual vision-language instruction-tuning dataset is a cornerstone of Palo's development. This dataset represents a critical effort in expanding the model's linguistic scope significantly. Utilizing a semi-automated translation pipeline grounded on a state-of-the-art LLM, the English dataset was adapted to the ten target languages, addressing common linguistic challenges through a mix of automated and manual verification processes. This meticulous dataset refinement allowed Palo to achieve enhanced linguistic accuracy and proficiency in generating content across the selected languages.

Experimental Findings and Implications

The findings from the Palo model underscore the viability of creating a unified multilingual LMM capable of high-performance across a variety of languages, including those with fewer resources. Specifically, Palo demonstrated substantial improvement in its ability to process and generate content for low-resource languages (e.g., Hindi, Arabic, Bengali, and Urdu) without detriment to its performance in high-resource languages. These advancements not only present practical implications for making AI more inclusive but also provide a significant leap toward bridging the linguistic divide in AI applications.

Future Directions

While Palo represents a significant stride towards inclusivity, the need for further linguistic expansion remains, as two-thirds of the world's languages are covered, leaving out a substantial number of languages and dialects. Future efforts could extend the model's linguistic repertoire, further closing the gap in multilingual VLM accessibility. Additionally, given the semi-automated nature of the translation process employed, further refinements in contextual and cultural nuance comprehension could enhance the model's applicability and effectiveness.

Challenges and Considerations

The endeavor of creating a globally accessible VLM like Palo does not come without its challenges and potential risks. Notably, the semi-automated translation process, while effective in scale, may not capture the full depth of cultural nuances across languages, potentially leading to biased interpretations. Rigorous evaluation and continuous refinement of the model are paramount to mitigate these risks and ensure Palo's responsible and beneficial use across diverse global communities.

Conclusion

Palo embodies the next evolutionary step in making AI technologies more accessible and inclusive. By effectively leveraging advanced translation methodologies and large-scale training approaches, this work paves the way for future advancements in the field. As the quest for truly global AI continues, Palo stands as a testament to the potential of multilingual and multimodal AI models to bridge the world's linguistic divides, opening new avenues for research and application in AI-driven technologies.