ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation (2407.06135v1)

Abstract: Previous open-source large multimodal models (LMMs) have faced several limitations: (1) they often lack native integration, requiring adapters to align visual representations with pre-trained LLMs; (2) many are restricted to single-modal generation; (3) while some support multimodal generation, they rely on separate diffusion models for visual modeling and generation. To mitigate these limitations, we present Anole, an open, autoregressive, native large multimodal model for interleaved image-text generation. We build Anole from Meta AI's Chameleon, adopting an innovative fine-tuning strategy that is both data-efficient and parameter-efficient. Anole demonstrates high-quality, coherent multimodal generation capabilities. We have open-sourced our model, training framework, and instruction tuning data.

Anole: Open Autoregressive Multimodal Models for Image-Text Generation (without Diffusion)

The paper presents Anole, an innovative large multimodal model (LMM) designed for the interleaved generation of images and text. Anole addresses significant limitations observed in previous open-source LMM projects by adopting an autoregressive, token-based approach that eliminates the dependency on diffusion models.

Background and Motivation

The landscape of open-source LLMs has rapidly evolved, giving rise to various autoregressive models like LLaMA, Alpaca, and Vicuna. However, progress in the development of LMMs has been considerably slower, with most models either focusing solely on multimodal understanding or relying on additional mechanisms such as diffusion models for vision generation.

Chameleon by Meta AI, a notable advancement in this field, combines early-fusion, token-based autoregressive techniques to model multimodal sequences effectively. However, the open-source version lacks image generation capabilities, which is where Anole makes a significant contribution by building on Chameleon’s foundation to enable robust image and multimodal generation.

Key Contributions

Anole introduces several innovations:

1. Full Open-Source Implementation: Anole provides a comprehensive open-source framework that enables vision and multimodal generation capabilities through an advanced fine-tuning approach. This release is designed to spur further research and development.
2. Efficient Fine-Tuning: The model is fine-tuned with fewer than 40M parameters using around 6,000 samples, demonstrating remarkable efficiency in incorporating complex functionality.
3. Training and Multimodal Framework: Anole includes a unified tokenizer-based multimodal training and inference framework, facilitating accessible development and experimentation.
4. Extensive Resources: The project provides a wealth of data resources and tutorials to support a broad range of researchers.

Methodology

Anole's architecture mirrors that of Chameleon, leveraging early-fusion, token-based autoregressive modeling. The model handles multimodal integration at the token level, streamlining image-text sequence generation. By freezing most of Chameleon's parameters and fine-tuning the logits corresponding to image token IDs in the transformer's output head layer, Anole effectively extends Chameleon's capabilities to cover image generation without compromising its existing strengths.

Anole's fine-tuning draws on a modest dataset yet demonstrates high quality and coherence in generating interleaved image-text sequences. For instance, the model is capable of generating detailed steps in a recipe and illustrating each step with relevant images.

Evaluation

Anole's performance is evaluated qualitatively through several scenarios:

• Image Generation: Anole produces high-quality images that are faithful to their textual prompts. Its ability to generate both realistic scenes and imaginative depictions highlights its versatility.
• Interleaved Image-Text Generation: The model excels in generating coherent sequences where text and images complement each other. Examples in the paper include detailed recipes and comprehensive descriptions of geographical and cultural subjects, enhanced with relevant imagery.

Implications and Future Directions

The contributions of Anole have practical and theoretical implications. Practically, the release of Anole democratizes access to advanced multimodal AI technologies, offering a robust, efficient tool for various applications, from educational content generation to interactive storytelling. Theoretically, Anole opens new research avenues. Future inquiries may explore the limits of vision generation using this unified token-based approach, develop optimal fine-tuning techniques, and ensure the ethical application of generated content.

Conclusion

Anole represents an important step in advancing LMMs, combining image and multimodal generation capabilities without relying on additional complex mechanisms like diffusion models. The open-source nature of Anole, paired with its efficient fine-tuning and robust performance, makes it a valuable asset for the research community, paving the way for further exploration and innovation in multimodal AI.