The Evolution of Multimodal Model Architectures (2405.17927v1)

Abstract: This work uniquely identifies and characterizes four prevalent multimodal model architectural patterns in the contemporary multimodal landscape. Systematically categorizing models by architecture type facilitates monitoring of developments in the multimodal domain. Distinct from recent survey papers that present general information on multimodal architectures, this research conducts a comprehensive exploration of architectural details and identifies four specific architectural types. The types are distinguished by their respective methodologies for integrating multimodal inputs into the deep neural network model. The first two types (Type A and B) deeply fuses multimodal inputs within the internal layers of the model, whereas the following two types (Type C and D) facilitate early fusion at the input stage. Type-A employs standard cross-attention, whereas Type-B utilizes custom-designed layers for modality fusion within the internal layers. On the other hand, Type-C utilizes modality-specific encoders, while Type-D leverages tokenizers to process the modalities at the model's input stage. The identified architecture types aid the monitoring of any-to-any multimodal model development. Notably, Type-C and Type-D are currently favored in the construction of any-to-any multimodal models. Type-C, distinguished by its non-tokenizing multimodal model architecture, is emerging as a viable alternative to Type-D, which utilizes input-tokenizing techniques. To assist in model selection, this work highlights the advantages and disadvantages of each architecture type based on data and compute requirements, architecture complexity, scalability, simplification of adding modalities, training objectives, and any-to-any multimodal generation capability.

• The paper identifies four distinct multimodal architectures based on unique fusion strategies within deep neural networks.
• It details the use of deep fusion versus early fusion techniques to balance computational cost and model flexibility.
• The study outlines a roadmap for future multimodal AI by analyzing trends in architecture design and integration methods.

The Evolution of Multimodal Model Architectures

The paper "The Evolution of Multimodal Model Architectures" by Shakti N. Wadekar et al. provides a comprehensive exploration and categorization of contemporary multimodal model architectures. This paper's primary contribution is the systematic identification of four distinct architectural types prevalent in the domain of multimodal models: Type-A, Type-B, Type-C, and Type-D. Each architectural type is characterized by its unique approach to integrating multimodal inputs within deep neural networks, offering a structured framework for understanding the evolution and development of multimodal models.

Categorization of Multimodal Model Architectures

The categorization is primarily based on the fusion stage of the multimodal inputs:

• Type-A and Type-B utilize deep fusion within the internal layers of the model.
• Type-C and Type-D emphasize early fusion at the input stage.

Type-A: Standard Cross-Attention based Deep Fusion (SCDF)

In Type-A, multimodal inputs are deeply fused using standard cross-attention within the internal layers of a pretrained LLM. This type is further divided into two subtypes:

• Subtype A.1 integrates cross-attention layers before each self-attention layer in the decoder (e.g., Flamingo, OpenFlamingo).
• Subtype A.2 places the cross-attention layers post self-attention in an encoder-decoder setup (e.g., VL-BART, VL-T5).

Type-A models generally require substantial computational resources and training data. Examples include the Flamingo and OpenFlamingo models, which have shown proficiency in handling vision-language tasks by incorporating image and text data via a robust transformer architecture.

Type-B: Custom Layer based Deep Fusion (CLDF)

Type-B models also achieve deep fusion within internal layers but through custom-designed layers rather than standard cross-attention. This type is categorized into:

• Subtype B.1 uses custom cross-attention layers (e.g., CogVLM, LLaMA-Adapter).
• Subtype B.2 employs other custom learnable layers (e.g., InternLM-XComposer2, MoE-LLaVA).

Type-B architecture balances the need for computational efficiency and fine-grained modality control, often requiring fewer resources compared to Type-A while offering improved flexibility in customizing layer behavior.

Type-C: Non-Tokenized Early Fusion (NTEF)

Type-C represents the most prevalent and modular architectural type, characterized by early fusion at the input stage without tokenizing the inputs. This type is divided based on the type of module connecting the modality encoders to the LLM:

• Subtype C.1 uses Linear Layer/MLP (e.g., LLaVA, PaLM-E).
• Subtype C.2 employs Q-former and Linear Layer/MLP (e.g., BLIP-2, MiniGPT-4).
• Subtype C.3 utilizes Perceiver Resampler (e.g., Kosmos-G, Monkey).
• Subtype C.4 includes custom learnable layers (e.g., Video-ChatGPT, Qwen-VL).

Type-C architectures are known for their modularity, ease of construction, and efficient training processes, often leveraging pre-trained components for vision and text alignment. They offer an effective balance of simplicity and performance across diverse multimodal tasks.

Type-D: Tokenized Early Fusion (TEF)

Type-D involves tokenizing the multimodal inputs and feeding them directly into the model, accommodating autoregressive training objectives across modalities. This type splits into:

• Subtype D.1 utilizing LLM (e.g., CM3Leon, TEAL).
• Subtype D.2 based on encoder-decoder style transformers (e.g., Unified-IO, 4M).

These models are trained to generate discrete tokens for multiple modalities, allowing comprehensive training using a standard autoregressive objective. However, this often demands extensive computational resources and sophisticated training strategies.

Implications and Future Directions

The categorization of multimodal architectures not only facilitates a systematic understanding of existing models but also provides a foundation for tracking and predicting future trends in multimodal AI.

Implications:

• Practical: The detailed taxonomy assists researchers and practitioners in selecting appropriate models based on specific requirements such as data and compute efficiency, scalability, and modality fusion complexity.
• Theoretical: The distinctions between deep versus early fusion architectures provide insights into the trade-offs between model flexibility and computational efficiency.

Future Directions:

• The paper suggests that future developments in multimodal AI may see a convergence towards more integrated architectures, where the benefits of deep and early fusion are combined.
• The exploration of SSMs (State-Space Models) as alternatives to transformer-based architectures presents an exciting avenue for addressing the quadratic complexity challenges of attention mechanisms, potentially leading to more efficient any-to-any multimodal models.

Conclusion

This paper represents a pivotal work in characterizing and understanding multimodal model architectures, offering a valuable framework for both academic research and practical implementation. By elucidating the taxonomy and comparative advantages of Type-A, B, C, and D architectures, the authors provide a roadmap for the evolution of multimodal models, fostering advancements in the creation and deployment of sophisticated AI systems capable of seamlessly integrating and processing diverse modalities. The paper's impact is likely to extend beyond current state-of-the-art models, informing the design of next-generation architectures and applications in multimodal AI.