ViT-Lens: Towards Omni-modal Representations (2311.16081v2)

Abstract: Aiming to advance AI agents, large foundation models significantly improve reasoning and instruction execution, yet the current focus on vision and language neglects the potential of perceiving diverse modalities in open-world environments. However, the success of data-driven vision and LLMs is costly or even infeasible to be reproduced for rare modalities. In this paper, we present ViT-Lens-2 that facilitates efficient omni-modal representation learning by perceiving novel modalities with a pretrained ViT and aligning them to a pre-defined space. Specifically, the modality-specific lens is tuned to project any-modal signals to an intermediate embedding space, which are then processed by a strong ViT with pre-trained visual knowledge. The encoded representations are optimized toward aligning with the modal-independent space, pre-defined by off-the-shelf foundation models. ViT-Lens-2 provides a unified solution for representation learning of increasing modalities with two appealing advantages: (i) Unlocking the great potential of pretrained ViTs to novel modalities effectively with efficient data regime; (ii) Enabling emergent downstream capabilities through modality alignment and shared ViT parameters. We tailor ViT-Lens-2 to learn representations for 3D point cloud, depth, audio, tactile and EEG, and set new state-of-the-art results across various understanding tasks, such as zero-shot classification. By seamlessly integrating ViT-Lens-2 into Multimodal Foundation Models, we enable Any-modality to Text and Image Generation in a zero-shot manner. Code and models are available at https://github.com/TencentARC/ViT-Lens.

Analyzing ViT-Lens: An Approach to Omni-modal Intelligence

The paper, "ViT-Lens: Gateway to Omni-modal Intelligence," introduces a novel framework designed to facilitate efficient omni-modal representation learning by leveraging Vision Transformers (ViTs). The premise of this work is predicated on the transformative potential of pretrained ViTs, which traditionally excel in visual and language tasks, to extend their capabilities into lesser-explored modalities such as 3D point clouds, depth, audio, tactile, and EEG data. This extension invites possibilities for handling diverse sensory inputs, aligning with the overarching goal of creating versatile AI agents capable of interacting with the world in a manner analogous to human perception.

Core Contributions and Methodology

The central contribution of the paper is the ViT-Lens framework, which ingeniously integrates modality-specific encoders with pretrained ViTs to map any-modal data into a cohesive feature space. A key component is the concept of a modality-specific Lens. This Lens serves to project various sensory inputs into an intermediate embedding space, which the ViT then processes using its pretrained visual knowledge.

The training regimen hinges on aligning features from novel modalities with those defined by established foundation models, such as CLIP, which serve as a common feature space for this integration. This approach not only capitalizes on the extensive training and generalization prowess of ViTs but also circumvents the prohibitive data demands typically associated with training new models from scratch for each modality.

Results and Implications

The empirical results underscore the efficacy of ViT-Lens across diverse modalities, where it sets new benchmarks in tasks such as zero-shot classification. For instance, in 3D point cloud zero-shot classification, ViT-Lens surpasses existing state-of-the-art results by a significant margin, achieving a top-1 accuracy increase of 11.0%. This performance suggests substantial potential for pretrained ViTs to generalize across traditionally distinct input modalities.

Further implications of ViT-Lens are explored through its integration with Multimodal Foundation Models (MFMs) like InstructBlip and SEED. These integrations demonstrate an emergent capability: ViT-Lens can facilitate functions such as Any-modality Captioning, Question Answering (QA), and Image Generation without requiring additional instruction tuning. This capability is notably achieved in a zero-shot manner, highlighting the model’s adaptability and underscoring its potential as a plug-and-play component in broader AI systems.

Theoretical and Practical Impacts

From a theoretical standpoint, ViT-Lens challenges traditional views of modality-specific model architectures, proposing instead a unified framework that efficiently leverages existing networks' pretrained knowledge. This work contributes to ongoing discourse regarding the scalability and adaptability of pretrained foundations in AI models.

Practically, the approach holds promise for applications where real-time multimodal understanding is critical, such as autonomous vehicles, assistive technologies, and cross-sensory virtual interfaces. The proposed framework reduces the data-hungry nature of training across modalities and offers a more integrative method to approach AI development for practical, multi-sensory applications.

Future Directions

Looking forward, further exploration could involve scaling ViT-Lens by incorporating larger foundation models or integrating with additional modalities beyond those tested. Moreover, while the paper demonstrates strong results across selected benchmarks, exploring real-world applications and understanding potential limitations in dynamic, open-world environments will be crucial for advancing this research.

In summary, the ViT-Lens framework is a noteworthy stride towards omni-modal intelligence, illustrating how pretrained visual knowledge can be effectively harnessed and extended to facilitate comprehensive perception and interaction across a diverse array of modalities. This foundation paves the way for significant advancements in AI's capacity to process and understand the rich tapestry of sensory experiences.