Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution (2409.12961v4)

Abstract: Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to a fixed resolution for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These design features enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously. Our work is open-sourced at https://github.com/Oryx-mLLM/Oryx.

• The paper presents a novel architecture combining adaptive visual encoding (OryxViT) and a dynamic compressor to process arbitrary resolution images and videos.
• It employs a two-stage training strategy on diverse datasets, achieving superior performance on general and long-form temporal benchmarks compared to larger models.
• The model preserves high-resolution details while optimizing computational efficiency, making it valuable for applications like medical imaging and autonomous driving.

Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution

The paper "Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution" proposes a novel architecture for multi-modal LLMs (MLLMs) that offers significant advancements in understanding and processing visual data. The authors introduce Oryx, a unified multi-modal model designed to address inefficiencies in current MLLM frameworks, particularly in handling diverse visual inputs that vary significantly in spatial sizes and temporal lengths.

Core Innovations

The Oryx model presents two main innovations:

1. OryxViT: A pre-trained visual encoder developed to generate visual representations compatible with LLMs at arbitrary resolutions. This component utilizes adaptive positional embeddings and variable-length self-attention to manage images of different sizes efficiently.
2. Dynamic Compressor Module: This module supports adjustable compression ratios (from 1x to 16x) on visual tokens, enabling the efficient processing of extensive visual contexts like videos with high compression while maintaining high precision for high-resolution tasks like document understanding.

Architectural Benefits

The primary architectural strength of Oryx lies in its ability to accommodate and efficiently process visual inputs of arbitrary resolutions and lengths. This flexibility allows Oryx to:

• Preserve important information by avoiding resolution downgrades in visual encoding.
• Enhance computational efficiency through on-demand compression tailored to the specific needs of diverse visual tasks.
• Achieve higher accuracy in multimodal tasks by better aligning visual inputs with LLMs, facilitated by the proposed OryxViT and dynamic compression techniques.

Training Strategy

The authors employ a two-stage training strategy:

1. Text-Image Pre-training and Supervised Fine-tuning: Initial training involving 558k images to pre-train the dynamic compressor, followed by fine-tuning with 4 million image-text pairs sourced from diverse datasets to develop high-quality knowledge alignment.
2. Joint Supervised Fine-tuning: A comprehensive training phase incorporating approximately 1.2 million data points, including images, videos, and 3D frames. This stage involves training the model on diverse multimodal datasets to enhance its spatial-temporal understanding capabilities.

Experimental Results

The Oryx model was rigorously tested across a variety of benchmarks:

• General Temporal Understanding: Oryx consistently outperformed existing 7B parameter models and matched or exceeded the performance of much larger models (up to 72B parameters) on benchmarks such as VideoMME, NextQA, MVBench, and others.
• Long-Form Temporal Understanding: The model showcased superior performance on long-video benchmarks such as MLVU and LongVideoBench, demonstrating Oryx's proficiency in handling extended video content.
• 2D and 3D Spatial Understanding: Oryx excelled in image-specific tasks, achieving high marks on MMBench, MMMU, DocVQA, and more. The model also performed exceedingly well on 3D spatial understanding tasks, surpassing both 3D-specific and general-purpose MLLMs.

Conclusion and Implications

The Oryx model represents a significant advancement in the field of multimodal learning, particularly in handling variable resolutions and temporal lengths in visual inputs. The proposed architecture and training approach strategically address the limitations of previous models, presenting a more flexible and efficient solution for diverse multimodal tasks.

Practically, the enhanced capabilities of Oryx in efficiently processing high-resolution images and long-form video content pave the way for its application in domains requiring detailed visual analysis, such as medical imaging, video surveillance, and autonomous driving. Theoretically, the introduction of on-demand compression and native resolution processing can inspire future research to explore more nuanced and adaptable architectures for multimodal understanding.

Future directions for this line of research might include the development of more sophisticated data curation methods to further enhance model performance, as well as the exploration of additional compression techniques to optimize computational efficiency without compromising on accuracy. The Oryx model sets a new standard for spatial-temporal understanding in MLLMs, providing a robust foundation for subsequent innovations in the field.