LLaVA-Next: Advancements in Multimodal Language and Vision Models

Last updated: June 11, 2025

The LLaVA ° (Large Language and Vision Assistant) framework has emerged as a significant open-source initiative in the field of multimodal LLMs ° (MLLMs °), enabling capabilities that combine visual understanding ° with natural language interaction °. Initial LLaVA models ° demonstrated strong performance on image-based visual question answering and instruction following tasks ° by connecting a pre-trained vision encoder ° (like CLIP) to a LLM ° via a projection layer ° and fine-tuning the system on multimodal instruction data ° (Liu et al., 2023 ° ). The concept referred to as "LLaVA-Next" represents the subsequent phase of this research, focusing on extending these foundational capabilities to handle greater complexity, efficiency, new modalities, and advanced reasoning while addressing limitations of earlier versions. This evolution is marked by a series of research efforts exploring different architectural modifications °, training methodologies, and applications (Li et al., 10 Jul 2024 ° ).

Significance and Background

The success of LLMs ° in text-based tasks spurred interest in extending similar capabilities to understand and interact with the visual world. LLaVA provided a key open-source framework ° by demonstrating that connecting established vision models ° with powerful LLMs, and training on appropriate instruction-following data, could yield capable multimodal assistants (Liu et al., 2023 ° ). Early LLaVA models primarily focused on single-image understanding ° and dialogue (Zhu et al., 4 Jan 2024 ° , Munasinghe et al., 2023 ° ). However, real-world applications often involve sequences of images, video, 3D data, complex reasoning, and interactions with external tools, alongside demands for greater efficiency and robustness (Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° ). The LLaVA-Next generation aims to address these limitations, expanding the model's versatility and practicality. This includes efforts to handle multi-image and video input (Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° , Gao et al., 5 Sep 2024 ° , Xu et al., 25 Apr 2024 ° ), improve efficiency and scalability (Zhu et al., 4 Jan 2024 ° , Lin et al., 29 Jan 2024 ° , Lan et al., 20 Sep 2024 ° , Wang et al., 11 Dec 2024 ° ), enhance specific reasoning skills ° like mathematics (Shi et al., 25 Jun 2024 ° ), enable tool use ° (Liu et al., 2023 ° ), facilitate continual learning (Qiao et al., 8 Oct 2024 ° ), and develop evaluation capabilities ° (Xiong et al., 3 Oct 2024 ° ).

Architectural Advancements and Core Concepts

At its core, the LLaVA framework connects a vision encoder, a projector, and a LLM. The LLaVA-Next evolution introduces several architectural modifications and concepts to enhance this structure:

• Vision Encoder and Projection: While CLIP-based vision encoders ° remain common (e.g., CLIP ViT-L/14) (Zhu et al., 4 Jan 2024 ° , Wang et al., 11 Dec 2024 ° ), some work explores alternatives like SigLIP ° for improved performance (Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° ). The projection layer, often a simple MLP °, continues to map visual features to the LLM's embedding space (Zhu et al., 4 Jan 2024 ° , Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° ). For example, PG-Video-LLaVA uses an MLP for this mapping:

  $Q_v = g(v_i) \in \mathbb{R}^{(T+N)\times K}$

  where $v_i$ are video features, $g$ is the MLP, and $K$ is the embedding dimension ° (Munasinghe et al., 2023 ° ).

• LLM Backbone °: The choice of LLM is a significant factor. While early LLaVA used models like Vicuna ° (Zhu et al., 4 Jan 2024 ° , Munasinghe et al., 2023 ° ), later iterations explore other powerful open-source LLMs, including Qwen-1.5 and Qwen-2 (Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° ). Notably, LLaVA-Phi demonstrates that even smaller LLMs like Phi-2 ° (2.7B parameters) can serve as effective backbones when trained with high-quality data, enabling resource-efficient models suitable for deployment in time-sensitive environments (Zhu et al., 4 Jan 2024 ° ). LLaVA-Phi's fusion incorporates visual embeddings ° $E_v$ and text embeddings ° $E_t$ as:

  $\text{Input}_{\text{LLM}} = [E_t; E_v]$

  where $E_v = \text{MLP}(\text{CLIP}(I))$ (Zhu et al., 4 Jan 2024 ° ).

• Handling Varied Input Modalities: A key advancement is the ability to handle more complex visual inputs ° than single, fixed-resolution images.
  • Interleaved Format: LLaVA-NeXT °-Interleave proposes a unified interleaved data ° format where sequences of images, video frames, or 3D views are presented mixed with textual tokens, providing a general template for diverse scenarios (Li et al., 10 Jul 2024 ° ). LLaVA-OneVision ° also processes visual inputs as tokens interleaved with language tokens ° (Li et al., 6 Aug 2024 ° ). This format supports inputs like <Image1> "Describe this." <Image2> "How does it differ?" (Li et al., 10 Jul 2024 ° ).
  • Variable Resolution ° (AnyRes): LLaVA-OneVision incorporates an "AnyRes" scheme to accommodate various input resolutions ° and aspect ratios, processing single images by splitting them into multiple crops to maximize the use of original resolution (Li et al., 6 Aug 2024 ° ).
• Mixture of Experts ° (MoE): MoE-LLaVA integrates Mixture of Experts layers into the LLM backbone (Lin et al., 29 Jan 2024 ° ). This allows the model to have a large total parameter count (e.g., 5.3B) while only activating a small, constant number of "experts" per token during inference (e.g., 3.6B activated parameters for top-k=2 with 4 experts), improving computational efficiency and enabling performance comparable to or exceeding larger dense models ° (Lin et al., 29 Jan 2024 ° ). Each MoE layer ° contains multiple expert networks ° and a router ° that selects which experts process each token, defined as:

  $\text{MoE}(x) = \sum_{i \in \text{Top-}k} P(x)_i \cdot E_i(x)$

  where $P(x)_i$ is the routing probability for expert $E_i$ (Lin et al., 29 Jan 2024 ° ). Load balancing is used to encourage expert utilization (Lin et al., 29 Jan 2024 ° ).

• Adaptive Visual Token Handling: To manage the large number of tokens generated by vision encoders, especially for high-resolution images ° or multiple images/frames:
  • Adaptive Granularity: AVG-LLaVA ° introduces a visual granularity scaler ° (using pooling) to produce visual tokens ° at different resolutions and a router that dynamically selects the appropriate granularity based on the image and instruction (Lan et al., 20 Sep 2024 ° ). This leads to significant token reduction ° (e.g., 85.3% on AI2D) and speedup (e.g., 2.53x inference speed on AI2D) (Lan et al., 20 Sep 2024 ° ).
  • Dynamic Compression: LLaVA-Zip proposes Dynamic Feature Map Reduction ° (DFMR °), which dynamically compresses visual tokens based on the intrinsic information content (e.g., standard deviation of feature map patches) of the image, freeing up token capacity and improving performance in token-limited scenarios (Wang et al., 11 Dec 2024 ° ).

These architectural shifts enable LLaVA-Next models to move beyond static image understanding towards more dynamic, efficient, and versatile multimodal processing.

Key Developments and Findings

The research within the LLaVA-Next theme presents several key developments and empirical findings:

• Tool Use: LLaVA-Plus demonstrates that MLLMs can learn to use and orchestrate external vision and vision-language tools (like object detectors, segmenters, image generators, OCR). This is achieved through instruction tuning on data that includes structured "thoughts," "actions," and "values" for tool invocation (Liu et al., 2023 ° ). This expands LLaVA's capabilities beyond its pre-trained scope and enables new scenarios like interactive visual assistance and compositional workflows (Liu et al., 2023 ° ).
• Video Understanding: Adapting image-based LLaVA models to video is a major focus. Challenges include capturing temporal dynamics and managing the high number of tokens from frames.
  • Pixel Grounding in Video: PG-Video-LLaVA is presented as the first LMM ° for video with pixel-level grounding, enabling localization and tracking of objects within video frames by leveraging an ensemble of models ° like GroundingDINO, DEVA ° Tracker, and SAM, coordinated by the LLM (Munasinghe et al., 2023 ° ). It also integrates audio cues by transcribing them to text to enrich video context (Munasinghe et al., 2023 ° ).
  • Temporal Considerations in Attention °: TC-LLaVA ° introduces modifications to the LLM's attention mechanism to explicitly model temporal dynamics. This includes Temporal-Aware Dual RoPE ° for temporal positional encoding ° and a Frame-wise Block Causal Attention Mask ° that allows intra-frame interaction while maintaining causal flow across frames (Gao et al., 5 Sep 2024 ° ). This leads to state-of-the-art performance on video benchmarks ° like MVBench ° (Gao et al., 5 Sep 2024 ° ).
  • Pooling for Video: PLLaVA ° uses a parameter-free ° adaptive average pooling ° strategy on visual features, primarily in the spatial dimension, to smooth feature distributions ° and reduce high-norm token dominance, improving robustness and output generation for video tasks (Xu et al., 25 Apr 2024 ° ). PLLaVA achieves state-of-the-art results on benchmarks like VideoChatGPT ° and MVBench (Xu et al., 25 Apr 2024 ° ).
• Efficiency and Scalability:
  • LLaVA-Phi shows that models with as few as 2.7B parameters can achieve competitive multimodal performance ° (e.g., 59.8 on MMBench for 2.7B Phi-2 vs 64.3 for 7B Vicuna) (Zhu et al., 4 Jan 2024 ° ).
  • MoE-LLaVA demonstrates that large total parameter counts ° (e.g., 5.3B) can be achieved with constant computational costs (e.g., 3.6B activated parameters) while maintaining or surpassing the performance of dense models (e.g., 68.5 on ScienceQA-IMG vs 66.8 for LLaVA-1.5 °-7B) (Lin et al., 29 Jan 2024 ° ).
  • AVG-LLaVA and LLaVA-Zip show significant reductions in visual tokens and increases in inference speed by adaptively selecting or compressing visual granularity based on image content and instruction (Lan et al., 20 Sep 2024 ° , Wang et al., 11 Dec 2024 ° ). LLaVA-Zip's DFMR improves performance across various visual token lengths compared to standard LLaVA-1.5 (Wang et al., 11 Dec 2024 ° ).
• Enhanced Reasoning: Math-LLaVA significantly improves multimodal mathematical reasoning ° by fine-tuning on a large, diverse dataset ° (MathV360K) specifically curated and synthesized for multimodal math problems (Shi et al., 25 Jun 2024 ° ). It shows substantial gains on benchmarks like MathVista ° (46.6% vs 27.7% for LLaVA-1.5-13B) (Shi et al., 25 Jun 2024 ° ). TG-LLaVA enhances visual encoding ° itself by guiding it with text via learnable latent embeddings, leading to improved performance across various benchmarks (e.g., +1.5% avg. on 10 benchmarks for 7B) without extra data (Yan et al., 15 Sep 2024 ° ).
• Generalization and Transfer: LLaVA-NeXT-Interleave and LLaVA-OneVision demonstrate that training a single model on diverse scenarios (single-image, multi-image, video, 3D) using unified formats enables strong transfer learning and leads to new emergent capabilities, such as transferring tasks learned on images to video (e.g., generating Twitter posts for videos when only trained on multi-image Twitter posts) (Li et al., 10 Jul 2024 ° ). LLaVA-OneVision unifies SoTA performance ° across single-image, multi-image, and video scenarios ° (Li et al., 6 Aug 2024 ° ).
• Evaluation Capabilities: LLaVA-Critic ° is introduced as the first open-source LMM specifically trained as a generalist evaluator for multimodal tasks. It provides reliable scores and justifications for LMM outputs, performing comparably to proprietary models ° (matching or surpassing GPT models on multiple benchmarks), and can generate reward signals ° for preference learning ° (e.g., for DPO) (Xiong et al., 3 Oct 2024 ° ).
• Continual Learning: LLaCA ° addresses catastrophic forgetting in continual multimodal instruction tuning ° by proposing a dynamic, self-adaptive Exponential Moving Average ° (EMA) update policy (Qiao et al., 8 Oct 2024 ° ). This allows the model to learn from sequential datasets while significantly reducing forgetting (e.g., reducing forgetting from 22.67 to 2.68) and boosting average accuracy on learned tasks (e.g., from 41.31 to 61.89) (Qiao et al., 8 Oct 2024 ° ).

These developments collectively push the boundaries of what open-source MLLMs ° can achieve in terms of capability, efficiency, and versatility.

Current Applications and State of the Art

LLaVA-Next models, building upon the LLaVA framework, enable a wider range of practical applications:

• Interactive Multimodal Assistants: LLaVA-Plus serves as a foundation for interactive agents ° that can use tools for tasks like object manipulation in images, detailed visual analysis, and content generation (Liu et al., 2023 ° ). Similarly, Purrfessor demonstrates the use of a fine-tuned LLaVA model ° for a personalized dietary health chatbot, integrating visual meal analysis ° with contextual advice and focusing on user experience and perceived care (Lu et al., 22 Nov 2024 ° ).
• Video Analysis and Understanding: PG-Video-LLaVA's pixel grounding allows for precise object localization ° and tracking in videos (Munasinghe et al., 2023 ° ). PLLaVA and TC-LLaVA achieve state-of-the-art performance on video QA ° and dense captioning ° tasks, enabling applications requiring detailed temporal descriptions or answers to questions about video content (Xu et al., 25 Apr 2024 ° , Gao et al., 5 Sep 2024 ° ). LLaVA-OneVision and LLaVA-NeXT-Interleave unify capabilities across multi-image, video, and 3D, suggesting applications in areas like surveillance, video editing, and spatial understanding (Li et al., 6 Aug 2024 ° , Li et al., 10 Jul 2024 ° ). GeoLLaVA fine-tunes LLaVA-NeXT-Video ° for temporal change detection ° in remote sensing data, crucial for environmental monitoring ° and urban planning ° (Elgendy et al., 25 Oct 2024 ° ).
• Efficient and Resource-Constrained Deployment: LLaVA-Phi's smaller size (2.7B parameters) makes it suitable for deployment in time-sensitive environments and on devices with limited resources, such as embodied agents ° and robotics (Zhu et al., 4 Jan 2024 ° ). MoE-LLaVA provides a path for scaling model capacity without proportional increases in compute, beneficial for deployment where throughput is critical (Lin et al., 29 Jan 2024 ° ). AVG-LLaVA and LLaVA-Zip's token efficiency ° improves inference speed and reduces memory requirements (Lan et al., 20 Sep 2024 ° , Wang et al., 11 Dec 2024 ° ), making resource-constrained academic research more feasible and enabling industry data augmentation (Wang et al., 11 Dec 2024 ° ).
• Guided Content Generation and Editing: Leveraging LLaVA to generate prompts for image-to-image generation ° pipelines (like Stable Diffusion) enhances visual coherence ° and provides greater control over the creative output (Ding et al., 4 Jun 2024 ° ).
• Multimodal Evaluation: LLaVA-Critic functions as an automated judge for LMMs, offering a scalable and cost-effective way to evaluate model performance across diverse multimodal tasks, and generating feedback for model alignment ° through preference learning (Xiong et al., 3 Oct 2024 ° ).
• Continual Assistants: LLaCA's ability to continually learn from new data streams without significant forgetting enables the development of lifelong multimodal assistants that can adapt to new instructions and domains over time (Qiao et al., 8 Oct 2024 ° ).

On standard benchmarks, LLaVA-Next models frequently match or surpass prior state-of-the-art open-source models and, in some cases, achieve performance comparable to or exceeding proprietary models like GPT-4V/o ° and Gemini on specific tasks and benchmarks, such as MathVista (Shi et al., 25 Jun 2024 ° ), MVBench (Xu et al., 25 Apr 2024 ° ), and LLaVA-Interleave Bench (Li et al., 10 Jul 2024 ° ).

Emerging Trends and Future Directions

The advancements in LLaVA-Next research highlight several emerging trends and suggest potential future directions:

• Greater Efficiency and Scalability: The focus on smaller models (LLaVA-Phi) (Zhu et al., 4 Jan 2024 ° ) and sparse architectures ° (MoE-LLaVA) (Lin et al., 29 Jan 2024 ° ), along with adaptive token handling (AVG-LLaVA, LLaVA-Zip) (Lan et al., 20 Sep 2024 ° , Wang et al., 11 Dec 2024 ° ), indicates a strong trend towards making powerful MLLMs more accessible and deployable. Future work may explore more advanced learned compression and pooling techniques (Lan et al., 20 Sep 2024 ° , Wang et al., 11 Dec 2024 ° , Xu et al., 25 Apr 2024 ° ), as well as more efficient training strategies like further optimized LoRA/QLoRA (Zhu et al., 4 Jan 2024 ° , Elgendy et al., 25 Oct 2024 ° ) and attention mechanisms (Gao et al., 5 Sep 2024 ° ).
• Unified and Generalist Models: LLaVA-NeXT-Interleave and LLaVA-OneVision represent a move towards single models capable of handling diverse visual inputs (single image, multi-image, video, 3D) and tasks (Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° ). Future research will likely continue to improve compositional generalization ° and task transfer ° across modalities (Li et al., 10 Jul 2024 ° , Li et al., 6 Aug 2024 ° ).
• Enhanced Reasoning and Planning: Improving specific reasoning skills, as demonstrated by Math-LLaVA (Shi et al., 25 Jun 2024 ° ) and the tool-use capabilities of LLaVA-Plus (Liu et al., 2023 ° ), is crucial. Future models could integrate more sophisticated symbolic reasoning, planning modules, and external knowledge sources (Shi et al., 25 Jun 2024 ° , Liu et al., 2023 ° ) to handle complex, multi-step multimodal tasks. TG-LLaVA's text-guided vision encoding (Yan et al., 15 Sep 2024 ° ) is another step in this direction.
• Temporal Modeling ° in LLMs: TC-LLaVA's architectural changes within the LLM layers to explicitly handle temporal information in video (Gao et al., 5 Sep 2024 ° ) are a significant step. This principle could be extended to other sequential or temporal data ° types and potentially combined with long-context modeling techniques ° to understand hours of video or sequences of interactions (Gao et al., 5 Sep 2024 ° , Xu et al., 25 Apr 2024 ° ).
• AI °-Driven Evaluation and Alignment: LLaVA-Critic establishes open-source MLLMs as capable evaluators (Xiong et al., 3 Oct 2024 ° ). This opens avenues for scalable, AI-driven feedback mechanisms (e.g., DPO, RLHF) (Xiong et al., 3 Oct 2024 ° ) to align MLLMs more effectively and efficiently, potentially leading to superhuman alignment in evaluation (Xiong et al., 3 Oct 2024 ° ). Future work could explore the use of AI critics in real-time feedback loops for training.
• Continual and Adaptive Learning °: LLaCA's approach to continual learning addresses a fundamental challenge for deploying models in dynamic environments (Qiao et al., 8 Oct 2024 ° ). Future research may focus on extending continual learning capabilities to broader domains, including pretraining from new modalities or languages, and developing more nuanced adaptive learning strategies ° (Qiao et al., 8 Oct 2024 ° ).
• Integration with Real-World Systems: The demonstrated applications in robotics (LLaVA-Phi) (Zhu et al., 4 Jan 2024 ° ), health (Purrfessor) (Lu et al., 22 Nov 2024 ° ), remote sensing (GeoLLaVA) (Elgendy et al., 25 Oct 2024 ° ), and content creation ° (Ding et al., 4 Jun 2024 ° ) highlight the increasing integration of LLaVA-based models into real-world systems. Future work will involve addressing robustness, safety, and real-time interaction ° requirements for broader deployment (Zhu et al., 4 Jan 2024 ° , Liu et al., 2023 ° , Lu et al., 22 Nov 2024 ° ).
• Human-Centric Design: The insights from Purrfessor regarding the importance of interaction design, persona, responsiveness, and personalization (Lu et al., 22 Nov 2024 ° ) underscore the need for future LLaVA models to be not just technically capable but also user-friendly and engaging for effective human-AI collaboration °.
• Open Data ° and Reproducibility: The continued release of datasets (M4-Instruct, MathV360K, Critic dataset, GeoLLaVA data) and code bases is crucial for accelerating research and fostering the open-source ecosystem in multimodal AI ° (Li et al., 10 Jul 2024 ° , Shi et al., 25 Jun 2024 ° , Xiong et al., 3 Oct 2024 ° , Elgendy et al., 25 Oct 2024 ° ).

The collective progress represented by LLaVA-Next research paints a picture of MLLMs becoming more capable, efficient, versatile, and aligned, moving towards more general-purpose visual assistants capable of handling complex real-world tasks across multiple modalities.