LLaVA-OneVision: Advanced Multimodal Framework

• LLaVA-OneVision is an open-source family of large multimodal models that unifies single-image, multi-image, and video tasks with a single set of weights.
• It employs innovative training strategies, such as a staged curriculum and AnyRes visual representation, to maximize cross-scenario transfer.
• LLaVA-OneVision-1.5 enhances efficiency and reproducibility through advanced data curation, optimized training, and cost-effective deployment.

LLaVA-OneVision is an open-source family of Large Multimodal Models (LMMs) designed to deliver state-of-the-art performance across single-image, multi-image, and video visual-language tasks using a single set of weights. Developed through consolidation of insights from the LLaVA-NeXT series, LLaVA-OneVision introduces new advancements in training methodology, visual representation, and data curation, enabling strong cross-scenario transfer and emergent capabilities. A subsequent release, LLaVA-OneVision-1.5, demonstrates further architectural, dataset, and training efficiency improvements, facilitating democratized multimodal training within modest computational and financial constraints (Li et al., 6 Aug 2024, An et al., 28 Sep 2025).

1. Technical Motivation and Conceptual Framework

The LLaVA-OneVision architecture extends the LLaVA modeling paradigm by emphasizing simultaneous competence on three key computer vision scenarios: single-image, multi-image, and video understanding. Motivated by observations from LLaVA-NeXT—such as the effective transfer of single-image instruction tuning to video and the value of interleaved multi-image/video data—LLaVA-OneVision adopts a recipe consisting of a vision encoder, a projector, and a LLM. This configuration leverages (i) high-quality, diverse instruction data (3.2M single-image, 0.56M multi-image, 0.35M video samples), (ii) an "AnyRes" visual representation mechanism to equalize token budgets across modalities, and (iii) a staged curriculum to maximize transfer within practical compute constraints (Li et al., 6 Aug 2024).

The LLaVA-OneVision-1.5 follow-up addresses the need for reproducibility and cost-efficiency by providing a fully open, end-to-end framework that builds competitive LMMs entirely from scratch, leveraging curated datasets (85M mid-training, 26M instruction-tuning samples, 64B compressed tokens), efficient data packing, and optimized training (An et al., 28 Sep 2025).

2. Model Architecture and Visual Representation

LLaVA-OneVision Core Pipeline

The architecture comprises three primary components:

• Vision Encoder ($g(\cdot)$): Utilizes a SigLIP ViT backbone ("SO400M"), 24 Transformer layers, hidden size 1024. Inputs are crops ($384 \times 384$) for images/multi-image; video frames resized similarly. Feature extraction yields $T$ tokens per crop ($T=729$ at base resolution), subject to pooling.
• Projector ($p(\cdot)$): A two-layer MLP maps ViT output vectors (dimension 1024) to the LLM embedding space (e.g., 1024 or 2048).
• LLM ($f(\cdot)$): Adopts Qwen-2 family variants with 0.5B, 7B, or 72B parameters, comprising a Transformer decoder with 32–64 layers.

The visual token budget is enforced via the "AnyResMax" scheme. For each image partitioned into $a \times b$ crops, the total token count ($L$) is capped at $\tau = 9 \times 729$ using bilinear pooling:

$$T_\mathrm{new} = \begin{cases} \tau / (a \cdot b + 1) & \text{if } (a \cdot b + 1) T > \tau \ T & \text{otherwise} \end{cases}$$

For single images (up to $6\times6$ crops), multi-image inputs (≤12), and video (≤32 frames at 196 tokens/frame), the total always remains $\approx$7,300 tokens (Li et al., 6 Aug 2024).

LLaVA-OneVision-1.5 Enhancements

LLaVA-OneVision-1.5 reimplements the pipeline with:

• Vision Encoder ($\Phi_v$): RICE-ViT (L-14-560px), generating region-aware patch embeddings with 2D rotary positional encoding.
• Projector ($\Pi$): Aggregates $2\times2$ patch blocks, concatenates, and projects via MLP.
• LLM ($\mathcal{H}$): Qwen3 decoder, featuring self- and cross-attention between language and visual tokens, and native multimodal representation alignment (An et al., 28 Sep 2025).

3. Training Data, Datasets, and Tokenization

Curated Data Stages

LLaVA-OneVision employs a staged curriculum:

• Stage 1: Projector-only language–image alignment, using LCS ($\sim$0.56M image-text pairs).
• Stage 1.5: High-quality knowledge learning (COCO, OCR, Chinese).
• Stage 2: Visual instruction tuning in two steps: 3.2M single-image (from 60+ sources and 5 categories), followed by 1.6M OneVision (multi-image, video, and a balanced subset of single-image).

All samples are cast into a consistent chat format via 24 formatting prompts, managing instruction/response structure and use of special tokens (Li et al., 6 Aug 2024).

LLaVA-OneVision-1.5 augments this regimen with:

• 85M concept-balanced mid-training corpus: Sources include COYO-700M, Obelics, DataComp-1B, LAION-CN, ImageNet-21K, SAM-1B, MINT, Zero250M. Concept balancing is achieved by embedding each image and concepts (500K vocabulary) with MetaCLIP encoders, then sampling for histogram uniformity.
• Instruction set of ≈22M samples: Aggregated from 124 sources, span captioning, chart/table, code/math, VQA, grounding/counting, OCR/science, and domain-specific data (An et al., 28 Sep 2025).

Tokenization uses Qwen3’s SentencePiece (32K subword tokens). Offline data packing achieves a compression ratio of $R_\text{pack}\approx11\times$, resulting in approximately 64B compressed tokens (An et al., 28 Sep 2025).

4. Training Objectives, Methodology, and Efficiency

End-to-End Loss: Both releases optimize the standard autoregressive cross-entropy loss on answer tokens:

$$\mathcal{L}_{\mathrm{CE}} = -\sum_{i=1}^L \log p(x_i|v, q, <i, a_{<i})$$

No contrastive or temporal-consistency losses are employed; the diversity and coverage of the instruction set promote cross-modal alignment (Li et al., 6 Aug 2024).

Curriculum and Hyperparameters: For LLaVA-OneVision, the multi-stage approach involves initial projector-only alignment, partial freezing of the vision encoder during subsequent phases (with different learning rates for visual and textual modules), and progressive token capacity scaling.

LLaVA-OneVision-1.5 introduces advanced efficiency techniques:

• Offline Parallel Data Packing: Solves a bin-packing problem to maximize GPU sequence utilization and minimize padding, with ~90% success and a 3.5× GPU utilization increase, reducing cross-batch communication overhead by ≈88%.
• Hybrid Parallelism: Employs Megatron-LM for integrated data and optimizer parallelism with uniform recomputation. All training fits within a $16,000 budget: 128 A800 GPUs × 3.7 days for mid-training, with regularization strategies including gradient clipping (1.0) and label smoothing (0.1) (An et al., 28 Sep 2025).

Stage	Modules Tuned	Batch Size/GPU	LR Init	Optimizer
Stage-1	Projector only	32	$5\times10^{-5}$$</td> <td>AdamW</td> </tr> <tr> <td>Stage-1.5</td> <td>All parameters</td> <td>4</td> <td>$$3\times10^{-5}$$</td> <td>AdamW</td> </tr> <tr> <td>Stage-2</td> <td>All parameters</td> <td>4→8</td> <td>$$2\times10^{-5}$	AdamW

5. Benchmark Performance and Evaluation

LLaVA-OneVision achieves competitive or superior results compared to proprietary models (e.g., GPT-4V) on a collection of standardized vision-language benchmarks (LMMs-Eval) in 0-shot greedy mode (Li et al., 6 Aug 2024):

Scenario	Benchmark	72B-Model (%)	GPT-4V (%)
Single-Image	AI2D	85.6	78.2
	ChartQA	83.7	78.5
	DocVQA	91.3	88.4
	MathVista	67.5	49.9
Multi-Image	IEI	95.3	52.0
	NLVR2	93.8	88.8
Video	ActivityNetQA	62.3	57.0
	MLVU	68.0	49.2

Across over 60 benchmarks, the 72B model matches or exceeds GPT-4V in a majority of tasks, notably outperforming in joint single-image/video scenarios.

LLaVA-OneVision-1.5-8B outperforms Qwen2.5-VL-7B on 18 of 27 benchmarks, with a mean accuracy of 76.0% vs 74.2% (An et al., 28 Sep 2025).

6. Emerging Abilities and Cross-Scenario Transfer

LLaVA-OneVision demonstrates nine emergent cross-scenario abilities unexpected from instruction tuning alone (Li et al., 6 Aug 2024):

1. Joint diagram + chart reasoning from multi-image inputs.
2. GUI-to-action instructions on iPhone screenshots.
3. Set-of-marks referential understanding in images (e.g., "refer to mark #4,5,7").
4. Image-to-video editing: generating multi-frame prompts from a static image.
5. Video-to-video difference detection.
6. Multi-camera driving scene analysis and next-action planning.
7. Vertical sub-scene comprehension in video sequences.
8. Visual prompt grounding across frames (e.g., "What number is circled?").
9. Multi-modal referring (identify a person in image, confirm in video).

Ablation studies confirm the efficacy of AnyResMax cropping, the importance of LLM scale (notably for reasoning and video), and the value of multi-image/video tuning for improving multi-view task performance by +10–25 points without harming single-image accuracy.

7. Future Directions and Reinforcement Learning Extensions

The LLaVA-OneVision-1.5 framework includes a forthcoming reinforcement learning-based variant ("1.5-RL") that introduces:

• Human preference collection on 1M multimodal responses.
• Reward model training via Proximal Policy Optimization (PPO).
• Decoder policy fine-tuning:

$$\min_\theta -\mathbb{E}_{(I, Q)\sim D, a\sim\pi}[R(a|I, Q)] + \beta \cdot KL(\pi_\theta || \pi_0)$$

Pseudo-code for PPO optimization is provided, and the release includes all model checkpoints, reward data, and scripts required for open replication and further research (An et al., 28 Sep 2025).

---

LLaVA-OneVision models consolidate advances in scalable cross-modal training, efficient visual token management, diverse instruction tuning, and cost-effective large-scale deployment, while remaining open-source to facilitate research reproducibility and further advancement in the multimodal foundation model domain (Li et al., 6 Aug 2024, An et al., 28 Sep 2025).