LLaVA-OneVision-1.5 Framework

• LLaVA-OneVision-1.5 is a fully open framework for constructing robust large multimodal models through concept balancing and efficient data utilization.
• The framework leverages an 85M image–text corpus and a 22–26M instruction set to achieve state-of-the-art results across 27 diverse vision-language benchmarks.
• Its innovative offline parallel data packing strategy minimizes training costs and maximizes GPU efficiency, democratizing access to high-quality model development.

LLaVA-OneVision-1.5 defines a fully open framework that enables the construction of high-quality Large Multimodal Models (LMMs) with efficient data usage, cost-effective training recipes, and competitive benchmark performance. Distinguished from prior approaches by its reproducible, democratized methodology, LLaVA-OneVision-1.5 leverages a concept-balanced, large-scale pretraining corpus and meticulously curated instruction data while introducing a highly compressed, offline data packing strategy. The framework demonstrates state-of-the-art performance on diverse vision-language benchmarks, setting a new standard for open multimodal model development with robust results across model scales (An et al., 28 Sep 2025).

1. Dataset Construction and Concept Balancing

LLaVA-OneVision-1.5 relies on two primary datasets: an 85M concept-balanced image–text pair set for mid-training, and a 22–26M instruction-tuning corpus. The 85M corpus comprises approximately 65M English and 20M Chinese examples sourced from COYO-700M, Obelics, DataComp-1B, LAION-CN, ImageNet-21K, SAM-1B, MINT, and Zero250M.

Concept balancing mitigates long-tail bias inherent to raw web data. Each image $i$ from set $\mathcal{I}$ and each concept $v$ from predefined vocabulary $\mathcal{V}$ are embedded via pretrained MetaCLIP-H/14-Full-CC2.5B, yielding $E_i = \{\Phi_v(i)\}$ and $E_t = \{\Phi_t(v)\}$, respectively. Cosine similarity ($\operatorname{cos\_sim}(\Phi_v(i), \Phi_t(v))$) assigns top-$k$ closest concepts to each image, followed by inverse-frequency reweighting to upsample rare concepts. This process results in a semantically balanced dataset optimized for robust visual representation learning.

The instruction-tuning set (reported at 22M and 26M instructions in various places) encompasses broad task categories: Captioning, Chart/Tabular reasoning, Code, Math, Domain-specific VQA, Grounding, OCR, and Science, supporting generalized cross-domain instruction following.

2. Efficient Training Framework

The framework introduces a cost-centric, offline parallel data packing strategy. Rather than dynamic padding for each batch, training data are prepacked into hash buckets and merged into fixed-length, multi-threaded batches offline. This method achieves up to $11\times$ compression of the 85M pretraining set, effectively increasing throughput and reducing wasted computation.

Mathematical details for concept balancing:

• For $\mathcal{I}$ (images) and $\mathcal{V}$ (concepts),
• Compute $E_i = \{\Phi_v(i)\}$, $E_t = \{\Phi_t(v)\}$,
• Assign top-$k$ concepts via cosine similarity on L2-normalized vectors,
• Weight sampling by $w_i = 1/\operatorname{freq}(v_{i})$ for rare concept upweighting.

The offline packing ensures maximal GPU utilization and minimal runtime overhead, enabling full model training within a reported USD $16,000$ budget.

3. Model Architecture and Training Pipeline

The model architecture is modular: a vision encoder (typically SigLIP or CLIP-derived visual backbone) produces embeddings; a compact MLP projector aligns visual tokens with the LLM’s input space. The system supports both English and Chinese, owing to the data’s bimodal distribution.

Training unfolds in sequential stages:

• Pretraining with the concept-balanced image–text corpus,
• Instruction-tuning leveraging curated multimodal tasks,
• (Forthcoming) RL-based refinement (LLaVA-OneVision-1.5-RL), suggesting potential integration of reward modeling or reinforcement alignment.

4. Benchmark Evaluation and Performance

LLaVA-OneVision-1.5 models are evaluated on 27 diverse benchmarks, covering general VQA (MMStar, MMBench), scientific reasoning, chart/table understanding, OCR, and document tasks.

• LLaVA-OneVision-1.5-8B outperforms Qwen2.5-VL-7B on 18 of 27 tasks.
• LLaVA-OneVision-1.5-4B exceeds Qwen2.5-VL-3B on all 27 tasks.

These results demonstrate robust cross-task generalization, with benchmark performance comparable or superior to strong proprietary systems under substantially reduced resource budgets.

5. Offline Parallel Data Packing: Implementation & Impact

Offline parallel data packing is executed via hash bucket allocation and multi-threaded, strategy-aware batching. Data samples of varying length are consolidated offline, minimizing padding during runtime. The process achieves an $11\times$ reduction in wasted sequence space.

This approach:

• Maximizes hardware utilization,
• Compresses token representation (totaling 64B multimodal tokens across the two main corpora),
• Enables democratized access to large-scale training by reducing both financial and computational barriers.

6. Open Source and Future Extensions

The framework is released fully open-source, with code, data recipes, and trained models available for reproducibility and extension. The team explicitly anticipates a release of LLaVA-OneVision-1.5-RL, likely involving reinforcement learning or reward-model alignment strategies, though details remain forthcoming.

A plausible implication is that the framework’s modular design, unified data preprocessing, and cost-efficient recipe enable wide community participation in multimodal model training and benchmarking.

7. Significance and Broader Impact

LLaVA-OneVision-1.5 establishes a standard for open, efficient multimodal model training that is both reproducible and economically accessible. The adoption of concept-balanced sampling and offline parallel packing strongly influences model robustness and scalability. By outperforming strong baselines on a broad array of benchmarks under a constrained budget, the framework lowers the entry threshold for both academic and industrial researchers in vision–language modeling.

This suggests future research may build upon these methods to further optimize data utilization, explore reinforcement learning for multimodal policy optimization, and extend the paradigm to richer and more diverse multimodal tasks, including cross-linguistic reasoning and multi-domain synthesis.