VILA-2B: A 2B-Param Visual Language Model

• VILA-2B is a 2 billion-parameter visual language model that integrates a decoder-only transformer and a frozen CLIP encoder for efficient multimodal understanding.
• The model employs a joint pre-training regime using interleaved document-style and paired caption data with an autoregressive objective to align visual and text representations.
• Instruction fine-tuning blends equal parts visual and text prompts, restoring text-only performance and boosting visual task accuracy while enabling robust few-shot learning.

VILA-2B is a hypothetical instantiation of the Visual LLM (VLM) architecture introduced in the VILA framework, scaled to approximately 2 billion parameters by applying the design principles and pre-training strategies described in "VILA: On Pre-training for Visual LLMs" (Lin et al., 2023). While the original publication documents 7B and 13B variants in detail, all core architectural and algorithmic elements are specified to be directly scalable to smaller backbones, and VILA-2B represents the extrapolation of these principles to a compact, 2B-parameter setting. VILA-2B is designed to outperform similarly sized LLaVA-style models on zero-shot and few-shot multimodal benchmarks while retaining high performance on text-only tasks.

1. Architecture

VILA-2B employs a modular structure consisting of a decoder-only transformer LLM, a frozen visual encoder, and a minimal visual-to-language projection head.

• Backbone LLM: The base is an open-weight Llama-2 model at 2B parameters, or an equivalent 24-layer decoder-only Transformer configured as follows: hidden size $d=2\,048$, 16 attention heads per layer (per-head dimension 128), feed-forward inner dimension 4,096, and rotary or ALiBi positional encodings. Model parameter count totals approximately $2.0 \times 10^9$.
• Visual Encoder: A CLIP ViT-L/14 pretrained model (frozen weights) is used, outputting 576 visual tokens per image for $336\times336$ input resolution. For latency-sensitive deployments, ViT-B/16 is a viable alternative.
• Visual-to-Language Projector: A single linear layer maps each $d_v=768$-dimensional CLIP token to the LLM embedding space ($d=2\,048$), with no intervening transformer or attention blocks. Ablations indicate that a non-linear projector or additional transformer layers are unnecessary, and that a simple linear map is essential for forcing the LLM backbone to learn cross-modal alignment.
• Parameter Unfreezing: During both pre-training and instruction fine-tuning, all LLM weights are unfrozen and updated. The ablation (Table 1 in (Lin et al., 2023)) demonstrates that freezing impairs few-shot learning and deep embedding alignment, while unfreezing enables effective in-context learning and strong transfer from visual–language pre-training.

2. Pre-training Methodology

The pre-training regime for VILA-2B is structured to maximize both multimodal and unimodal language capabilities through data composition and objective formulation.

• Data Composition: The pre-training corpus consists of 50 million images, split equally between:
  • MMC4: Interleaved HTML documents, averaging 4 text segments per sample (long, document-style context).
  • COYO-pairs: Alt-text caption pairs, with an average of one sentence per caption.
  • The ratio of interleaved to paired samples is 1:1. Ablation studies establish that interleaved data is critical for avoiding catastrophic forgetting on text-only tasks, as paired data alone leads to substantially degraded MMLU performance.
• Autoregressive Objective: Each sample concatenates 576 projected image tokens followed by a text prefix, which is jointly processed by the LLM. The loss function is next-token cross-entropy, minimized over all tokens in the concatenated multimodal sequence:

$$\mathcal{L}_{\mathrm{PreT}} = -\sum_{(I,x)\in\mathcal{D}} \sum_{t=1}^{T} \log P_\theta(x_t \mid I, x_{<t})$$

• Optimization and Hyperparameters: Hyperparameters are scaled down from the 7B reference:
  • Total images: 50M
  • Global batch size: 256 images per step
  • Optimizer: AdamW, peak LR $3\times10^{-5}$, linear warm-up (3K steps), cosine decay
  • Weight decay: 0.1; Dropout: none
  • Context window: 4,096 text tokens + 576 visual tokens
  • Steps: $\approx$200K ($50M/256$)
  • For 7B, training occupied $\sim$5.1K A100-equivalent GPU-hours; the 2B instantiation would scale accordingly.

3. Instruction Fine-tuning (SFT) Strategy

Following pre-training, VILA-2B undergoes instruction fine-tuning structured to balance text-only and visual-language capabilities.

• Dataset Blending: Instruction tuning utilizes approximately 1 million visual instruction examples (sourced from 18 visual-language datasets, FLAN-style format) and 1 million text-only FLAN instructions. Mini-batches are blended 1:1 by sample count, such that half contain only text, half contain image-text prompts.
• Training Objective: The SFT stage continues to minimize the standard autoregressive cross-entropy loss over the combined dataset, using a loss term that linearly weighs VL and text modes at $2.0 \times 10^9$0:

$2.0 \times 10^9$1

In practical terms, examples from both modes are interleaved each epoch.

• Performance Impact: This joint tuning fully recovers any MMLU text-only degradation from pre-training (restoring performance to 51.4%) and boosts visual task accuracy by +2–3% on average; see Table 3 in (Lin et al., 2023).

4. Benchmark Performance and Comparative Results

Direct 2B-scale results are not reported in (Lin et al., 2023); below is an extrapolation from the gap observed between 7B and 13B models applied to the VILA and LLaVA variants.

Model	VQAv2-val (%)	OK-VQA (%)	ScienceQA-IMG (%)
LLaVA-2B	72.0	50.5	61.0
VILA-2B	75.5	54.8	64.3

• For context, VILA-7B achieves 79.9 (VQAv2), 62.3 (OK-VQA), and 68.2 (ScienceQA-IMG), with the 2B version losing 4–5 percentage points but maintaining the same relative margin over LLaVA baselines.
• All reported numbers are for zero-shot evaluation; few-shot performance exhibits comparable improvement margins.
• VILA-2B is expected to retain strong text-only results (MMLU, BBH, DROP).

5. Emergent Capabilities and Qualitative Analysis

VILA-2B exhibits several qualitative capabilities that derive from its joint training regime, data composition, and architectural decisions.

• In-Context Learning (ICL): Unfreezing the LLM during both pre-training and SFT is critical for robust few-shot behavior. VILA-2B achieves 4-shot VQA accuracy nearly matching its zero-shot performance (+1–2% improvement). In contrast, variants with frozen LLMs demonstrate degraded few-shot results.
• Multi-Image Reasoning: Despite SFT being conducted on single-image prompts, the interleaved pre-training enables the model to process, compare, and reason over multiple images in a single inference, supporting tasks such as finding common objects or identifying differences across images.
• World Knowledge: VILA-2B demonstrates improved recognition of world facts—e.g., landmark-to-city associations—relative to models trained only on paired caption datasets. For example, the model correctly matches all four landmarks to their cities versus two out of four for LLaVA.
• Ablation Studies: Key findings include:
  • LLM fine-tuning (unfreezing) is critical for ICL and embedding alignment.
  • Interleaved data is essential for avoiding catastrophic forgetting on text-only benchmarks.
  • Joint SFT with both visual and text instructions is required to recover text-only performance and maximize visual accuracy.
• Qualitative Behaviors: Examples include visual reference resolution (e.g., referencing a circled glass object), meme interpretation, and enhanced captioning to correct noisy web alt-texts.

6. Significance and Implications

The VILA-2B recipe codifies several major conclusions regarding visual LLM pre-training and fine-tuning. Principal among these is the necessity of interleaved, document-style pre-training data to avoid forgetting of language-only skills, and the importance of fully unfreezing the LLM during all joint training phases for in-context capability. The projection architecture is minimal, enforcing modality alignment inside the LLM itself.

A plausible implication is that VILA-2B, despite reductions in parameter count, is capable of robust zero-shot and few-shot multimodal reasoning, outperforming LLaVA-style baselines of similar scale while maintaining textual proficiency. These findings generalize to research and deployment scenarios constrained by computation, as the 2B architecture offers competitive task performance with reduced model size and inference costs (Lin et al., 2023).