Delta-LLaVA: Token-Efficient Vision-Language Model

• Delta-LLaVA is a modular, token-efficient visual projector that uses a two-stage 'base-then-specialize' strategy to compress dense vision features into a compact semantic subspace.
• It employs a low-rank DeltaProjection for aligning multi-level vision features and lightweight Transformer blocks for refining tokens for downstream reasoning.
• Empirical benchmarks demonstrate improved computational efficiency and competitive task performance compared to dense token pipelines in multimodal large language models.

Delta-LLaVA is a modular, token-efficient visual projector for vision-language modeling, designed to address the prohibitive computational costs of dense visual token pipelines in Multimodal LLMs (MLLMs). Instead of mapping high-resolution vision encoder outputs directly and exhaustively into the LLM’s embedding space, Delta-LLaVA constrains the visual token budget—often to as low as 144 tokens—while preserving fine-grained semantic cues. This is achieved by a two-stage "base-then-specialize" strategy in which a low-rank DeltaProjection first aligns and compresses multi-level vision features into a compact semantic subspace, followed by a lightweight Transformer-based specialization cascade that further harmonizes these tokens for robust downstream reasoning. The system yields significant improvements in both computational efficiency and task performance across standard benchmarks (Zamini et al., 21 Dec 2025).

1. Architectural Principles

Delta-LLaVA employs a three-stage pipeline designed to maximize efficiency under a tight visual token constraint. First, a frozen or pretrained vision encoder (e.g., CLIP-ViT-L/14) extracts dense patch embeddings from each input image of shape $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$, resulting in $N=(H/P)(W/P)$ patch tokens $\{\mathbf{z}_p\in\mathbb{R}^C\}_{p=1}^N$. Conventionally, these would be projected into text space and fed, in bulk, to the LLM—introducing high quadratic costs. Delta-LLaVA addresses this by introducing the projector $\Gamma_{I \rightarrow T}$, which compresses these features into a small set of $M$ tokens (where $M \ll N$, e.g., $M = 144$). These tokens, after further specialization, are concatenated with textual tokens and processed by the LLM (e.g., Vicuna-7B, Qwen-7B).

The modular projector consists of:

• Base Alignment (DeltaProjection): A low-rank linear adaptation that merges and compresses features across multiple vision model layers into a compact semantic representation.
• Specialization Layers: Lightweight Transformer blocks—efficient multi-head self-attention (EMHSA), multi-head convolutional attention (MHCA), and position-wise MLPs—further refine token quality, capturing both global and local cues.

The system’s core design principle is "align first, interact second," concentrating model capacity on generating meaningfully aligned visual tokens prior to any self-attentive refinement.

2. DeltaProjection: Low-Rank Alignment Mechanism

The DeltaProjection module is central to Delta-LLaVA's parameter efficiency and semantic fidelity under extreme token constraints. DeltaProjection implements a low-rank adaptation in the style of DeltaLLM, aligning dense vision features to the vision–LLM’s joint embedding space.

Process overview:

1. The raw patch grid is interpolated to a compact $g \times g$ lattice of $V = g^2 \ll N$ points.
2. Positional embeddings (fixed 2D sinusoidal) are applied.
3. The query matrix $\widehat{\mathbf{Q}} \in \mathbb{R}^{V \times C}$ is processed through MHCA and EMHSA to capture local and global structure.
4. Projection to the target space uses a base weight $\mathbf{W}_s$ and a layer-specific, rank-$r$ update:

$$\mathbf{Q}_0 = (\mathbf{W}_s + \Delta\mathbf{W}_q^{(\ell)})\,\widehat{\mathbf{Q}}$$

with $$\Delta\mathbf{W}_q^{(\ell)} = \mathbf{U}_q^{(\ell)}\,\mathbf{V}_q^{(\ell)\top}$$, $$\mathbf{U}_q^{(\ell)} \in \mathbb{R}^{d_v \times r}$$, $\mathbf{V}_q^{(\ell)} \in \mathbb{R}^{C \times r}$.

For general multi-level features, this low-rank factorization reduces per-layer cost to $O(r(C_i + d_v))$ instead of $O(C_i d_v)$, enabling scalable adaptation even as $C_i$ and $d_v$ grow.

Identical machinery projects compact memory tokens (e.g., from different ViT layers) into the required key and value spaces for subsequent cross-attention, ensuring both feature reuse and parameter parsimony.

3. Specialization: Transformer Block Cascade

Following alignment, the Next-Token Block (NTB) applies lightweight Transformer and convolutional blocks to infuse the projected tokens with structure necessary for downstream language modeling:

1. Efficient Multi-Head Self-Attention (EMHSA): Captures long-range dependencies with optional spatial reduction by a factor $s \geq 1$, reducing attention complexity from $O(HV^2)$ to $O(HV^2/s^2)$.
2. Multi-Head Convolutional Attention (MHCA): Depthwise grouped $3\times3$ convolution per head provides local context in linear cost $O(V d k^2)$, omitting the softmax for efficiency.
3. Position-Wise MLP: Standard two-layer network with hidden size $h \approx 4096$ for nonlinearity and residual refinement.

Windowed cross-attention further anchors each $w\times w$ local block in the lattice to the full set of memory tokens, enabling local-global fusion while keeping within the $M=144$ token ceiling.

4. Token Formation and Memory Strategy

Delta-LLaVA constructs its queries by interpolating the initial patch grid down to a reduced resolution, preserving positional correspondences. The low-rank DeltaProjection aligns these queries, while parallel multi-level outputs (potentially from different depths of the vision encoder) serve as the compact memory. Windowed cross-attention integrates these for each spatial block, enabling a controlled trade-off between spatial detail and computational cost.

The $M \ll V$ token regime is central for efficiency, forcing the alignment and memory formation stages to be more semantically aware, in contrast to prior approaches that over-rely on self-attentive processing of large token sets.

5. Efficiency, Scaling, and Performance

Delta-LLaVA achieves linear complexity scaling with the reduced number of visual tokens $V$ ($V=N/s^2$), as opposed to global self-attention’s quadratic scaling. Key computational costs in the projector and LLM prefill phases are dominated by MLP terms, thus also roughly linear in $V$.

Empirical results demonstrate:

• Reducing $V$ from 576 to 144 lowers projector + prefill FLOPs from 6.72 to 2.16 TFLOPs.
• Inference throughput increases from ~24 tokens/s to ~37 tokens/s (~55% improvement).
• Training efficiency in paired pretraining improves from 1h24min to 33min (~4x speedup); finetuning from 4h34min to 3h21min (>1.5x).
• On the GQA benchmark, Delta-LLaVA achieves 62.34 vs. 61.9 (TokenPacker).
• For MMB: 65.95 vs. 65.1; for SQA: 68.7 vs. 67.5. At extreme compression (36, 16, 4, 1 visual tokens) Delta-LLaVA retains ≥85% of full-token performance, frequently outperforming alternatives.

6. Ablation Findings and Benchmarking

Ablation studies confirm that the low-rank DeltaProjection is the principal driver of Delta-LLaVA’s performance. Removal of this module yields the steepest declines, e.g., on GQA, dropping from 62.34 to 61.47. In contrast, removing specialized refinement modules (EMHSA or entire Transformer blocks) incurs smaller, task-dependent degradations.

Delta-LLaVA consistently outperforms or matches prior token-efficient methods across a suite of eight vision-language benchmarks (GQA, MMB, MME, POPE, SQA, TextVQA, VizWiz, VQAv2). Performance at minimal token budgets is notably robust, in contrast to comparator models that deteriorate precipitously.

Benchmark	Delta-LLaVA (144)	TokenPacker (144)
GQA	62.34	61.9
MMB	65.95	65.1
POPE	86.77	87.0
SQA	68.7	67.5

7. Insights, Limitations, and Prospective Extensions

Delta-LLaVA validates the principle that, under severe token constraints, the quality and semantic richness of initial token alignment ("alignment before interaction") is of greater consequence than stacking deeper or wider self-attentive blocks. Efficient, low-rank DeltaProjection preserves key multi-level semantics, enabling lightweight refinement modules to suffice for competitive downstream reasoning.

Potential extensions include application to ultra-high-resolution imagery or video via more aggressive multi-level summary formation, and exploration of input-conditional rank or spatial scale adaptation. A noted limitation is that, for very lengthy generated sequences ($G \gg T+V$), decode time dominates and further visual token compression yields diminishing latency returns.

Delta-LLaVA exemplifies a scalable approach to bridging visual and textual modalities by prioritizing principled token formation, charting a direction for future multimodal alignment pipelines that are both computationally sustainable and semantically precise (Zamini et al., 21 Dec 2025).