SwiftVLM: Efficient VLM Inference

• SwiftVLM is a training-free, plug-and-play inference acceleration scheme for vision-language models that uses a cross-layer token bypass paradigm.
• It selectively prunes and merges visual tokens while preserving critical information via a bypass path to ensure accurate text-conditioned reasoning.
• Experimental evaluations demonstrate significant latency speedup and high accuracy retention on both localization and non-localization tasks.

SwiftVLM is a training-free, plug-and-play inference acceleration scheme for vision-LLMs (VLMs) based on a cross-layer token bypass paradigm. It addresses the inefficiency of dense visual token processing in transformer-based VLMs by selectively pruning and merging tokens with a mechanism that preserves and re-evaluates unselected tokens across multiple layers. This approach circumvents critical information loss that arises from premature or irreversible early pruning in conventional methods, enabling significant reductions in latency and computational cost while maintaining high fidelity in tasks requiring fine-grained visual-textual reasoning (Qian et al., 3 Feb 2026).

1. Motivation and Problem Context

Transformer-based vision-LLMs achieve state-of-the-art results on multimodal tasks by jointly processing large numbers of visual and textual tokens. This token explosion, particularly from high-resolution images, imposes substantial inference-time computation and latency burdens. Existing approaches to alleviate this—such as text-agnostic merging (ToMe, VisionZip) or text-aware pruning (FastV, PDrop, SparseVLM, FEATHER)—suffer notable shortcomings. Text-agnostic methods indiscriminately merge visual tokens based on spatial or feature similarity, often sacrificing the fine-grained details necessary for precise, text-conditioned reasoning. Text-aware pruning, conversely, drops tokens early in the network based on early attention scores, but this discards information that could become critical for fine semantic alignment at later layers.

Layer-wise analysis in VLMs such as LLaVA-1.5-7B demonstrates that the importance of visual tokens is highly non-stationary: tokens considered unimportant at shallow layers may acquire high relevance for aligned reasoning at greater depth. Empirical results show that up to 59% of tokens ranked in the bottom 50% at layers 1–9 become top 10% important in layers 10–20 on TextVQA, highlighting the unreliability of early importance scores.

2. Cross-Layer Token Bypass Paradigm

To prevent irreversible loss of critical tokens, SwiftVLM introduces the bypass paradigm. At designated pruning layers, tokens are split along two orthogonal axes: a merge path and a bypass path. The merge path retains the top-K% visual tokens (by cross-modal attention score) for immediate progression. The remaining (bottom 100−K%) tokens are grouped by feature similarity and merged within each group into a representative token, then advanced along the standard pathway. Importantly, the unmerged original tokens are also preserved in a parallel buffer (the bypass path), and periodically reintroduced for re-ranking and inference at subsequent pruning layers.

At later pruning layers, merged tokens are processed as usual, while bypassed tokens undergo lightweight representation alignment (offset correction) using precomputed differences based on the unpruned model (“vanilla offsets”). Thereafter, these aligned bypassed tokens are concatenated back for re-ranking, enabling layer-specific, independent token selection with maximized discriminative power.

3. Algorithmic Details and Mathematical Formulation

Pruning-Layer Selection

Let $L$ denote the total number of transformer layers. The method computes a layer-wise performance curve $x_\ell$, evaluating the model when only the top $V\%$ tokens are retained at layer $\ell$. To optimally allocate a pruning budget $m$ across the network, layers are selected ($\{i_1, ..., i_m\}$) via dynamic programming to maximize the sum of area gains $AU(i|i_{k-1},j) = (x_i - x_{i_{k-1}})\cdot(j-i)$ between non-overlapping intervals.

Token Pruning and Bypass Procedure

At each selected pruning layer $x$:

• Compute cross-modal attention scores $$s_i^{(x)} = \text{Softmax}(h_\text{text}^{(x-1)} W_Q \cdot h_i^{(x-1)} W_K / \sqrt{d})$$.
• Retain indices $S_x$ of the top-K tokens; the remainder $B_x$ are grouped by cosine similarity and merged: $$sim(i,j) = \langle h_i, h_j \rangle / (\|h_i\| \|h_j\|)$$.
• The merged token for group $g$ is $\bar{h}_g = (1/|g|) \sum_{i\in g} h_i^{(x-1)}$. Unmerged $h_i$'s are preserved in a bypass path.

Representation Alignment

Before reintroduction at subsequent pruning layer $y$, each bypassed token $h_i$ is aligned using precomputed offsets: $$\Delta h_g = (1/|g|) \sum_{i\in g}\left(h_i^{\text{vanilla},(y-1)} - h_i^{\text{vanilla},(x)}\right)$$. The aligned token is $\hat{h}_i^{(y-1)} = h_i^{(x)} + \Delta h_g$.

Pseudocode Outline

Given: pretrained VLM with L layers, pruning layers {i1,…,im}, thresholds {τ_{ik}} Extract vanilla offsets Δh_g for each group over (ik → i_{k+1}−1) Initialize T = all visual tokens for pruning layer i in {i1,…,im}: Compute cross-modal attention s = Attention(last_text, T) S_keep = top-{τ_i}% tokens by s S_drop = T \ S_keep Group S_drop by feature similarity into {G_g} For each G_g: compute mean h̄_g, store bypassed {h_i} T = S_keep ∪ {h̄_g} Before next pruning layer: For each bypassed token i in G_g: h_i ← h_i + Δh_g T = T ∪ {aligned bypassed tokens} Continue inference with final T

4. Efficiency and Complexity Analysis

Assume $n_\text{vis}$ visual tokens, $n_\text{txt}$ textual tokens, $T$ layers, and $K$ the deepest pruning layer. The computational requirement is:

$$F = K\cdot C(n_\text{vis} + n_\text{txt}) + (T-K)\cdot C(n' + n_\text{txt})$$

where $n' = (1-D) n_\text{vis} + n_\text{txt}$ after fraction $D$ tokens are pruned, and $C(n) \approx 4n d^2 + 2 n^2 d + 3 n d m$ for embedding $d$ and FFN dimension $m$.

SwiftVLM introduces additional group merging ($O(R Z d)$), alignment ($O(R d)$), and explicit attention for the final text token ($O(2 n' d^2 + 2 n' d)$). With a 33% visual token retention on LLaVA-1.5-7B, latency speedup reaches up to $1.48\times$ (end-to-end) and $1.79\times$ (prefill) over vanilla inference, outperforming FastV and SparseVLM.

5. Experimental Evaluation

SwiftVLM is assessed on LLaVA-1.5-7B, LLaVA-NeXT-7B, and nine benchmarks, including localization (RefCOCO, RefCOCO+, RefCOCOg) and non-localization (TextVQA, GQA, SQA, MME, MMB, POPE) tasks. Under a 192-token budget ($\sim66.7\%$ retention), SwiftVLM sustains $96.6\%$ non-localization accuracy and $87.7\%$ localization accuracy, substantially ahead of FastV, VisionZip, PDrop, SparseVLM, and FEATHER, which fall as low as 30–68%. Even under a 128-token budget ($\sim77.8\%$ retention), SwiftVLM delivers $96.7\%$ and $55.2\%$ accuracy in non-localization and localization, respectively. On LLaVA-NeXT-7B, accuracy retention reaches $98\%$ and $97.1\%$ for 33.3% and 22.2% budgets.

Token selection fidelity is quantified by overlap with vanilla top-K tokens: SwiftVLM’s bypass path yields 15–20% higher average overlap than drop-based pruning, underlying its superior reasoning reliability.

Token Budget	Task Type	SwiftVLM Accuracy (%)	FastV/SparseVLM/FEATHER Accuracy (%)
192 (66.7%)	Non-localization	96.6	30–68
192 (66.7%)	Localization	87.7	30–68
128 (77.8%)	Non-localization	96.7	N/A
128 (77.8%)	Localization	55.2	N/A

6. Mechanistic Insights and Ablations

The cross-layer bypass yields several advantages: irreversible token loss is mitigated, each pruning layer makes selection decisions at its peak discriminative capacity, and representation misalignments are remedied via lightweight offset application rather than expensive full-dimensional transformations. Ablation studies indicate that strategic layer selection maximizes performance at moderate pruning rates, merging proves useful only under tight budgets, and bypass consistently enhances selection stability.

7. Limitations and Potential Extensions

SwiftVLM, despite its plug-and-play and training-free deployment, depends on pre-computed offsets (“vanilla offsets”), necessitating an offline pass over the unpruned model. This suggests a limitation for extremely dynamic or online applications. Potential extensions include online offset estimation, end-to-end fine-tuning for optimal grouping, dynamic data-driven pruning layer selection, or integration with neuron-level sparsification. These avenues plausibly offer additional efficiency gains and adaptability.

SwiftVLM introduces the cross-layer token bypass paradigm and achieves state-of-the-art efficiency-accuracy trade-offs for multimodal transformers, especially on fine-grained, text-conditioned reasoning tasks, while remaining agnostic to model architecture and task specification (Qian et al., 3 Feb 2026).