METEOR: Multi-Encoder Token Pruning

• METEOR is a multi-stage token pruning framework for multi-encoder multimodal LLMs that progressively prunes redundant tokens during encoding, fusion, and decoding.
• It employs rank-guided collaborative token assignment, cooperative cross-encoder fusion pruning, and text-conditioned adaptive pruning to optimize token retention.
• The framework achieves up to a 76% reduction in visual tokens and increases throughput by 46% with only a minimal performance drop across various benchmarks.

Searching arXiv for the METEOR paper and closely related token-pruning work to ground the article. Multi-Encoder Collaborative Token Pruning (METEOR) is a progressive multi-stage token pruning framework for multi-encoder multimodal LLMs (MLLMs), introduced to make multi-vision architectures efficient without discarding the complementary strengths that motivate their use in the first place (Liu et al., 28 Jul 2025). It targets the characteristic inefficiency of systems such as EAGLE or Cambrian, where several vision encoders produce large visual token sets that are concatenated and processed by a decoder-only LLM. METEOR addresses redundancy at three distinct loci—within individual vision encoders, across encoders after projection into a shared space, and during LLM decoding—through rank-guided collaborative token assignment, cooperative fusion-stage pruning, and text-conditioned adaptive pruning, respectively (Liu et al., 28 Jul 2025). The method is positioned as the first successful attempt to achieve an efficient multi-encoder vision LLM with multi-stage pruning strategies, and it reports that, relative to EAGLE, it reduces 76% visual tokens with only 0.3% performance drop in average across 11 benchmarks (Liu et al., 28 Jul 2025).

1. Conceptual setting and motivation

METEOR is designed for the specific regime in which single-encoder MLLMs are no longer sufficient for broad multimodal competence, yet naïve multi-encoder fusion becomes computationally prohibitive (Liu et al., 28 Jul 2025). The motivating observation is that CLIP-style single-encoder systems are weak at OCR, fine-grained grounding, and dense or domain-specific reasoning, whereas multi-encoder systems improve robustness by combining encoders with complementary inductive biases, such as CLIP, ConvNeXt, Pix2Struct, and EVA-02/EVA-CLIP (Liu et al., 28 Jul 2025). The cost of this improvement is substantial token growth: multi-encoder models often produce thousands of tokens, and self-attention complexity scales as $O(N^2 d)$ in token length $N$ (Liu et al., 28 Jul 2025).

The framework therefore treats token redundancy as a multi-stage phenomenon rather than a single pruning problem. Redundancy arises during encoding, because high-resolution encoders produce many background or low-frequency tokens; during fusion, because different encoders often produce overlapping representations for the same regions; and during decoding, because many visual tokens are irrelevant to a specific prompt and many attention heads contribute little to vision grounding (Liu et al., 28 Jul 2025). This staged diagnosis distinguishes METEOR from single-encoder pruning methods and from pruning schemes that act only inside the LLM.

This suggests that METEOR should be understood not merely as an efficiency technique but as an architectural coordination mechanism for multi-encoder MLLMs. The “collaborative” aspect refers both to coordination among encoders and to coordination among pruning stages, each of which uses a different signal: visual statistics during encoding, cross-encoder similarity during fusion, and vision–text attention during decoding (Liu et al., 28 Jul 2025).

2. System architecture and multi-stage pipeline

METEOR is built on an EAGLE-style multi-encoder architecture comprising multiple vision encoders, dedicated projectors, post-projection token fusion, and a decoder-only LLM such as Vicuna-7B, LLaMA-2, or LLaMA-3 (Liu et al., 28 Jul 2025). In the main setup, the vision side includes four encoders: CLIP ViT, ConvNeXt, Pix2Struct, and EVA-02/EVA-CLIP (Liu et al., 28 Jul 2025). Each encoder $l$ produces tokens $\boldsymbol{g}^l$, which are mapped into a shared semantic space by a dedicated 2-layer MLP projector $G^l$: $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$ The projected tokens are then concatenated: $$\boldsymbol{z} = [\boldsymbol{z}^1, \dots, \boldsymbol{z}^L].$$ These fused visual tokens are prepended or concatenated with text tokens and processed by the LLM (Liu et al., 28 Jul 2025).

The three METEOR stages are structurally distinct. Stage 1 prunes tokens inside each vision encoder progressively across depth. Stage 2 prunes across encoders after projection by removing mutually redundant tokens in the shared space. Stage 3 applies text-aware adaptive pruning during LLM decoding using attention from the last instruction token to visual tokens and dynamically adjusting the number of retained tokens according to inferred visual contribution (Liu et al., 28 Jul 2025).

A concise summary of the pipeline is given below.

Stage	Location	Core signal
1	Within each vision encoder	Average-token similarity in shallow blocks; cls-attention in deep blocks; rank-guided per-encoder budgets
2	After per-encoder projectors, before LLM	Cross-encoder mutual redundancy in shared space
3	During LLM decoding	Vision–text attention from selected heads and adaptive token budget

This multi-stage structure bears comparison to later collaborative pruning work for LVLMs such as ViTCoP, which also distributes pruning across the vision encoder and LLM depth, though in a two-encoder vision-plus-language setting rather than a true multi-encoder visual fusion setting (Luo et al., 25 Jan 2026). A plausible implication is that METEOR helped define the design space for later collaborative pruning frameworks that combine encoder-local saliency, cross-stage coordination, and prompt-aware late pruning.

3. Stage 1: rank-guided collaborative token assignment in multi-vision encoding

Stage 1 accelerates the vision stack itself by progressively pruning tokens inside each vision encoder rather than waiting until all encoder outputs have been formed (Liu et al., 28 Jul 2025). For block $b$ in encoder $l$, the token output is

$$\boldsymbol{g}_b^l \in \mathbb{R}^{N_b^l \times D},$$

where $N$0 is the token count and $N$1 is the feature dimension (Liu et al., 28 Jul 2025). Instead of a single end-of-encoder pruning event, METEOR discards tokens at multiple blocks, relying on subsequent self-attention to propagate information and reduce abrupt information loss (Liu et al., 28 Jul 2025).

A central design choice is the use of different token-importance criteria at different depths. In shallow blocks, attention is regarded as unreliable because shallow layers exhibit high-entropy attention distributions and unstable top-$N$2 token indices across adjacent layers (Liu et al., 28 Jul 2025). METEOR therefore computes the average token

$N$3

and evaluates cosine similarity between each token and this average (Liu et al., 28 Jul 2025). Tokens most similar to the average are treated as background-like or low-information, and the pruning operator retains those least similar to the average: $N$4 In deeper blocks, where attention becomes sparser and more semantically meaningful, METEOR uses cls-token attention. If $N$5 is the cls query and $N$6 the keys of $N$7, then

$N$8

and the top-attended tokens are retained (Liu et al., 28 Jul 2025). The encoder is equally split into three phases: the first phase uses average-token similarity, while the last two phases use cls-attention (Liu et al., 28 Jul 2025).

The “collaborative” element at this stage is the allocation of token budgets across encoders according to feature-map rank. METEOR performs SVD on $N$9,

$l$0

and interprets higher rank as higher information richness (Liu et al., 28 Jul 2025). Because online SVD would be expensive, the expected ranks $l$1 are precomputed offline on a small batch of images and found to be stable across inputs (Liu et al., 28 Jul 2025). Given a global budget $l$2 for block $l$3, METEOR assigns

$l$4

Thus high-rank encoders retain more tokens, while lower-rank encoders are pruned more aggressively (Liu et al., 28 Jul 2025).

This rank-guided collaborative token assignment is one of METEOR’s distinctive contributions. It differs from equal per-encoder budgets and from simply pruning each encoder independently, because it treats the ensemble of encoders as a shared information budget whose allocation depends on measured representational richness (Liu et al., 28 Jul 2025).

4. Stage 2: cooperative pruning in the multi-vision fusion stage

After Stage 1, the pruned encoder outputs are projected into a shared space by per-encoder projectors and then fused (Liu et al., 28 Jul 2025). METEOR uses post-projection fusion rather than concatenating encoder outputs before projection. The post-projection design is described as advantageous because each encoder can be independently adapted to the shared semantic space, and because similarity comparisons across encoders become more meaningful after alignment (Liu et al., 28 Jul 2025). Table 3 in the paper reports that post-projection fusion improves accuracy even without additional pruning (Liu et al., 28 Jul 2025).

Stage 2 then targets cross-encoder redundancy explicitly. Even after within-encoder pruning, two encoders may contribute highly similar tokens for the same region or concept. For token $l$5 from encoder $l$6, METEOR defines mutual redundancy as

$l$7

where $l$8 is cosine similarity and $l$9 is the token count from encoder $\boldsymbol{g}^l$0 (Liu et al., 28 Jul 2025). Tokens with high $\boldsymbol{g}^l$1 are similar to many tokens from other encoders and are therefore treated as redundant. The pruning operator keeps the top-$\boldsymbol{g}^l$2 tokens with lowest mutual redundancy: $\boldsymbol{g}^l$3 The criterion is cooperative because the importance of a token is defined relationally, by what other encoders already contribute, rather than by its standalone saliency (Liu et al., 28 Jul 2025).

The paper reports a diversity analysis via nuclear norm showing that cooperative pruning yields higher diversity than no pruning, random pruning, or separate per-encoder pruning (Liu et al., 28 Jul 2025). It also compares cooperative pruning with parameter-based fusion or compression modules such as MLP, Resampler, and PixelShuffle, finding that cooperative pruning achieves higher accuracy at the same final token count of 576 without introducing additional learned parameters (Liu et al., 28 Jul 2025).

This stage is conceptually close to the broader idea of semantic collaborative pruning later explored in ViTCoP, where clustering preserves diversity and a second signal preserves task relevance (Luo et al., 25 Jan 2026). The difference is that METEOR’s Stage 2 is explicitly cross-encoder: it removes overlap among independently informative visual streams rather than diversity collapse within a single stream.

5. Stage 3: adaptive token pruning during LLM decoding

Stage 3 performs text-aware pruning during LLM decoding to address prompt specificity and task heterogeneity (Liu et al., 28 Jul 2025). The motivating observation is that visually guided pruning alone is insufficient because many retained tokens are irrelevant for a given textual instruction, and different tasks require different numbers of visual tokens. OCR and document-style tasks generally need more visual evidence than coarse-grained QA tasks such as AI2D (Liu et al., 28 Jul 2025).

At several LLM layers, exemplified by layers $\boldsymbol{g}^l$4, METEOR computes attention from the last instruction token $\boldsymbol{g}^l$5 to visual tokens for each head (Liu et al., 28 Jul 2025). For attention head $\boldsymbol{g}^l$6, the method defines the Visual Attention Value (VAV): $\boldsymbol{g}^l$7 where $\boldsymbol{g}^l$8 is the attention from $\boldsymbol{g}^l$9 to visual token $G^l$0 in head $G^l$1 (Liu et al., 28 Jul 2025). The paper argues that averaging across all heads is suboptimal because many heads have near-zero total attention to visual tokens and some heads are misleading (Liu et al., 28 Jul 2025). METEOR therefore sorts heads by VAV, selects the top-$G^l$2 heads,

$G^l$3

and aggregates their attention to compute token importance: $G^l$4 Visual tokens are ranked by $G^l$5, and the lowest-importance tokens are pruned (Liu et al., 28 Jul 2025).

The second key element is dynamic budget selection. METEOR defines the visual contribution level

$G^l$6

and sets the instance-specific number of retained visual tokens to

$G^l$7

where $G^l$8 is a scaling constant (Liu et al., 28 Jul 2025). Higher VC results in more retained tokens, and lower VC results in more aggressive pruning. The paper reports that, for the same average token count of around 242, adaptive pruning clearly outperforms fixed-ratio pruning, especially on OCR, with a reported $G^l$9 accuracy improvement (Liu et al., 28 Jul 2025). Top-$\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$0 head filtering itself yields an OCR improvement of about $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$1 over using all heads at the same token count (Liu et al., 28 Jul 2025).

This stage places METEOR in a broader line of text-guided and prompt-aware token pruning methods, but it differs from single-encoder approaches by applying adaptivity after multi-encoder fusion. Earlier work such as PuMer also used text-informed scores to reduce vision tokens, but did so in a cross-modal encoder rather than in a multi-encoder visual fusion stack (Cao et al., 2023). A plausible implication is that METEOR generalizes the principle of prompt-conditioned visual sparsification to the more difficult case where the visual evidence originates from multiple heterogeneous encoders.

6. Training protocol, efficiency, results, and relation to adjacent work

METEOR is trained in three phases aligned with its staged design (Liu et al., 28 Jul 2025). During pretraining for vision–language alignment, on the same 558k image–text pairs as LLaVA-1.5, all LLM and vision encoders are frozen and only the projectors $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$2 are trained; Stage 1 pruning is already used in multi-vision encoding (Liu et al., 28 Jul 2025). During supervised fine-tuning, using 1M instruction data in the base setup and 1.8M in the advanced setup, the whole model becomes trainable and Stage 2 cooperative pruning is integrated, with a fixed 576-token budget at fusion during training for stability (Liu et al., 28 Jul 2025). Stage 3 is training-free and is applied only at inference; its tunable components are the top-$\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$3 number of attention heads, the scaling constant $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$4, and the pruning layers (Liu et al., 28 Jul 2025).

The principal empirical result is that, relative to EAGLE, METEOR reduces around 76% of visual tokens on average, reduces TFLOPs by 49%, increases throughput by 46% on Ascend 910B, and incurs only 0.3% absolute performance drop in average across 11 benchmarks (Liu et al., 28 Jul 2025). In the efficient-method comparison based on EAGLE, METEOR uses approximately 242 adaptive tokens and reports TFLOPs of 13.42 versus 26.43 for EAGLE, throughput 1.18 times EAGLE’s, and average accuracy 69.0, compared with 64.9 for FastV, 55.5 for VTW, 63.5 for Pdrop, 63.9 for SparseVLM, 64.3 for PixelShuffle, and 64.1 for DeCo (Liu et al., 28 Jul 2025). The paper further reports that, using Vicuna-7B and around 242 tokens, METEOR outperforms LLaVA-Next with 2880 tokens on 10 of 11 benchmarks, and that with LLaMA-3-8B it surpasses EAGLE on 7 of 11 benchmarks while using less than 25% of EAGLE’s tokens (Liu et al., 28 Jul 2025).

The benchmark suite includes ScienceQA, AI2D, OKVQA, GQA, POPE, SEEDBench, MMBench, TextVQA, DocVQA, ChartQA, and OCRBench, among others depending on the evaluation setting (Liu et al., 28 Jul 2025). One notable result is that, using the same vision encoders as Cambrian-1—SigLIP, ConvNeXt, CLIP, and DINOv2—METEOR improves OCR tasks by $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$5 over MGM-HD and $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$6 over Cambrian-1 while using 44% fewer visual tokens than Cambrian-1 and only about 11% of MGM-HD’s tokens (Liu et al., 28 Jul 2025).

Ablation studies support each stage. Average-token similarity in shallow layers combined with attention in deep layers outperforms using either criterion alone. Rank-based sparsity allocation outperforms equal allocation and rank-inverse allocation, especially on OCR. Post-projection fusion outperforms pre-projection fusion. Cooperative cross-encoder pruning outperforms separate per-encoder pruning and random pruning. Top-$\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$7 head filtering and adaptive pruning ratios outperform using all heads and fixed ratios, respectively (Liu et al., 28 Jul 2025).

Within the broader literature, METEOR occupies a specific niche. PuMer provides a text-informed pruning and modality-aware merging framework for cross-modal encoders, with up to $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$8 throughput improvement and over 50% memory reduction while incurring less than a 1% accuracy drop, but it does not address multi-encoder fusion or rank-based budget allocation (Cao et al., 2023). ViTCoP later advances collaborative pruning for large vision-LLMs by combining vision-encoder pruning with LLM-side step-wise co-pruning and a FlashAttention-friendly K-vector $\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).$9-norm saliency metric, reporting especially strong performance under extreme pruning rates (Luo et al., 25 Jan 2026). In contrast, METEOR is specialized for the multi-encoder MLLM regime, where the central challenge is not only prompt-conditioned token saliency but also how to distribute pruning budgets across encoders with unequal information richness and how to remove redundancy after those encoders have been aligned into a shared space (Liu et al., 28 Jul 2025).

A common misconception is to treat METEOR as merely another prompt-aware LLM-side visual token dropper. The paper’s methodology shows otherwise: its primary novelty lies in combining encoder-internal progressive pruning, fusion-stage cross-encoder redundancy suppression, and inference-time text-aware adaptivity in a single coordinated pipeline (Liu et al., 28 Jul 2025). Another possible misconception is that the framework depends on a learned pruning controller throughout; in fact, Stage 3 is explicitly training-free, and Stage 2 cooperative pruning avoids extra learned parameters beyond the per-encoder projectors already used for fusion (Liu et al., 28 Jul 2025).

From a systems perspective, METEOR demonstrates that multi-encoder MLLMs need not choose between complementary visual representations and practical efficiency. Its contribution is the formulation of multi-encoder token pruning as a staged collaborative optimization problem in which information-rich encoders are protected, cross-encoder overlap is explicitly suppressed, and prompt-conditioned visual evidence is retained only to the degree demanded by the instance (Liu et al., 28 Jul 2025).