---
title: 'METEOR: Multi-Encoder Token Pruning'
url: https://www.emergentmind.com/topics/multi-encoder-collaborative-token-pruning-meteor
type: topic
---

# METEOR: Multi-Encoder Token Pruning

Searching arXiv for the METEOR paper and closely related token-pruning work to ground the article.
Multi-Encoder Collaborative Token Pruning (METEOR) is a progressive multi-stage token pruning framework for multi-encoder multimodal large language models (MLLMs), introduced to make multi-vision architectures efficient without discarding the complementary strengths that motivate their use in the first place [2507.20842]. It targets the characteristic inefficiency of systems such as EAGLE or Cambrian, where several vision encoders produce large visual token sets that are concatenated and processed by a decoder-only LLM. METEOR addresses redundancy at three distinct loci—within individual vision encoders, across encoders after projection into a shared space, and during LLM decoding—through rank-guided collaborative token assignment, cooperative fusion-stage pruning, and text-conditioned adaptive pruning, respectively [2507.20842]. The method is positioned as the first successful attempt to achieve an efficient multi-encoder vision language model with multi-stage pruning strategies, and it reports that, relative to EAGLE, it reduces 76% visual tokens with only 0.3% performance drop in average across 11 benchmarks [2507.20842].

## 1. Conceptual setting and motivation

METEOR is designed for the specific regime in which single-encoder MLLMs are no longer sufficient for broad multimodal competence, yet naïve multi-encoder fusion becomes computationally prohibitive [2507.20842]. The motivating observation is that CLIP-style single-encoder systems are weak at OCR, fine-grained grounding, and dense or domain-specific reasoning, whereas multi-encoder systems improve robustness by combining encoders with complementary inductive biases, such as CLIP, ConvNeXt, Pix2Struct, and EVA-02/EVA-CLIP [2507.20842]. The cost of this improvement is substantial token growth: multi-encoder models often produce thousands of tokens, and self-attention complexity scales as \(O(N^2 d)\) in token length \(N\) [2507.20842].

The framework therefore treats token redundancy as a multi-stage phenomenon rather than a single pruning problem. Redundancy arises during encoding, because high-resolution encoders produce many background or low-frequency tokens; during fusion, because different encoders often produce overlapping representations for the same regions; and during decoding, because many visual tokens are irrelevant to a specific prompt and many attention heads contribute little to vision grounding [2507.20842]. This staged diagnosis distinguishes METEOR from single-encoder pruning methods and from pruning schemes that act only inside the LLM.

This suggests that METEOR should be understood not merely as an efficiency technique but as an architectural coordination mechanism for multi-encoder MLLMs. The “collaborative” aspect refers both to coordination among encoders and to coordination among pruning stages, each of which uses a different signal: visual statistics during encoding, cross-encoder similarity during fusion, and vision–text attention during decoding [2507.20842].

## 2. System architecture and multi-stage pipeline

METEOR is built on an EAGLE-style multi-encoder architecture comprising multiple vision encoders, dedicated projectors, post-projection token fusion, and a decoder-only LLM such as Vicuna-7B, LLaMA-2, or LLaMA-3 [2507.20842]. In the main setup, the vision side includes four encoders: CLIP ViT, ConvNeXt, Pix2Struct, and EVA-02/EVA-CLIP [2507.20842]. Each encoder \(l\) produces tokens \(\boldsymbol{g}^l\), which are mapped into a shared semantic space by a dedicated 2-layer MLP projector \(G^l\):
\[
\boldsymbol{z}^l = G^l(\boldsymbol{g}^l).
\]
The projected tokens are then concatenated:
\[
\boldsymbol{z} = [\boldsymbol{z}^1, \dots, \boldsymbol{z}^L].
\]
These fused visual tokens are prepended or concatenated with text tokens and processed by the LLM [2507.20842].

The three METEOR stages are structurally distinct. Stage 1 prunes tokens inside each vision encoder progressively across depth. Stage 2 prunes across encoders after projection by removing mutually redundant tokens in the shared space. Stage 3 applies text-aware adaptive pruning during LLM decoding using attention from the last instruction token to visual tokens and dynamically adjusting the number of retained tokens according to inferred visual contribution [2507.20842].

A concise summary of the pipeline is given below.

| Stage | Location | Core signal |
|---|---|---|
| 1 | Within each vision encoder | Average-token similarity in shallow blocks; cls-attention in deep blocks; rank-guided per-encoder budgets |
| 2 | After per-encoder projectors, before LLM | Cross-encoder mutual redundancy in shared space |
| 3 | During LLM decoding | Vision–text attention from selected heads and adaptive token budget |

This multi-stage structure bears comparison to later collaborative pruning work for LVLMs such as ViTCoP, which also distributes pruning across the vision encoder and LLM depth, though in a two-encoder vision-plus-language setting rather than a true multi-encoder visual fusion setting [2601.17818]. A plausible implication is that METEOR helped define the design space for later collaborative pruning frameworks that combine encoder-local saliency, cross-stage coordination, and prompt-aware late pruning.

## 3. Stage 1: rank-guided collaborative token assignment in multi-vision encoding

Stage 1 accelerates the vision stack itself by progressively pruning tokens inside each vision encoder rather than waiting until all encoder outputs have been formed [2507.20842]. For block \(b\) in encoder \(l\), the token output is
\[
\boldsymbol{g}_b^l \in \mathbb{R}^{N_b^l \times D},
\]
where \(N_b^l\) is the token count and \(D\) is the feature dimension [2507.20842]. Instead of a single end-of-encoder pruning event, METEOR discards tokens at multiple blocks, relying on subsequent self-attention to propagate information and reduce abrupt information loss [2507.20842].

A central design choice is the use of different token-importance criteria at different depths. In shallow blocks, attention is regarded as unreliable because shallow layers exhibit high-entropy attention distributions and unstable top-\(k\) token indices across adjacent layers [2507.20842]. METEOR therefore computes the average token
\[
\bar{\boldsymbol{g}_b^l} = \frac{1}{N_b^l} \sum_{i=1}^{N_b^l} \boldsymbol{g}_{b,i}^l
\]
and evaluates cosine similarity between each token and this average [2507.20842]. Tokens most similar to the average are treated as background-like or low-information, and the pruning operator retains those least similar to the average:
\[
T(\boldsymbol{g}_b^l; k) = \operatorname{gather}\left[\boldsymbol{g}_b^l, \operatorname{sort}\big(-\mathcal{S}(\boldsymbol{g}_b^l,\textstyle\sum^{n}\boldsymbol{g}_b^l/n)\big) < k\right].
\]
In deeper blocks, where attention becomes sparser and more semantically meaningful, METEOR uses cls-token attention. If \(\mathbf{q}^{\mathrm{cls}}\) is the cls query and \(\mathbf{K}\) the keys of \(\boldsymbol{g}_b^l\), then
\[
\boldsymbol{a} = \operatorname{softmax}\left(\frac{\mathbf{q}^{\mathrm{cls}} \mathbf{K}^\top}{\sqrt{d_k}}\right),
\]
and the top-attended tokens are retained [2507.20842]. The encoder is equally split into three phases: the first phase uses average-token similarity, while the last two phases use cls-attention [2507.20842].

The “collaborative” element at this stage is the allocation of token budgets across encoders according to feature-map rank. METEOR performs SVD on \(\boldsymbol{g}_b^l\),
\[
\boldsymbol{g}_b^l = \sum_{i=1}^{r} \sigma_i \alpha_i w_i^\top,
\]
and interprets higher rank as higher information richness [2507.20842]. Because online SVD would be expensive, the expected ranks \(r_b^l\) are precomputed offline on a small batch of images and found to be stable across inputs [2507.20842]. Given a global budget \(k_b\) for block \(b\), METEOR assigns
\[
k_b^l = k_b \cdot \frac{r_b^l}{\sum_{l=1}^L r_b^l}.
\]
Thus high-rank encoders retain more tokens, while lower-rank encoders are pruned more aggressively [2507.20842].

This rank-guided collaborative token assignment is one of METEOR’s distinctive contributions. It differs from equal per-encoder budgets and from simply pruning each encoder independently, because it treats the ensemble of encoders as a shared information budget whose allocation depends on measured representational richness [2507.20842].

## 4. Stage 2: cooperative pruning in the multi-vision fusion stage

After Stage 1, the pruned encoder outputs are projected into a shared space by per-encoder projectors and then fused [2507.20842]. METEOR uses post-projection fusion rather than concatenating encoder outputs before projection. The post-projection design is described as advantageous because each encoder can be independently adapted to the shared semantic space, and because similarity comparisons across encoders become more meaningful after alignment [2507.20842]. Table 3 in the paper reports that post-projection fusion improves accuracy even without additional pruning [2507.20842].

Stage 2 then targets cross-encoder redundancy explicitly. Even after within-encoder pruning, two encoders may contribute highly similar tokens for the same region or concept. For token \(z_i^j\) from encoder \(j\), METEOR defines mutual redundancy as
\[
\mathcal{R}_i^j = \sum_{\substack{l=1 \\ l \neq j} }^L \sum_{m=1}^{n_l} \mathcal{S}(z_i^j, z_m^l),
\]
where \(\mathcal{S}\) is cosine similarity and \(n_l\) is the token count from encoder \(l\) [2507.20842]. Tokens with high \(\mathcal{R}_i^j\) are similar to many tokens from other encoders and are therefore treated as redundant. The pruning operator keeps the top-\(k\) tokens with lowest mutual redundancy:
\[
T(\boldsymbol{z}; k) = \operatorname{gather}\big[\boldsymbol{z}, \operatorname{sort}(-\boldsymbol{\mathcal{R}}) < k\big].
\]
The criterion is cooperative because the importance of a token is defined relationally, by what other encoders already contribute, rather than by its standalone saliency [2507.20842].

The paper reports a diversity analysis via nuclear norm showing that cooperative pruning yields higher diversity than no pruning, random pruning, or separate per-encoder pruning [2507.20842]. It also compares cooperative pruning with parameter-based fusion or compression modules such as MLP, Resampler, and PixelShuffle, finding that cooperative pruning achieves higher accuracy at the same final token count of 576 without introducing additional learned parameters [2507.20842].

This stage is conceptually close to the broader idea of semantic collaborative pruning later explored in ViTCoP, where clustering preserves diversity and a second signal preserves task relevance [2601.17818]. The difference is that METEOR’s Stage 2 is explicitly cross-encoder: it removes overlap among independently informative visual streams rather than diversity collapse within a single stream.

## 5. Stage 3: adaptive token pruning during LLM decoding

Stage 3 performs text-aware pruning during LLM decoding to address prompt specificity and task heterogeneity [2507.20842]. The motivating observation is that visually guided pruning alone is insufficient because many retained tokens are irrelevant for a given textual instruction, and different tasks require different numbers of visual tokens. OCR and document-style tasks generally need more visual evidence than coarse-grained QA tasks such as AI2D [2507.20842].

At several LLM layers, exemplified by layers \([4, 12, 20]\), METEOR computes attention from the last instruction token \(t\) to visual tokens for each head [2507.20842]. For attention head \(h\), the method defines the Visual Attention Value (VAV):
\[
\operatorname{VAV}_h \triangleq \sum_{i=1}^{N} a_{i,h},
\]
where \(a_{i,h}\) is the attention from \(t\) to visual token \(z_i\) in head \(h\) [2507.20842]. The paper argues that averaging across all heads is suboptimal because many heads have near-zero total attention to visual tokens and some heads are misleading [2507.20842]. METEOR therefore sorts heads by VAV, selects the top-\(k\) heads,
\[
\boldsymbol{I} = \arg\max(\operatorname{VAV}_1, \dots, \operatorname{VAV}_H)[:k],
\]
and aggregates their attention to compute token importance:
\[
a'_i = \sum_{h=1}^{k} a_{i,\boldsymbol{I}(h)}.
\]
Visual tokens are ranked by \(a'_i\), and the lowest-importance tokens are pruned [2507.20842].

The second key element is dynamic budget selection. METEOR defines the visual contribution level
\[
\text{VC} = \sum_{h=1}^{k} \sum_{i=1}^{N} a_{i,\boldsymbol{I}(h)},
\]
and sets the instance-specific number of retained visual tokens to
\[
K = \lambda \cdot \text{VC},
\]
where \(\lambda\) is a scaling constant [2507.20842]. Higher VC results in more retained tokens, and lower VC results in more aggressive pruning. The paper reports that, for the same average token count of around 242, adaptive pruning clearly outperforms fixed-ratio pruning, especially on OCR, with a reported \(+2.6\%\) accuracy improvement [2507.20842]. Top-\(k\) head filtering itself yields an OCR improvement of about \(+0.8\%\) over using all heads at the same token count [2507.20842].

This stage places METEOR in a broader line of text-guided and prompt-aware token pruning methods, but it differs from single-encoder approaches by applying adaptivity after multi-encoder fusion. Earlier work such as PuMer also used text-informed scores to reduce vision tokens, but did so in a cross-modal encoder rather than in a multi-encoder visual fusion stack [2305.17530]. A plausible implication is that METEOR generalizes the principle of prompt-conditioned visual sparsification to the more difficult case where the visual evidence originates from multiple heterogeneous encoders.

## 6. Training protocol, efficiency, results, and relation to adjacent work

METEOR is trained in three phases aligned with its staged design [2507.20842]. During pretraining for vision–language alignment, on the same 558k image–text pairs as LLaVA-1.5, all LLM and vision encoders are frozen and only the projectors \(G^l\) are trained; Stage 1 pruning is already used in multi-vision encoding [2507.20842]. During supervised fine-tuning, using 1M instruction data in the base setup and 1.8M in the advanced setup, the whole model becomes trainable and Stage 2 cooperative pruning is integrated, with a fixed 576-token budget at fusion during training for stability [2507.20842]. Stage 3 is training-free and is applied only at inference; its tunable components are the top-\(k\) number of attention heads, the scaling constant \(\lambda\), and the pruning layers [2507.20842].

The principal empirical result is that, relative to EAGLE, METEOR reduces around 76% of visual tokens on average, reduces TFLOPs by 49%, increases throughput by 46% on Ascend 910B, and incurs only 0.3% absolute performance drop in average across 11 benchmarks [2507.20842]. In the efficient-method comparison based on EAGLE, METEOR uses approximately 242 adaptive tokens and reports TFLOPs of 13.42 versus 26.43 for EAGLE, throughput 1.18 times EAGLE’s, and average accuracy 69.0, compared with 64.9 for FastV, 55.5 for VTW, 63.5 for Pdrop, 63.9 for SparseVLM, 64.3 for PixelShuffle, and 64.1 for DeCo [2507.20842]. The paper further reports that, using Vicuna-7B and around 242 tokens, METEOR outperforms LLaVA-Next with 2880 tokens on 10 of 11 benchmarks, and that with LLaMA-3-8B it surpasses EAGLE on 7 of 11 benchmarks while using less than 25% of EAGLE’s tokens [2507.20842].

The benchmark suite includes ScienceQA, AI2D, OKVQA, GQA, POPE, SEEDBench, MMBench, TextVQA, DocVQA, ChartQA, and OCRBench, among others depending on the evaluation setting [2507.20842]. One notable result is that, using the same vision encoders as Cambrian-1—SigLIP, ConvNeXt, CLIP, and DINOv2—METEOR improves OCR tasks by \(+11.7\%\) over MGM-HD and \(+3.3\%\) over Cambrian-1 while using 44% fewer visual tokens than Cambrian-1 and only about 11% of MGM-HD’s tokens [2507.20842].

Ablation studies support each stage. Average-token similarity in shallow layers combined with attention in deep layers outperforms using either criterion alone. Rank-based sparsity allocation outperforms equal allocation and rank-inverse allocation, especially on OCR. Post-projection fusion outperforms pre-projection fusion. Cooperative cross-encoder pruning outperforms separate per-encoder pruning and random pruning. Top-\(k\) head filtering and adaptive pruning ratios outperform using all heads and fixed ratios, respectively [2507.20842].

Within the broader literature, METEOR occupies a specific niche. PuMer provides a text-informed pruning and modality-aware merging framework for cross-modal encoders, with up to \(2\times\) throughput improvement and over 50% memory reduction while incurring less than a 1% accuracy drop, but it does not address multi-encoder fusion or rank-based budget allocation [2305.17530]. ViTCoP later advances collaborative pruning for large vision-language models by combining vision-encoder pruning with LLM-side step-wise co-pruning and a FlashAttention-friendly K-vector \(L_2\)-norm saliency metric, reporting especially strong performance under extreme pruning rates [2601.17818]. In contrast, METEOR is specialized for the multi-encoder MLLM regime, where the central challenge is not only prompt-conditioned token saliency but also how to distribute pruning budgets across encoders with unequal information richness and how to remove redundancy after those encoders have been aligned into a shared space [2507.20842].

A common misconception is to treat METEOR as merely another prompt-aware LLM-side visual token dropper. The paper’s methodology shows otherwise: its primary novelty lies in combining encoder-internal progressive pruning, fusion-stage cross-encoder redundancy suppression, and inference-time text-aware adaptivity in a single coordinated pipeline [2507.20842]. Another possible misconception is that the framework depends on a learned pruning controller throughout; in fact, Stage 3 is explicitly training-free, and Stage 2 cooperative pruning avoids extra learned parameters beyond the per-encoder projectors already used for fusion [2507.20842].

From a systems perspective, METEOR demonstrates that multi-encoder MLLMs need not choose between complementary visual representations and practical efficiency. Its contribution is the formulation of multi-encoder token pruning as a staged collaborative optimization problem in which information-rich encoders are protected, cross-encoder overlap is explicitly suppressed, and prompt-conditioned visual evidence is retained only to the degree demanded by the instance [2507.20842].

Source: https://www.emergentmind.com/topics/multi-encoder-collaborative-token-pruning-meteor