Token-Pooling MoE: Efficient Visual Compression

• The paper introduces a token-pooling-based Mixture-of-Experts (QMoP) that adaptively compresses visual tokens by coordinating pooling, resampling, and pruning branches.
• It leverages a query-guided router to dynamically fuse expert outputs, balancing global semantics, high-level abstraction, and fine-grained detail retention.
• Empirical evaluations on VTCBench and multimodal benchmarks demonstrate that QMoP achieves efficient visual token compression while maintaining strong reasoning and counting performance.

Searching arXiv for papers on token-pooling-based mixture-of-experts and QMoP. Token-pooling-based Mixture-of-Experts, in the formulation of "QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression," is a visual token compression architecture for multimodal LLMs in which a pooling-based branch for coarse-grained global semantics, a resampler branch for extracting high-level semantic representations, and a pruning-based branch for fine-grained token selection are coordinated by a Query Guided Router and combined through a Mixture-of-Experts-style fusion mechanism (Li et al., 22 Mar 2026). The design is motivated by the observation that multimodal LLMs suffer from severe computational and memory bottlenecks because the number of visual tokens far exceeds that of textual tokens, while fixed compression heuristics limit adaptability across diverse scenarios.

1. Problem Setting and Conceptual Scope

In QMoP, the input visual sequence is the raw visual tokens from a frozen ViT encoder,

$Z_v \in \mathbb{R}^{N\times d},$

where $N=H\cdot W$ is the number of patch tokens and $d$ is the per-token embedding dimension (Li et al., 22 Mar 2026). The objective is not generic expert specialization over feed-forward capacity, but adaptive compression of $Z_v$ into $M$ tokens with $M\ll N$.

The central idea is to distribute compression across three complementary specialists rather than a single fixed operator. The pooling-based branch targets coarse-grained global semantics; the resampler branch targets high-level semantic abstraction; and the pruning-based branch targets fine-grained token selection to preserve critical visual detail. The router then dynamically selects and weights the outputs from different branches based on both visual input and textual queries, and the final compressed representation is obtained by a weighted sum over expert outputs (Li et al., 22 Mar 2026).

This suggests a narrower and more task-specific use of Mixture-of-Experts than in conventional Transformer MoE literature. Here, the experts are compression modules operating on visual tokens, and the routing signal is explicitly multimodal because it depends on both image and text. A plausible implication is that token-pooling-based MoE is most naturally interpreted as an adaptive token compressor rather than merely a sparse parameter-scaling device.

2. Three Expert Branches for Visual Token Compression

QMoP builds three experts that each compress $Z_v$ into $M$ tokens in complementary ways (Li et al., 22 Mar 2026).

Branch	Role	Output
Pooling-based branch	Coarse-grained global semantics	$E_{\rm pool}\in\mathbb{R}^{M\times d}$
Resampler branch	High-level semantic representations	$E_{\rm resamp}\in\mathbb{R}^{M\times d}$
Pruning-based branch	Fine-grained token selection	$N=H\cdot W$0

The pooling-based branch begins by reshaping $N=H\cdot W$1 to a 2D feature map

$N=H\cdot W$2

with $N=H\cdot W$3. A spatial down-sampling factor $N=H\cdot W$4 is chosen so that $N=H\cdot W$5 and $N=H\cdot W$6, with $N=H\cdot W$7. A learnable 2D query map

$N=H\cdot W$8

is introduced, and for each grid cell $N=H\cdot W$9 the local window is

$d$0

Keys and values are produced by learned linear projections,

$d$1

and one token per grid cell is computed through a local attention or pooling head:

$d$2

Collecting all cells yields

$d$3

The resampler branch introduces $d$4 learnable query tokens

$d$5

and applies cross-attention against all $d$6 visual tokens:

$d$7

with $d$8. The compressed representation is

$d$9

The pruning-based branch combines image-side importance and text-side relevance. From the visual encoder it takes the penultimate class token $Z_v$0, and from the CLIP text encoder it takes the EOS token $Z_v$1. For each token $Z_v$2, importance is defined by the class token’s attention weight to token $Z_v$3,

$Z_v$4

and relevance is defined by similarity to the EOS text embedding, for example

$Z_v$5

or negative $Z_v$6-distance. These are fused as

$Z_v$7

The top-$Z_v$8 tokens above a threshold $Z_v$9 (or fixed ratio) form

$M$0

Taken together, the three branches instantiate a division of labor over global structure, semantic abstraction, and detail preservation. The paper’s summary characterizes them as token-pooling (global), token-resampling (semantic), and token-pruning (detail) specialists (Li et al., 22 Mar 2026).

3. Query-Guided Routing and MoE-Style Fusion

The Query Guided Router produces a weight $M$1 for each expert $M$2 conditioned on both image and text (Li et al., 22 Mar 2026). It extracts $M$3 from the vision encoder and $M$4 from the CLIP text encoder, concatenates them into a joint query feature

$M$5

and passes $M$6 through a two-layer MLP with hidden width $M$7 and temperature $M$8:

$M$9

where

$M\ll N$0

$M\ll N$1 is e.g. GELU, and Gumbel noise $M\ll N$2 may be injected to encourage discrete decisions during training.

The final token set is obtained by a weighted sum over expert outputs. Let

$M\ll N$3

all in $M\ll N$4, or re-projected to a common size. Then

$M\ll N$5

The mixture is described as one that “Harnesses” the branch whose $M\ll N$6 is large, “Suppresses” noise from less relevant branches, and allows dynamic adaptation per input pair $M\ll N$7 (Li et al., 22 Mar 2026).

A common misconception is to treat this routing step as equivalent to standard top-$M\ll N$8 sparse MoE gating over neural experts. In QMoP, the routed objects are compressed token sets produced by heterogeneous projector branches. The routing decision is therefore over compression strategies rather than over parallel FFNs.

4. Training Stages, Hyperparameters, and Inference Behavior

The implementation details are explicit. Feature dimensions are set to $M\ll N$9, and the hidden width in QGR is $Z_v$0. In pruning, $Z_v$1, and the pruning ratio is set so $Z_v$2–$Z_v$3. The parameters $Z_v$4, $Z_v$5, $Z_v$6, $Z_v$7, $Z_v$8, and $Z_v$9 are all learnable (Li et al., 22 Mar 2026).

Training proceeds in two stages. During Stage 1, described as projector pre-training, QGR is disabled; all three outputs are concatenated and passed through an MLP to align to text embedding space with a simple $M$0 reconstruction loss. During Stage 2, described as joint fine-tuning with LLM, QGR is activated; Gumbel-softmax with annealed $M$1 encourages the router to sharpen (Li et al., 22 Mar 2026).

Inference introduces explicit sparsity. During inference $M$2, corresponding to hard selection, and only the top-2 experts by $M$3 are kept; weights are re-normalized over those two. The summary therefore describes QMoP as a lightweight Mixture-of-Experts in which “a small Query-Guided Router steers computation to the two most relevant experts per image-text pair” and a weighted-sum fusion yields “a compact, noise-suppressed set of $M$4 tokens perfectly aligned for the downstream LLM” (Li et al., 22 Mar 2026).

This staged schedule clarifies that token-pooling-based MoE in QMoP is not purely a routing mechanism from the outset. The projector branches are first aligned to text embedding space, and only then is query-conditioned specialization activated. A plausible implication is that the method seeks to stabilize branch representations before imposing competitive routing.

5. Empirical Characterization and Evaluation

QMoP introduces VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression (Li et al., 22 Mar 2026). The benchmark is used to isolate how different compression operators affect retained information, rather than measuring only downstream task accuracy.

On the VTCBench ablation reported in Tab. 3, the pooling-only expert (“B”) achieves the best Global Understanding with $M$5 and strong Spatial/Counting with $M$6, outperforming pruning-only and resampling-only. Combining pooling+pruning (B + C) without QGR yields $M$7 but lower reasoning (RC) and style (SE). Full QMoP with router yields balanced performance across all five dimensions with $M$8 versus baseline $M$9 (Li et al., 22 Mar 2026).

On standard benchmarks, including VQAv2, GQA, and TextVQA, QMoP matches or exceeds all single-paradigm compressors at the same token budget, with the example given as $E_{\rm pool}\in\mathbb{R}^{M\times d}$0 tokens. The abstract further states that, despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time (Li et al., 22 Mar 2026).

These results are notable because the pooling-only expert already performs strongly on Global Understanding and Spatial/Counting, yet the full routed mixture is preferred for balanced behavior. This suggests that token-pooling-based MoE is not simply a mechanism for selecting the strongest single compressor, but for reconciling partially conflicting desiderata across global semantics, detail retention, reasoning, and style-sensitive performance.

6. Relation to Broader Token-Centric MoE Research

A useful comparison is provided by "Mixture of Tokens: Continuous MoE through Cross-Example Aggregation" (Antoniak et al., 2023). MoT replaces each standard Transformer feed-forward sub-layer by a Mixture-of-Tokens layer with $E_{\rm pool}\in\mathbb{R}^{M\times d}$1 parallel experts, where each expert receives a mixed token

$E_{\rm pool}\in\mathbb{R}^{M\times d}$2

formed from a group $E_{\rm pool}\in\mathbb{R}^{M\times d}$3 of tokens, computes $E_{\rm pool}\in\mathbb{R}^{M\times d}$4, and redistributes expert outputs back to tokens through the same continuous weights:

$E_{\rm pool}\in\mathbb{R}^{M\times d}$5

Because $E_{\rm pool}\in\mathbb{R}^{M\times d}$6 is produced by a softmax, the operations are fully differentiable. The paper emphasizes that this architecture is fully compatible with autoregressive training and generation, and that in the $E_{\rm pool}\in\mathbb{R}^{M\times d}$7 limit it recovers exactly Expert-Choice sparse MoE in the forward pass (Antoniak et al., 2023).

The comparison helps separate two distinct notions of token-level MoE. In QMoP, experts are heterogeneous visual token compression branches and routing is conditioned on both image and text. In MoT, experts are FFNs, and the defining operation is cross-example token aggregation within grouped positions across the batch. MoT reports that its best models achieve a $E_{\rm pool}\in\mathbb{R}^{M\times d}$8 increase in training speed over dense Transformer models in language pretraining, reach the dense model’s final training loss in only $E_{\rm pool}\in\mathbb{R}^{M\times d}$9 of steps, achieve the same loss in about $E_{\rm resamp}\in\mathbb{R}^{M\times d}$0 of the training time, and are on par with the dense baseline in perplexity at convergence (Antoniak et al., 2023).

The distinction matters for terminology. Token-pooling-based Mixture-of-Experts, as exemplified by QMoP, is best understood as MoE over token compression operators. MoT, by contrast, is a continuous MoE through token aggregation in the Transformer’s feed-forward pathway. The shared structure is the use of soft routing, expert mixing, and weighted redistribution; the differing structure is where the experts intervene and what they are meant to optimize. A plausible implication is that the two lines of work are complementary rather than interchangeable: one addresses visual token compression for multimodal LLMs, while the other addresses scalable continuous expert computation in autoregressive Transformers.