---
title: 'Token-Pooling MoE: Efficient Visual Compression'
url: https://www.emergentmind.com/topics/token-pooling-based-mixture-of-experts
type: topic
---

# Token-Pooling MoE: Efficient Visual Compression

Searching arXiv for recent papers on token-pooling-based mixture-of-experts and QMoP.
Token-pooling-based Mixture-of-Experts, in the formulation of "QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression," is a visual token compression architecture for multimodal large language models in which a pooling-based branch for coarse-grained global semantics, a resampler branch for extracting high-level semantic representations, and a pruning-based branch for fine-grained token selection are coordinated by a Query Guided Router and combined through a Mixture-of-Experts-style fusion mechanism [2603.21232]. The design is motivated by the observation that multimodal large language models suffer from severe computational and memory bottlenecks because the number of visual tokens far exceeds that of textual tokens, while fixed compression heuristics limit adaptability across diverse scenarios.

## 1. Problem Setting and Conceptual Scope

In QMoP, the input visual sequence is the raw visual tokens from a frozen ViT encoder,
$$
Z_v \in \mathbb{R}^{N\times d},
$$
where $N=H\cdot W$ is the number of patch tokens and $d$ is the per-token embedding dimension [2603.21232]. The objective is not generic expert specialization over feed-forward capacity, but adaptive compression of $Z_v$ into $M$ tokens with $M\ll N$.

The central idea is to distribute compression across three complementary specialists rather than a single fixed operator. The pooling-based branch targets coarse-grained global semantics; the resampler branch targets high-level semantic abstraction; and the pruning-based branch targets fine-grained token selection to preserve critical visual detail. The router then dynamically selects and weights the outputs from different branches based on both visual input and textual queries, and the final compressed representation is obtained by a weighted sum over expert outputs [2603.21232].

This suggests a narrower and more task-specific use of Mixture-of-Experts than in conventional Transformer MoE literature. Here, the experts are compression modules operating on visual tokens, and the routing signal is explicitly multimodal because it depends on both image and text. A plausible implication is that token-pooling-based MoE is most naturally interpreted as an adaptive token compressor rather than merely a sparse parameter-scaling device.

## 2. Three Expert Branches for Visual Token Compression

QMoP builds three experts that each compress $Z_v$ into $M$ tokens in complementary ways [2603.21232].

| Branch | Role | Output |
|---|---|---|
| Pooling-based branch | Coarse-grained global semantics | $E_{\rm pool}\in\mathbb{R}^{M\times d}$ |
| Resampler branch | High-level semantic representations | $E_{\rm resamp}\in\mathbb{R}^{M\times d}$ |
| Pruning-based branch | Fine-grained token selection | $E_{\rm prune}\in\mathbb{R}^{M\times d}$ |

The pooling-based branch begins by reshaping $Z_v$ to a 2D feature map
$$
X_{\rm img2d}\in\mathbb{R}^{H\times W\times d},
$$
with $H\cdot W=N$. A spatial down-sampling factor $s$ is chosen so that $h=H/s$ and $w=W/s$, with $M=h\cdot w$. A learnable 2D query map
$$
Q_{2d}\in\mathbb{R}^{h\times w\times d}
$$
is introduced, and for each grid cell $(i,j)$ the local window is
$$
X_{i,j}=X_{\rm img2d}[s\cdot i:s(i+1),\, s\cdot j:s(j+1)]\in\mathbb{R}^{s^2\times d}.
$$
Keys and values are produced by learned linear projections,
$$
K_{i,j}=\phi_k(X_{i,j}),\qquad V_{i,j}=\phi_v(X_{i,j}),
$$
and one token per grid cell is computed through a local attention or pooling head:
$$
e^{\rm (pool)}_{i,j}
=\mathrm{Softmax}\!\bigl(Q_{2d}[i,j]\;K_{i,j}^{T}/\sqrt{d}\bigr)\;V_{i,j}
\in\mathbb{R}^{1\times d}.
$$
Collecting all cells yields
$$
E_{\rm pool}=\bigl[e^{\rm (pool)}_{i,j}\bigr]_{i=1..h,j=1..w}\in\mathbb{R}^{M\times d}.
$$

The resampler branch introduces $M$ learnable query tokens
$$
Q\in\mathbb{R}^{M\times d}
$$
and applies cross-attention against all $N$ visual tokens:
$$
A=\mathrm{Softmax}\bigl(QK^T/\sqrt d\bigr),\qquad
K=Z_vW_K,\qquad
V=Z_vW_V,
$$
with $W_K,W_V\in\mathbb{R}^{d\times d}$. The compressed representation is
$$
E_{\rm resamp}=AV\in\mathbb{R}^{M\times d}.
$$

The pruning-based branch combines image-side importance and text-side relevance. From the visual encoder it takes the penultimate class token $v_{CLS}\in\mathbb{R}^d$, and from the CLIP text encoder it takes the EOS token $t_{EOS}\in\mathbb{R}^d$. For each token $Z_v^{(i)}$, importance is defined by the class token’s attention weight to token $i$,
$$
s^{(i)}_{\rm imp}=\mathrm{AttentionWeight}(v_{CLS}\rightarrow Z_v^{(i)}),
$$
and relevance is defined by similarity to the EOS text embedding, for example
$$
s^{(i)}_{\rm rel}=\langle Z_v^{(i)},\,t_{EOS}\rangle
$$
or negative $\ell_2$-distance. These are fused as
$$
s^{(i)}=\lambda\,s^{(i)}_{\rm imp}+(1-\lambda)\,s^{(i)}_{\rm rel},\qquad \lambda=0.5.
$$
The top-$M$ tokens above a threshold $\eta$ (or fixed ratio) form
$$
E_{\rm prune}=\{\,Z_v^{(i)}\mid s^{(i)}\ge \eta\,\}\in\mathbb{R}^{M\times d}.
$$

Taken together, the three branches instantiate a division of labor over global structure, semantic abstraction, and detail preservation. The paper’s summary characterizes them as token-pooling (global), token-resampling (semantic), and token-pruning (detail) specialists [2603.21232].

## 3. Query-Guided Routing and MoE-Style Fusion

The Query Guided Router produces a weight $\alpha_i$ for each expert $i\in\{\text{pool},\text{resamp},\text{prune}\}$ conditioned on both image and text [2603.21232]. It extracts $v_{CLS}\in\mathbb{R}^{d}$ from the vision encoder and $t_{EOS}\in\mathbb{R}^{d}$ from the CLIP text encoder, concatenates them into a joint query feature
$$
f=[\,v_{CLS};\,t_{EOS}\,]\in\mathbb{R}^{2d},
$$
and passes $f$ through a two-layer MLP with hidden width $h_r$ and temperature $\tau$:
$$
l = W_2\,\sigma\bigl(W_1 f + b_1\bigr) + b_2 \in\mathbb R^{3},\qquad
\alpha = \mathrm{softmax}\!\bigl(l/\tau + g\bigr),
$$
where
$$
W_1\in\mathbb{R}^{h_r\times 2d},\; b_1\in\mathbb{R}^{h_r},\;
W_2\in\mathbb{R}^{3\times h_r},\; b_2\in\mathbb{R}^{3},
$$
$\sigma$ is e.g. GELU, and Gumbel noise $g$ may be injected to encourage discrete decisions during training.

The final token set is obtained by a weighted sum over expert outputs. Let
$$
e_1=E_{\rm pool},\qquad e_2=E_{\rm resamp},\qquad e_3=E_{\rm prune},
$$
all in $\mathbb{R}^{M\times d}$, or re-projected to a common size. Then
$$
E=\sum_{i=1}^3 \alpha_i\,e_i\in\mathbb{R}^{M\times d}.
$$
The mixture is described as one that “Harnesses” the branch whose $\alpha_i$ is large, “Suppresses” noise from less relevant branches, and allows dynamic adaptation per input pair $(I_{\rm img},I_{\rm text})$ [2603.21232].

A common misconception is to treat this routing step as equivalent to standard top-$K$ sparse MoE gating over neural experts. In QMoP, the routed objects are compressed token sets produced by heterogeneous projector branches. The routing decision is therefore over compression strategies rather than over parallel FFNs.

## 4. Training Stages, Hyperparameters, and Inference Behavior

The implementation details are explicit. Feature dimensions are set to $d=1024$, and the hidden width in QGR is $h_r=512$. In pruning, $\lambda=0.5$, and the pruning ratio is set so $M\approx 64$–$144$. The parameters $\phi_k$, $\phi_v$, $W_K$, $W_V\in\mathbb{R}^{d\times d}$, $Q_{2d}\in\mathbb{R}^{h\times w\times d}$, and $Q\in\mathbb{R}^{M\times d}$ are all learnable [2603.21232].

Training proceeds in two stages. During Stage 1, described as projector pre-training, QGR is disabled; all three outputs are concatenated and passed through an MLP to align to text embedding space with a simple $\ell_2$ reconstruction loss. During Stage 2, described as joint fine-tuning with LLM, QGR is activated; Gumbel-softmax with annealed $\tau\in[1\rightarrow 0.1]$ encourages the router to sharpen [2603.21232].

Inference introduces explicit sparsity. During inference $\tau\rightarrow 0$, corresponding to hard selection, and only the top-2 experts by $\alpha$ are kept; weights are re-normalized over those two. The summary therefore describes QMoP as a lightweight Mixture-of-Experts in which “a small Query-Guided Router steers computation to the two most relevant experts per image-text pair” and a weighted-sum fusion yields “a compact, noise-suppressed set of $M$ tokens perfectly aligned for the downstream LLM” [2603.21232].

This staged schedule clarifies that token-pooling-based MoE in QMoP is not purely a routing mechanism from the outset. The projector branches are first aligned to text embedding space, and only then is query-conditioned specialization activated. A plausible implication is that the method seeks to stabilize branch representations before imposing competitive routing.

## 5. Empirical Characterization and Evaluation

QMoP introduces VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression [2603.21232]. The benchmark is used to isolate how different compression operators affect retained information, rather than measuring only downstream task accuracy.

On the VTCBench ablation reported in Tab. 3, the pooling-only expert (“B”) achieves the best Global Understanding with $GU = 56.5$ and strong Spatial/Counting with $SC = 47.8$, outperforming pruning-only and resampling-only. Combining pooling+pruning (B + C) without QGR yields $GU = 58.6$ but lower reasoning (RC) and style (SE). Full QMoP with router yields balanced performance across all five dimensions with $Avg = 61.8$ versus baseline $61.0$ [2603.21232].

On standard benchmarks, including VQAv2, GQA, and TextVQA, QMoP matches or exceeds all single-paradigm compressors at the same token budget, with the example given as $144$ tokens. The abstract further states that, despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time [2603.21232].

These results are notable because the pooling-only expert already performs strongly on Global Understanding and Spatial/Counting, yet the full routed mixture is preferred for balanced behavior. This suggests that token-pooling-based MoE is not simply a mechanism for selecting the strongest single compressor, but for reconciling partially conflicting desiderata across global semantics, detail retention, reasoning, and style-sensitive performance.

## 6. Relation to Broader Token-Centric MoE Research

A useful comparison is provided by "Mixture of Tokens: Continuous MoE through Cross-Example Aggregation" [2310.15961]. MoT replaces each standard Transformer feed-forward sub-layer by a Mixture-of-Tokens layer with $E$ parallel experts, where each expert receives a mixed token
$$
m_e=\sum_{i\in G}\alpha_{i,e}x_i
$$
formed from a group $G$ of tokens, computes $z_e=FFN_e(m_e)$, and redistributes expert outputs back to tokens through the same continuous weights:
$$
y_i=\sum_{e=1}^E \alpha_{i,e}z_e.
$$
Because $\alpha_{\cdot,e}$ is produced by a softmax, the operations are fully differentiable. The paper emphasizes that this architecture is fully compatible with autoregressive training and generation, and that in the $\tau\rightarrow 0$ limit it recovers exactly Expert-Choice sparse MoE in the forward pass [2310.15961].

The comparison helps separate two distinct notions of token-level MoE. In QMoP, experts are heterogeneous visual token compression branches and routing is conditioned on both image and text. In MoT, experts are FFNs, and the defining operation is cross-example token aggregation within grouped positions across the batch. MoT reports that its best models achieve a $3\times$ increase in training speed over dense Transformer models in language pretraining, reach the dense model’s final training loss in only $24\%$ of steps, achieve the same loss in about $33\%$ of the training time, and are on par with the dense baseline in perplexity at convergence [2310.15961].

The distinction matters for terminology. Token-pooling-based Mixture-of-Experts, as exemplified by QMoP, is best understood as MoE over token compression operators. MoT, by contrast, is a continuous MoE through token aggregation in the Transformer’s feed-forward pathway. The shared structure is the use of soft routing, expert mixing, and weighted redistribution; the differing structure is where the experts intervene and what they are meant to optimize. A plausible implication is that the two lines of work are complementary rather than interchangeable: one addresses visual token compression for multimodal LLMs, while the other addresses scalable continuous expert computation in autoregressive Transformers.

Source: https://www.emergentmind.com/topics/token-pooling-based-mixture-of-experts