Informative Token Aggregation Module

• Informative token aggregation module is a neural component that compresses large sets of tokens or patches into a smaller, information-rich format while maintaining key details.
• It integrates operations like local pooling, explicit bottlenecking, graph-based summarization, and frequency-domain gating to optimize efficiency and performance.
• This approach is applied in diverse fields such as point-cloud processing, pathology image analysis, and multimodal retrieval, proving its utility in preserving critical information under high compression.

An informative token aggregation module is a neural component that compresses a set of token-, patch-, point-, or multimodal representations into a smaller set of tokens or into a sample-level embedding while attempting to preserve the information most relevant to the downstream task. Across recent work, this idea appears in point-cloud tokenization and token-to-point projection, explicit bottleneck tokens for multimodal retrieval, semantic-slot routing for gigapixel pathology, graph-based visual token summarization for multimodal LLMs, and frequency-domain gating before final pooling (Xu et al., 2021, Sun et al., 13 Apr 2026, Chen et al., 1 Mar 2026, Jiang et al., 25 Aug 2025, Li et al., 6 Jun 2026).

1. Definition and Scope

The term covers a family of mechanisms rather than a single standardized architecture. In some settings, the module reduces a large set of local entities to a compact set of latent representatives; in others, it constructs a single retrieval or classification embedding; in yet others, it compresses token representations themselves by replacing monolithic embeddings with compositional semantic building blocks. What unifies these variants is that aggregation is treated as a learned or structured information bottleneck rather than as a purely incidental pooling step (V et al., 22 Sep 2025, Behrendt et al., 21 May 2025).

A recurring motivation is that conventional pooling or repeated local grouping either wastes computation on redundant elements or discards fine-grained information. Point-cloud pipelines such as PointNet++ repeatedly resample and regroup points; decoder-only retrieval systems often overload a single vocabulary token such as <EOS> as a sequence-level embedding; whole-slide pathology models face more than $10^5$ patches per slide; multimodal LLMs can be dominated by visual tokens; and ViT-based retrieval or adaptation pipelines suffer when token compression removes information that later modules cannot reconstruct (Xu et al., 2021, Sun et al., 13 Apr 2026, Chen et al., 1 Mar 2026, Jiang et al., 25 Aug 2025, Xiong et al., 5 Aug 2025).

For that reason, recent modules tend to combine four operations: selecting or synthesizing representative tokens, enriching them by relational modeling, preserving discarded information through residual, projection, or aggregation paths, and exposing a fixed-capacity interface to downstream heads. This suggests a broad but technically coherent notion of informative aggregation: compression is acceptable only if the compressed representation remains structurally aligned with the target task and retains access to the salient information that would otherwise be spread across many local tokens (Xu et al., 2021, Sun et al., 13 Apr 2026, Rios et al., 2024).

2. Canonical Operations and Formal Patterns

Across domains, informative aggregation is usually instantiated through a small set of recurring operators. Some methods begin with pooling or routing into latent representatives; some introduce explicit bottleneck tokens; some aggregate by graph propagation or frequency-domain reweighting; and some aggregate evidence across layers or across generated tokens rather than across spatial positions.

Operator family	Representative expression	Representative works
Local pooling into tokens	$$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$	YOGO (Xu et al., 2021)
Explicit bottleneck pooling	$e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$	BToks (Sun et al., 13 Apr 2026)
Sparse routing into slots	$$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$	TC-SSA (Chen et al., 1 Mar 2026)
Graph-based token summarization	$$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$	VISA (Jiang et al., 25 Aug 2025)
Frequency-domain gating	$\tilde{F}=F\odot(1+g)$	FLaG (Li et al., 6 Jun 2026)
Sequential evidence aggregation	$$\Lambda^{(K)}=\sum_{\ell=1}^{K} m_\ell \log\frac{p_\ell}{1-p_\ell}$$	MTRE (Zollicoffer et al., 16 May 2025)

These operators differ in where the bottleneck is imposed. YOGO compresses point neighborhoods into region tokens and then projects token information back to all points, so computation occurs on tokens but point-wise detail is preserved through cross-attention (Xu et al., 2021). BToks append a small set of learnable vectors after the multimodal input and define the embedding explicitly as the mean of their final-layer hidden states; the bottleneck is architectural and fixed-capacity (Sun et al., 13 Apr 2026). TC-SSA routes every patch to at most two semantic slots by sparse Top-2 gating and uses weighted aggregation, so all patches contribute while the token budget remains small (Chen et al., 1 Mar 2026).

Other modules define informativeness through reweighting rather than direct pooling. FLaG first transforms token sequences with the real FFT, uses latent queries to summarize spectral components, and applies a channel-wise gate before inverse transformation and final pooling, thereby making the aggregation sensitive to low- and high-frequency structure (Li et al., 6 Jun 2026). MTRE does not aggregate spatial tokens at all; instead, it aggregates reliability evidence across the first ten generated token logits, showing that “informative token aggregation” can also mean temporal aggregation of diagnostic signals rather than compression of encoder outputs (Zollicoffer et al., 16 May 2025).

3. Spatial and Geometric Instantiations

In geometric and spatial settings, informative aggregation is often tied to locality, redundancy reduction, and recovery of fine detail. YOGO exemplifies this by grouping a point cloud only once, creating a small set of tokens from FPS-defined sub-regions, running self-attention over those tokens, and then using cross-attention to project the enriched token features back to all points. The paper reports at least $3.0\times$ speedup over point-based baselines while maintaining competitive classification and segmentation performance, and its ablations show that removing self-attention or replacing cross-attention with naive fusion reduces ShapeNetPart mIoU (Xu et al., 2021).

Set-Mixer pursues a different objective: robustness to noise corruption in point clouds. It imposes a deterministic spatial order through sorting, then applies token-mixing MLPs over local point sets so that each mixed feature depends on all points in the set. On ModelNet40-C, Set-Mixer-APS reports $\mathrm{ER}_{\text{noise}}=10.0$ and Set-Mixer-PCS $\mathrm{ER}_{\text{noise}}=11.0$, compared with $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$0 for PointNet++ and $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$1 for PCT, indicating that aggregation over ordered sets can dilute the effect of individual noisy points more effectively than max-pooling (Zhang et al., 2024).

Cross-scale aggregation appears in image restoration. The ACT super-resolution network combines CNN and Transformer branches and introduces a cross-scale token attention module in which small-scale and large-scale tokens exchange key-value pairs. In its ablations, MHSA + CSTA outperforms MHSA-only and CSTA-only variants, and the two-scale setting $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$2 works better than a single scale or three scales, which suggests that informative aggregation across scales is beneficial but should remain selective rather than indiscriminately multiscale (Yoo et al., 2022).

Long-form video-language understanding raises the same issue in spatiotemporal form. TESTA aggregates similar frames and similar patches within frames using divided temporal and spatial token aggregation in each video encoder block. The method reduces the number of visual tokens by $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$3, improves computing efficiency by $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$4 times, and yields gains such as $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$5 R@1 on QuerYD and $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$6 R@1 on Condensed Movie by enabling denser frame sampling under a practical compute budget (Ren et al., 2023).

In document understanding, token-level correlation-guided compression uses two correlation measures: patch–patch correlation to estimate sub-image information density and [CLS]–patch correlation to mine globally and locally informative tokens. Integrated into mPLUG-DocOwl1.5, the adaptive compressor achieves an average compression ratio of about $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$7, with some sub-images compressed to $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$8, while preserving comparable document understanding performance and clearly outperforming PruMerge and PruMerge+ on the same backbone (Zhang et al., 2024).

4. Explicit Bottlenecks, Semantic Slots, and Soft Tokens

A second major lineage makes the bottleneck explicit and learnable. In unified multimodal retrieval, BToks are a small set of learnable non-vocabulary vectors appended after the input sequence in a decoder-only MLLM. Their hidden states are the sole source of the retrieval embedding, and a Condensation Mask blocks direct query-to-target attention during a next-token prediction auxiliary objective so that predictive information must pass through the bottleneck tokens. On MMEB-V2, this combination reaches an Overall score of $$\mathbf{T}^{(j)} = G\big(\mathrm{maxpool}_i(\{\mathbf{f}_i^{(j)}\})\big)$$9, which is $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$0 over VLM2Vec-V2, with especially large gains on semantically demanding tasks such as Video-QA $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$1 (Sun et al., 13 Apr 2026).

TC-SSA applies the same fixed-capacity logic to gigapixel pathology. A gated routing module assigns patches to a fixed number of semantic slots using sparse Top-2 routing, and each slot aggregates its assigned patch features by a weighted mean. The method reduces the number of visual tokens to $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$2 of the original sequence, achieves $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$3 overall accuracy and $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$4 on the diagnosis subset of SlideBench(TCGA), and reaches AUCs of $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$5, $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$6, and $e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$7 on TCGA-BRCA, TCGA-NSCLC, and PANDA, respectively (Chen et al., 1 Mar 2026).

Token Factory transports this idea into recommendation systems. Each Token Maker transforms heterogeneous raw features into a concatenated feature vector,

$e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$8

and then maps it into a fixed number of soft tokens,

$e(x)=\frac{1}{K}\sum_{k=1}^{K} h^{(L)}_{N+k}$9

Because the output token budget is fixed, prompt length becomes independent of the number of raw features. In the reported ranking setup, prompt length falls from $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$0 to $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$1 tokens for $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$2 watched items; in generative retrieval, it falls from $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$3 to $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$4 tokens, while the online system reports gains including $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$5 Unique Impressions and $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$6 Unique Impressions for one-day-fresh videos (Chen et al., 17 Jun 2026).

ASG addresses informativeness at the embedding layer itself. Instead of storing one monolithic vector per token, it splits each token embedding into $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$7 sub-vectors, quantizes each subspace with a codebook, and reconstructs the token representation by concatenating shared concept vectors: $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$8 Across mBERT, XLM-R, mT5, and BioBERT, ASG compresses embedding parameters to $$c_k=\frac{\sum_{j=1}^{N}\tilde{P}_{j,k}x_j}{\sum_{j=1}^{N}\tilde{P}_{j,k}+\delta}$$9 while maintaining $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$0 task performance relative to the base models, showing that informative aggregation can also be compositional rather than purely sequence-level (V et al., 22 Sep 2025).

5. Layer-, Cluster-, and Cache-Based Aggregation

Not all informative aggregation modules reduce tokens immediately; some aggregate information across layers, clusters, or cached intermediate states before a final prediction head. MaxPoolBERT is the clearest example in text classification. It retains a standard BERT encoder and modifies only the aggregation step by max-pooling the [CLS] token across the last $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$1 layers, adding an extra MHA layer in which [CLS] attends to the full sequence, or combining sequence-wise max-pooling with MHA. On GLUE, the full MaxPoolBERT variant reaches an average score of $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$2 compared with $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$3 for the BERT-base baseline, with especially noticeable improvements on smaller tasks (Behrendt et al., 21 May 2025).

Cross-layer aggregation appears in ultra-fine-grained recognition under aggressive token reduction. CLCA combines a Cross-Layer Aggregation head, which stacks CLS tokens from encoder groups and aggregates them with depth-wise convolution, and a Cross-Layer Cache, which stores GAP summaries and register tokens before token reduction and re-injects them afterward. In the reported comparisons, CLCA at $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$4 keep rate achieves $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$5 top-1 on SoyAgeing at $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$6 FLOPs, and the framework is designed to preserve discriminative detail when token reduction would otherwise remove it (Rios et al., 2024).

WeiAD and WeiToP develop an explicit cluster-weighting view of informativeness for visual place recognition. WeiAD uses OT-based token-to-cluster assignment with bidirectional dustbins, computes cluster importance from transported mass and ghost mass, ranks clusters into tiers, and weights them during descriptor formation. Token importance is then defined as

$$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$7

and WeiToP distills this importance into an early-layer pruning module. After a single joint training phase, the model can vary the retention ratio $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$8 at inference and outperform general-purpose pruning methods adapted from vision tasks (Zeng et al., 19 May 2026).

VISA performs a closely related operation for multimodal LLM inference. Group-wise Token Selection uses the last text token’s attention to visual tokens across several layers and heads to define importance scores, while graph-based Visual Token Aggregation builds a cosine-similarity graph over visual tokens and propagates information from removed tokens to kept tokens through a normalized adjacency. On LLaVA-1.5-7B, retaining $$x^{\text{vis}}_k=x^{\text{vis}}_k+\alpha\,\hat{G}_A x^{\text{vis}}_r$$9 tokens, corresponding to $\tilde{F}=F\odot(1+g)$0 pruning, still preserves $\tilde{F}=F\odot(1+g)$1 of baseline performance, and throughput increases substantially relative to the uncompressed model (Jiang et al., 25 Aug 2025).

6. Reliability, Adaptation, and Frequency-Domain Aggregation

Informative aggregation also appears in modules whose purpose is not representation compression per se, but reliability estimation or adaptation under distribution shift. MTRE aggregates token-level diagnostic evidence from the first ten generated logits of a vision-LLM. A reliability head estimates per-token truthfulness probabilities $\tilde{F}=F\odot(1+g)$2, and the final statistic is a cumulative log-likelihood ratio,

$\tilde{F}=F\odot(1+g)$3

Across MAD-Bench, MM-SafetyBench, MathVista, and four compositional-geometry benchmarks, this multi-token aggregation improves AUROC by $\tilde{F}=F\odot(1+g)$4 points over single-token linear probing and by $\tilde{F}=F\odot(1+g)$5 points over P(True), indicating that informativeness may emerge only after several generated tokens rather than in the first logit alone (Zollicoffer et al., 16 May 2025).

NAVIA approaches aggregation from a mutual-information perspective in efficient test-time adaptation. The analysis states that token aggregation in ViTs causes strict information loss and that LayerNorm tuning cannot recover that loss. NAVIA therefore augments the [CLS] token embedding and adds shallow-layer [CLS] biases optimized by entropy minimization and feature-statistics alignment. Across out-of-distribution benchmarks, the method is reported to outperform prior methods by over $\tilde{F}=F\odot(1+g)$6 while reducing inference latency by more than $\tilde{F}=F\odot(1+g)$7, effectively treating the [CLS] pathway as an information-recovering aggregation interface under compression (Xiong et al., 5 Aug 2025).

FLaG extends the aggregation design space beyond the original token domain. The module applies rFFT along the sequence axis, summarizes spectral components with learnable latent queries, computes a channel-wise gate, reconstructs enhanced time-domain tokens by inverse FFT, and then applies final pooling. In the AMP setting, low-frequency bands contribute the most overall, the remaining higher-band pattern is more sample-specific, the gate acts as a broadly shared spectral reweighting stage, and higher-helix peptides show stronger average spectral sensitivity. Empirically, FLaG gives its clearest gains on ESM2-8M antimicrobial peptide prediction and on CIFAR-100 while remaining competitive on IMDB and GLUE (Li et al., 6 Jun 2026).

7. Efficiency Regimes, Misconceptions, and Open Problems

A common misconception is that token aggregation is equivalent to token dropping. Several of the surveyed systems explicitly reject that equivalence. TESTA reports that token aggregation preserves performance better than pruning at similar compute, YOGO shows that attention-based token-to-point projection outperforms several naive token-to-point fusion alternatives, and VISA is framed as superior to pruning because removed visual tokens are aggregated into kept ones rather than deleted outright (Ren et al., 2023, Xu et al., 2021, Jiang et al., 25 Aug 2025).

A second misconception is that a single summary token is always sufficient. BToks finds that $\tilde{F}=F\odot(1+g)$8 is better than $\tilde{F}=F\odot(1+g)$9 for multimodal retrieval, MTRE shows that later-token logits can carry decisive reliability information absent from the first token, and MaxPoolBERT improves over a last-layer [CLS]-only baseline by aggregating across both layers and token positions (Sun et al., 13 Apr 2026, Zollicoffer et al., 16 May 2025, Behrendt et al., 21 May 2025).

The literature also converges on several limitations. Static grouping or fixed token budgets may underfit non-uniform inputs: YOGO notes the limitation of fixed grouping reused across layers, TC-SSA notes that a fixed slide-level slot budget $$\Lambda^{(K)}=\sum_{\ell=1}^{K} m_\ell \log\frac{p_\ell}{1-p_\ell}$$0 may be suboptimal across slides, and VISA reports sensitivity to group size $$\Lambda^{(K)}=\sum_{\ell=1}^{K} m_\ell \log\frac{p_\ell}{1-p_\ell}$$1, layer count $$\Lambda^{(K)}=\sum_{\ell=1}^{K} m_\ell \log\frac{p_\ell}{1-p_\ell}$$2, and aggregation strength $$\Lambda^{(K)}=\sum_{\ell=1}^{K} m_\ell \log\frac{p_\ell}{1-p_\ell}$$3 (Xu et al., 2021, Chen et al., 1 Mar 2026, Jiang et al., 25 Aug 2025). Compression can preserve global performance while still harming tasks that require precise localization or very large contexts: TC-SSA explicitly notes loss of fine-grained spatial geometry, FLaG adds overhead relative to simple pooling, and NAVIA formalizes that downstream norm-tuning cannot undo information destroyed by aggregation itself (Chen et al., 1 Mar 2026, Li et al., 6 Jun 2026, Xiong et al., 5 Aug 2025).

Possible extensions proposed in the surveyed works are correspondingly consistent. They include adaptive token sizes and dynamic grouping, multi-scale token hierarchies, sparse token-point interactions, hybrid point-token or CNN-Transformer processing, task-aware cross-modal token selection, adaptive slot budgeting, and richer guidance signals for how information should flow through bottleneck tokens rather than only what final embeddings should match (Xu et al., 2021, Chen et al., 1 Mar 2026, Yoo et al., 2022, Ren et al., 2023, Sun et al., 13 Apr 2026). This suggests that the field is moving from simple pooling toward modules that treat aggregation as an explicit modeling problem: deciding which information should be compressed, where the bottleneck should live, and how compressed summaries should remain aligned with the downstream objective.