Learnable Aggregation Tokens

• Learnable aggregation tokens are parameterized embedding vectors that adaptively summarize information in deep learning architectures across modalities such as vision, language, and graphs.
• They are incorporated into models via attention mechanisms like self-attention and cross-attention, with initialization methods ranging from random to data-driven approaches like K-means.
• Empirical results demonstrate that these tokens enhance performance in tasks such as image retrieval, segmentation, and multimodal fusion while reducing computational complexity.

Learnable aggregation tokens are parameterized embedding vectors introduced into deep learning architectures to perform adaptive and efficient information pooling within structured data—such as images, text, graphs, or point clouds. These tokens replace or augment traditional hard-coded aggregation mechanisms by participating in self-attention, cross-attention, or specialized pooling operations, thereby enabling dynamic, learnable summaries that propagate contextual information and mitigate representational bottlenecks. Their adoption spans modalities (vision, language, multimodal, graph) and use cases (retrieval, recognition, segmentation, fusion, knowledge distillation), with strong empirical performance and computational benefits demonstrated across benchmarks.

1. Mathematical Definition and Fundamental Properties

Given an input sequence or set of features $X \in \mathbb{R}^{N \times D}$, a set of $M$ learnable aggregation tokens $A = [a_1;\dots;a_M] \in \mathbb{R}^{M \times D}$ is introduced as independent parameters. These tokens are initialized either randomly (e.g., Gaussian or uniform) (Wang et al., 2023, Im et al., 17 Sep 2025, Jiang et al., 16 May 2024, Georgiou et al., 15 Apr 2025, Kim et al., 29 Jan 2025) or by data-driven priors such as K-means cluster centers of pretrained features (Lu et al., 8 Nov 2025). Aggregation tokens participate in attention-based mixing, where they interact with data-derived tokens (patches, node embeddings, feature descriptors) to accumulate global or regional statistics.

Across designs, the core operation is the contextual update of aggregation tokens via (multi-head) attention:

$Q_A = A W^Q, \quad K_X = X W^K, \quad V_X = X W^V$

$$A^{attn} = \mathrm{Softmax}\left(Q_A K_X^T/\sqrt{d}\right) V_X$$

$A' = f(A, A^{attn})$

with $f$ denoting an update mechanism (potentially including residual connections, normalization, or further cross-modality fusion). This enables aggregation tokens to absorb dataset or task-specific information throughout the forward pass.

2. Methodological Variants across Modalities

Vision Transformers and Image Representation

Learnable aggregation tokens are prepended to patch or feature tokens and passed through a subset of transformer blocks using full self-attention (Lu et al., 8 Nov 2025), as in ImAge (robust place recognition). Strategies are studied for insertion (e.g., introducing tokens after frozen blocks or at deeper layers) and initialization (K-means vs. random) to optimize both downstream accuracy and training efficiency. These tokens implicitly aggregate global and local context, outperforming explicit aggregators on VPR (Recall@1 of 94.0% on Pitts30k vs. 92.8% for NetVLAD) with zero added aggregator parameters and faster inference.

In METransformer (Wang et al., 2023), multiple learnable "expert" tokens are used to encourage disjoint, complementary image summaries. Orthogonality losses enforce token diversity, and expert voting is used for output aggregation. Expert tokens attend jointly to patches and each other, facilitating ensemble-like, parameter-efficient diagnosis.

LeMeViT (Jiang et al., 16 May 2024) exploits a small set of meta tokens initialized by cross-attention to image tokens, then alternates image/meta pooling using Dual Cross-Attention (DCA). This yields linear (in token count) complexity for early blocks, significantly improving throughput (1.7× speedup and competitive accuracy relative to PVTv2) in remote sensing and dense prediction.

Few-Shot and Prototype-Based Aggregation

For few-shot 3D point cloud segmentation, the White Aggregation and Restoration Module (WARM) (Im et al., 17 Sep 2025) combines learnable prototypical tokens with data whitening: support features are whitened, tokens query support points (via cross-attention as queries), then output tokens are "colored" (inverse transform) to restore original statistics. This yields robust multi-prototype class summaries, with S3DIS mIoU improving to 55.66 (from 51.03/47.21). Whitening not only aligns feature distributions but also increases attention entropy and convergence speed.

Multimodal and Fusion Contexts

In DeepMLF (Georgiou et al., 15 Apr 2025), a small set ($8 \leq M \leq 20$) of learnable fusion tokens is appended to the decoder of a pre-trained LLM to mediate cross-modal fusion. These tokens absorb linguistic context using causal self-attention and, at designated MM Blocks, gather audiovisual features using gated cross-attention. Only these tokens transmit multimodal content; unimodal flows are preserved. The number of tokens and fusion depth (5–7 layers) are crucial, as empirically validated by +1–2% gains in sentiment analysis benchmarks.

Graph-Based Representation

The Learnable Graph Pooling Token (LGPT) (Kim et al., 29 Jan 2025) compresses $k$-node graph embeddings into $n \ll k$ prompt tokens for LLMs. Each token softly aggregates over all nodes via attention, ensuring a controllable tradeoff between scalability and structural fidelity. Early Query Fusion, in which the query is injected into the graph prior to pooling, further enhances downstream accuracy (+4.13% on GraphQA).

Mixture of Tokens for Expert Routing

The Mixture of Tokens (MoT) framework (Antoniak et al., 2023) redefines MoE routing: instead of sparsely dispatching single tokens to experts, MoT employs a continuous gating network to compute soft mixtures over grouped tokens, aggregates to each expert, processes, and redistributes outputs. This is fully differentiable, compatible with autoregressive decoding, supports transition to sparse MoE by annealing the softmax temperature, and yields 3× faster training convergence in language pretraining.

3. Design Principles: Token Count, Placement, and Initialization

The number of aggregation tokens ($M$, $K$, or $n$) is a tightly controlled hyperparameter, selected via grid search or ablation to balance capacity and representation diversity. Vision tasks find $M=4\ldots8$ optimal for compact global descriptors (Lu et al., 8 Nov 2025, Wu et al., 2021), while multimodal and graph settings may benefit from $M=8\ldots20$ for richer fusion (Georgiou et al., 15 Apr 2025, Kim et al., 29 Jan 2025). Over-parameterization with excessive token counts can plateau or degrade performance (Wang et al., 2023, Im et al., 17 Sep 2025).

Insertion point is domain-sensitive. For frozen-backbone transformers, introducing aggregation tokens just before the first trainable block maximizes downstream retrieval while minimizing compute (Lu et al., 8 Nov 2025). Gradual or multi-block insertion can be explored but may not outperform single-block methods.

Initialization by unsupervised methods (K-means) on existing features improves downstream R@1 compared to random assignment (Lu et al., 8 Nov 2025). Normalization (L2) of the initial centers preserves scale invariance and stability.

4. Attention Mechanisms and Specialized Aggregation

Aggregation tokens interact with data tokens via standard or modified attention operators. In the class of transformer models, attention may be:

• Full multi-head self-attention among all tokens, so agg tokens participate identically to other tokens (ViT-style) (Lu et al., 8 Nov 2025, Wang et al., 2023).
• Cross-attention, where aggregation tokens serve as queries and data tokens as keys/values (Im et al., 17 Sep 2025, Kim et al., 29 Jan 2025, Wu et al., 2021).
• Alternating dual cross-attention (image-to-meta, then meta-to-image) (Jiang et al., 16 May 2024).

Specialized mechanisms (e.g., whitening/coloring (Im et al., 17 Sep 2025), orthogonality regularization (Wang et al., 2023), gating and residual connections (Georgiou et al., 15 Apr 2025)) further promote diversity, stability, and information disentanglement.

In multimodal architectures, learnable fusion tokens exploit masked or gated attention, ensuring only those tokens carry modality-fused content, thereby preserving unimodal attribute flows (Georgiou et al., 15 Apr 2025).

In flexible pooling designs (MoT (Antoniak et al., 2023), LGPT (Kim et al., 29 Jan 2025)), soft weighting via continuous gating or attention avoids discrete bottlenecks and provides end-to-end differentiability.

5. Empirical Results and Benchmark Comparisons

Learnable aggregation tokens consistently outperform explicit and hand-crafted aggregation schemes in sample efficiency, downstream accuracy, and runtime across tasks:

Task/Domain	Model	# Agg Tokens	Metric	Performance	Comparison
Visual Place Retrieval	ImAge (Lu et al., 8 Nov 2025)	8	Recall@1 (Pitts30k)	94.0%	NetVLAD: 92.8%
3D FS Segmentation	WARM (Im et al., 17 Sep 2025)	100 (FG/BG)	mIoU (S3DIS 1w1s)	55.66	FPS+min-dist: 51.03
Med. Report Generation	METransformer (Wang et al., 2023)	5–7	BLEU-4, CIDEr (IU-Xray/MIMIC)	+empirical gains	Single-[CLS] or head-only
Multimodal Sentiment	DeepMLF (Georgiou et al., 15 Apr 2025)	8–20	Acc2 (MOSEI)	87.15%	Prior SOTA: –1–2%
GraphQA	LGPT (Kim et al., 29 Jan 2025)	8	Hits@1 (Mean)	82.25 (+4.13%)	G-Retriever

Performance gains are observed for moderate token counts; excessive token proliferation delivers diminishing returns or may promote redundancy. Convergence is accelerated by normalization steps (e.g., whitening (Im et al., 17 Sep 2025)) and K-means initialization (Lu et al., 8 Nov 2025).

In computational terms, aggregation tokens can sharply reduce memory and FLOPs, particularly when replacing $O(N^2)$ self-attention with $O(NM)$ cross-attention (as in LeMeViT (Jiang et al., 16 May 2024), with $M \ll N$). MoT (Antoniak et al., 2023) achieves 3× faster training vs. dense transformers.

6. Generalizations, Limitations, and Future Directions

Learnable aggregation tokens provide a unifying, adaptive abstraction for summarizing structured data in deep models. They subsume rigid operators such as mean/sum pooling, NetVLAD (Lu et al., 8 Nov 2025), or hard-coded graph pooling (Kim et al., 29 Jan 2025). Whitewashing (input feature whitening) prior to aggregation, as in WARM (Im et al., 17 Sep 2025), is a transferable technique for robust prototype learning in limited-data environments.

Open issues remain concerning optimal token insertion policy, initialization, token scalability, and overparameterization, as well as adaptation to very large scales and new modalities. Further, specialized regularization or diversity-encouraging objectives may be required to prevent redundancy (e.g., orthogonality penalties (Wang et al., 2023)).

A plausible implication is that the architecture-agnostic nature of learnable aggregation tokens—requiring only that the model support token-wise attention or pooling—facilitates cross-domain transfer and plug-in extensibility. As design principles are refined and empirically validated, aggregation tokens are likely to constitute a canonical component in future scalable, multi-modal, and few-shot learning systems.